StarRocks version 3.1

3.1.3

Release date: September 25, 2023

New Features

• The aggregate function group_concat supports the DISTINCT keyword and the ORDER BY clause. #28778
• Stream Load, Broker Load, Kafka Connector, Flink Connector, and Spark Connector support partial updates in column mode on a Primary Key table. #28288
• Data in partitions can be automatically cooled down over time. (This feature is not supported for list partitioning.) #29335 #29393

Improvements

Executing SQL commands with invalid comments now returns results consistent with MySQL. #30210

Bug Fixes

Fixed the following issues:

• If the BITMAP or HLL data type is specified in the WHERE clause of a DELETE statement to be executed, the statement cannot be properly executed. #28592
• After a follower FE is restarted, CpuCores statistics are not up-to-date, resulting in query performance degradation. #28472 #30434
• The execution cost of the to_bitmap() function is incorrectly calculated. As a result, an inappropriate execution plan is selected for the function after materialized views are rewritten. #29961
• In certain use cases of the shared-data architecture, after a follower FE is restarted, queries submitted to the follower FE return an error that reads "Backend node not found. Check if any backend node is down". #28615
• If data is continuously loaded into a table that is being altered by using the ALTER TABLE statement, an error "Tablet is in error state" may be thrown. #29364
• Modifying the FE dynamic parameter max_broker_load_job_concurrency using the ADMIN SET FRONTEND CONFIG command does not take effect. #29964 #29720
• BEs crash if the time unit in the date_diff() function is a constant but the dates are not constants. #29937
• In the shared-data architecture, automatic partitioning does not take effect after asynchronous load is enabled. #29986
• If users create a Primary Key table by using the CREATE TABLE LIKE statement, an error "Unexpected exception: Unknown properties: {persistent_index_type=LOCAL}" is thrown. #30255
• Restoring Primary Key tables causes metadata inconsistency after BEs are restarted. #30135
• If users load data into a Primary Key table on which truncate operations and queries are concurrently performed, an error "java.lang.NullPointerException" is thrown in certain cases. #30573
• If predicate expressions are specified in materialized view creation statements, the refresh results of those materialized views are incorrect. #29904
• After users upgrade their StarRocks cluster to v3.1.2, the storage volume properties of the tables created before the upgrade are reset to null. #30647
• If checkpointing and restoration are concurrently performed on tablet metadata, some tablet replicas will be lost and cannot be retrieved. #30603
• If users use CloudCanal to load data into table columns that are set to NOT NULL but have no default value specified, an error "Unsupported dataFormat value is : \N" is thrown. #30799

3.1.2

Release date: August 25, 2023

Bug Fixes

Fixed the following issues:

• If a user specifies which database is to be connected by default and the user only has permissions on tables in the database but does not have permissions on the database, an error stating that the user does not have permissions on the database is thrown. #29767
• The values returned by the RESTful API action show_data for cloud-native tables are incorrect. #29473
• BEs crash if queries are canceled while the array_agg() function is being run. #29400
• The Default field values returned by the SHOW FULL COLUMNS statement for columns of the BITMAP or HLL data type are incorrect. #29510
• If the array_map() function in queries involves multiple tables, the queries fail due to pushdown strategy issues. #29504
• Queries against ORC-formatted files fail because the bugfix ORC-1304 (apache/orc#1299) from Apache ORC is not merged. #29804

Behavior Change

For a newly deployed StarRocks v3.1 cluster, you must have the USAGE privilege on the destination external catalog if you want to run SET CATALOG to switch to that catalog. You can use GRANT to grant the required privileges.

For a v3.1 cluster upgraded from an earlier version, you can run SET CATALOG with inherited privilege.

3.1.1

Release date: August 18, 2023

New Features

• Supports Azure Blob Storage for shared-data clusters.
• Supports List partitioning for shared-data clusters.
• Supports aggregate functions COVAR_SAMP, COVAR_POP, and CORR.
• Supports the following window functions: COVAR_SAMP, COVAR_POP, CORR, VARIANCE, VAR_SAMP, STD, and STDDEV_SAMP.

Improvements

Supports implicit conversions for all compound predicates and for all expressions in the WHERE clause. You can enable or disable implicit conversions by using the session variable enable_strict_type. The default value of this session variable is false.

Bug Fixes

Fixed the following issues:

• When data is loaded into tables that have multiple replicas, a large number of invalid log records are written if some partitions of the tables are empty. #28824
• Inaccurate estimation of average row size causes partial updates in column mode on Primary Key tables to occupy excessively large memory. #27485
• If clone operations are triggered on tablets in an ERROR state, disk usage increases. #28488
• Compaction causes cold data to be written to the local cache. #28831

3.1.0

Release date: August 7, 2023

New Features

Shared-data cluster

• Added support for Primary Key tables, on which persistent indexes cannot be enabled.
• Supports the AUTO_INCREMENT column attribute, which enables a globally unique ID for each data row and thus simplifies data management.
• Supports automatically creating partitions during loading and using partitioning expressions to define partitioning rules, thereby making partition creation easier to use and more flexible.
• Supports abstraction of storage volumes, in which users can configure storage location and authentication information, in StarRocks shared-data clusters. Users can directly reference an existing storage volume when creating a database or table, making authentication configuration easier.

Data Lake analytics

• Supports accessing views created on tables within Hive catalogs.
• Supports accessing Parquet-formatted Iceberg v2 tables.
• Supports sinking data to Parquet-formatted Iceberg tables.
• [Preview] Supports accessing data stored in Elasticsearch by using Elasticsearch catalogs. This simplifies the creation of Elasticsearch external tables.
• [Preview] Supports performing analytics on streaming data stored in Apache Paimon by using Paimon catalogs.

Storage engine, data ingestion, and query

• Upgraded automatic partitioning to expression partitioning. Users only need to use a simple partition expression (either a time function expression or a column expression) to specify a partitioning method at table creation, and StarRocks will automatically create partitions based on the data characteristics and the rule defined in the partition expression during data loading. This method of partition creation is suitable for most scenarios and is more flexible and user-friendly.
• Supports list partitioning. Data is partitioned based on a list of values predefined for a particular column, which can accelerate queries and manage clearly categorized data more efficiently.
• Added a new table named loads to the Information_schema database. Users can query the results of Broker Load and Insert jobs from the loads table.
• Supports logging the unqualified data rows that are filtered out by Stream Load, Broker Load, and Spark Load jobs. Users can use the log_rejected_record_num parameter in their load job to specify the maximum number of data rows that can be logged.
• Supports random bucketing. With this feature, users do not need to configure bucketing columns at table creation, and StarRocks will randomly distribute the data loaded into it to buckets. Using this feature together with the capability of automatically setting the number of buckets (BUCKETS) that StarRocks has provided since v2.5.7, users no longer need to consider bucket configurations, and table creation statements are greatly simplified. In big data and high performance-demanding scenarios, however, we recommend that users continue using hash bucketing, because this way they can use bucket pruning to accelerate queries.
• Supports using the table function FILES() in INSERT INTO to directly load the data of Parquet- or ORC-formatted data files stored in AWS S3. The FILES() function can automatically infer the table schema, which relieves the need to create external catalogs or file external tables before data loading and therefore greatly simplifies the data loading process.
• Supports generated columns. With the generated column feature, StarRocks can automatically generate and store the values of column expressions and automatically rewrite queries to improve query performance.
• Supports loading data from Spark to StarRocks by using Spark connector. Compared to Spark Load, the Spark connector provides more comprehensive capabilities. Users can define a Spark job to perform ETL operations on the data, and the Spark connector serves as the sink in the Spark job.
• Supports loading data into columns of the MAP and STRUCT data types, and supports nesting Fast Decimal values in ARRAY, MAP, and STRUCT.

SQL reference

• Added the following storage volume-related statements: CREATE STORAGE VOLUME, ALTER STORAGE VOLUME, DROP STORAGE VOLUME, SET DEFAULT STORAGE VOLUME, DESC STORAGE VOLUME, SHOW STORAGE VOLUMES.

• Supports altering table comments using ALTER TABLE. #21035

• Added the following functions:

  • Struct functions: struct (row), named_struct
  • Map functions: str_to_map, map_concat, map_from_arrays, element_at, distinct_map_keys, cardinality
  • Higher-order Map functions: map_filter, map_apply, transform_keys, transform_values
  • Array functions: array_agg supports ORDER BY, array_generate, element_at, cardinality
  • Higher-order Array functions: all_match, any_match
  • Aggregate functions: min_by, percentile_disc
  • Table functions: FILES, generate_series
  • Date functions: next_day, previous_day, last_day, makedate, date_diff
  • Bitmap functions：bitmap_subset_limit, bitmap_subset_in_range
  • Math functions: cosine_similarity, cosine_similarity_norm

Privileges and security

Added privilege items related to storage volumes and privilege items related to external catalogs, and supports using GRANT and REVOKE to grant and revoke these privileges.

Improvements

Shared-data cluster

Optimized the data cache in StarRocks shared-data clusters. The optimized data cache allows for specifying the range of hot data. It can also prevent queries against cold data from occupying the local disk cache, thereby ensuring the performance of queries against hot data.

Materialized view

• Optimized the creation of an asynchronous materialized view:
  • Supports random bucketing. If users do not specify bucketing columns, StarRocks adopts random bucketing by default.
  • Supports using ORDER BY to specify a sort key.
  • Supports specifying attributes such as colocate_group, storage_medium, and storage_cooldown_time.
  • Supports using session variables. Users can configure these variables by using the properties("session.<variable_name>" = "<value>") syntax to flexibly adjust view refreshing strategies.
  • Enables the spill feature for all asynchronous materialized views and implements a query timeout duration of 1 hour by default.
  • Supports creating materialized views based on views. This makes materialized views easier to use in data modeling scenarios, because users can flexibly use views and materialized views based on their varying needs to implement layered modeling.
• Optimized query rewrite with asynchronous materialized views:
  • Supports Stale Rewrite, which allows materialized views that are not refreshed within a specified time interval to be used for query rewrite regardless of whether the base tables of the materialized views are updated. Users can specify the time interval by using the mv_rewrite_staleness_second property at materialized view creation.
  • Supports rewriting View Delta Join queries against materialized views that are created on Hive catalog tables (a primary key and a foreign key must be defined).
  • Optimized the mechanism for rewriting queries that contain union operations, and supports rewriting queries that contain joins or functions such as COUNT DISTINCT and time_slice.
• Optimized the refreshing of asynchronous materialized views:
  • Optimized the mechanism for refreshing materialized views that are created on Hive catalog tables. StarRocks now can perceive partition-level data changes, and refreshes only the partitions with data changes during each automatic refresh.
  • Supports using the REFRESH MATERIALIZED VIEW WITH SYNC MODE syntax to synchronously invoke materialized view refresh tasks.
• Enhanced the use of asynchronous materialized views:
  • Supports using ALTER MATERIALIZED VIEW {ACTIVE | INACTIVE} to enable or disable a materialized view. Materialized views that are disabled (in the INACTIVE state) cannot be refreshed or used for query rewrite, but can be directly queried.
  • Supports using ALTER MATERIALIZED VIEW SWAP WITH to swap two materialized views. Users can create a new materialized view and then perform an atomic swap with an existing materialized view to implement schema changes on the existing materialized view.
• Optimized synchronous materialized views:
  • Supports direct queries against synchronous materialized views using SQL hints [_SYNC_MV_], allowing for walking around issues that some queries cannot be properly rewritten in rare circumstances.
  • Supports more expressions, such as CASE-WHEN, CAST, and mathematical operations, which make materialized views suitable for more business scenarios.

Data Lake analytics

• Optimized metadata caching and access for Iceberg to improve Iceberg data query performance.
• Optimized the data cache to further improve data lake analytics performance.

Storage engine, data ingestion, and query

• Announced the general availability of the spill feature, which supports spilling the intermediate computation results of some blocking operators to disk. With the spill feature enabled, when a query contains aggregate, sort, or join operators, StarRocks can cache the intermediate computation results of the operators to disk to reduce memory consumption, thereby minimizing query failures caused by memory limits.
• Supports pruning on cardinality-preserving joins. If users maintain a large number of tables which are organized in the star schema (for example, SSB) or the snowflake schema (for example, TCP-H) but they query only a small number of these tables, this feature helps prune unnecessary tables to improve the performance of joins.
• Supports partial updates in column mode. Users can enable the column mode when they perform partial updates on Primary Key tables by using the UPDATE statement. The column mode is suitable for updating a small number of columns but a large number of rows, and can improve the updating performance by up to 10 times.
• Optimized the collection of statistics for the CBO. This reduces the impact of statistics collection on data ingestion and increases statistics collection performance.
• Optimized the merge algorithm to increase the overall performance by up to 2 times in permutation scenarios.
• Optimized the query logic to reduce dependency on database locks.

SQL reference

• Conditional functions case, coalesce, if, ifnull, and nullif support the ARRAY, MAP, STRUCT, and JSON data types.
• The following Array functions support nested types MAP, STRUCT, and ARRAY:
  • array_agg
  • array_contains, array_contains_all, array_contains_any
  • array_slice, array_concat
  • array_length, array_append, array_remove, array_position
  • reverse, array_distinct, array_intersect, arrays_overlap
  • array_sortby
• The following Array functions support the Fast Decimal data type:
  • array_agg
  • array_append, array_remove, array_position, array_contains
  • array_length
  • array_max, array_min, array_sum, array_avg
  • arrays_overlap, array_difference
  • array_slice, array_distinct, array_sort, reverse, array_intersect, array_concat
  • array_sortby, array_contains_all, array_contains_any

Bug Fixes

Fixed the following issues:

• Requests to reconnect to Kafka for Routine Load jobs cannot be properly processed. #23477
• For SQL queries that involve multiple tables and contain a WHERE clause, if these SQL queries have the same semantics but the order of the tables in each SQL query is different, some of these SQL queries may fail to be rewritten to benefit from the related materialized views. #22875
• Duplicate records are returned for queries that contain a GROUP BY clause. #19640
• Invoking the lead() or lag() function may cause BE crashes. #22945
• Rewriting partial partition queries based on materialized views that are created on external catalog tables fail. #19011
• SQL statements that contain both a backward slash (\) and a semicolon (;) cannot be properly parsed. #16552
• A table cannot be truncated if a materialized view created on the table is removed. #19802

Behavior Change

• The storage_cache_ttl parameter is deleted from the table creation syntax used for StarRocks shared-data clusters. Now the data in the local cache is evicted based on the LRU algorithm.
• The BE configuration items disable_storage_page_cache and alter_tablet_worker_count and the FE configuration item lake_compaction_max_tasks are changed from immutable parameters to mutable parameters.
• The default value of the BE configuration item block_cache_checksum_enable is changed from true to false.
• The default value of the BE configuration item enable_new_load_on_memory_limit_exceeded is changed from false to true.
• The default value of the FE configuration item max_running_txn_num_per_db is changed from 100 to 1000.
• The default value of the FE configuration item http_max_header_size is changed from 8192 to 32768.
• The default value of the FE configuration item tablet_create_timeout_second is changed from 1 to 10.
• The default value of the FE configuration item max_routine_load_task_num_per_be is changed from 5 to 16, and error information will be returned if a large number of Routine Load tasks are created.
• The FE configuration item quorom_publish_wait_time_ms is renamed as quorum_publish_wait_time_ms, and the FE configuration item async_load_task_pool_size is renamed as max_broker_load_job_concurrency.
• The BE configuration item routine_load_thread_pool_size is deprecated. Now the routine load thread pool size per BE node is controlled only by the FE configuration item max_routine_load_task_num_per_be.
• The BE configuration item txn_commit_rpc_timeout_ms and the system variable tx_visible_wait_timeout are deprecated. Now the time_out parameter is used to specify the transaction timeout duration.
• The FE configuration items max_broker_concurrency and load_parallel_instance_num are deprecated.
• The FE configuration item max_routine_load_job_num is deprecated. Now StarRocks dynamically infers the maximum number of Routine Load tasks supported by each individual BE node based on the max_routine_load_task_num_per_be parameter and provides suggestions on task failures.
• The CN configuration item thrift_port is renamed as be_port.
• Two new Routine Load job properties, task_consume_second and task_timeout_second, are added to control the maximum amount of time to consume data and the timeout duration for individual load tasks within a Routine Load job, making job adjustment more flexible. If users do not specify these two properties in their Routine Load job, the FE configuration items routine_load_task_consume_second and routine_load_task_timeout_second prevail.
• The session variable enable_resource_group is deprecated because the Resource Group feature is enabled by default since v3.1.0.
• Two new reserved keywords, COMPACTION and TEXT, are added.