Skip to main content

Releases of StarRocks Connector for Flink

Notifications​

User guide:

Source codes: starrocks-connector-for-apache-flink

Naming format of the JAR file:

  • Flink 1.15 and later: flink-connector-starrocks-${connector_version}_flink-${flink_version}.jar
  • Prior to Flink 1.15: flink-connector-starrocks-${connector_version}_flink-${flink_version}_${scala_version}.jar

Methods to obtain the JAR file:

  • Directly download the the Flink connector JAR file from the Maven Central Repository.
  • Add the Flink connector as a dependency in your Maven project's pom.xml file and download it. For specific instructions, see user guide.
  • Compile the source codes into Flink connector JAR file. For specific instructions, see user guide.

Version requirements:

ConnectorFlinkStarRocksJavaScala
1.2.91.15,1.16,1.17,1.182.1 and later82.11,2.12
1.2.81.13,1.14,1.15,1.16,1.172.1 and later82.11,2.12
1.2.71.11,1.12,1.13,1.14,1.152.1 and later82.11,2.12

NOTICE

In general, the latest version of the Flink connector only maintains compatibility with the three most recent versions of Flink.

Release notes​

1.2​

1.2.10​

Features

  • Supports reading JSON columns. #334
  • Supports reading ARRAY, STRUCT, and MAP columns. #347
  • Supports LZ4 compression when sinking data with the JSON format. #354
  • Supports Flink 1.19. #379

Improvements

  • Supports configuring socket timeout. #319
  • The Stream Load transaction interface supports asynchronous prepare and commit operations. #328
  • Supports mapping a subset of columns in a StarRocks table to a Flink source table. #352
  • Supports setting a specific warehouse when using the Stream Load transaction interface. #361

Bug Fixes

Fixed the following issues:

  • StarRocksSourceBeReader in StarRocksDynamicLookupFunction is not closed after data reading completes. #351
  • An exception was thrown when loading an empty JSON string into a JSON column. #380

1.2.9​

This release includes some features and bug fixes. The notable change is that the Flink connector is integrated with Flink CDC 3.0 to easily build a streaming ELT pipeline from CDC sources (such as MySQL and Kafka) to StarRocks. You can see Synchronize data with Flink CDC 3.0 (with schema change supported) for details.

Features

  • Implement catalog to support Flink CDC 3.0. #295
  • Implement new sink API in FLP-191 to support Flink CDC 3.0. #301
  • Support Flink 1.18. #305

Bug Fixes

  • Fix misleading thread name and log. #290
  • Fix wrong stream-load-sdk configurations used for writing to multiple tables. #298

1.2.8​

This release includes some improvements and bug fixes. The notable changes are as follows:

  • Support Flink 1.16 and 1.17.
  • Recommend to set sink.label-prefix when the sink is configured to guarantee the exactly-once semantics. For the specific instructions, see Exactly Once.

Improvements

  • Support to configure whether to use Stream Load transaction interface to guarantee at-least-once. #228
  • Add retry metrics for sink V1. #229
  • No need to getLabelState when EXISTING_JOB_STATUS is FINISHED. #231
  • Remove useless stack trace log for sink V1. #232
  • [Refactor] Move StarRocksSinkManagerV2 to stream-load-sdk. #233
  • Automatically detect partial updates according to a Flink table's schema instead of the sink.properties.columns parameter explicitly specified by users. #235
  • [Refactor] Move probeTransactionStreamLoad to stream-load-sdk. #240
  • Add git-commit-id-plugin for stream-load-sdk. #242
  • Use info log for DefaultStreamLoader#close. #243
  • Support to generate stream-load-sdk JAR file without dependencies. #245
  • Replace fastjson with jackson in stream-load-sdk. #247
  • Support to process update_before record. #250
  • Add the Apache license into files. #251
  • Support to get the exception in stream-load-sdk. #252
  • Enable strip_outer_array and ignore_json_size by default. #259
  • Try to cleanup lingering transactions when a Flink job restores and the sink semantics is exactly-once. #271
  • Return the first exception after the retrying fails. #279

Bug Fixes

  • Fix typos in StarRocksStreamLoadVisitor. #230
  • Fix the fastjson classloader leak. #260

Tests

  • Add the test framework for loading from Kafka to StarRocks. #249

Doc

  • Refactor the docs. #262
  • Improve the doc for the sink. #268 #275
  • Add examples of DataStream API for the sink. #253