Cross-cluster Data Migration Tool
The StarRocks Cross-cluster Data Migration Tool is provided by StarRocks Community. You can use this tool to easily migrate data from the source cluster to the target cluster.
- The StarRocks Cross-cluster Data Migration Tool only supports migrating data from a shared-nothing cluster to either another shared-nothing cluster or a shared-data cluster.
- The StarRocks version of the target cluster must be v3.1.8, v3.2.3, or later.
Preparations
The following preparations must be performed on the target cluster for data migration.
Enable Legacy Compatibility for Replication
StarRocks may behave differently between the old and new versions, causing problems during cross-cluster data migration. Therefore, you must enable Legacy Compatibility for the target cluster before data migration and disable it after data migration is completed.
-
You can check whether Legacy Compatibility for Replication is enabled by using the following statement:
ADMIN SHOW FRONTEND CONFIG LIKE 'enable_legacy_compatibility_for_replication';
If
true
is returned, it indicates that Legacy Compatibility for Replication is enabled. -
Dynamically enable Legacy Compatibility for Replication:
ADMIN SET FRONTEND CONFIG("enable_legacy_compatibility_for_replication"="true");
-
To prevent Legacy Compatibility for Replication from automatically disabling during the data migration process in case of cluster restart, you also need to add the following configuration item in the FE configuration file fe.conf:
enable_legacy_compatibility_for_replication = true
After the data migration is completed, you need to remove the configuration enable_legacy_compatibility_for_replication = true
from the configuration file, and dynamically disable Legacy Compatibility for Replication using the following statement:
ADMIN SET FRONTEND CONFIG("enable_legacy_compatibility_for_replication"="false");
Disable Compaction
If the target cluster for data migration is a shared-data cluster, you need to manually disable Compaction before starting the data migration and re-enable it after the data migration is completed.
-
You can check whether Compaction is enabled by using the following statement:
ADMIN SHOW FRONTEND CONFIG LIKE 'lake_compaction_max_tasks';
If
0
is returned, it indicates that Compaction is disabled. -
Dynamically disable Compaction:
ADMIN SET FRONTEND CONFIG("lake_compaction_max_tasks"="0");
-
To prevent Compaction from automatically enabling during the data migration process in case of cluster restart, you also need to add the following configuration item in the FE configuration file fe.conf:
lake_compaction_max_tasks = 0
After the data migration is completed, you need to remove the configuration lake_compaction_max_tasks = 0
from the configuration file, and dynamically enable Compaction using the following statement:
ADMIN SET FRONTEND CONFIG("lake_compaction_max_tasks"="-1");