Authenticate to Google Cloud Storage

Authentication methods

From v3.0 onwards, StarRocks supports using one of the following authentication methods to access Google Cloud Storage (GCS):

• VM-based authentication

  Use the credential attached to Google Cloud Compute Engine to authenticate GCS.

• Service account-based authentication

  Use a service account to authenticate GCS.

• Impersonation-based authentication

  Make a service account or virtual machine (VM) instance impersonate another service account.

Scenarios

StarRocks can authenticate to GCS in the following scenarios:

• Batch load data from GCS.
• Back up data from and restore data to GCS.
• Query Parquet and ORC files in GCS.
• Query Hive, Iceberg, Hudi, and Delta Lake tables in GCS.

In this topic, Hive catalog, file external table, and Broker Load are used as examples to show how StarRocks integrates with GCS in different scenarios. For information about StorageCredentialParams in the examples, see the "Parameters" section of this topic.

NOTE

StarRocks supports loading data or directly querying files from GCS only according to the gs protocol. Therefore, when you load data or query files from GCS, you must include gs as a prefix in the file path.

External catalog

Use the CREATE EXTERNAL CATALOG statement to create a Hive catalog named hive_catalog_gcs as follows, in order to query files from GCS:

CREATE EXTERNAL CATALOG hive_catalog_gcs
PROPERTIES
(
    "type" = "hive", 
    "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083",
    StorageCredentialParams
);

File external table

Use the CREATE EXTERNAL TABLE statement to create a file external table named external_table_gcs as follows, in order to query a data file named test_file_external_tbl from GCS without any metastore:

CREATE EXTERNAL TABLE external_table_gcs
(
    id varchar(65500),
    attributes map<varchar(100), varchar(2000)>
) 
ENGINE=FILE
PROPERTIES
(
    "path" = "gs:////test-gcs/test_file_external_tbl",
    "format" = "ORC",
    StorageCredentialParams
);

Broker load

Use the LOAD LABEL statement to create a Broker Load job whose label is test_db.label000, in order to batch load data from GCS into the StarRocks table target_table:

LOAD LABEL test_db.label000
(
    DATA INFILE("gs://bucket_gcs/test_brokerload_ingestion/*")
    INTO TABLE target_table
    FORMAT AS "parquet"
)
WITH BROKER
(
    StorageCredentialParams
);

Parameters

StorageCredentialParams represents a parameter set that describes how to authenticate to GCS with different authentication methods.

VM-based authentication

If your StarRocks cluster is deployed on a VM instance hosted on Google Cloud Platform (GCP) and you want to use that VM instance to authenticate GCS, configure StorageCredentialParams as follows:

"gcp.gcs.use_compute_engine_service_account" = "true"

The following table describes the parameters you need to configure in StorageCredentialParams.

Parameter	Default value	Value example	Description
gcp.gcs.use_compute_engine_service_account	false	true	Specifies whether to directly use the service account that is bound to your Compute Engine.

Service account-based authentication

If you directly use a service account to authenticate GCS, configure StorageCredentialParams as follows:

"gcp.gcs.service_account_email" = "<google_service_account_email>",
"gcp.gcs.service_account_private_key_id" = "<google_service_private_key_id>",
"gcp.gcs.service_account_private_key" = "<google_service_private_key>"

The following table describes the parameters you need to configure in StorageCredentialParams.

Parameter	Default value	Value example	Description
gcp.gcs.service_account_email	""	"user@hello.iam.gserviceaccount.com"	The email address in the JSON file generated at the creation of the service account.
gcp.gcs.service_account_private_key_id	""	"61d257bd8479547cb3e04f0b9b6b9ca07af3b7ea"	The private key ID in the JSON file generated at the creation of the service account.
gcp.gcs.service_account_private_key	""	"-----BEGIN PRIVATE KEY----xxxx-----END PRIVATE KEY-----\n"	The private key in the JSON file generated at the creation of the service account.

Impersonation-based authentication

Make a VM instance impersonate a service account

If your StarRocks cluster is deployed on a VM instance hosted on GCP and you want to make that VM instance impersonate a service account, so as to make StarRocks inherit the privileges from the service account to access GCS, configure StorageCredentialParams as follows:

"gcp.gcs.use_compute_engine_service_account" = "true",
"gcp.gcs.impersonation_service_account" = "<assumed_google_service_account_email>"

The following table describes the parameters you need to configure in StorageCredentialParams.

Parameter	Default value	Value example	Description
gcp.gcs.use_compute_engine_service_account	false	true	Specifies whether to directly use the service account that is bound to your Compute Engine.
gcp.gcs.impersonation_service_account	""	"hello"	The service account that you want to impersonate.

Make a service account impersonate another service account

If you want to make a service account (temporarily named as meta service account) impersonate another service account (temporarily named as data service account) and make StarRocks inherit the privileges from the data service account to access GCS, configure StorageCredentialParams as follows:

"gcp.gcs.service_account_email" = "<google_service_account_email>",
"gcp.gcs.service_account_private_key_id" = "<meta_google_service_account_email>",
"gcp.gcs.service_account_private_key" = "<meta_google_service_account_email>",
"gcp.gcs.impersonation_service_account" = "<data_google_service_account_email>"

The following table describes the parameters you need to configure in StorageCredentialParams.

Parameter	Default value	Value example	Description
gcp.gcs.service_account_email	""	"user@hello.iam.gserviceaccount.com"	The email address in the JSON file generated at the creation of the meta service account.
gcp.gcs.service_account_private_key_id	""	"61d257bd8479547cb3e04f0b9b6b9ca07af3b7ea"	The private key ID in the JSON file generated at the creation of the meta service account.
gcp.gcs.service_account_private_key	""	"-----BEGIN PRIVATE KEY----xxxx-----END PRIVATE KEY-----\n"	The private key in the JSON file generated at the creation of the meta service account.
gcp.gcs.impersonation_service_account	""	"hello"	The data service account that you want to impersonate.