Skip to main content
Version: Latest-3.5

TPC-DS Benchmarking

TPC-DS is a decision support benchmark developed by the Transaction Processing Performance Council (TPC). It uses more comprehensive test datasets and complex SQL queries than TPC-H.

TPC-DS models several generally applicable aspects of a decision support system, including queries and data maintenance. TPC-DS aims to provide a comprehensive and realistic workload for testing and evaluating the performance of database systems in a retail environment. The TPC-DS benchmark simulates the sales and return data of three sales channels (stores, Internet, and catalog) in a retail enterprise. In addition to creating tables for sales and return data models, it also includes a simple inventory system and a promotion system.

This benchmark tests a total of 99 complex SQL queries against 24 tables whose data size ranges from 1 GB to 3 TB. The main performance metric is the response time of each query, which is the duration between the time a query is submitted to the time the result is returned.

Test Conclusions​

The test is performed against the TPC-DS 1 TB dataset on StarRocks and Trino. The unit of the following results are milliseconds.

TPC-DS-SR

For StarRocks, the test queries are performed on its native tables (under both shared-nothing and shared-data architecture) and Iceberg Catalog (under shared-data architecture). For Trino, the test queries are performed on the same Iceberg Catalog. Both tests for StarRocks and Trino on the Iceberg Catalog use AWS Glue as the metastore, and Parquet-formatted data with ZSTD compression.

The test results are:

  • Queries performed on the OLAP table in the StarRocks shared-nothing cluster took 314 seconds.
  • Queries performed on the OLAP table in the StarRocks shared-data cluster took 326 seconds.
  • Queries performed on Iceberg Catalog in the StarRocks shared-data cluster took 368 seconds.
  • Queries performed on Iceberg Catalog in Trino took 2552 seconds.

The conclusions are:

  • StarRocks delivers comparable performance in both shared-nothing and shared-data architecture

    In OLAP table query scenarios, StarRocks in the shared-nothing mode achieves 1.04× the performance of the shared-data mode, indicating minimal performance overhead from storage-compute separation.

  • StarRocks significantly outperforms Trino in Lakehouse query scenarios out of the box, with even greater advantages in OLAP table queries

    • In Iceberg Catalog query scenarios, StarRocks delivers 6.93× the performance of Trino.
    • In OLAP table queries, the performance gap widens further—StarRocks achieves 8.13× (under shared-nothing mode) and 7.82× (under shared-data mode) the performance of Trino.

Test Preparation​

Hardware​

StarRocks and Trino are deployed on hosts of the same configurations - AWS m6id.4xlarge.

Spec
Instance Number5
vCPU16
Memory (GiB)64
Instance Storage (GB)1x950 NVMe SSD
Network Bandwidth (Gbps)Up to 12.5
EBS Bandwidth (Gbps)Up to 10

Software​

StarRocksTrino
Cluster SizeOne FE, four BEOne Coordinator, four Workers
Version3.5.0475
Release Date2025.6.132025.6.5
Configurationset query_mem_limit=66571993088; (62 GB)query.max-memory=224GB
query.max-memory-per-node=35GB

Test results​

The following table shows the performance test results on 99 queries. The unit of query latency is milliseconds. All queries are warmed up 1 time and then executed 3 times to take the average value as the result. Trino vs StarRocks on Iceberg in the table header means using the query response time of Trino to divide the query response time of StarRocks. A larger value indicates better performance of StarRocks. Before the test on Iceberg Catalog, ANALYZE TABLE and EXPLAIN COST statements are executed on StarRocks to collect statistics.

QueryStarRocksTrinoTrino vs StarRocks on Iceberg
OLAP TableIceberg CatalogIceberg Catalog
Shared-nothingShared-dataShare-data
SUM31358232616336840425520766.93
Q0167563672722763.13
Q0293592519492128910.92
Q0322014039917644.42
Q0417185162061674420024911.96
Q05417543172455743.23
Q0620018034329098.48
Q07814959179536382.03
Q0823518131925347.94
Q097097650323153361251.56
Q1029734349724714.97
Q111122910249977713323013.63
Q1212813020710675.15
Q134174472218182978.25
Q141511215055153101431729.35
Q1545547452521584.11
Q164765239751403514.39
Q17895948164554873.34
Q18802973154447363.07
Q1920926850718703.69
Q2014717625312574.97
Q2183804059452.33
Q2214621823238478713.30
Q238288784426507194886539.63
Q247649906411902434363.65
Q25795919140443473.10
Q2637651186829613.41
Q27458532139742193.02
Q286489600418727246781.32
Q29129621682129106405.00
Q3029533052927705.24
Q31217221622535102604.05
Q3211022527710353.74
Q3323524573427433.74
Q3440242269624553.53
Q3513531393156280235.14
Q36553559119333422.80
Q37263243217617828.47
Q38500252075519384086.96
Q3935636782056966.95
Q4012413457515542.70
Q414350762713.57
Q42811041979965.06
Q4348773372241085.69
Q44208721029364164681.76
Q4543147051422124.30
Q4614681829197647802.42
Q472921312031304270413.64
Q484675731441124648.65
Q49496547171538272.23
Q50376851353899217715.58
Q51482646333244122473.78
Q528710621910074.60
Q5344663595517681.85
Q541982667633186441.76
Q558210522111875.37
Q5617518649921334.27
Q571848178719162604213.59
Q5824124550330326.03
Q59393539083160208196.59
Q6026229264327494.28
Q6133034491320542.25
Q6250051984051046.08
Q6344862993717191.83
Q64450888868602169901.98
Q65569056175445206163.79
Q66387390133535112.63
Q67312862924021439894304.17
Q6824034168328834.22
Q6926029250121814.35
Q70290430123751214995.73
Q71272111163125784.09
Q72164218994969504573101.54
Q7318622838819334.98
Q74943391338890780718.78
Q75644677189082262702.89
Q76235623315028112662.24
Q7723824170127853.97
Q78175831659018614514972.77
Q7922012474159755723.49
Q80800855267247621.78
Q8151853376940405.25
Q826236138541162213.61
Q8315821740422745.63
Q8419543435130868.79
Q85455733108754244.99
Q8672573681934224.18
Q87481650045535413007.46
Q888712844419465307381.58
Q8953073091522272.43
Q90587581115038903.38
Q91959622720358.96
Q92781402289944.36
Q93372347055781229263.97
Q9460196396258846.12
Q951747310625752960711.50
Q9614441428256249051.91
Q97479548445383227264.22
Q9828432336519115.24
Q99113211841594109706.88