Query Profile Structure and Metrics
Structure of Query Profile
The structure of a Query Profile is closely related to the design of StarRocks' execution engine and consists of the following five parts:
- Fragment: Execution tree. A query is composed of one or more fragments.
- FragmentInstance: Each fragment can have multiple instances, each instance is called a FragmentInstance, and is executed by different computing nodes.
- Pipeline: A FragmentInstance is split into multiple pipelines. A pipeline is an execution chain consisting of a group of connected Operator instances.
- PipelineDriver: A Pipeline can have multiple instances, each instance is called a PipelineDriver, to fully utilize multiple computing cores.
- Operator: A PipelineDriver consists of multiple Operator instances.
Query Profile Merging Strategy
By analyzing the structure of Query Profile, you can easily observe that multiple FragmentInstances associated with the same Fragment have a high degree of similarity in structure. Similarly, multiple PipelineDrivers belonging to the same Pipeline also exhibit similar structural features. To reduce the volume of the Query Profile, StarRocks by default merges the FragmentInstance layer and the PipelineDriver layer. As a result, the original five-layer structure is simplified to three layers:
- Fragment
- Pipeline
- Operator
You can control this merging behavior through a session variable pipeline_profile_level
, which has two valid values:
1
(Default): StarRocks merges the metrics into a three-layer structure.2
: StarRocks does not merge the metrics. The original five-layer structure is retained.- Any other value will be treated as the default value
1
.
Generally, we do not recommend setting this parameter to 2
because the Query Profile with the five-layer structure has many limitations. For example, you cannot perform visualized analysis on the profile using any tools. Therefore, unless the merging process leads to the loss of crucial information, you do not need to adjust this parameter.
Metric Merging and MIN/MAX Values
When merging FragmentInstance and PipelineDriver, all metrics with the same name are merged. Only the minimum and maximum values of each metric in all concurrent instances are recorded. Different types of metrics use different merging strategies:
- Time-related metrics take the average. For example:
OperatorTotalTime
is the average time consumption of all concurrent instances.__MAX_OF_OperatorTotalTime
is the maximum time consumption among all concurrent instances.__MIN_OF_OperatorTotalTime
is the minimum time consumption among all concurrent instances.
- OperatorTotalTime: 2.192us
- __MAX_OF_OperatorTotalTime: 2.502us
- __MIN_OF_OperatorTotalTime: 1.882us
- Non-time-related metrics are summed. For example:
PullChunkNum
is the sum of this metric in all concurrent instances.__MAX_OF_PullChunkNum
is the maximum value of this metric among all concurrent instances.__MIN_OF_PullChunkNum
is the minimum value of this metric among all concurrent instances.
- PullChunkNum: 146.66K (146660)
- __MAX_OF_PullChunkNum: 24.45K (24450)
- __MIN_OF_PullChunkNum: 24.435K (24435)
- Some metrics without extreme values have the same value in all concurrent instances, for example:
DegreeOfParallelism
.
Usually, if there is a significant difference between MIN and MAX values, it indicates a high probability of data skew. Possible scenarios include aggregation and join operations.
- OperatorTotalTime: 2m48s
- __MAX_OF_OperatorTotalTime: 10m30s
- __MIN_OF_OperatorTotalTime: 279.170us
Query Profile Metrics List
The Query Profile includes a multitude of metrics providing detailed information about query execution. In most cases, you only need to focus on the execution time of operators and the size of processed data. Once you identify bottlenecks, you can address them specifically.
Summary Metrics
Total
Description: The total time consumed by the query, including Planning, Executing, and Profiling phase durations.
Query State
Description: Query state, possible states include Finished, Error, and Running.
Execution Overview Metrics
FrontendProfileMergeTime
Description: Query Profile processing time on the Frontend (FE) side.
QueryAllocatedMemoryUsage
Description: Cumulative allocated memory across all compute nodes.
QueryDeallocatedMemoryUsage
Description: Cumulative deallocated memory across all compute nodes.
QueryPeakMemoryUsagePerNode
Description: Maximum peak memory across all compute nodes.
QuerySumMemoryUsage
Description: Summary of peak memory across all compute nodes.
QueryExecutionWallTime
Description: Wall time of the execution.
QueryCumulativeCpuTime
Description: Cumulative CPU time across all compute nodes.
QueryCumulativeOperatorTime
Description: Cumulative time across all nodes. This is a simple linear accumulation, but in reality, execution times of different operators may overlap. This parameter serves as the denominator for calculating the percentage of time spent on each operator.
QueryCumulativeNetworkTime
Description: Cumulative network time of all Exchange nodes. Similar to cumulative operator time, actual execution times of different Exchanges may overlap.
QueryCumulativeScanTime
Description: Cumulative IO time of all Scan nodes. Similar to cumulative operator time, actual execution times of different Scan operations may overlap.
QueryPeakScheduleTime
Description: Maximum ScheduleTime metric across all Pipelines.
QuerySpillBytes
Description: Size of data spilled to local disks.
ResultDeliverTime
Description: Additional time to transfer results. For query statements, this parameter refers to the time it takes to send data back to the client; for insert statements, it refers to the time it takes to write data to the storage layer.
Fragment Metrics
InstanceNum
Description: Number of all FragmentInstances for this Fragment.
InstanceIds
Description: IDs of all FragmentInstances for this Fragment.
BackendNum
Description: Number of BEs participating in the execution of this Fragment.
BackendAddresses
Description: Addresses of all BEs participating in the execution of this Fragment.
FragmentInstancePrepareTime
Description: Time spent in the Fragment Prepare phase.
InstanceAllocatedMemoryUsage
Description: Cumulative allocated memory for all FragmentInstances under this Fragment.
InstanceDeallocatedMemoryUsage
Description: Cumulative deallocated memory for all FragmentInstances under this Fragment.
InstancePeakMemoryUsage
Description: The peak memory usage across all FragmentInstances under this Fragment.
Pipeline Metrics
The relationship between core metrics is illustrated in the following diagram:
- DriverTotalTime = ActiveTime + PendingTime + ScheduleTime
- ActiveTime = ∑ OperatorTotalTime + OverheadTime
- PendingTime = InputEmptyTime + OutputFullTime + PreconditionBlockTime + PendingFinishTime
- InputEmptyTime = FirstInputEmptyTime + FollowupInputEmptyTime
DegreeOfParallelism
Description: Degree of pipeline execution parallelism.
TotalDegreeOfParallelism
Description: Sum of degrees of parallelism. Since the same Pipeline may execute on multiple machines, this item aggregates all values.
DriverPrepareTime
Description: Time taken by the Prepare phase. This metric is not included in DriverTotalTime.
DriverTotalTime
Description: Total execution time of the Pipeline, excluding the time spent in the Prepare phase.
ActiveTime
Description: Execution time of the Pipeline, including the execution time of each operator and the overall framework overhead, such as time spent in invoking methods like has_output, need_input, etc.
PendingTime
Description: Time the Pipeline is blocked from being scheduled for various reasons.
InputEmptyTime
Description: Time the Pipeline is blocked due to an empty input queue.
FirstInputEmptyTime
Description: Time the Pipeline is first blocked due to an empty input queue. The first blocking time is separately calculated because the first blocking is mainly caused by Pipeline dependencies.
FollowupInputEmptyTime
Description: Time the Pipeline is subsequently blocked due to an empty input queue.
OutputFullTime
Description: Time the Pipeline is blocked due to a full output queue.
PreconditionBlockTime
Description: Time the Pipeline is blocked due to unmet dependencies.
PendingFinishTime
Description: Time the Pipeline is blocked waiting for asynchronous tasks to finish.
ScheduleTime
Description: Scheduling time of the Pipeline, from entering the ready queue to being scheduled for execution.
BlockByInputEmpty
Description: Number of times the pipeline is blocked due to InputEmpty.
BlockByOutputFull
Description: Number of times the pipeline is blocked due to OutputFull.
BlockByPrecondition
Description: Number of times the pipeline is blocked due to unmet preconditions.
Operator General Metrics
PrepareTime
Description: Time spent on preparation.
OperatorTotalTime
Description: Total time consumed by the Operator. It satisfies the equation: OperatorTotalTime = PullTotalTime + PushTotalTime + SetFinishingTime + SetFinishedTime + CloseTime. It excludes time spent on preparation.
PullTotalTime
Description: Total time the Operator spends executing push_chunk.
PushTotalTime
Description: Total time the Operator spends executing pull_chunk.
SetFinishingTime
Description: Total time the Operator spends executing set_finishing.
SetFinishedTime
Description: Total time the Operator spends executing set_finished.
PushRowNum
Description: Cumulative number of input rows for the Operator.
PullRowNum
Description: Cumulative number of output rows for the Operator.
JoinRuntimeFilterEvaluate
Description: Number of times Join Runtime Filter is evaluated.
JoinRuntimeFilterHashTime
Description: Time spent computing hash for Join Runtime Filter.
JoinRuntimeFilterInputRows
Description: Number of input rows for Join Runtime Filter.
JoinRuntimeFilterOutputRows
Description: Number of output rows for Join Runtime Filter.
JoinRuntimeFilterTime
Description: Time spent on Join Runtime Filter.
Unique Metrics
Scan Operator
The Scan Operator utilizes an additional thread pool for executing IO tasks. Therefore, the relationship between time metrics for this node is illustrated below:
OLAP Scan Operator
To facilitate a better understanding of the various metrics within the Scan Operator, the following diagram demonstrates the associations between these metrics and storage structures.
Table
- Description: Table name.
- Level: Primary metric
Rollup
- Description: Materialized view name. If no materialized view is hit, it is equivalent to the table name.
- Level: Primary metric
SharedScan
- Description: Whether the enable_shared_scan session variable is enabled.
- Level: Primary metric
TabletCount
- Description: Number of tablets.
- Level:
Primary metric
MorselsCount
- Description: Number of morsels.
- Level: Primary metric
PushdownPredicates
- Description: Number of pushdown predicates.
- Level: Primary metric
Predicates
- Description: Predicate expressions.
- Level: Primary metric
BytesRead
- Description: Size of data read.
- Level: Primary metric
CompressedBytesRead
- Description: Size of compressed data read from disk.
- Level: Primary metric
UncompressedBytesRead
- Description: Size of uncompressed data read from disk.
- Level: Primary metric
RowsRead
- Description: Number of rows read (after predicate filtering).
- Level: Primary metric
RawRowsRead
- Description: Number of raw rows read (before predicate filtering).
- Level: Primary metric
ReadPagesNum
- Description: Number of pages read.
- Level: Primary metric
CachedPagesNum
- Description: Number of cached pages.
- Level: Primary metric
ChunkBufferCapacity
- Description: Capacity of the Chunk Buffer.
- Level: Primary metric
DefaultChunkBufferCapacity
- Description: Default capacity of the Chunk Buffer.
- Level: Primary metric
PeakChunkBufferMemoryUsage
- Description: Peak memory usage of the Chunk Buffer.
- Level: Primary metric
PeakChunkBufferSize
- Description: Peak size of the Chunk Buffer.
- Level: Primary metric
PrepareChunkSourceTime
- Description: Time spent preparing the Chunk Source.
- Level: Primary metric
ScanTime
- Description: Cumulative scan time. Scan operations are completed in an asynchronous I/O thread pool.
- Level: Primary metric
IOTaskExecTime
- Description: Execution time of IO tasks.
- Level: Primary metric
- Sub-metrics: CreateSegmentIter, DictDecode, GetDelVec, GetDeltaColumnGroup, GetRowsets, IOTime, LateMaterialize, ReadPKIndex, SegmentInit, SegmentRead
CreateSegmentIter
- Description: Time spent creating the Segment Iterator.
- Level: Secondary metric
DictDecode
- Description: Time spent on decoding dictionary for low cardinality optimization.
- Level: Secondary metric
GetDelVec
- Description: Time spent loading DelVec (delete vector).
- Level: Secondary metric
GetDeltaColumnGroup
- Description: Time spent loading DelVecColumnGroup.
- Level: Secondary metric
GetRowsets
- Description: Time spent loading RowSet.
- Level: Secondary metric
IOTime
- Description: Time spent on file I/O.
- Level: Secondary metric
LateMaterialize
- Description: Time spent on late materialization.
- Level: Secondary metric
ReadPKIndex
- Description: Time spent reading Primary Key index.
- Level: Secondary metric
SegmentInit
- Description: Time spent initializing the Segment.
- Level: Secondary metric
- Sub-metrics: BitmapIndexFilter, BitmapIndexFilterRows, BloomFilterFilter, BloomFilterFilterRows, ColumnIteratorInit, ShortKeyFilter, ShortKeyFilterRows, ShortKeyRangeNumber, RemainingRowsAfterShortKeyFilter, ZoneMapIndexFilter, ZoneMapIndexFilterRows, SegmentZoneMapFilterRows, SegmentRuntimeZoneMapFilterRows
BitmapIndexFilter
- Description: Time spent on Bitmap index filtering.
- Level: Tertiary metric
BitmapIndexFilterRows
- Description: Number of rows filtered by Bitmap index.
- Level: Tertiary metric
BloomFilterFilter
- Description: Time spent on Bloom filter indexing.
- Level: Tertiary metric
BloomFilterFilterRows
- Description: Number of rows filtered by Bloom filter.
- Level: Tertiary metric
ColumnIteratorInit
- Description: Time spent initializing the Column Iterator.
- Level: Tertiary metric
ShortKeyFilter
- Description: Time spent on ShortKey index filtering.
- Level: Tertiary metric
ShortKeyFilterRows
- Description: Number of rows filtered by ShortKey index.
- Level: Tertiary metric
ShortKeyRangeNumber
- Description: Number of ShortKey ranges.
- Level: Tertiary metric
RemainingRowsAfterShortKeyFilter
- Description: Number of rows remaining after ShortKey index filtering.
- Level: Tertiary metric
ZoneMapIndexFiter
- Description: Time spent on ZoneMap index filtering.
- Level: Tertiary metric
ZoneMapIndexFilterRows
- Description: Number of rows filtered by ZoneMap index.
- Level: Tertiary metric
SegmentZoneMapFilterRows
- Description: Number of rows filtered by Segment ZoneMap index.
- Level: Tertiary metric
SegmentRuntimeZoneMapFilterRows
- Description: Number of rows filtered by Segment Runtime ZoneMap index.
- Level: Tertiary metric
SegmentRead
- Description: Time spent reading the Segment.
- Level: Secondary metric
- Sub-metrics: BlockFetch, BlockFetchCount, BlockSeek, BlockSeekCount, ChunkCopy, DecompressT, DelVecFilterRows, PredFilter, PredFilterRows, RowsetsReadCount, SegmentsReadCount, TotalColumnsDataPageCount
BlockFetch
- Description: Time spent on Block fetch.
- Level: Tertiary metric
BlockFetchCount
- Description: Number of Block fetches.
- Level: Tertiary metric
BlockSeek
- Description: Time spent on Block seek.
- Level: Tertiary metric
BlockSeekCount
- Description: Number of Block seeks.
- Level: Tertiary metric
ChunkCopy
- Description: Time spent on Chunk copy.
- Level: Tertiary metric
DecompressT
- Description: Time spent on decompression.
- Level: Tertiary metric
DelVecFilterRows
- Description: Number of rows filtered by DELETE vector.
- Level: Tertiary metric
PredFilter
- Description: Time spent on predicate filtering.
- Level: Tertiary metric
PredFilterRows
- Description: Number of rows filtered by predicate.
- Level: Tertiary metric