- Introduction
- Quick Start
- Table Design
- Data Loading
- Data Export
- Using StarRocks
- Reference
- SQL Reference
- User Account Management
- Cluster Management
- ADMIN CANCEL REPAIR
- ADMIN CHECK TABLET
- ADMIN REPAIR
- ADMIN SET CONFIG
- ADMIN SET REPLICA STATUS
- ADMIN SHOW CONFIG
- ADMIN SHOW REPLICA DISTRIBUTION
- ADMIN SHOW REPLICA STATUS
- ALTER SYSTEM
- CANCEL DECOMMISSION
- CREATE FILE
- DROP FILE
- INSTALL PLUGIN
- SHOW BACKENDS
- SHOW BROKER
- SHOW FILE
- SHOW FRONTENDS
- SHOW FULL COLUMNS
- SHOW INDEX
- SHOW PLUGINS
- SHOW TABLE STATUS
- UNINSTALL PLUGIN
- DDL
- ALTER DATABASE
- ALTER TABLE
- ALTER VIEW
- BACKUP
- CANCEL BACKUP
- CANCEL RESTORE
- CREATE DATABASE
- CREATE INDEX
- CREATE MATERIALIZED VIEW
- CREATE REPOSITORY
- CREATE RESOURCE
- CREATE TABLE AS SELECT
- CREATE TABLE LIKE
- CREATE TABLE
- CREATE VIEW
- CREATE FUNCTION
- DROP DATABASE
- DROP INDEX
- DROP MATERIALIZED VIEW
- DROP REPOSITORY
- DROP RESOURCE
- DROP TABLE
- DROP VIEW
- DROP FUNCTION
- HLL
- RECOVER
- RESTORE
- SHOW RESOURCES
- SHOW FUNCTION
- TRUNCATE TABLE
- DML
- ALTER ROUTINE LOAD
- BROKER LOAD
- CANCEL LOAD
- DELETE
- EXPORT
- GROUP BY
- INSERT
- PAUSE ROUTINE LOAD
- RESUME ROUTINE LOAD
- ROUTINE LOAD
- SELECT
- SHOW ALTER
- SHOW BACKUP
- SHOW DATA
- SHOW DATABASES
- SHOW DELETE
- SHOW DYNAMIC PARTITION TABLES
- SHOW EXPORT
- SHOW LOAD
- SHOW PARTITIONS
- SHOW PROPERTY
- SHOW REPOSITORIES
- SHOW RESTORE
- SHOW ROUTINE LOAD
- SHOW ROUTINE LOAD TASK
- SHOW SNAPSHOT
- SHOW TABLES
- SHOW TABLET
- SHOW TRANSACTION
- SPARK LOAD
- STOP ROUTINE LOAD
- STREAM LOAD
- Data Type
- Auxiliary Commands
- Function Reference
- Date Functions
- Geographic Functions
- String Functions
- Aggregation Functions
- Bitmap Functions
- Array Functions
- cast function
- hash function
- Crytographic Functions
- Math Functions
- Utility Functions
- System variables
- Error code
- System limits
- SQL Reference
- Administration
- FAQ
- Deployment
- Data Migration
- SQL
- Others FAQs
- Benchmark
- Developers
- Contribute to StarRocks
- Code Style Guides
- Use the debuginfo file for debugging
- Development Environment
- Trace Tools
Edit
Memory Management
This section briefly introduces memory classification and StarRocks’ methods of managing memory.
Memory Classification
Explanation:
| Indicator | Name | Description |
|---|---|---|
| process | Total memory used of BE | |
| query_pool | Querying memory | Consists of two parts: (1) memory used by the execution layer (2) memory used by the storage layer. |
| load | Importing memory | Generally MemTable |
| table_meta | Metadata memory | S Schema, Tablet metadata, RowSet metadata, Column metadata, ColumnReader, IndexReader |
| compaction | Multi-version memory compaction | compaction that happens after data import is complete |
| snapshot | Snapshot memory | Generally used for clone, little memory usage |
| column_pool | Column pool memory | Request to release column cache for accelerated column |
| page_cache | BE's own PageCache | The default is off, the user can turn it on by modifying the BE file |
Memory-related configuration
- BE Configuration
| Name | Default | Description |
|---|---|---|
| vector_chunk_size | 4096 | Number of chunk rows |
| tc_use_memory_min | 10737418240 | minimum reserved memory for TCmalloc, exceeding which StarRocks will return free memory to the operating system |
| mem_limit | 80% | The percentage of total memory that BE can use. If BE is deployed as a standalone, there is no need to configure it. If it is deployed with other services that consume more memory, it should be configured separately. |
| disable_storage_page_cache | true | Whether to enable the storage limit of PageCachestorage_page_cache_limit0PageCache |
| write_buffer_size | 104857600 | The capacity limit of a single MemTable, exceeding which a disk swipe will be performed. |
| load_process_max_memory_limit_bytes | 107374182400 | The total import memory limit min(mem_limit * load_process_max_memory_limit_bytes, load_process_max_memory_limit_bytes). It is the actual available import memory threshold, reaching which a disk swipe will be triggered. |
| load_process_max_memory_limit_percent | 30 | The total import memory limit min(mem_limit * load_process_max_memory_limit_percent, load_process_max_memory_limit_bytes). It is the actual available import memory threshold, reaching which the swipe will be triggered. |
| default_load_mem_limit | 2147483648 | If the memory limit on the receiving side is reached for a single import instance, a disk swipe will be triggered. This needs to be modified with the Session variable load_mem_limit to take effect. |
| max_compaction_concurrency | 10 | Disable the compaction |
| cumulative_compaction_check_interval_seconds | 1 | Interval of compaction check |
- Session variables
| Name | Default | Description |
|---|---|---|
| exec_mem_limit | 2147483648 | Memory limit of a single instance |
| load_mem_limit | 0 | Memory limit of a single import task. If the value is 0, exec_mem_limit will be taken |
View memory usage
- mem_tracker
//View the overall memory statistics
<http://be_ip:be_web_port/mem_tracker>
// View fine-grained memory statistics
<http://be_ip:be_web_port/mem_tracker?type=query_pool&upper_level=3>- tcmalloc
<http://be_ip:be_web_port/memz>------------------------------------------------
MALLOC: 777276768 ( 741.3 MiB) Bytes in use by application
MALLOC: + 8851890176 ( 8441.8 MiB) Bytes in page heap freelist
MALLOC: + 143722232 ( 137.1 MiB) Bytes in central cache freelist
MALLOC: + 21869824 ( 20.9 MiB) Bytes in transfer cache freelist
MALLOC: + 832509608 ( 793.9 MiB) Bytes in thread cache freelists
MALLOC: + 58195968 ( 55.5 MiB) Bytes in malloc metadata
MALLOC: ------------
MALLOC: = 10685464576 (10190.5 MiB) Actual memory used (physical + swap)
MALLOC: + 25231564800 (24062.7 MiB) Bytes released to OS (aka unmapped)
MALLOC: ------------
MALLOC: = 35917029376 (34253.1 MiB) Virtual address space used
MALLOC:
MALLOC: 112388 Spans in use
MALLOC: 335 Thread heaps in use
MALLOC: 8192 Tcmalloc page size
------------------------------------------------
Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()).
Bytes released to the OS take up virtual address space but no physical memory.The memory queried by this method is accurate. However, some memory in StarRocks is reserved but not in use. TcMalloc counts the memory that is reserved, not the memory used.
Here Bytes in use by application refers to the memory currently in use.
- metrics
curl -XGET http://be_ip:be_web_port/metrics | grep 'mem'
curl -XGET http://be_ip:be_web_port/metrics | grep 'column_pool'The value of metrics is updated every 10 seconds. It is possible to monitor some of the memory statistics with older versions.