oom even if set menory

[elderbig]

databend version

mysql> status
--------------
mysql  Ver 8.0.31-0ubuntu0.22.04.1 for Linux on x86_64 ((Ubuntu))

Connection id:		43
Current database:	gh_log
Current user:		'root'@'127.0.0.1'
SSL:			Not in use
Current pager:		stdout
Using outfile:		''
Using delimiter:	;
Server version:		8.0.26-v0.8.107-nightly-9b25785(rust-1.67.0-nightly-2022-11-10T16:28:51.106788692Z) 0
Protocol version:	10
Connection:		127.0.0.1 via TCP/IP
Server characterset:	0
Db     characterset:	0
Client characterset:	0
Conn.  characterset:	0
TCP port:		3307
Binary data as:		Hexadecimal

config query

table_engine_memory_enabled = true
database_engine_github_enabled = true

table_cache_enabled = true
table_memory_cache_mb_size = 1024
table_disk_cache_root = "_cache"
table_disk_cache_mb_size = 1024
table_cache_bloom_index_meta_count=3000
table_cache_bloom_index_data_bytes=1073741824

killed by os on ubuntu 22.0.4 when oom

Out of memory: Killed process 3481 (databend-query) total-vm:25368832kB, anon-rss:5251212kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:45556kB oom_score_adj:0

what is wrong?

[elderbig]

My compute has 8G memory and 2G swap space,when I input gharchive using curl XPUT method, It seems memory usage by databend-query grow up slowly,and killed by os in a long time late,maybe 20 hours, when oom came, anyone meet this ?
Next, I want collect VmStatus for more detail about memory usage.

[Xuanwo]

cc @youngsofun for a look.

[elderbig]

log databend-query memory usage :

while true; do cat /proc/385287/status|awk   -v t="$(date +%Y-%m-%d:%H:%M:%S)" '/Vm/{ print t " "$0}'; sleep 5; done>>memory_usage.log

head and tail of memory_usage.log file bellow:

2022-11-13:17:48:24 VmPeak:	10649796 kB
2022-11-13:17:48:24 VmSize:	10647752 kB
2022-11-13:17:48:24 VmLck:	       0 kB
2022-11-13:17:48:24 VmPin:	       0 kB
2022-11-13:17:48:24 VmHWM:	 3068340 kB
2022-11-13:17:48:24 VmRSS:	 2046408 kB
2022-11-13:17:48:24 VmData:	10563364 kB
2022-11-13:17:48:24 VmStk:	     132 kB
2022-11-13:17:48:24 VmExe:	   83452 kB
2022-11-13:17:48:24 VmLib:	       8 kB
2022-11-13:17:48:24 VmPTE:	   19596 kB
2022-11-13:17:48:24 VmSwap:	   24416 kB

.........

2022-11-13:18:40:52 VmPeak:	18857348 kB
2022-11-13:18:40:52 VmSize:	18857348 kB
2022-11-13:18:40:52 VmLck:	       0 kB
2022-11-13:18:40:52 VmPin:	       0 kB
2022-11-13:18:40:52 VmHWM:	 4719080 kB
2022-11-13:18:40:52 VmRSS:	 4454988 kB
2022-11-13:18:40:52 VmData:	18554536 kB
2022-11-13:18:40:52 VmStk:	     132 kB
2022-11-13:18:40:52 VmExe:	   83452 kB
2022-11-13:18:40:52 VmLib:	       8 kB
2022-11-13:18:40:52 VmPTE:	   32992 kB
2022-11-13:18:40:52 VmSwap:	  262056 kB

oom came more earler this time, less 1 hour,RSS from 2G up to 4.7G, VmData segment from 10G up to 18.5G.

[elderbig]

upload gharchive for 2022 using curl in a loop for every hour,each command like this:

curl -XPUT 'http://root:@127.0.0.1:8000/v1/streaming_load' -H 'insert_sql: insert into gh_log.github_data format NDJson' -H 'format_compression:auto' -H 'format_skip_header: 0'  -F '[email protected]'

[youngsofun]

can it be produced with uploading the same file 2022-10-31-22.json.gz many times in a loop?

[BohuTANG]

My compute has 8G memory and 2G swap space

For small memory sizes, XPUT may not work stable, because databend-query uses parallel streaming load internally, and each worker's memory usage is not easy to control.
Try set global max_threads=2 to make 2 parallel workers, this may be in a low CPU usage but not OOM.

[elderbig]

set global max_threads=2 by environment or config file?
I set global max_threads in mysql client, it seems do work.
Is there possible databend adjust max_threads or other things automatically to survive self from oom?if databend run in product environment, we must do something prevent oom,all of all, memory resource is hard limit in any machine.

[BohuTANG]

It's possible and we are working on it

[elderbig]

Thanks for your reply.
Data input successfully on set global max_threads=2, memory usage by databend-query in the end:

2022-11-14:07:49:31 VmPeak:	10304660 kB
2022-11-14:07:49:31 VmSize:	10265128 kB
2022-11-14:07:49:31 VmLck:	       0 kB
2022-11-14:07:49:31 VmPin:	       0 kB
2022-11-14:07:49:31 VmHWM:	 4292200 kB
2022-11-14:07:49:31 VmRSS:	 2735472 kB
2022-11-14:07:49:31 VmData:	 9962520 kB
2022-11-14:07:49:31 VmStk:	     132 kB
2022-11-14:07:49:31 VmExe:	   83452 kB
2022-11-14:07:49:31 VmLib:	       8 kB
2022-11-14:07:49:31 VmPTE:	   18692 kB
2022-11-14:07:49:31 VmSwap:	  166156 kB

here is some data file in gz:

/gharchive$ ls -lh  *.json.gz|tail -10
-rwxrwxrwx 1 sh sh 103M 11月  1 07:05 2022-10-31-22.json.gz
-rwxrwxrwx 1 sh sh  92M 11月  1 08:05 2022-10-31-23.json.gz
-rwxrwxrwx 1 sh sh  79M 10月 31 11:05 2022-10-31-2.json.gz
-rwxrwxrwx 1 sh sh  87M 10月 31 12:05 2022-10-31-3.json.gz
-rwxrwxrwx 1 sh sh  84M 10月 31 13:05 2022-10-31-4.json.gz
-rwxrwxrwx 1 sh sh  79M 10月 31 14:05 2022-10-31-5.json.gz
-rwxrwxrwx 1 sh sh  89M 10月 31 15:05 2022-10-31-6.json.gz
-rwxrwxrwx 1 sh sh  97M 10月 31 16:05 2022-10-31-7.json.gz
-rwxrwxrwx 1 sh sh 106M 10月 31 17:05 2022-10-31-8.json.gz
-rwxrwxrwx 1 sh sh 113M 10月 31 18:05 2022-10-31-9.json.gz

Memory used may be a little more, because this data can be put into a data stream,I thought!
other wise, if curl put request run in client block mode, should cut down threads and rescue memory for it's self, when only on client is exists,parallel or background processing is unnecessary .

[BohuTANG]

Now databend-query has soft OOM and it works well, you can try the latest version:
https://docs.databend.com/download