[Spce-user] Influxdb spiking CPU usage

[Dian van Antwerpen]

Hello Everyone,

Apologies if any mistakes in logging this I'm new to the mailing list. I am
experiencing issues where our sip servers are getting high load due to
influxdb causing spikes in CPU usage. This is a recurring issue with no
real pattern.

Both machines are on hosted and have the following assigned:
2 CPU's sharing 4GHz
4GB RAM
60GB disk space

After stopping influxdb the CPU normalizes. A reboot does resolve this for
a few days then it acts up again. I did not configure this setup but I am
the one maintaining it now.

If I run grep influx /var/log/syslog I do see Out of memory reports, but
the box is not reporting any RAM issues if I run top command etc.

This is an output of the issue:
Jul 10 08:18:50 server2 systemd[1]: influxdb.service: Service
RestartSec=100ms expired, scheduling restart.
Jul 10 08:18:50 server2 systemd[1]: influxdb.service: Scheduled restart
job, restart counter is at 9370.
Jul 10 08:19:26 server2 kernel:[521238.794572] [18933]   108 18933  1062514
  151509     600       9        0             0 influxd
Jul 10 08:19:26 server2 kernel:[521238.794605] Out of memory: Kill process
18933 (influxd) score 150 or sacrifice child
Jul 10 08:19:26 server2 kernel:[521238.795841] Killed process 18933
(influxd) total-vm:4250056kB, anon-rss:606036kB, file-rss:0kB, shmem-rss:0kB
Jul 10 08:19:26 server2 systemd[1]: influxdb.service: Main process exited,
code=killed, status=9/KILL
Jul 10 08:19:26 server2 systemd[1]: influxdb.service: Failed with result
'signal'.
Jul 10 08:19:26 server2 kernel:[521239.028308] oom_reaper: reaped process
18933 (influxd), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Jul 10 08:19:26 server2 systemd[1]: influxdb.service: Service
RestartSec=100ms expired, scheduling restart.
Jul 10 08:19:26 server2 systemd[1]: influxdb.service: Scheduled restart
job, restart counter is at 9371.
Jul 10 08:19:51 server2 kernel:[521263.101876] [19105]   108 19105  1082726
  153351     594      10        0             0 influxd
Jul 10 08:19:51 server2 kernel:[521263.101883] Out of memory: Kill process
19105 (influxd) score 152 or sacrifice child
Jul 10 08:19:51 server2 kernel:[521263.103223] Killed process 19105
(influxd) total-vm:4330904kB, anon-rss:613404kB, file-rss:0kB, shmem-rss:0kB
Jul 10 08:19:51 server2 kernel:[521263.201835] oom_reaper: reaped process
19105 (influxd), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Jul 10 08:19:51 server2 systemd[1]: influxdb.service: Main process exited,
code=killed, status=9/KILL
Jul 10 08:19:51 server2 systemd[1]: influxdb.service: Failed with result
'signal'.
Jul 10 08:19:51 server2 systemd[1]: influxdb.service: Service
RestartSec=100ms expired, scheduling restart.
Jul 10 08:19:51 server2 systemd[1]: influxdb.service: Scheduled restart
job, restart counter is at 9372.
Jul 10 08:20:02 server2 kernel:[521274.267604] [19261]   108 19261   734490
  154879     502       8        0             0 influxd
Jul 10 08:20:02 server2 kernel:[521274.267614] Out of memory: Kill process
19261 (influxd) score 153 or sacrifice child
Jul 10 08:20:02 server2 kernel:[521274.268636] Killed process 19261
(influxd) total-vm:2937960kB, anon-rss:619516kB, file-rss:0kB, shmem-rss:0kB
Jul 10 08:20:02 server2 systemd[1]: influxdb.service: Main process exited,
code=killed, status=9/KILL
Jul 10 08:20:02 server2 systemd[1]: influxdb.service: Failed with result
'signal'.
Jul 10 08:20:02 server2 kernel:[521274.363562] oom_reaper: reaped process
19261 (influxd), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Jul 10 08:20:02 server2 systemd[1]: influxdb.service: Service
RestartSec=100ms expired, scheduling restart.
Jul 10 08:20:02 server2 systemd[1]: influxdb.service: Scheduled restart
job, restart counter is at 9373.

Kind regards,
Dian
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sipwise.com/pipermail/spce-user_lists.sipwise.com/attachments/20230710/e6690a26/attachment.html>