[Spce-user] Performance problem with ngcp-witnessd

[Henk]

Hi, we are having a performance problem with ngcp-witnessd. It starts 
after the cpu load is higher during nightly cron jobs like backup. Port 
9800 is opened multiple times (>100) like this:

root at spce:~# lsof -i TCP|  grep 9800
victoria-     607 _victoria-metrics   28u  IPv4 34459176 0t0  TCP 
localhost:45000->localhost:9800 (ESTABLISHED)
ngcp-witn 1300009              root    5u  IPv4 33636424 0t0  TCP 
localhost:9800 (LISTEN)
ngcp-witn 1300009              root    9u  IPv4 34442250 0t0  TCP 
localhost:9800->localhost:49954 (CLOSE_WAIT)
ngcp-witn 1300009              root   11u  IPv4 34442251 0t0  TCP 
localhost:9800->localhost:49964 (CLOSE_WAIT)
ngcp-witn 1300009              root   16u  IPv4 34442256 0t0  TCP 
localhost:9800->localhost:51952 (CLOSE_WAIT)
ngcp-witn 1300009              root   17u  IPv4 34442257 0t0  TCP 
localhost:9800->localhost:51958 (CLOSE_WAIT)
ngcp-witn 1300009              root   20u  IPv4 34442260 0t0  TCP 
localhost:9800->localhost:45752 (CLOSE_WAIT)
ngcp-witn 1300009              root   21u  IPv4 34442261 0t0  TCP 
localhost:9800->localhost:45764 (CLOSE_WAIT)
ngcp-witn 1300009              root   24u  IPv4 34442264 0t0  TCP 
localhost:9800->localhost:48518 (CLOSE_WAIT)
ngcp-witn 1300009              root   25u  IPv4 34442265 0t0  TCP 
localhost:9800->localhost:48520 (CLOSE_WAIT)
ngcp-witn 1300009              root   28u  IPv4 34442268 0t0  TCP 
localhost:9800->localhost:58756 (CLOSE_WAIT)
ngcp-witn 1300009              root   29u  IPv4 34442269 0t0  TCP 
localhost:9800->localhost:58758 (CLOSE_WAIT)
ngcp-witn 1300009              root   30u  IPv4 34442270 0t0  TCP 
localhost:9800->localhost:45948 (CLOSE_WAIT)
ngcp-witn 1300009              root   31u  IPv4 34442271 0t0  TCP 
localhost:9800->localhost:45954 (CLOSE_WAIT)
ngcp-witn 1300009              root   34u  IPv4 34442274 0t0  TCP 
localhost:9800->localhost:50588 (CLOSE_WAIT)
ngcp-witn 1300009              root   35u  IPv4 34442275 0t0  TCP 
localhost:9800->localhost:50590 (CLOSE_WAIT)
ngcp-witn 1300009              root   37u  IPv4 34442277 0t0  TCP 
localhost:9800->localhost:52274 (CLOSE_WAIT)
ngcp-witn 1300009              root   52u  IPv4 34442290 0t0  TCP 
localhost:9800->localhost:56316 (CLOSE_WAIT)
ngcp-witn 1300009              root   54u  IPv4 34442292 0t0  TCP 
localhost:9800->localhost:56322 (CLOSE_WAIT)
ngcp-witn 1300009              root   55u  IPv4 34442293 0t0  TCP 
localhost:9800->localhost:56332 (CLOSE_WAIT)
ngcp-witn 1300009              root   56u  IPv4 34442294 0t0  TCP 
localhost:9800->localhost:58530 (CLOSE_WAIT)
ngcp-witn 1300009              root   57u  IPv4 34442295 0t0  TCP 
localhost:9800->localhost:58532 (CLOSE_WAIT)
ngcp-witn 1300009              root   60u  IPv4 34442298 0t0  TCP 
localhost:9800->localhost:56404 (CLOSE_WAIT)
ngcp-witn 1300009              root   61u  IPv4 34442299 0t0  TCP 
localhost:9800->localhost:56414 (CLOSE_WAIT)
ngcp-witn 1300009              root   62u  IPv4 34442300 0t0  TCP 
localhost:9800->localhost:46702 (CLOSE_WAIT)
ngcp-witn 1300009              root   63u  IPv4 34442301 0t0  TCP 
localhost:9800->localhost:46708 (CLOSE_WAIT)
ngcp-witn 1300009              root   64u  IPv4 34442302 0t0  TCP 
localhost:9800->localhost:50426 (CLOSE_WAIT)
ngcp-witn 1300009              root   65u  IPv4 34442303 0t0  TCP 
localhost:9800->localhost:50432 (CLOSE_WAIT)
ngcp-witn 1300009              root   68u  IPv4 34442306 0t0  TCP 
localhost:9800->localhost:50944 (CLOSE_WAIT)
ngcp-witn 1300009              root   69u  IPv4 34442307 0t0  TCP 
localhost:9800->localhost:50956 (CLOSE_WAIT)
ngcp-witn 1300009              root   70u  IPv4 34442308 0t0  TCP 
localhost:9800->localhost:45806 (CLOSE_WAIT)
ngcp-witn 1300009              root   71u  IPv4 34442309 0t0  TCP 
localhost:9800->localhost:45812 (CLOSE_WAIT)
ngcp-witn 1300009              root   73u  IPv4 34442311 0t0  TCP 
localhost:9800->localhost:53530 (CLOSE_WAIT)
ngcp-witn 1300009              root   74u  IPv4 34442312 0t0  TCP 
localhost:9800->localhost:47184 (CLOSE_WAIT)

Restarting ngcp-witnessd fixes the problem temporary. The witnessd log 
doesn't give a clue, only some uninitialized values:

Dec 28 16:47:36 spce (warning) ngcp-witnessd[1548076]: Use of 
uninitialized value $lv in substitution (s///) at 
/usr/share/perl5/Prometheus/Tiny.pm line 28.
Dec 28 16:47:36 spce (warning) ngcp-witnessd[1548076]: Use of 
uninitialized value $lv in substitution (s///) at 
/usr/share/perl5/Prometheus/Tiny.pm line 29.
Dec 28 16:47:36 spce (warning) ngcp-witnessd[1548076]: Use of 
uninitialized value $lv in concatenation (.) or string at 
/usr/share/perl5/Prometheus/Tiny.pm line 30.
Dec 28 16:47:36 spce (warning) ngcp-witnessd[1548076]: Use of 
uninitialized value $lv in substitution (s///) at 
/usr/share/perl5/Prometheus/Tiny.pm line 28.
Dec 28 16:47:36 spce (warning) ngcp-witnessd[1548076]: Use of 
uninitialized value $lv in substitution (s///) at 
/usr/share/perl5/Prometheus/Tiny.pm line 29.
Dec 28 16:47:36 spce (warning) ngcp-witnessd[1548076]: Use of 
uninitialized value $lv in concatenation (.) or string at 
/usr/share/perl5/Prometheus/Tiny.pm line 30.
Dec 28 16:47:36 spce (warning) ngcp-witnessd[1548076]: Use of 
uninitialized value $lv in substitution (s///) at 
/usr/share/perl5/Prometheus/Tiny.pm line 28.
Dec 28 16:47:36 spce (warning) ngcp-witnessd[1548076]: Use of 
uninitialized value $lv in substitution (s///) at 
/usr/share/perl5/Prometheus/Tiny.pm line 29.
Dec 28 16:47:36 spce (warning) ngcp-witnessd[1548076]: Use of 
uninitialized value $lv in concatenation (.) or string at 
/usr/share/perl5/Prometheus/Tiny.pm line 30.
Dec 28 16:47:36 spce (warning) ngcp-witnessd[1548076]: Use of 
uninitialized value $lv in substitution (s///) at 
/usr/share/perl5/Prometheus/Tiny.pm line 28.
Dec 28 16:47:36 spce (warning) ngcp-witnessd[1548076]: Use of 
uninitialized value $lv in substitution (s///) at 
/usr/share/perl5/Prometheus/Tiny.pm line 29.
Dec 28 16:47:36 spce (warning) ngcp-witnessd[1548076]: Use of 
uninitialized value $lv in concatenation (.) or string at 
/usr/share/perl5/Prometheus/Tiny.pm line 30.

Sipwise version is  mr9.5.8  with the latest patches, a dual-core with 8 
GBmemory.

Any hints?

Thanks



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sipwise.com/pipermail/spce-user_lists.sipwise.com/attachments/20231228/4fadf93b/attachment-0001.html>