[Spce-user] Performance problem with ngcp-witnessd

[Guillem Jover]

Hi!

On Thu, 2023-12-28 at 17:02:54 +0100, Henk wrote:
> Hi, we are having a performance problem with ngcp-witnessd. It starts after
> the cpu load is higher during nightly cron jobs like backup. Port 9800 is
> opened multiple times (>100) like this:
> 
> root at spce:~# lsof -i TCP|  grep 9800
> victoria-     607 _victoria-metrics   28u  IPv4 34459176 0t0  TCP
> localhost:45000->localhost:9800 (ESTABLISHED)
> ngcp-witn 1300009              root    5u  IPv4 33636424 0t0  TCP
> localhost:9800 (LISTEN)
> ngcp-witn 1300009              root    9u  IPv4 34442250 0t0  TCP
> localhost:9800->localhost:49954 (CLOSE_WAIT)
> ngcp-witn 1300009              root   11u  IPv4 34442251 0t0  TCP
> localhost:9800->localhost:49964 (CLOSE_WAIT)
[…]

This looks like the issue with VictoriMetrics and its keep-alive
handling, that was reported and fixed in
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3293. For
ngcp-witnessd we applied a workaround to match the Golang based
exporters, which I checked now and does not seem to be in mr9.5.8, so
I'll backport that one. For now you might want to check hot-applying
the following patch on your system and report back whether that fixes
your issue:

,--- /usr/sbin/ngcp-witnessd ---
diff --git a/ngcp-witnessd b/ngcp-witnessd
index c63f40a..974844a 100755
--- a/ngcp-witnessd
+++ b/ngcp-witnessd
@@ -1753,7 +1753,7 @@ $r->get('/metrics', sub {
 my $d = Mojo::Server::Daemon->new(
     app => $app,
     listen => [ "http://$config{listen_host}:$config{listen_port}" ],
-    max_requests => 0,
+    keep_alive_timeout => 30,
 );
 
 notify_ready();
`---

> Restarting ngcp-witnessd fixes the problem temporary. The witnessd log
> doesn't give a clue, only some uninitialized values:
> 
> Dec 28 16:47:36 spce (warning) ngcp-witnessd[1548076]: Use of uninitialized
> value $lv in substitution (s///) at /usr/share/perl5/Prometheus/Tiny.pm line
> 28.
> Dec 28 16:47:36 spce (warning) ngcp-witnessd[1548076]: Use of uninitialized
> value $lv in substitution (s///) at /usr/share/perl5/Prometheus/Tiny.pm line
> 29.
> Dec 28 16:47:36 spce (warning) ngcp-witnessd[1548076]: Use of uninitialized
> value $lv in concatenation (.) or string at
> /usr/share/perl5/Prometheus/Tiny.pm line 30.
[…]

Hmm, these warnings from the Prometheus::Tiny module are not very
helpful to try to track this down. Could you either upgrade the
libprometheus-tiny-perl package to version 0.011-1 (from for example
<https://packages.debian.org/trixie/all/libprometheus-tiny-perl/download>).
Or alternatively hot-applying the patch for the .pm module from
<https://github.com/robn/Prometheus-Tiny/commit/8de6ba18830ff27806033cff90273474acdd0519>.

Then restarting the daemon and reporting back the label involved from
the new log entries, so that we can try to track this down?

> Sipwise version is  mr9.5.8  with the latest patches, a dual-core with 8
> GBmemory.

Thanks,
Guillem