[Spce-user] Performance problem with ngcp-witnessd

[Henk]

Hi Guillem,

Thank you for looking into this. I missed your answer, but made the 
change now and it's much better, although the metrics are still in the 
top-cpu list. The patch wasn't working as my version has a lot more 
lines and the patch is around line 1968. It seems in my version InfluxDB 
code is still present.

I also replaced the Tiny.pm with the latest version and the log now 
stops after startup, so the variable messages are gone.

Thank you!

Regards,

Henk

> Hi!
>
> On Thu, 2023-12-28 at 17:02:54 +0100, Henk wrote:
> >/Hi, we are having a performance problem with ngcp-witnessd. It starts 
> after />/the cpu load is higher during nightly cron jobs like backup. Port 
> 9800 is />/opened multiple times (>100) like this: />//>/root at spce 
> <http://lists.sipwise.com/mailman/listinfo/spce-user_lists.sipwise.com>:~# 
> lsof -i TCP|  grep 9800 />/victoria-     607 _victoria-metrics   28u  IPv4 34459176 0t0  TCP />/localhost:45000->localhost:9800 (ESTABLISHED) />/ngcp-witn 1300009              root    5u  IPv4 33636424 0t0  TCP />/localhost:9800 (LISTEN) />/ngcp-witn 1300009              root    9u  IPv4 34442250 0t0  TCP />/localhost:9800->localhost:49954 (CLOSE_WAIT) />/ngcp-witn 1300009              root   11u  IPv4 34442251 0t0  TCP />/localhost:9800->localhost:49964 (CLOSE_WAIT) /[…]
>
> This looks like the issue with VictoriMetrics and its keep-alive
> handling, that was reported and fixed in
> https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3293.  For
> ngcp-witnessd we applied a workaround to match the Golang based
> exporters, which I checked now and does not seem to be in mr9.5.8, so
> I'll backport that one. For now you might want to check hot-applying
> the following patch on your system and report back whether that fixes
> your issue:
>
> ,--- /usr/sbin/ngcp-witnessd ---
> diff --git a/ngcp-witnessd b/ngcp-witnessd
> index c63f40a..974844a 100755
> --- a/ngcp-witnessd
> +++ b/ngcp-witnessd
> @@ -1753,7 +1753,7 @@ $r->get('/metrics', sub {
>   my $d = Mojo::Server::Daemon->new(
>       app => $app,
>       listen => [ "http://$config{listen_host}:$config{listen_port}" ],
> -    max_requests => 0,
> +    keep_alive_timeout => 30,
>   );
>   
>   notify_ready();
> `---
>
> >/Restarting ngcp-witnessd fixes the problem temporary. The witnessd log />/doesn't give a clue, only some uninitialized values: />//>/Dec 28 16:47:36 spce (warning) ngcp-witnessd[1548076]: Use of 
> uninitialized />/value $lv in substitution (s///) at 
> /usr/share/perl5/Prometheus/Tiny.pm line />/28. />/Dec 28 16:47:36 spce (warning) ngcp-witnessd[1548076]: Use of 
> uninitialized />/value $lv in substitution (s///) at 
> /usr/share/perl5/Prometheus/Tiny.pm line />/29. />/Dec 28 16:47:36 spce (warning) ngcp-witnessd[1548076]: Use of 
> uninitialized />/value $lv in concatenation (.) or string at />//usr/share/perl5/Prometheus/Tiny.pm line 30. /[…]
>
> Hmm, these warnings from the Prometheus::Tiny module are not very
> helpful to try to track this down. Could you either upgrade the
> libprometheus-tiny-perl package to version 0.011-1 (from for example
> <https://packages.debian.org/trixie/all/libprometheus-tiny-perl/download>).
> Or alternatively hot-applying the patch for the .pm module from
> <https://github.com/robn/Prometheus-Tiny/commit/8de6ba18830ff27806033cff90273474acdd0519>.
>
> Then restarting the daemon and reporting back the label involved from
> the new log entries, so that we can try to track this down?
>
> >/Sipwise version is  mr9.5.8  with the latest patches, a dual-core with 8 />/GBmemory. /
> Thanks,
> Guillem

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sipwise.com/pipermail/spce-user_lists.sipwise.com/attachments/20240120/21f0f265/attachment.html>