<!DOCTYPE html><html><head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body>
<p>Hi Guillem,</p>
<p>Thank you for looking into this. I missed your answer, but made
the change now and it's much better, although the metrics are
still in the top-cpu list. The patch wasn't working as my version
has a lot more lines and the patch is around line 1968. It seems
in my version InfluxDB code is still present.</p>
<p>I also replaced the Tiny.pm with the latest version and the log
now stops after startup, so the variable messages are gone.</p>
<p>Thank you!</p>
<p>Regards,</p>
<p>Henk<br>
<blockquote type="cite">
<pre>Hi!
On Thu, 2023-12-28 at 17:02:54 +0100, Henk wrote:
><i> Hi, we are having a performance problem with ngcp-witnessd. It starts after
</i>><i> the cpu load is higher during nightly cron jobs like backup. Port 9800 is
</i>><i> opened multiple times (>100) like this:
</i>><i>
</i>><i> <a href="http://lists.sipwise.com/mailman/listinfo/spce-user_lists.sipwise.com">root at spce</a>:~# lsof -i TCP| grep 9800
</i>><i> victoria- 607 _victoria-metrics 28u IPv4 34459176 0t0 TCP
</i>><i> localhost:45000->localhost:9800 (ESTABLISHED)
</i>><i> ngcp-witn 1300009 root 5u IPv4 33636424 0t0 TCP
</i>><i> localhost:9800 (LISTEN)
</i>><i> ngcp-witn 1300009 root 9u IPv4 34442250 0t0 TCP
</i>><i> localhost:9800->localhost:49954 (CLOSE_WAIT)
</i>><i> ngcp-witn 1300009 root 11u IPv4 34442251 0t0 TCP
</i>><i> localhost:9800->localhost:49964 (CLOSE_WAIT)
</i>[…]
This looks like the issue with VictoriMetrics and its keep-alive
handling, that was reported and fixed in
<a href="https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3293." class="moz-txt-link-freetext">https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3293.</a> For
ngcp-witnessd we applied a workaround to match the Golang based
exporters, which I checked now and does not seem to be in mr9.5.8, so
I'll backport that one. For now you might want to check hot-applying
the following patch on your system and report back whether that fixes
your issue:
,--- /usr/sbin/ngcp-witnessd ---
diff --git a/ngcp-witnessd b/ngcp-witnessd
index c63f40a..974844a 100755
--- a/ngcp-witnessd
+++ b/ngcp-witnessd
@@ -1753,7 +1753,7 @@ $r->get('/metrics', sub {
my $d = Mojo::Server::Daemon->new(
app => $app,
listen => [ "<a href="http://$config{listen_host" class="moz-txt-link-freetext">http://$config{listen_host</a>}:$config{listen_port}" ],
- max_requests => 0,
+ keep_alive_timeout => 30,
);
notify_ready();
`---
><i> Restarting ngcp-witnessd fixes the problem temporary. The witnessd log
</i>><i> doesn't give a clue, only some uninitialized values:
</i>><i>
</i>><i> Dec 28 16:47:36 spce (warning) ngcp-witnessd[1548076]: Use of uninitialized
</i>><i> value $lv in substitution (s///) at /usr/share/perl5/Prometheus/Tiny.pm line
</i>><i> 28.
</i>><i> Dec 28 16:47:36 spce (warning) ngcp-witnessd[1548076]: Use of uninitialized
</i>><i> value $lv in substitution (s///) at /usr/share/perl5/Prometheus/Tiny.pm line
</i>><i> 29.
</i>><i> Dec 28 16:47:36 spce (warning) ngcp-witnessd[1548076]: Use of uninitialized
</i>><i> value $lv in concatenation (.) or string at
</i>><i> /usr/share/perl5/Prometheus/Tiny.pm line 30.
</i>[…]
Hmm, these warnings from the Prometheus::Tiny module are not very
helpful to try to track this down. Could you either upgrade the
libprometheus-tiny-perl package to version 0.011-1 (from for example
<<a href="https://packages.debian.org/trixie/all/libprometheus-tiny-perl/download" class="moz-txt-link-freetext">https://packages.debian.org/trixie/all/libprometheus-tiny-perl/download</a>>).
Or alternatively hot-applying the patch for the .pm module from
<<a href="https://github.com/robn/Prometheus-Tiny/commit/8de6ba18830ff27806033cff90273474acdd0519" class="moz-txt-link-freetext">https://github.com/robn/Prometheus-Tiny/commit/8de6ba18830ff27806033cff90273474acdd0519</a>>.
Then restarting the daemon and reporting back the label involved from
the new log entries, so that we can try to track this down?
><i> Sipwise version is mr9.5.8 with the latest patches, a dual-core with 8
</i>><i> GBmemory.
</i>
Thanks,
Guillem</pre>
</blockquote>
<br>
</p>
</body>
</html>