[Spce-user] Memory Utilization very high since yesterday mr5.5.2

[Henk]

Hi Matthias,

I cannot solve the cause of your problem, but I have a working solution 
for the growing perl-fcgi memory which I run for a few months now. I 
started with the script found here : 
http://www.catalystframework.org/calendar/2007/18 made by Mark Blythe. 
You can also find a good explanation here about memory use of perl-fcgi.
As I was not able to start it from cron I adopted it a little bit, but 
basically it works. I use a 720MB memory limit and run it every hour.
If anyone is interested, I attached the modified script. You can store 
it in /usr/local/sbin

The cron I'm using to run every hour on :40 looks like this:
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
40 * * * *   root fcgi-kill /var/run/fastcgi/ngcp-panel.pid 720000 > 
/dev/null 2>&1

Best regards,

Henk Plessius
VoipDigit

On 8-3-2018 13:57, Hohl Matthias wrote:
>
> Hello!
>
> We have noticed that storage usage has increased massively since 
> yesterday 18:30. The RAM was populated with 3 additional GB of RAM 
> within 15 minutes and there's no reason to see why this happened. 
> Also, the RAM has not degraded since then.
>
> The system runs since the last update to 5.5.2 for 50 days without 
> problems and always with the same utilization / use of the services.
>
> We looked at the logs but cannot detect any irregularities. In the 
> Grafana we saw that at that time the CPU utilization went up to 100%. 
> What we still recognized was that the database SELECT queries have 
> massively increased. Usually we have between 30-50 queries but from 
> 18:30 to 18:45 there were nearly 2,000 queries.
>
> We then looked at the syslog, but it is empty and contains no data. We 
> also looked at the slow-queries, but again, nothing remarkable was 
> noticed at this time. We also tried to clear the RAM, but this has not 
> brought any improvement.
>
> Normally, the RAM utilization should normalize again when the queries 
> are finished, right?
>
> Where else can we look to investigate the cause of this massive 
> increase in SELECT queries?
>
> It would be really great if you could help us.
>
>
>
> _______________________________________________
> Spce-user mailing list
> Spce-user at lists.sipwise.com
> https://lists.sipwise.com/listinfo/spce-user

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sipwise.com/mailman/private/spce-user_lists.sipwise.com/attachments/20180308/ed36a325/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.jpg
Type: image/jpeg
Size: 36003 bytes
Desc: not available
URL: <http://lists.sipwise.com/mailman/private/spce-user_lists.sipwise.com/attachments/20180308/ed36a325/attachment.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.jpg
Type: image/jpeg
Size: 21466 bytes
Desc: not available
URL: <http://lists.sipwise.com/mailman/private/spce-user_lists.sipwise.com/attachments/20180308/ed36a325/attachment-0001.jpg>
-------------- next part --------------
#!/usr/bin/perl
 
use strict;
 
my ($fnam_pid, $bytes) = @ARGV;
my $ppid;
 
die "Usage: fcgi-kill filename_with_pid ram_limit_in_bytes \n" unless $fnam_pid && $bytes;

my $kids;

if (open(my $fil, "<", "$fnam_pid"))
{
	$ppid = <$fil>;
	chomp $ppid;
}
else
{
	die "Process not found";
}

if (open($kids, "/bin/ps -o pid=,vsz= --ppid $ppid|")) {
   my @goners;
 
   while (<$kids>) {
      chomp;
      my ($pid, $mem) = split;
 
      if ($mem >= $bytes) {
         push @goners, $pid;
      }
   }
 
   close($kids);
 
   if (@goners) {
      # kill them slowly, so that all connection serving
      # children don't suddenly die at once.
 
      foreach my $victim (@goners) {
         kill 'HUP', $victim;
         sleep 10;
      }
   }
   die "ready cleaning up\n";
}
else {
   die "Can't get process list: $!\n";
}