[Spce-user] Memory Utilization very high since yesterday mr5.5.2

Thu Mar 8 09:03:14 EST 2018

BTW: is there maybe a cronjob running at every hour and 30 minutes?

Crontab –e shows me just our copy script for the CDR directory, which is running at this time “tar czpf /backup/$DEST.tgz -C / home/jail/home/cdrexport”

I see that this normally use some CPU load, when it is running, but no DB SELECT queries.

Von: Spce-user [mailto:spce-user-bounces at lists.sipwise.com] Im Auftrag von Hohl Matthias
Gesendet: Donnerstag, 8. März 2018 14:56
An: 'Alex Lutay' <alutay at sipwise.com>; spce-user at lists.sipwise.com
Betreff: Re: [Spce-user] Memory Utilization very high since yesterday mr5.5.2

Hello,

for DDOS also my network traffic should be increased or?

But I can’t see there any peak at this time…

I also searched the log files but can’t see something special in it or a abnormal size to identify any problems. L

What I see in panel.log are a lot of warnings the whole day over…

Mar  7 16:03:34 spce ngcp-panel: WARN: could not apply rewrite: no rewrite rule set found.

Mar  7 16:03:35 spce ngcp-panel: WARN: could not apply rewrite: no rewrite rule set found.

…

Mar  7 18:26:01 spce ngcp-panel: ERROR: Failed to find path a2billing/common/javascript/misc.js

Mar  7 19:08:58 spce ngcp-panel: ERROR: Failed to find path robots.txt

Mar  7 18:30:57 spce ngcp-panel: DEBUG: calling voicemails_slice in SubscriberVmOverview

Mar  7 18:30:57 spce ngcp-panel: DEBUG: *** Root::auto skip authn, grant access to login/admin

Mar  7 18:30:57 spce ngcp-panel: DEBUG: *** Root::auto grant access for authenticated user

Mar  7 18:30:57 spce ngcp-panel: DEBUG: calling cfs in SubscriberCFOverview

Mar  7 18:30:57 spce ngcp-panel: DEBUG: *** Root::auto grant access for authenticated user

Mar  7 18:30:57 spce ngcp-panel: DEBUG: calling registrations_slice in SubscriberRegisterOverview

Mar  7 18:30:57 spce ngcp-panel: DEBUG: *** Root::auto grant access for authenticated user

Mar  7 18:30:57 spce ngcp-panel: DEBUG: calling voicemails_count in SubscriberVmOverview

Mar  7 18:30:59 spce ngcp-panel: DEBUG: *** Root::auto grant access for authenticated user

Mar  7 18:30:59 spce ngcp-panel: DEBUG: calling registrations_count in SubscriberRegisterOverview

Mar  7 18:30:59 spce ngcp-panel: DEBUG: *** Root::auto grant access for authenticated user

Mar  7 18:30:59 spce ngcp-panel: DEBUG: calling calls_count in SubscriberCallsOverview

Mar  7 18:30:59 spce ngcp-panel: DEBUG: *** Root::auto grant access for authenticated user

Mar  7 18:30:59 spce ngcp-panel: DEBUG: NGCP::Panel::Controller::Subscriber::auto

I also searched kamailio proxy/lb log but also can’t see there anything…

Also we have strict scanning/dos policies. So if there was a scanning process, the ip get banned immediately.

Is there somewhere a DB log file too?

I can’ find something in this direction…

Yah restarting ngcp-panel was also my first idea.. but I will wait until tonight, to perform this task.

Do you have any other idea too?

Thanks for your help alex,

Von: Spce-user [mailto:spce-user-bounces at lists.sipwise.com] Im Auftrag von Alex Lutay
Gesendet: Donnerstag, 8. März 2018 14:33
An: spce-user at lists.sipwise.com <mailto:spce-user at lists.sipwise.com> 
Betreff: Re: [Spce-user] Memory Utilization very high since yesterday mr5.5.2

Hi,

Is you system publicly accessible? Maybe you have been scanned or
received some (D)DOS attack? It is hard to say something from here.

Check the log files size to see the abnormal size to know the type of
the load SIP/WEB/... you need to understand who uses all right now RAM.
Restart this component to free the used RAM.

Sure, report here your findings.

On 03/08/2018 01:57 PM, Hohl Matthias wrote:
> ... SELECT queries have massively increased. Usually we have between 30-50 queries but from
> 18:30 to 18:45 there were nearly 2,000 queries.

Maybe someone dumped you DB massively?
A lot of SELECTS can happens if someone was scanning you with a lot of
REGISTER/INVITE SIP messages, which forces kamailio to perform selects, etc.

> We then looked at the syslog, but it is empty and contains no data. We
> also looked at the slow-queries, but again, nothing remarkable was
> noticed at this time.

> We also tried to clear the RAM, but this has not
> brought any improvement.

I suppose mysql/ngcp-panel restart should clean memory usage.
You should perform this during off-peak only.

> Normally, the RAM utilization should normalize again when the queries
> are finished, right?

It depends on the service.

-- 
Alex Lutay
_______________________________________________
Spce-user mailing list
Spce-user at lists.sipwise.com <mailto:Spce-user at lists.sipwise.com> 
https://lists.sipwise.com/listinfo/spce-user

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sipwise.com/pipermail/spce-user_lists.sipwise.com/attachments/20180308/48f834b3/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5532 bytes
Desc: not available
URL: <http://lists.sipwise.com/pipermail/spce-user_lists.sipwise.com/attachments/20180308/48f834b3/attachment-0001.p7s>