<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
Hi,
<div class=""><br class="">
</div>
<div class="">MySQL is not locking up other than due to anti-fraud script which runs every half an hour.</div>
<div class=""><br class="">
</div>
<div class="">I think I can also rule out DDOS because it’s a steady 300-350 packets per second that go to unknown port.</div>
<div class=""><br class="">
</div>
<div class="">What I have not figured out yet is how the heck I find out which packets are the actual culprits… Even doing a tcpdump on UDP packets only and excluding the hosts I know are legit and the ports I know are legit, still gives me a heck of a lot
of traffic, probably actual payload traffic of ongoing voice calls (around 250 concurrent)…</div>
<div class=""><br class="">
</div>
<div class="">Now the packets to unknown port could also be some equipment sending some garbage (Grandstream ATA’s like to do this) to keep the NAT port open, and it may not actually be a problem, but I still can’t seem to figure out what causes the RcvbufErrors
which periodically happen and when I listen to for instance the conference bridge music, it will break for a while…</div>
<div class=""><br class="">
</div>
<div class="">How to find out when the rcvbuferror occurs, what application is causing it?</div>
<div class=""><br class="">
</div>
<div class="">Thanks for any suggestions.</div>
<div class="">Walter</div>
<div class=""><br class="">
<div>
<blockquote type="cite" class="">
<div class="">On 23 Jun 2016, at 4:25 PM, Skyler <<a href="mailto:skchopperguy@gmail.com" class="">skchopperguy@gmail.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<p dir="ltr" class="">Dang these thumbs..now to the list.</p>
<div class="gmail_quote">On Jun 23, 2016 2:06 AM, "Skyler" <<a href="mailto:skchopperguy@gmail.com" class="">skchopperguy@gmail.com</a>> wrote:<br type="attribution" class="">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<p dir="ltr" class="">Sorry, in the list now.</p>
<p dir="ltr" class="">I had a similar issue last month. Basically mysql locking up the box. I think there's an update for hackers out there. Kamailio is tuff...but mysql can be broken..
</p>
<p dir="ltr" class="">It was resolved by exiting/dropping on common hacker UA which were retreived from logs and the IP's. Eventually they gave up and moves along.</p>
<p dir="ltr" class="">Ddos type attack.</p>
<p dir="ltr" class="">-Skyler</p>
<div class="gmail_quote">On Jun 23, 2016 1:59 AM, "Skyler" <<a href="mailto:skchopperguy@gmail.com" target="_blank" class="">skchopperguy@gmail.com</a>> wrote:<br type="attribution" class="">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<p dir="ltr" class="">Looks like a flood to me. Yer spec is 2 days here, are you seeing anything in lb or proxy log when tailing?</p>
<p dir="ltr" class="">- Skyler</p>
<div class="gmail_quote">On Jun 22, 2016 9:01 PM, "Walter Klomp" <<a href="mailto:walter@myrepublic.com.sg" target="_blank" class="">walter@myrepublic.com.sg</a>> wrote:<br type="attribution" class="">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word" class="">Hi,
<div class=""><br class="">
</div>
<div class="">Running SPCE 3.8.5 on dedicated ESXi host (Dell R320 with Xeon E2460 & 16GB RAM) with ~30.000 registered subscribers (and online).</div>
<div class=""><br class="">
</div>
<div class="">Last week we were having horrible statistics and packet-loss galore… After tweaking the network settings with the below, I have managed to minimize the packet-loss.. but still there is.</div>
<div class="">
<div class=""><br class="">
</div>
<div class="">sysctl -w net.core.rmem_max=33554432</div>
<div class="">sysctl -w net.core.wmem_max=33554432</div>
<div class="">sysctl -w net.core.rmem_default=65536</div>
<div class="">sysctl -w net.core.wmem_default=65536</div>
<div class="">sysctl -w net.ipv4.tcp_mem='8388608 8388608 8388608'</div>
<div class="">sysctl -w net.ipv4.udp_mem='4096 174760 33554432'</div>
<div class="">sysctl -w net.ipv4.tcp_rmem='4096 87380 8388608'</div>
<div class="">sysctl -w net.ipv4.tcp_wmem='4096 65536 8388608'</div>
<div class="">sysctl -w net.ipv4.route.flush=1</div>
</div>
<div class=""><br class="">
</div>
<div class="">I am currently still seeing around 300 packets per second going to unknown ports. Below are the statistics. That’s about 1/5th of all the packets received are not being processed… That seems a lot to me.</div>
<div class=""><br class="">
</div>
<div class=""> 10:43:40 up 2 days, 5:11, 3 users, load average: 1.52, 2.05, 2.17</div>
<div class=""><br class="">
</div>
<div class="">
<div class="">Every 1.0s: netstat -anus|grep -A 7 Udp: Thu Jun 23 10:40:45 2016</div>
<div class=""><br class="">
</div>
<div class="">Udp:</div>
<div class=""> 310870895 packets received</div>
<div class=""> 61212884 packets to unknown port received.</div>
<div class=""> 103338 packet receive errors</div>
<div class=""> 312245302 packets sent</div>
<div class=""> RcvbufErrors: 103249</div>
<div class=""> SndbufErrors: 765</div>
<div class=""> InCsumErrors: 75</div>
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">I had to do a lot of buffer tweaking to get the RcvbufErrors down and even the SndbufErrors as every time it happens (at bursts - sporadically every 10 minutes, but definitely every half hour), one would get silence and the packet receive errors
would should up by about between 200 and 800 packets.</div>
<div class=""><br class="">
</div>
<div class="">The load average can shoot up to 4.x at times. Knowing that Sipwise Pro is on the same hardware, and they support up to 50.000 users, what am I missing?</div>
<div class=""><br class="">
</div>
<div class="">rtpengine is running in kernel. major contributor of CPU usage is actually MySQL regularly maxing out at 100%. Especially when it’s doing the fraud check. Below is a snapshot of top….</div>
<div class=""><br class="">
</div>
<div class="">
<div class=""><font face="Courier" class="">top - 10:56:53 up 2 days, 5:24, 3 users, load average: 2.39, 2.14, 1.94</font></div>
<div class=""><font face="Courier" class="">Tasks: 184 total, 1 running, 183 sleeping, 0 stopped, 0 zombie</font></div>
<div class=""><font face="Courier" class="">%Cpu(s): 25.3 us, 7.0 sy, 0.0 ni, 63.7 id, 1.0 wa, 0.0 hi, 2.9 si, 0.0 st</font></div>
<div class=""><font face="Courier" class="">KiB Mem: 12334464 total, 12157676 used, 176788 free, 144944 buffers</font></div>
<div class=""><font face="Courier" class="">KiB Swap: 2096124 total, 0 used, 2096124 free, 4430336 cached</font></div>
<div class=""><font face="Courier" class=""><br class="">
</font></div>
<div class=""><font face="Courier" class=""> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND </font></div>
<div class=""><font face="Courier" class=""> 4063 mysql 20 0 6127m 5.6g 7084 S 54.7 47.7 809:35.18 mysqld </font></div>
<div class=""><font face="Courier" class=""> 2576 root 20 0 253m 7176 1816 S 9.9 0.1 164:02.97 rsyslogd </font></div>
<div class=""><font face="Courier" class=""> 5058 root 20 0 67176 11m 5308 S 6.0 0.1 7:05.16 rate-o-mat </font></div>
<div class=""><font face="Courier" class="">15432 root 20 0 276m 12m 3696 S 5.0 0.1 117:56.92 rtpengine </font></div>
<div class=""><font face="Courier" class=""> 5257 sems 20 0 873m 37m 7624 S 4.0 0.3 139:44.03 ngcp-sems </font></div>
<div class=""><font face="Courier" class="">30996 kamailio 20 0 539m 100m 53m S 4.0 0.8 6:02.68 kamailio </font></div>
</div>
<div class=""><br class="">
</div>
<div class="">Does anybody have any pointers I can try to completely eliminate the packet loss and where do these unknown port packets go to?</div>
<div class=""><br class="">
</div>
<div class="">Thanks</div>
<div class="">Walter.</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
</div>
<br class="">
_______________________________________________<br class="">
Spce-user mailing list<br class="">
<a href="mailto:Spce-user@lists.sipwise.com" target="_blank" class="">Spce-user@lists.sipwise.com</a><br class="">
<a href="https://lists.sipwise.com/listinfo/spce-user" rel="noreferrer" target="_blank" class="">https://lists.sipwise.com/listinfo/spce-user</a><br class="">
<br class="">
</blockquote>
</div>
</blockquote>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</body>
</html>