<p dir="ltr">Dang these thumbs..now to the list.</p>
<div class="gmail_quote">On Jun 23, 2016 2:06 AM, "Skyler" <<a href="mailto:skchopperguy@gmail.com">skchopperguy@gmail.com</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><p dir="ltr">Sorry, in the list now.</p>
<p dir="ltr"> I had a similar issue last month. Basically mysql locking up the box. I think there's an update for hackers out there. Kamailio is tuff...but mysql can be broken.. </p>
<p dir="ltr"> It was resolved by exiting/dropping on common hacker UA which were retreived from logs and the IP's. Eventually they gave up and moves along.</p>
<p dir="ltr">Ddos type attack.</p>
<p dir="ltr">-Skyler</p>
<div class="gmail_quote">On Jun 23, 2016 1:59 AM, "Skyler" <<a href="mailto:skchopperguy@gmail.com" target="_blank">skchopperguy@gmail.com</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><p dir="ltr">Looks like a flood to me. Yer spec is 2 days here, are you seeing anything in lb or proxy log when tailing?</p>
<p dir="ltr"> - Skyler</p>
<div class="gmail_quote">On Jun 22, 2016 9:01 PM, "Walter Klomp" <<a href="mailto:walter@myrepublic.com.sg" target="_blank">walter@myrepublic.com.sg</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word">
Hi,
<div><br>
</div>
<div>Running SPCE 3.8.5 on dedicated ESXi host (Dell R320 with Xeon E2460 & 16GB RAM) with ~30.000 registered subscribers (and online).</div>
<div><br>
</div>
<div>Last week we were having horrible statistics and packet-loss galore… After tweaking the network settings with the below, I have managed to minimize the packet-loss.. but still there is.</div>
<div>
<div><br>
</div>
<div>sysctl -w net.core.rmem_max=33554432</div>
<div>sysctl -w net.core.wmem_max=33554432</div>
<div>sysctl -w net.core.rmem_default=65536</div>
<div>sysctl -w net.core.wmem_default=65536</div>
<div>sysctl -w net.ipv4.tcp_mem='8388608 8388608 8388608'</div>
<div>sysctl -w net.ipv4.udp_mem='4096 174760 33554432'</div>
<div>sysctl -w net.ipv4.tcp_rmem='4096 87380 8388608'</div>
<div>sysctl -w net.ipv4.tcp_wmem='4096 65536 8388608'</div>
<div>sysctl -w net.ipv4.route.flush=1</div>
</div>
<div><br>
</div>
<div>I am currently still seeing around 300 packets per second going to unknown ports. Below are the statistics. That’s about 1/5th of all the packets received are not being processed… That seems a lot to me.</div>
<div><br>
</div>
<div> 10:43:40 up 2 days, 5:11, 3 users, load average: 1.52, 2.05, 2.17</div>
<div><br>
</div>
<div>
<div>Every 1.0s: netstat -anus|grep -A 7 Udp: Thu Jun 23 10:40:45 2016</div>
<div><br>
</div>
<div>Udp:</div>
<div> 310870895 packets received</div>
<div> 61212884 packets to unknown port received.</div>
<div> 103338 packet receive errors</div>
<div> 312245302 packets sent</div>
<div> RcvbufErrors: 103249</div>
<div> SndbufErrors: 765</div>
<div> InCsumErrors: 75</div>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div>I had to do a lot of buffer tweaking to get the RcvbufErrors down and even the SndbufErrors as every time it happens (at bursts - sporadically every 10 minutes, but definitely every half hour), one would get silence and the packet receive errors
would should up by about between 200 and 800 packets.</div>
<div><br>
</div>
<div>The load average can shoot up to 4.x at times. Knowing that Sipwise Pro is on the same hardware, and they support up to 50.000 users, what am I missing?</div>
<div><br>
</div>
<div>rtpengine is running in kernel. major contributor of CPU usage is actually MySQL regularly maxing out at 100%. Especially when it’s doing the fraud check. Below is a snapshot of top….</div>
<div><br>
</div>
<div>
<div><font face="Courier">top - 10:56:53 up 2 days, 5:24, 3 users, load average: 2.39, 2.14, 1.94</font></div>
<div><font face="Courier">Tasks: 184 total, 1 running, 183 sleeping, 0 stopped, 0 zombie</font></div>
<div><font face="Courier">%Cpu(s): 25.3 us, 7.0 sy, 0.0 ni, 63.7 id, 1.0 wa, 0.0 hi, 2.9 si, 0.0 st</font></div>
<div><font face="Courier">KiB Mem: 12334464 total, 12157676 used, 176788 free, 144944 buffers</font></div>
<div><font face="Courier">KiB Swap: 2096124 total, 0 used, 2096124 free, 4430336 cached</font></div>
<div><font face="Courier"><br>
</font></div>
<div><font face="Courier"> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND </font></div>
<div><font face="Courier"> 4063 mysql 20 0 6127m 5.6g 7084 S 54.7 47.7 809:35.18 mysqld </font></div>
<div><font face="Courier"> 2576 root 20 0 253m 7176 1816 S 9.9 0.1 164:02.97 rsyslogd </font></div>
<div><font face="Courier"> 5058 root 20 0 67176 11m 5308 S 6.0 0.1 7:05.16 rate-o-mat </font></div>
<div><font face="Courier">15432 root 20 0 276m 12m 3696 S 5.0 0.1 117:56.92 rtpengine </font></div>
<div><font face="Courier"> 5257 sems 20 0 873m 37m 7624 S 4.0 0.3 139:44.03 ngcp-sems </font></div>
<div><font face="Courier">30996 kamailio 20 0 539m 100m 53m S 4.0 0.8 6:02.68 kamailio </font></div>
</div>
<div><br>
</div>
<div>Does anybody have any pointers I can try to completely eliminate the packet loss and where do these unknown port packets go to?</div>
<div><br>
</div>
<div>Thanks</div>
<div>Walter.</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
</div>
<br>_______________________________________________<br>
Spce-user mailing list<br>
<a href="mailto:Spce-user@lists.sipwise.com" target="_blank">Spce-user@lists.sipwise.com</a><br>
<a href="https://lists.sipwise.com/listinfo/spce-user" rel="noreferrer" target="_blank">https://lists.sipwise.com/listinfo/spce-user</a><br>
<br></blockquote></div>
</blockquote></div>
</blockquote></div>