[Spce-user] Help - many packet loss and packets to unknown port

Skyler skchopperguy at gmail.com
Thu Jun 23 17:46:37 EDT 2016 

If it were my box I'd have iptables accept only tcp/udp 5060, 5061 and TCP
80, 443 and admin, xmlrpc ports then drop all of the rest.

Could start here for good examples:

https://www.kamailio.org/wiki/tutorials/security/kamailio-security

-- Skyler
On Jun 23, 2016 3:27 PM, "Skyler" <skchopperguy at gmail.com> wrote:

> Hi,
>
> On Jun 23, 2016 3:18 AM, "Walter Klomp" <walter at myrepublic.com.sg> wrote:
> >
> > Hi,
> >
> > MySQL is not locking up other than due to anti-fraud script which runs
> every half an hour.
> >
>
> Oh you mentioned mysql pinning cpu so I assumed we may have had the same
> problem.
>
> > I think I can also rule out DDOS because it’s a steady 300-350 packets
> per second that go to unknown port.
> >
>
> Wow, so one device is doing that you figure? How do you know it's that
> many pps if the port is unknown?
>
> If they are udp, I'd set kamailio lb to listen on that unknown port and
> look in the logs to see what shows up.
>
> > What I have not figured out yet is how the heck I find out which packets
> are the actual culprits…  Even doing a tcpdump on UDP packets only and
> excluding the hosts I know are legit and the ports I know are legit, still
> gives me a heck of a lot of traffic, probably actual payload traffic of
> ongoing voice calls (around 250 concurrent)…
> >
> > Now the packets to unknown port could also be some equipment sending
> some garbage (Grandstream ATA’s like to do this) to keep the NAT port open,
> and it may not actually be a problem, but I still can’t seem to figure out
> what causes the RcvbufErrors which periodically happen and when I listen to
> for instance the conference bridge music, it will break for a while…
> >
>
> I've never heard of grandstream devices sending that kind of pps before.
> Unless it's like 3000 of them all misconfigured and pointing at you. All
> UA's do nat ping on the port configured on the device, so 5060 usually.
> Can't see devices being the problem here. The pps is too high.
>
> > How to find out when the rcvbuferror occurs, what application is causing
> it?
>
> First find out where the packets are coming from and why. Then you'll know
> if it can be dropped or what app to look at.
>
> > Thanks for any suggestions.
> > Walter
> >
> >> On 23 Jun 2016, at 4:25 PM, Skyler <skchopperguy at gmail.com> wrote:
> >>
> >> Dang these thumbs..now to the list.
> >>
> >> On Jun 23, 2016 2:06 AM, "Skyler" <skchopperguy at gmail.com> wrote:
> >>>
> >>> Sorry, in the list now.
> >>>
> >>> I had a similar issue last month. Basically mysql locking up the box.
> I think there's an update for hackers out there. Kamailio is tuff...but
> mysql can be broken..
> >>>
> >>> It was resolved by exiting/dropping on common hacker UA which were
> retreived from logs and the IP's. Eventually they gave up and moves along.
> >>>
> >>> Ddos type attack.
> >>>
> >>> -Skyler
> >>>
> >>> On Jun 23, 2016 1:59 AM, "Skyler" <skchopperguy at gmail.com> wrote:
> >>>>
> >>>> Looks like a flood to me. Yer spec is 2 days here, are you seeing
> anything in lb or proxy log when tailing?
> >>>>
> >>>> - Skyler
> >>>>
> >>>> On Jun 22, 2016 9:01 PM, "Walter Klomp" <walter at myrepublic.com.sg>
> wrote:
> >>>>>
> >>>>> Hi,
> >>>>>
> >>>>> Running SPCE 3.8.5 on dedicated ESXi host (Dell R320 with Xeon E2460
> & 16GB RAM) with ~30.000 registered subscribers (and online).
> >>>>>
> >>>>> Last week we were having horrible statistics and packet-loss galore…
> After tweaking the network settings with the below, I have managed to
> minimize the packet-loss.. but still there is.
> >>>>>
> >>>>> sysctl -w net.core.rmem_max=33554432
> >>>>> sysctl -w net.core.wmem_max=33554432
> >>>>> sysctl -w net.core.rmem_default=65536
> >>>>> sysctl -w net.core.wmem_default=65536
> >>>>> sysctl -w net.ipv4.tcp_mem='8388608 8388608 8388608'
> >>>>> sysctl -w net.ipv4.udp_mem='4096 174760 33554432'
> >>>>> sysctl -w net.ipv4.tcp_rmem='4096 87380 8388608'
> >>>>> sysctl -w net.ipv4.tcp_wmem='4096 65536 8388608'
> >>>>> sysctl -w net.ipv4.route.flush=1
> >>>>>
> >>>>> I am currently still seeing around 300 packets per second going to
> unknown ports. Below are the statistics.  That’s about 1/5th of all the
> packets received are not being processed… That seems a lot to me.
> >>>>>
> >>>>>  10:43:40 up 2 days,  5:11,  3 users,  load average: 1.52, 2.05, 2.17
> >>>>>
> >>>>> Every 1.0s: netstat -anus|grep -A 7 Udp:
>
>           Thu Jun 23 10:40:45 2016
> >>>>>
> >>>>> Udp:
> >>>>>     310870895 packets received
> >>>>>     61212884 packets to unknown port received.
> >>>>>     103338 packet receive errors
> >>>>>     312245302 packets sent
> >>>>>     RcvbufErrors: 103249
> >>>>>     SndbufErrors: 765
> >>>>>     InCsumErrors: 75
> >>>>>
> >>>>>
> >>>>>
> >>>>> I had to do a lot of buffer tweaking to get the RcvbufErrors down
> and even the SndbufErrors as every time it happens (at bursts -
> sporadically every 10 minutes, but definitely every half hour), one would
> get silence and the packet receive errors would should up by about between
> 200 and 800 packets.
> >>>>>
> >>>>> The load average can shoot up to 4.x at times.   Knowing that
> Sipwise Pro is on the same hardware, and they support up to 50.000 users,
> what am I missing?
> >>>>>
> >>>>> rtpengine is running in kernel. major contributor of CPU usage is
> actually MySQL regularly maxing out at 100%. Especially when it’s doing the
> fraud check. Below is a snapshot of top….
> >>>>>
> >>>>> top - 10:56:53 up 2 days,  5:24,  3 users,  load average: 2.39,
> 2.14, 1.94
> >>>>> Tasks: 184 total,   1 running, 183 sleeping,   0 stopped,   0 zombie
> >>>>> %Cpu(s): 25.3 us,  7.0 sy,  0.0 ni, 63.7 id,  1.0 wa,  0.0 hi,  2.9
> si,  0.0 st
> >>>>> KiB Mem:  12334464 total, 12157676 used,   176788 free,   144944
> buffers
> >>>>> KiB Swap:  2096124 total,        0 used,  2096124 free,  4430336
> cached
> >>>>>
> >>>>>   PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+
>  COMMAND
> >>>>>  4063 mysql     20   0 6127m 5.6g 7084 S  54.7 47.7 809:35.18 mysqld
>
> >>>>>  2576 root      20   0  253m 7176 1816 S   9.9  0.1 164:02.97
> rsyslogd
> >>>>>  5058 root      20   0 67176  11m 5308 S   6.0  0.1   7:05.16
> rate-o-mat
> >>>>> 15432 root      20   0  276m  12m 3696 S   5.0  0.1 117:56.92
> rtpengine
> >>>>>  5257 sems      20   0  873m  37m 7624 S   4.0  0.3 139:44.03
> ngcp-sems
> >>>>> 30996 kamailio  20   0  539m 100m  53m S   4.0  0.8   6:02.68
> kamailio
> >>>>>
> >>>>> Does anybody have any pointers I can try to completely eliminate the
> packet loss and where do these unknown port packets go to?
> >>>>>
> >>>>> Thanks
> >>>>> Walter.
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> _______________________________________________
> >>>>> Spce-user mailing list
> >>>>> Spce-user at lists.sipwise.com
> >>>>> https://lists.sipwise.com/listinfo/spce-user
> >>>>>
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sipwise.com/pipermail/spce-user_lists.sipwise.com/attachments/20160623/e518fc13/attachment-0001.html>

More information about the Spce-user mailing list