[Spce-user] 500 Internal Server error
Tabi Tabe Tabi
tabi.tabe at gmail.com
Wed Mar 18 13:10:49 EDT 2015
Hi Andre, Daniel, Alex and all,
Finally we found the cause. It suffices to say the SPCE now runs clean. See
the SQL processlist below, compared to the thousands we had opened
previously.
mysql> show processlist;
+------+------------+-----------+--------------+---------+------+-------+------------------+
| Id | User | Host | db | Command | Time | State |
Info |
+------+------------+-----------+--------------+---------+------+-------+------------------+
| 70 | nagios | localhost | NULL | Sleep | 10 | |
NULL |
| 143 | mediator | localhost | accounting | Sleep | 5 | |
NULL |
| 144 | kamailio | localhost | kamailio | Sleep | 5 | |
NULL |
| 148 | rateomat | localhost | billing | Sleep | 4 | |
NULL |
| 149 | rateomat | localhost | accounting | Sleep | 4 | |
NULL |
| 185 | kamailio | localhost | kamailio | Sleep | 5 | |
NULL |
| 5006 | kamailio | localhost | kamailio | Sleep | 34 | |
NULL |
| 5011 | kamailio | localhost | kamailio | Sleep | 35 | |
NULL |
| 5012 | kamailioro | localhost | kamailio | Sleep | 35 | |
NULL |
| 5013 | kamailio | localhost | kamailio | Sleep | 35 | |
NULL |
| 5020 | kamailio | localhost | kamailio | Sleep | 13 | |
NULL |
| 5027 | mediator | localhost | provisioning | Sleep | 29 | |
NULL |
| 5031 | kamailio | localhost | kamailio | Sleep | 17 | |
NULL |
| 5032 | kamailioro | localhost | kamailio | Sleep | 17 | |
NULL |
| 5033 | kamailio | localhost | kamailio | Sleep | 13 | |
NULL |
| 5034 | kamailioro | localhost | kamailio | Sleep | 13 | |
NULL |
| 5039 | root | localhost | NULL | Query | 0 | NULL |
show processlist |
+------+------------+-----------+--------------+---------+------+-------+------------------+
17 rows in set (0.00 sec)
We approached our cloud service provider in a nice way and found the
underlying openstack infrastructure was maxed out. Once the node was
cleaned up, the errors stopped and the total experience better.
We will put together a document and share with the forum as this has been a
hard learned lesson for using cloud infrastructure for real-time service.
Thank you all for your contribution. It helped us focus on the problem and
finally solved it.
Regards,
Tabi
On Wed, Mar 18, 2015 at 1:47 PM, Andreas Granig <agranig at sipwise.com> wrote:
> Hi,
>
> Yes, InnoDB is required. You probably don't have enough RAM for MySQL to
> start, try to decrease bufferpoolsize in your config.yml and do an
> "ngcpcfg apply". It usually takes half your RAM available during
> installation. Try reducing it to 512M or so to see if it helps.
>
> In regards to inserting acc taking very very long, it's most likely
> rather a system issue (storage got stuck or something) on your VM
> cluster than an SPCE issue. Just speculating here though.
>
> Andreas
>
> On 03/18/2015 11:09 AM, Tabi Tabe Tabi wrote:
> > Is InoDB required for SPCE?
> >
> > What is the standard MySQL config for SPCE? I would imagine that the
> > installation would configure MySql accordingly, unless some tuning is
> > required.
> > Any ideas?
> >
> > Regards,
> >
> > Tabi
> >
> > On Wed, Mar 18, 2015 at 11:56 AM, Tabi Tabe Tabi <tabi.tabe at gmail.com
> > <mailto:tabi.tabe at gmail.com>> wrote:
> >
> > Hi Daniel,
> >
> > Looking into the Mysql.err log file, this is what I see:
> >
> > InnoDB: Unable to lock /var/lib/mysql/ibdata1, error: 11
> > InnoDB: Check that you do not already have another mysqld process
> >
> >
> > 150318 1:44:50 [ERROR] Plugin 'InnoDB' init function returned error.
> > 150318 1:44:50 [ERROR] Plugin 'InnoDB' registration as a STORAGE
> > ENGINE failed.
> > 150318 1:44:50 [ERROR] Unknown/unsupported storage engine: InnoDB
> > 150318 1:44:50 [ERROR] Aborting
> >
> >
> >
> > and this one
> >
> >
> > 150318 2:26:51 [Warning] 'proxies_priv' entry '@ root at sbc-1'
> > ignored in --skip-name-resolve mode.
> > 150318 2:26:51 [Warning] 'user' entry 'root at sbc-1' ignored in
> > --skip-name-resolve mode.
> >
> >
> >
> > Not sure if this tells of any possible cause.
> >
> >
> > Regards,
> >
> > Tabi
> >
> > On Wed, Mar 18, 2015 at 11:24 AM, Tabi Tabe Tabi
> > <tabi.tabe at gmail.com <mailto:tabi.tabe at gmail.com>> wrote:
> >
> > Hi Daniel,
> >
> > We run SPCE of a VM (openstack) with 6GB RAM, 80GB HDD and
> > 10gbps NIC.
> >
> >
> > What's interesting is that it ran for a long time with no issues
> > and suddenly started have that error. Here is the resource
> > utilization screen dump
> >
> >
> > root at sbc-1:~-> # df -h
> > Filesystem Size
> > Used Avail Use% Mounted on
> > rootfs 79G
> > 5.5G 70G 8% /
> > udev 10M
> > 0 10M 0% /dev
> > tmpfs 599M
> > 184K 599M 1% /run
> > /dev/disk/by-uuid/07741205-d163-44df-ad67-d1b19bf0afb3 79G
> > 5.5G 70G 8% /
> > tmpfs 5.0M
> > 0 5.0M 0% /run/lock
> > tmpfs 1.2G
> > 0 1.2G 0% /run/shm
> >
> >
> >
> > root at sbc-1:~-> # df -i
> > Filesystem Inodes
> > IUsed IFree IUse% Mounted on
> > rootfs 5242880
> > 87584 5155296 2% /
> > udev 763931
> > 290 763641 1% /dev
> > tmpfs 765904
> > 213 765691 1% /run
> > /dev/disk/by-uuid/07741205-d163-44df-ad67-d1b19bf0afb3 5242880
> > 87584 5155296 2% /
> > tmpfs 765904
> > 2 765902 1% /run/lock
> > tmpfs 765904
> > 2 765902 1% /run/shm
> >
> >
> >
> > root at sbc-1:~-> # free -m
> > total used free shared buffers
> > cached
> > Mem: 5983 4803 1180 0 70
> > 2297
> > -/+ buffers/cache: 2435 3548
> > Swap: 0 0 0
> >
> > Regards,
> >
> > Tabi
> >
> >
> > On Wed, Mar 18, 2015 at 11:16 AM, Daniel Grotti
> > <dgrotti at sipwise.com <mailto:dgrotti at sipwise.com>> wrote:
> >
> > What are your system values ?
> > Are they satisfying our minimum requirements ?
> >
> > Daniel
> >
> >
> >
> > --
> > Meet us @ ANGACOM: Hall 10.1/booth N10
> > Exhibition and Congress for Broadband, Cable & Satellite:
> > 9-11 June
> > 2015, Cologne
> > --
> >
> > Daniel Grotti
> > VoIP Engineer
> >
> > Phone: +43(0)1 301 2032 <tel:%2B43%280%291%20301%202032>
> > Email: dgrotti at sipwise.com <mailto:dgrotti at sipwise.com>
> > Website: www.sipwise.com <http://www.sipwise.com>
> >
> > Particulars according Austrian Companies Code paragraph 14
> > "Sipwise GmbH“ - Europaring F15 – 2345 Brunn am Gebirge
> > FN:305595f, Commercial Court Vienna, ATU64002206
> >
> > On 03/18/2015 10:14 AM, Tabi Tabe Tabi wrote:
> > > Hi Daniel,
> > >
> > > It is certainly a MySql problem. However we installed a
> new version of
> > > SPCE and have exactly the same issue. The version of MySql
> is 5.5.40 an
> > > SPCE is 3.7.1. Previous versions all exhibited the same
> problem.
> > >
> > > I am thinking of using a script to kill all queries with
> time longer
> > > than 200 secs as an interim work around. But would be nice
> to know why
> > > this happens.
> > >
> > >
> > > Regards,
> > >
> > > Tabi
> > >
> > > On Wed, Mar 18, 2015 at 10:44 AM, Daniel Grotti <
> dgrotti at sipwise.com <mailto:dgrotti at sipwise.com>
> > > <mailto:dgrotti at sipwise.com <mailto:dgrotti at sipwise.com>>>
> wrote:
> > >
> > > Hi,
> > >
> > > Those stuck query are not normal, you must have a
> problem in myslq.
> > > Can't be that an insert into acc table takes 101443sec!
> > > Maybe you can take a look also into slow-query.log and
> mysql error.log
> > >
> > >
> > > Daniel
> > >
> > >
> > >
> > >
> > >
> > > --
> > > Meet us @ ANGACOM: Hall 10.1/booth N10Exhibition and
> Congress for
> > > Broadband, Cable & Satellite: 9-11 June 2015, Cologne
> > > --
> > >
> > > Daniel Grotti
> > > VoIP Engineer
> > >
> > > Phone: +43(0)1 301 2032
> > <tel:%2B43%280%291%20301%202032>
> > <tel:%2B43%280%291%20301%202032>
> > > Email: dgrotti at sipwise.com
> > <mailto:dgrotti at sipwise.com> <mailto:dgrotti at sipwise.com
> > <mailto:dgrotti at sipwise.com>>
> > > Website: www.sipwise.com <http://www.sipwise.com>
> > <http://www.sipwise.com>
> > >
> > > Particulars according Austrian Companies Code
> paragraph 14
> > > "Sipwise GmbH“ - Europaring F15 – 2345 Brunn am Gebirge
> > > FN:305595f, Commercial Court Vienna, ATU64002206
> > >
> > > On 03/17/2015 10:50 PM, Tabi Tabe Tabi wrote:
> > > > Correct Alex.
> > > >
> > > > So far I have traced them to error writing
> communication packets. On of
> > > > the things I will do is to update the
> max_allowed_packet value to the
> > > > number of INSERT queries SPCE runs for each
> transaction. I will see if
> > > > this makes any difference.
> > > >
> > > >
> > > > Regards,
> > > >
> > > > Tabi
> > > >
> > > >
> > > >
> > > > On Tue, Mar 17, 2015 at 11:45 PM, Alexander Griesser
> <ag at anexia.at <mailto:ag at anexia.at> <mailto:ag at anexia.at
> > <mailto:ag at anexia.at>>
> > > > <mailto:ag at anexia.at <mailto:ag at anexia.at> <mailto:
> ag at anexia.at
> > <mailto:ag at anexia.at>>>> wrote:
> > > >
> > > > Well, the problem ist, that all the queries that
> are hanging here
> > > > are INSERT queries into the acc table, so
> they’re just hanging and
> > > > are not able to finish, so you will likely
> experience data loss
> > > > (i.e. calls not properly recorded, etc.).____
> > > >
> > > > __ __
> > > >
> > > > best,____
> > > >
> > > > __ __
> > > >
> > > > *Alexander Griesser____*
> > > >
> > > > Head of Systems Operations____
> > > >
> > > > __ __
> > > >
> > > > ANEXIA Internetdienstleistungs GmbH____
> > > >
> > > > __ __
> > > >
> > > > E-Mail: ag at anexia.at <mailto:ag at anexia.at>
> <mailto:ag at anexia.at
> > <mailto:ag at anexia.at>>
> > > <mailto:ag at anexia.at <mailto:ag at anexia.at>
> > <mailto:ag at anexia.at <mailto:ag at anexia.at>>>____
> > > >
> > > > Web: http://www.anexia.at <http://www.anexia.at/
> >____
> > > >
> > > > __ __
> > > >
> > > > Anschrift Hauptsitz Klagenfurt:
> Feldkirchnerstraße 140, 9020
> > > > Klagenfurt____
> > > >
> > > > Geschäftsführer: Alexander Windbichler____
> > > >
> > > > Firmenbuch: FN 289918a | Gerichtsstand:
> Klagenfurt | UID-Nummer: AT
> > > > U63216601____
> > > >
> > > > __ __
> > > >
> > > > *Von:*Spce-user [mailto:
> spce-user-bounces at lists.sipwise.com
> > <mailto:spce-user-bounces at lists.sipwise.com>
> > > <mailto:spce-user-bounces at lists.sipwise.com
> > <mailto:spce-user-bounces at lists.sipwise.com>>
> > > > <mailto:spce-user-bounces at lists.sipwise.com
> > <mailto:spce-user-bounces at lists.sipwise.com>
> > > <mailto:spce-user-bounces at lists.sipwise.com
> > <mailto:spce-user-bounces at lists.sipwise.com>>>] *Im Auftrag
> > von *Tabi
> > > > Tabe Tabi
> > > > *Gesendet:* Dienstag, 17. März 2015 22:43
> > > > *An:* Daniel Grotti
> > > > *Cc:* spce-user at lists.sipwise.com <mailto:
> spce-user at lists.sipwise.com>
> > > <mailto:spce-user at lists.sipwise.com
> > <mailto:spce-user at lists.sipwise.com>>
> > > <mailto:spce-user at lists.sipwise.com
> > <mailto:spce-user at lists.sipwise.com>
> > > <mailto:spce-user at lists.sipwise.com
> > <mailto:spce-user at lists.sipwise.com>>>
> > > > *Betreff:* Re: [Spce-user] 500 Internal Server
> error____
> > > >
> > > > __ __
> > > >
> > > > Update on this issue.____
> > > >
> > > > __ __
> > > >
> > > > I found that increasing the max_connections in
> mysql helped. I am
> > > > trying a few things and will update the forum
> after testing
> > > them.____
> > > >
> > > > __ __
> > > >
> > > > Regards,____
> > > >
> > > > __ __
> > > >
> > > > Tabi____
> > > >
> > > > ____
> > > >
> > > > __ __
> > > >
> > > > On Tue, Mar 17, 2015 at 7:14 PM, Daniel Grotti <
> dgrotti at sipwise.com <mailto:dgrotti at sipwise.com>
> > <mailto:dgrotti at sipwise.com <mailto:dgrotti at sipwise.com>>
> > > > <mailto:dgrotti at sipwise.com
> > <mailto:dgrotti at sipwise.com> <mailto:dgrotti at sipwise.com
> > <mailto:dgrotti at sipwise.com>>>>
> > > wrote:____
> > > >
> > > > Please grep the incident number in panel.log
> and panel-debig.log.
> > > >
> > > > Daniel____
> > > >
> > > > On 17 Mar 2015 18:04, Tabi Tabe Tabi <
> tabi.tabe at gmail.com <mailto:tabi.tabe at gmail.com>
> > <mailto:tabi.tabe at gmail.com <mailto:tabi.tabe at gmail.com>>
> > > > <mailto:tabi.tabe at gmail.com
> > <mailto:tabi.tabe at gmail.com> <mailto:tabi.tabe at gmail.com
> > <mailto:tabi.tabe at gmail.com>>>> wrote:
> > > > >
> > > > > Hi,
> > > > > I am running SPCE 3.7.1. I get 500
> > Internal server error when
> > > > login to the admin web portal. After logging
> > out and try to
> > > > login again, still get the same error below.
> > Any ideas on the
> > > > cause and how to fix it?
> > > > >
> > > > >
> > > > >
> > > > > 500 Internal server error
> > > > >
> > > > > We are terribly sorry, an exceptional
> > error has occured:
> > > > >
> > > > > incident number: 13CC57719176611F
> > > > > time of incident:
> > 2015-03-17T16:55:27.252979999Z
> > > > >
> > > > >
> > > > > If the login is successful, I get and
> > NGINX bad gateway or
> > > > timeout error.
> > > > >
> > > > > The server is a VM and has 6GB or RAM, 4
> > Core processor and
> > > > 80GB disk space. the resources are hardly
> > being used up.
> > > > >
> > > > >
> > > > > --
> > > > > ...Tabi
> > > > >____
> > > >
> > > >
> > > >
> > > > ____
> > > >
> > > > __ __
> > > >
> > > > -- ____
> > > >
> > > > ...Tabi____
> > > >
> > > > __ __
> > > >
> > > >
> > > >
> > > >
> > > > --
> > > > ...Tabi
> > > >
> > >
> > >
> > >
> > >
> > > --
> > > ...Tabi
> > >
> >
> >
> >
> >
> > --
> > ...Tabi
> >
> >
> >
> >
> > --
> > ...Tabi
> >
> >
> >
> >
> > --
> > ...Tabi
> >
> >
> >
> > _______________________________________________
> > Spce-user mailing list
> > Spce-user at lists.sipwise.com
> > https://lists.sipwise.com/listinfo/spce-user
> >
> _______________________________________________
> Spce-user mailing list
> Spce-user at lists.sipwise.com
> https://lists.sipwise.com/listinfo/spce-user
>
--
...Tabi
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sipwise.com/pipermail/spce-user_lists.sipwise.com/attachments/20150318/1eed932e/attachment-0001.html>
More information about the Spce-user
mailing list