<div dir="ltr">Hi Andre, Daniel, Alex and all,<div><br></div><div>Finally we found the cause. It suffices to say the SPCE now runs clean. See the SQL processlist below, compared to the thousands we had opened previously.</div><div><br></div><div><br></div><div><div>mysql> show processlist;</div><div>+------+------------+-----------+--------------+---------+------+-------+------------------+</div><div>| Id | User | Host | db | Command | Time | State | Info |</div><div>+------+------------+-----------+--------------+---------+------+-------+------------------+</div><div>| 70 | nagios | localhost | NULL | Sleep | 10 | | NULL |</div><div>| 143 | mediator | localhost | accounting | Sleep | 5 | | NULL |</div><div>| 144 | kamailio | localhost | kamailio | Sleep | 5 | | NULL |</div><div>| 148 | rateomat | localhost | billing | Sleep | 4 | | NULL |</div><div>| 149 | rateomat | localhost | accounting | Sleep | 4 | | NULL |</div><div>| 185 | kamailio | localhost | kamailio | Sleep | 5 | | NULL |</div><div>| 5006 | kamailio | localhost | kamailio | Sleep | 34 | | NULL |</div><div>| 5011 | kamailio | localhost | kamailio | Sleep | 35 | | NULL |</div><div>| 5012 | kamailioro | localhost | kamailio | Sleep | 35 | | NULL |</div><div>| 5013 | kamailio | localhost | kamailio | Sleep | 35 | | NULL |</div><div>| 5020 | kamailio | localhost | kamailio | Sleep | 13 | | NULL |</div><div>| 5027 | mediator | localhost | provisioning | Sleep | 29 | | NULL |</div><div>| 5031 | kamailio | localhost | kamailio | Sleep | 17 | | NULL |</div><div>| 5032 | kamailioro | localhost | kamailio | Sleep | 17 | | NULL |</div><div>| 5033 | kamailio | localhost | kamailio | Sleep | 13 | | NULL |</div><div>| 5034 | kamailioro | localhost | kamailio | Sleep | 13 | | NULL |</div><div>| 5039 | root | localhost | NULL | Query | 0 | NULL | show processlist |</div><div>+------+------------+-----------+--------------+---------+------+-------+------------------+</div><div>17 rows in set (0.00 sec)</div></div><div><br></div><div><br></div><div>We approached our cloud service provider in a nice way and found the underlying openstack infrastructure was maxed out. Once the node was cleaned up, the errors stopped and the total experience better.</div><div><br></div><div>We will put together a document and share with the forum as this has been a hard learned lesson for using cloud infrastructure for real-time service.</div><div><br></div><div>Thank you all for your contribution. It helped us focus on the problem and finally solved it.</div><div><br></div><div><br></div><div>Regards,</div><div><br></div><div>Tabi</div><img src="http://t.signaletre.com/e1t/o/5/f18dQhb0S7ks8dDMPbW2n0x6l2B9gXrN7sKj6v4LTTlW653lLR4X9P3HW64QWy01pctGFW2JNYSB1k1H6H0?si=6355392162955264&pi=157a6b8c-b2eb-4ba9-e8b1-8c6de0f512bb" style="display:none!important" height="1" width="1"></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Mar 18, 2015 at 1:47 PM, Andreas Granig <span dir="ltr"><<a href="mailto:agranig@sipwise.com" target="_blank">agranig@sipwise.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<br>
<br>
Yes, InnoDB is required. You probably don't have enough RAM for MySQL to<br>
start, try to decrease bufferpoolsize in your config.yml and do an<br>
"ngcpcfg apply". It usually takes half your RAM available during<br>
installation. Try reducing it to 512M or so to see if it helps.<br>
<br>
In regards to inserting acc taking very very long, it's most likely<br>
rather a system issue (storage got stuck or something) on your VM<br>
cluster than an SPCE issue. Just speculating here though.<br>
<br>
Andreas<br>
<span class=""><br>
On 03/18/2015 11:09 AM, Tabi Tabe Tabi wrote:<br>
> Is InoDB required for SPCE?<br>
><br>
> What is the standard MySQL config for SPCE? I would imagine that the<br>
> installation would configure MySql accordingly, unless some tuning is<br>
> required.<br>
> Any ideas?<br>
><br>
> Regards,<br>
><br>
> Tabi<br>
><br>
</span><span class="">> On Wed, Mar 18, 2015 at 11:56 AM, Tabi Tabe Tabi <<a href="mailto:tabi.tabe@gmail.com">tabi.tabe@gmail.com</a><br>
</span><span class="">> <mailto:<a href="mailto:tabi.tabe@gmail.com">tabi.tabe@gmail.com</a>>> wrote:<br>
><br>
> Hi Daniel,<br>
><br>
> Looking into the Mysql.err log file, this is what I see:<br>
><br>
> InnoDB: Unable to lock /var/lib/mysql/ibdata1, error: 11<br>
> InnoDB: Check that you do not already have another mysqld process<br>
><br>
><br>
> 150318 1:44:50 [ERROR] Plugin 'InnoDB' init function returned error.<br>
> 150318 1:44:50 [ERROR] Plugin 'InnoDB' registration as a STORAGE<br>
> ENGINE failed.<br>
> 150318 1:44:50 [ERROR] Unknown/unsupported storage engine: InnoDB<br>
> 150318 1:44:50 [ERROR] Aborting<br>
><br>
><br>
><br>
> and this one<br>
><br>
><br>
> 150318 2:26:51 [Warning] 'proxies_priv' entry '@ root@sbc-1'<br>
> ignored in --skip-name-resolve mode.<br>
> 150318 2:26:51 [Warning] 'user' entry 'root@sbc-1' ignored in<br>
> --skip-name-resolve mode.<br>
><br>
><br>
><br>
> Not sure if this tells of any possible cause.<br>
><br>
><br>
> Regards,<br>
><br>
> Tabi<br>
><br>
> On Wed, Mar 18, 2015 at 11:24 AM, Tabi Tabe Tabi<br>
</span><div><div class="h5">> <<a href="mailto:tabi.tabe@gmail.com">tabi.tabe@gmail.com</a> <mailto:<a href="mailto:tabi.tabe@gmail.com">tabi.tabe@gmail.com</a>>> wrote:<br>
><br>
> Hi Daniel,<br>
><br>
> We run SPCE of a VM (openstack) with 6GB RAM, 80GB HDD and<br>
> 10gbps NIC.<br>
><br>
><br>
> What's interesting is that it ran for a long time with no issues<br>
> and suddenly started have that error. Here is the resource<br>
> utilization screen dump<br>
><br>
><br>
> root@sbc-1:~-> # df -h<br>
> Filesystem Size<br>
> Used Avail Use% Mounted on<br>
> rootfs 79G<br>
> 5.5G 70G 8% /<br>
> udev 10M<br>
> 0 10M 0% /dev<br>
> tmpfs 599M<br>
> 184K 599M 1% /run<br>
> /dev/disk/by-uuid/07741205-d163-44df-ad67-d1b19bf0afb3 79G<br>
> 5.5G 70G 8% /<br>
> tmpfs 5.0M<br>
> 0 5.0M 0% /run/lock<br>
> tmpfs 1.2G<br>
> 0 1.2G 0% /run/shm<br>
><br>
><br>
><br>
> root@sbc-1:~-> # df -i<br>
> Filesystem Inodes<br>
> IUsed IFree IUse% Mounted on<br>
> rootfs 5242880<br>
> 87584 5155296 2% /<br>
> udev 763931<br>
> 290 763641 1% /dev<br>
> tmpfs 765904<br>
> 213 765691 1% /run<br>
> /dev/disk/by-uuid/07741205-d163-44df-ad67-d1b19bf0afb3 5242880<br>
> 87584 5155296 2% /<br>
> tmpfs 765904<br>
> 2 765902 1% /run/lock<br>
> tmpfs 765904<br>
> 2 765902 1% /run/shm<br>
><br>
><br>
><br>
> root@sbc-1:~-> # free -m<br>
> total used free shared buffers<br>
> cached<br>
> Mem: 5983 4803 1180 0 70<br>
> 2297<br>
> -/+ buffers/cache: 2435 3548<br>
> Swap: 0 0 0<br>
><br>
> Regards,<br>
><br>
> Tabi<br>
><br>
><br>
> On Wed, Mar 18, 2015 at 11:16 AM, Daniel Grotti<br>
</div></div><span class="">> <<a href="mailto:dgrotti@sipwise.com">dgrotti@sipwise.com</a> <mailto:<a href="mailto:dgrotti@sipwise.com">dgrotti@sipwise.com</a>>> wrote:<br>
><br>
> What are your system values ?<br>
> Are they satisfying our minimum requirements ?<br>
><br>
> Daniel<br>
><br>
><br>
><br>
> --<br>
> Meet us @ ANGACOM: Hall 10.1/booth N10<br>
> Exhibition and Congress for Broadband, Cable & Satellite:<br>
> 9-11 June<br>
> 2015, Cologne<br>
> --<br>
><br>
> Daniel Grotti<br>
> VoIP Engineer<br>
><br>
</span><span class="">> Phone: <a href="tel:%2B43%280%291%20301%202032" value="+4313012032">+43(0)1 301 2032</a> <tel:%2B43%280%291%20301%202032><br>
> Email: <a href="mailto:dgrotti@sipwise.com">dgrotti@sipwise.com</a> <mailto:<a href="mailto:dgrotti@sipwise.com">dgrotti@sipwise.com</a>><br>
> Website: <a href="http://www.sipwise.com" target="_blank">www.sipwise.com</a> <<a href="http://www.sipwise.com" target="_blank">http://www.sipwise.com</a>><br>
><br>
> Particulars according Austrian Companies Code paragraph 14<br>
> "Sipwise GmbH“ - Europaring F15 – 2345 Brunn am Gebirge<br>
> FN:305595f, Commercial Court Vienna, ATU64002206<br>
><br>
</span><span class="">> On 03/18/2015 10:14 AM, Tabi Tabe Tabi wrote:<br>
> > Hi Daniel,<br>
> ><br>
> > It is certainly a MySql problem. However we installed a new version of<br>
> > SPCE and have exactly the same issue. The version of MySql is 5.5.40 an<br>
> > SPCE is 3.7.1. Previous versions all exhibited the same problem.<br>
> ><br>
> > I am thinking of using a script to kill all queries with time longer<br>
> > than 200 secs as an interim work around. But would be nice to know why<br>
> > this happens.<br>
> ><br>
> ><br>
> > Regards,<br>
> ><br>
> > Tabi<br>
> ><br>
> > On Wed, Mar 18, 2015 at 10:44 AM, Daniel Grotti <<a href="mailto:dgrotti@sipwise.com">dgrotti@sipwise.com</a> <mailto:<a href="mailto:dgrotti@sipwise.com">dgrotti@sipwise.com</a>><br>
</span><span class="">> > <mailto:<a href="mailto:dgrotti@sipwise.com">dgrotti@sipwise.com</a> <mailto:<a href="mailto:dgrotti@sipwise.com">dgrotti@sipwise.com</a>>>> wrote:<br>
> ><br>
</span><span class="">> > Hi,<br>
> ><br>
> > Those stuck query are not normal, you must have a problem in myslq.<br>
> > Can't be that an insert into acc table takes 101443sec!<br>
> > Maybe you can take a look also into slow-query.log and mysql error.log<br>
> ><br>
> ><br>
> > Daniel<br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> > --<br>
> > Meet us @ ANGACOM: Hall 10.1/booth N10Exhibition and Congress for<br>
> > Broadband, Cable & Satellite: 9-11 June 2015, Cologne<br>
> > --<br>
> ><br>
> > Daniel Grotti<br>
> > VoIP Engineer<br>
> ><br>
> > Phone: <a href="tel:%2B43%280%291%20301%202032" value="+4313012032">+43(0)1 301 2032</a><br>
> <tel:%2B43%280%291%20301%202032><br>
> <tel:%2B43%280%291%20301%202032><br>
> > Email: <a href="mailto:dgrotti@sipwise.com">dgrotti@sipwise.com</a><br>
</span>> <mailto:<a href="mailto:dgrotti@sipwise.com">dgrotti@sipwise.com</a>> <mailto:<a href="mailto:dgrotti@sipwise.com">dgrotti@sipwise.com</a><br>
<div><div class="h5">> <mailto:<a href="mailto:dgrotti@sipwise.com">dgrotti@sipwise.com</a>>><br>
> > Website: <a href="http://www.sipwise.com" target="_blank">www.sipwise.com</a> <<a href="http://www.sipwise.com" target="_blank">http://www.sipwise.com</a>><br>
> <<a href="http://www.sipwise.com" target="_blank">http://www.sipwise.com</a>><br>
> ><br>
> > Particulars according Austrian Companies Code paragraph 14<br>
> > "Sipwise GmbH“ - Europaring F15 – 2345 Brunn am Gebirge<br>
> > FN:305595f, Commercial Court Vienna, ATU64002206<br>
> ><br>
> > On 03/17/2015 10:50 PM, Tabi Tabe Tabi wrote:<br>
> > > Correct Alex.<br>
> > ><br>
> > > So far I have traced them to error writing communication packets. On of<br>
> > > the things I will do is to update the max_allowed_packet value to the<br>
> > > number of INSERT queries SPCE runs for each transaction. I will see if<br>
> > > this makes any difference.<br>
> > ><br>
> > ><br>
> > > Regards,<br>
> > ><br>
> > > Tabi<br>
> > ><br>
> > ><br>
> > ><br>
> > > On Tue, Mar 17, 2015 at 11:45 PM, Alexander Griesser <<a href="mailto:ag@anexia.at">ag@anexia.at</a> <mailto:<a href="mailto:ag@anexia.at">ag@anexia.at</a>> <mailto:<a href="mailto:ag@anexia.at">ag@anexia.at</a><br>
> <mailto:<a href="mailto:ag@anexia.at">ag@anexia.at</a>>><br>
> > > <mailto:<a href="mailto:ag@anexia.at">ag@anexia.at</a> <mailto:<a href="mailto:ag@anexia.at">ag@anexia.at</a>> <mailto:<a href="mailto:ag@anexia.at">ag@anexia.at</a><br>
> <mailto:<a href="mailto:ag@anexia.at">ag@anexia.at</a>>>>> wrote:<br>
> > ><br>
> > > Well, the problem ist, that all the queries that are hanging here<br>
> > > are INSERT queries into the acc table, so they’re just hanging and<br>
> > > are not able to finish, so you will likely experience data loss<br>
> > > (i.e. calls not properly recorded, etc.).____<br>
> > ><br>
> > > __ __<br>
> > ><br>
> > > best,____<br>
> > ><br>
> > > __ __<br>
> > ><br>
> > > *Alexander Griesser____*<br>
> > ><br>
> > > Head of Systems Operations____<br>
> > ><br>
> > > __ __<br>
> > ><br>
> > > ANEXIA Internetdienstleistungs GmbH____<br>
> > ><br>
> > > __ __<br>
> > ><br>
> > > E-Mail: <a href="mailto:ag@anexia.at">ag@anexia.at</a> <mailto:<a href="mailto:ag@anexia.at">ag@anexia.at</a>> <mailto:<a href="mailto:ag@anexia.at">ag@anexia.at</a><br>
> <mailto:<a href="mailto:ag@anexia.at">ag@anexia.at</a>>><br>
> > <mailto:<a href="mailto:ag@anexia.at">ag@anexia.at</a> <mailto:<a href="mailto:ag@anexia.at">ag@anexia.at</a>><br>
> <mailto:<a href="mailto:ag@anexia.at">ag@anexia.at</a> <mailto:<a href="mailto:ag@anexia.at">ag@anexia.at</a>>>>____<br>
> > ><br>
> > > Web: <a href="http://www.anexia.at" target="_blank">http://www.anexia.at</a> <<a href="http://www.anexia.at/" target="_blank">http://www.anexia.at/</a>>____<br>
> > ><br>
> > > __ __<br>
> > ><br>
> > > Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020<br>
> > > Klagenfurt____<br>
> > ><br>
> > > Geschäftsführer: Alexander Windbichler____<br>
> > ><br>
> > > Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT<br>
> > > U63216601____<br>
> > ><br>
> > > __ __<br>
> > ><br>
> > > *Von:*Spce-user [mailto:<a href="mailto:spce-user-bounces@lists.sipwise.com">spce-user-bounces@lists.sipwise.com</a><br>
> <mailto:<a href="mailto:spce-user-bounces@lists.sipwise.com">spce-user-bounces@lists.sipwise.com</a>><br>
> > <mailto:<a href="mailto:spce-user-bounces@lists.sipwise.com">spce-user-bounces@lists.sipwise.com</a><br>
> <mailto:<a href="mailto:spce-user-bounces@lists.sipwise.com">spce-user-bounces@lists.sipwise.com</a>>><br>
> > > <mailto:<a href="mailto:spce-user-bounces@lists.sipwise.com">spce-user-bounces@lists.sipwise.com</a><br>
> <mailto:<a href="mailto:spce-user-bounces@lists.sipwise.com">spce-user-bounces@lists.sipwise.com</a>><br>
> > <mailto:<a href="mailto:spce-user-bounces@lists.sipwise.com">spce-user-bounces@lists.sipwise.com</a><br>
> <mailto:<a href="mailto:spce-user-bounces@lists.sipwise.com">spce-user-bounces@lists.sipwise.com</a>>>>] *Im Auftrag<br>
> von *Tabi<br>
> > > Tabe Tabi<br>
> > > *Gesendet:* Dienstag, 17. März 2015 22:43<br>
> > > *An:* Daniel Grotti<br>
> > > *Cc:* <a href="mailto:spce-user@lists.sipwise.com">spce-user@lists.sipwise.com</a> <mailto:<a href="mailto:spce-user@lists.sipwise.com">spce-user@lists.sipwise.com</a>><br>
> > <mailto:<a href="mailto:spce-user@lists.sipwise.com">spce-user@lists.sipwise.com</a><br>
> <mailto:<a href="mailto:spce-user@lists.sipwise.com">spce-user@lists.sipwise.com</a>>><br>
> > <mailto:<a href="mailto:spce-user@lists.sipwise.com">spce-user@lists.sipwise.com</a><br>
> <mailto:<a href="mailto:spce-user@lists.sipwise.com">spce-user@lists.sipwise.com</a>><br>
> > <mailto:<a href="mailto:spce-user@lists.sipwise.com">spce-user@lists.sipwise.com</a><br>
> <mailto:<a href="mailto:spce-user@lists.sipwise.com">spce-user@lists.sipwise.com</a>>>><br>
> > > *Betreff:* Re: [Spce-user] 500 Internal Server error____<br>
> > ><br>
> > > __ __<br>
> > ><br>
> > > Update on this issue.____<br>
> > ><br>
> > > __ __<br>
> > ><br>
> > > I found that increasing the max_connections in mysql helped. I am<br>
> > > trying a few things and will update the forum after testing<br>
> > them.____<br>
> > ><br>
> > > __ __<br>
> > ><br>
> > > Regards,____<br>
> > ><br>
> > > __ __<br>
> > ><br>
> > > Tabi____<br>
> > ><br>
> > > ____<br>
> > ><br>
> > > __ __<br>
> > ><br>
> > > On Tue, Mar 17, 2015 at 7:14 PM, Daniel Grotti <<a href="mailto:dgrotti@sipwise.com">dgrotti@sipwise.com</a> <mailto:<a href="mailto:dgrotti@sipwise.com">dgrotti@sipwise.com</a>><br>
> <mailto:<a href="mailto:dgrotti@sipwise.com">dgrotti@sipwise.com</a> <mailto:<a href="mailto:dgrotti@sipwise.com">dgrotti@sipwise.com</a>>><br>
> > > <mailto:<a href="mailto:dgrotti@sipwise.com">dgrotti@sipwise.com</a><br>
> <mailto:<a href="mailto:dgrotti@sipwise.com">dgrotti@sipwise.com</a>> <mailto:<a href="mailto:dgrotti@sipwise.com">dgrotti@sipwise.com</a><br>
> <mailto:<a href="mailto:dgrotti@sipwise.com">dgrotti@sipwise.com</a>>>>><br>
> > wrote:____<br>
> > ><br>
> > > Please grep the incident number in panel.log and panel-debig.log.<br>
> > ><br>
> > > Daniel____<br>
> > ><br>
> > > On 17 Mar 2015 18:04, Tabi Tabe Tabi <<a href="mailto:tabi.tabe@gmail.com">tabi.tabe@gmail.com</a> <mailto:<a href="mailto:tabi.tabe@gmail.com">tabi.tabe@gmail.com</a>><br>
> <mailto:<a href="mailto:tabi.tabe@gmail.com">tabi.tabe@gmail.com</a> <mailto:<a href="mailto:tabi.tabe@gmail.com">tabi.tabe@gmail.com</a>>><br>
> > > <mailto:<a href="mailto:tabi.tabe@gmail.com">tabi.tabe@gmail.com</a><br>
> <mailto:<a href="mailto:tabi.tabe@gmail.com">tabi.tabe@gmail.com</a>> <mailto:<a href="mailto:tabi.tabe@gmail.com">tabi.tabe@gmail.com</a><br>
> <mailto:<a href="mailto:tabi.tabe@gmail.com">tabi.tabe@gmail.com</a>>>>> wrote:<br>
> > > ><br>
> > > > Hi,<br>
> > > > I am running SPCE 3.7.1. I get 500<br>
> Internal server error when<br>
> > > login to the admin web portal. After logging<br>
> out and try to<br>
> > > login again, still get the same error below.<br>
> Any ideas on the<br>
> > > cause and how to fix it?<br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > > 500 Internal server error<br>
> > > ><br>
> > > > We are terribly sorry, an exceptional<br>
> error has occured:<br>
> > > ><br>
> > > > incident number: 13CC57719176611F<br>
> > > > time of incident:<br>
> 2015-03-17T16:55:27.252979999Z<br>
> > > ><br>
> > > ><br>
> > > > If the login is successful, I get and<br>
> NGINX bad gateway or<br>
> > > timeout error.<br>
> > > ><br>
> > > > The server is a VM and has 6GB or RAM, 4<br>
> Core processor and<br>
> > > 80GB disk space. the resources are hardly<br>
> being used up.<br>
> > > ><br>
> > > ><br>
> > > > --<br>
> > > > ...Tabi<br>
> > > >____<br>
> > ><br>
> > ><br>
> > ><br>
> > > ____<br>
> > ><br>
> > > __ __<br>
> > ><br>
> > > -- ____<br>
> > ><br>
> > > ...Tabi____<br>
> > ><br>
> > > __ __<br>
> > ><br>
> > ><br>
> > ><br>
> > ><br>
> > > --<br>
> > > ...Tabi<br>
> > ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> > --<br>
> > ...Tabi<br>
> ><br>
><br>
><br>
><br>
><br>
> --<br>
> ...Tabi<br>
><br>
><br>
><br>
><br>
> --<br>
> ...Tabi<br>
><br>
><br>
><br>
><br>
> --<br>
> ...Tabi<br>
><br>
><br>
><br>
</div></div>> _______________________________________________<br>
> Spce-user mailing list<br>
> <a href="mailto:Spce-user@lists.sipwise.com">Spce-user@lists.sipwise.com</a><br>
> <a href="https://lists.sipwise.com/listinfo/spce-user" target="_blank">https://lists.sipwise.com/listinfo/spce-user</a><br>
><br>
_______________________________________________<br>
Spce-user mailing list<br>
<a href="mailto:Spce-user@lists.sipwise.com">Spce-user@lists.sipwise.com</a><br>
<a href="https://lists.sipwise.com/listinfo/spce-user" target="_blank">https://lists.sipwise.com/listinfo/spce-user</a><br>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature">...Tabi<div><br></div></div>
</div>