[Spce-user] 500 Internal Server error
Alexander Griesser
ag at anexia.at
Tue Mar 17 13:57:16 EDT 2015
Please run the following command:
echo "show full processlist" | mysql --defaults-file=/etc/mysql/debian.cnf | gzip -c > pl.txt.gz
And then upload the resulting file pl.txt.gz to somewhere so I can have a look at what’s going on.
Best,
Alexander Griesser
Head of Systems Operations
ANEXIA Internetdienstleistungs GmbH
E-Mail: ag at anexia.at<mailto:ag at anexia.at>
Web: http://www.anexia.at<http://www.anexia.at/>
Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt
Geschäftsführer: Alexander Windbichler
Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
Von: Tabi Tabe Tabi [mailto:tabi.tabe at gmail.com]
Gesendet: Dienstag, 17. März 2015 18:55
An: Alexander Griesser
Cc: Daniel Grotti; spce-user at lists.sipwise.com
Betreff: Re: [Spce-user] 500 Internal Server error
Hi Alex,
You are correct.
The main process running is Kamailio.
mysql> show processlist;
+--------+------------+-----------+------------+---------+--------+-----------+---------------------------------------------------- --------------------------------------------------+
| Id | User | Host | db | Command | Time | State | Info |
+--------+------------+-----------+------------+---------+--------+-----------+---------------------------------------------------- --------------------------------------------------+
| 22 | nagios | localhost | NULL | Sleep | 1 | | NULL |
| 406181 | kamailio | localhost | kamailio | Query | 101443 | query end | insert into acc (method,from_tag,to_tag,callid,sip_ code,sip_reason,time,time_hires,src_user,src_doma |
| 409077 | kamailio | localhost | kamailio | Query | 95831 | query end | insert into acc (method,from_tag,to_tag,callid,sip_ code,sip_reason,time,time_hires,src_user,src_doma |
| 411835 | kamailio | localhost | kamailio | Query | 90758 | query end | insert into acc (method,from_tag,to_tag,callid,sip_ code,sip_reason,time,time_hires,src_user,src_doma |
This is what is shown from show processlist.
Not sure how prevent this from happening.
Regards,
Tabi
On Tue, Mar 17, 2015 at 7:50 PM, Alexander Griesser <ag at anexia.at<mailto:ag at anexia.at>> wrote:
Well, increasing won’t help in the long run.
Whatever is maxing out your connections, it will max out the new limit pretty soon again, so you need to find out where these connections are coming from.
What does show processlist show? Are there many connections in sleep state?
Best,
Alexander Griesser
Head of Systems Operations
ANEXIA Internetdienstleistungs GmbH
E-Mail: ag at anexia.at<mailto:ag at anexia.at>
Web: http://www.anexia.at<http://www.anexia.at/>
Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt
Geschäftsführer: Alexander Windbichler
Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
Von: Tabi Tabe Tabi [mailto:tabi.tabe at gmail.com<mailto:tabi.tabe at gmail.com>]
Gesendet: Dienstag, 17. März 2015 18:49
An: Alexander Griesser
Cc: Daniel Grotti; spce-user at lists.sipwise.com<mailto:spce-user at lists.sipwise.com>
Betreff: Re: [Spce-user] 500 Internal Server error
Lol
I am using 2049 concurrent connections, even when the server is idling.
Thanks. Will increase the number of concurrent connections. What is the recommended max connection?
Regards,
Tabi
On Tue, Mar 17, 2015 at 7:40 PM, Alexander Griesser <ag at anexia.at<mailto:ag at anexia.at>> wrote:
Yah, you’re having too many open connections to your mysql database.
The default config should be 2048 connections, so you’ve somehow exhausted them.
Try to login to your mysql server from the CLI and examine the show processlist output:
# mysql --defaults-file=/etc/mysql/debian.cnf
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 291683
Server version: 5.5.41-0+wheezy1-log (Debian)
Copyright (c) 2000, 2014, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> show processlist;
+--------+------------------+-----------+----------+---------+------+-------+------------------+
| Id | User | Host | db | Command | Time | State | Info |
+--------+------------------+-----------+----------+---------+------+-------+------------------+
| 6 | kamailio | localhost | kamailio | Sleep | 7 | | NULL |
| 40 | kamailio | localhost | kamailio | Sleep | 9 | | NULL |
| 116 | nagios | localhost | NULL | Sleep | 5 | | NULL |
| 291683 | debian-sys-maint | localhost | NULL | Query | 0 | NULL | show processlist |
+--------+------------------+-----------+----------+---------+------+-------+------------------+
4 rows in set (0.00 sec)
You will probably have one query hanging at the top of this output which blocks all other queries from going through, or there’s something else wrong, but this output will help you find out, what it is.
As a quick fix, while still in the mysql console, you can try to increase the number of available connections by entering the command:
mysql> set global max_connections = 3096;
Query OK, 0 rows affected (0.00 sec)
best,
Alexander Griesser
Head of Systems Operations
ANEXIA Internetdienstleistungs GmbH
E-Mail: ag at anexia.at<mailto:ag at anexia.at>
Web: http://www.anexia.at<http://www.anexia.at/>
Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt
Geschäftsführer: Alexander Windbichler
Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
Von: Tabi Tabe Tabi [mailto:tabi.tabe at gmail.com<mailto:tabi.tabe at gmail.com>]
Gesendet: Dienstag, 17. März 2015 18:37
An: Daniel Grotti; Alexander Griesser
Cc: spce-user at lists.sipwise.com<mailto:spce-user at lists.sipwise.com>
Betreff: Re: [Spce-user] 500 Internal Server error
Thanks Alex/Daniel,
This is what I found:
Mar 17 18:55:27 sbc-1 ngcp-panel: ERROR: fatal error, id=13CC57719176611F, timestamp=2015-03-17T16:55:27.252979999Z, error=DBIx::Class::ResultSet::first(): DBI Connection failed: DBI connect('database=provisioning;host=localhost;port=3306','soap',...) failed: Too many connections at /usr/share/perl5/DBIx/Class/Storage/DBI.pm line 1249 at /usr/share/perl5/Catalyst/Authentication/Store/DBIx/Class/User.pm line 77
I have noticed a other errors related to too many connections. It is also related to errors occurring when creating a new subscribers and failed drop calls with same 500 internal error.
Regards,
Tabi
On Tue, Mar 17, 2015 at 7:14 PM, Daniel Grotti <dgrotti at sipwise.com<mailto:dgrotti at sipwise.com>> wrote:
Please grep the incident number in panel.log and panel-debig.log.
Daniel
On 17 Mar 2015 18:04, Tabi Tabe Tabi <tabi.tabe at gmail.com<mailto:tabi.tabe at gmail.com>> wrote:
>
> Hi,
> I am running SPCE 3.7.1. I get 500 Internal server error when login to the admin web portal. After logging out and try to login again, still get the same error below. Any ideas on the cause and how to fix it?
>
>
>
> 500 Internal server error
>
> We are terribly sorry, an exceptional error has occured:
>
> incident number: 13CC57719176611F
> time of incident: 2015-03-17T16:55:27.252979999Z
>
>
> If the login is successful, I get and NGINX bad gateway or timeout error.
>
> The server is a VM and has 6GB or RAM, 4 Core processor and 80GB disk space. the resources are hardly being used up.
>
>
> --
> ...Tabi
>
--
...Tabi
--
...Tabi
--
...Tabi
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sipwise.com/pipermail/spce-user_lists.sipwise.com/attachments/20150317/dd917c6a/attachment-0001.html>
More information about the Spce-user
mailing list