[Spce-user] sip registration stuck 3.5.16

Raúl Alexis Betancor Santana rabs at dimension-virtual.com
Fri Nov 21 10:24:46 EST 2014


As I supposed ... if you get that error ... that means a BAD, VERY BAD configured SPCE, you should have messed with IP's or socks files ... you should look at kamailio-proxy.log and see the errors. 

----- Mensaje original -----

De: "Theo" <axessofficetheo at gmail.com> 
Para: "Andreas Granig" <agranig at sipwise.com> 
CC: spce-user at lists.sipwise.com 
Enviados: Viernes, 21 de Noviembre 2014 15:25:55 
Asunto: Re: [Spce-user] sip registration stuck 3.5.16 

ngcp-sercmd proxy ul.dump 
error: 500 - Internal error creating aor struct 

Any ideas on that? 

On Fri, Nov 21, 2014 at 5:10 PM, Andreas Granig < agranig at sipwise.com > wrote: 


Just to add my 2 cents: 

The proxy (kamailio) uses in-memory caching for registrations and writes 
them back to DB. The panel displays the registrations from the DB. 

Now if you delete something from the DB, it's not reflected in the 
proxy, because it still will be in the cache. So what we're doing is to 
delete a registration from the cache (by telling the proxy to purge it), 
and the proxy will propagate it down to the DB. 

The same happens if the proxy detects that a registration has expired. 
It will delete it from the cache and propagate the deletion down to the DB. 

Now here is where it's getting interesting: if the registration is no 
longer in the cache (for whatever reason) and it still is in the DB (for 
whatever else reason), you see it in the panel and can click on the 
delete button, but as the deletion request is sent to the proxy, it will 
just do nothing because it doesn't have it in the cache, so it will 
still persist in the DB. 

This scenario should never happen (famous last words), so to 
troubleshoot it, you'd need to dump the registrations from the cache 
using "ngcp-sercmd proxy ul.dump" and check the count (see 
"Stats"->"Records") and compare it to the db by doing "mysql kamailio -e 
"select id, username, domain, expires from location". 

Take into account that the cache count can/will potentially be lower 
than the db count because the proxy batches deletion in a separate 
process, which is executed every 60 sec. So to fetch the "real" expired 
registrations, do this: "mysql kamailio -e "select id, username, domain, 
expires from location where expires < date_sub(now(), interval 90 second)" 

So this will give you a representation of your current state. Note that 
if a registration is not in the cache but only in the DB, it's not a 
real operational issue, because the proxy will not know about it. 

It still doesn't answer the question how this can even happen, so you 
need to provide a full trace ("tcpdump -i any -s 0 -w /tmp/reg.pcap port 
5060") of a complete cycle (fresh registration and subsequent 
registrations, which will then cause the issue you described), along 
with the full kamailio-proxy.log, so we can check for errors etc. Also 
provide the output of "ngcp-sercmd proxy ul.dump" and "mysql kamailio -e 
"select id, username, domain, expires from location" along with the 
timestamps when you executed those commands. 

Andreas 

On 11/21/2014 03:41 PM, Raúl Alexis Betancor Santana wrote: 
> 
> Helo Aziz ... let's talk about ... 
> 
> 1- Doesn't matter ... if you have more than one register for the same 
> user, the only thing that happens it's that on an outbound request to 
> that subscriber, SPCE will launch 2 branches ... one will fail and the 
> other one will sucess. No big deal at all. 
> 2- That should not happens ... as expired registration should be 
> auto-erased. Have you installed the system from 0 ? .. or did you 
> upgrade from older version? 
> 3- That sounds as an upgrades SPCE version that didn't complete the upgrade. 
> 
> 
> ------------------------------------------------------------------------ 
> *De: *"Aziz Palmer" <Aziz.Palmer at AirFIBER.com> 
> *Para: *"Raúl Alexis Betancor Santana" < rabs at dimension-virtual.com >, 
> spce-user at lists.sipwise.com 
> *Enviados: *Viernes, 21 de Noviembre 2014 14:31:25 
> *Asunto: *RE: sip registration stuck 3.5.16 
> 
> Thank you Raul for your input and suggestions. Not to get into a long 
> discussion but I at first didn't think it was a big deal until the 
> following accours 
> 1. the network environment at subscriber changes (change of IP) - 
> asterisk registers and now you have 2 active registrations in the db. 
> 2. Customers logging into the user portal and see the expired time stamp 
> 3. Deleting the registration in the UI will not completely delete it in 
> the db but add a second and third and so on registration. 
> 
> all my test was with public IPs i've even tested it while both asterisk 
> and sipwise was connected to the same switch both in a virtual and hard 
> environment 
> 
> Thank you again Raul but before placing into production it's absolutely 
> imperative that the basic functionality work properly - register and 
> register. 
> 
> Warmest Regards, 
> Aziz 
> 
> 
> ------------------------------------------------------------------------ 
> *From:* Spce-user [ spce-user-bounces at lists.sipwise.com ] on behalf of 
> Raúl Alexis Betancor Santana [ rabs at dimension-virtual.com ] 
> *Sent:* Friday, November 21, 2014 6:06 AM 
> *To:* spce-user at lists.sipwise.com 
> *Subject:* Re: [Spce-user] sip registration stuck 3.5.16 
> 
> Just to add ... that we are using SPCE since it's first versions ... and 
> don't have that issue. With lot of asterisk customers with versions from 
> Asterisk 1.2 to Asterisk 12 . 
> 
> As it seems that the register doesn't get auto-expired (I don't 
> undestand why it hurts you that they are on the location table ... but 
> that another thing), if you are experiencing 'registration lost' with 
> Asterisk and SPCE ... 99.9% of the times it a fault on Asterisk or in a 
> middle router that 'modify' the SIP request. 
> 
> Best and faster thing you could do to track the issue ... it to capture 
> the traffic JUST on the Asterisk ... and on the SPCE and compare them, 
> if you see any difference ... then you probably have a sip-alg-activated 
> device on the middle of the path. 
> 
> If you don't have access to the Asterisk boxes ... just install one 
> yourself for testing, it's easy and fast. ... Faster than trying that 
> someneone give you the solution on a comunity list 
> 
> ------------------------------------------------------------------------ 
> *De: *"Daniel Grotti" < dgrotti at sipwise.com > 
> *Para: *"Aziz Palmer" <Aziz.Palmer at AirFIBER.com>, "Theo" 
> < axessofficetheo at gmail.com > 
> *CC: * spce-user at lists.sipwise.com 
> *Enviados: *Viernes, 21 de Noviembre 2014 14:08:31 
> *Asunto: *Re: [Spce-user] sip registration stuck 3.5.16 
> 
> Hi, 
> why are you so sure ? 
> Kamailio version is the same, so if there is an issue it's also in PRO. 
> I opened the internal ticket # 0010303. 
> 
> We will update you soon. 
> 
> Daniel 
> 
> 
> 
> On 11/21/2014 02:54 PM, Aziz Palmer wrote: 
> 
> Thank you Daniel! 
> 
> quick question - i'm sure the customers that are using the pro 
> version aren't having this issue. Why is it happening on the CE 
> version? (I've spent so much time on it and i'm trying to more 
> understand it) 
> 
> Thank you, 
> Aziz 
> ------------------------------------------------------------------------ 
> *From:* Daniel Grotti [ dgrotti at sipwise.com ] 
> *Sent:* Friday, November 21, 2014 1:17 AM 
> *To:* Aziz Palmer; Theo 
> *Cc:* spce-user at lists.sipwise.com 
> *Subject:* Re: [Spce-user] sip registration stuck 3.5.16 
> 
> Aziz, 
> I was able to have a look at the issue, and I opened an internal 
> ticket about that. 
> We will have a look at this and let you know. 
> 
> Daniel 
> 
> 
> 
> On 11/19/2014 12:41 AM, Aziz Palmer wrote: 
> 
> Theo, 
> can you send what you put in your sip trunk settings when you 
> connect to sipwise? (this part seems pretty straight forward but 
> would like to try someone else's settings.) 
> 
> Thank you 
> ------------------------------------------------------------------------ 
> *From:* Theo [ axessofficetheo at gmail.com ] 
> *Sent:* Thursday, November 13, 2014 5:40 AM 
> *To:* Aziz Palmer 
> *Cc:* Andreas Granig; spce-user at lists.sipwise.com 
> *Subject:* Re: [Spce-user] sip registration stuck 3.5.16 
> 
> Hi 
> 
> The problem is on the asterisk side so they are the ones to fix 
> it (assuming that is indeed your problem). The "fix" on the spce 
> is merely a work around to not ban the user and accept the 
> registration. 
> 
> On Thu, Nov 13, 2014 at 3:38 PM, Aziz Palmer 
> < Aziz.Palmer at airfiber.com <mailto: Aziz.Palmer at airfiber.com >> wrote: 
> 
> Thank you Theo! 
> 
> but what i don't understand if this is the case why wasn't 
> fixed ? 
> 
> what should i be doing instead when having users register to 
> avoid this issue? 
> 
> Is this happening to you also? if so what did you do to fix 
> it? (follow the steps below?) 
> 
> Thank you! 
> 
> 
> ------------------------------------------------------------------------ 
> *From:* Theo [ axessofficetheo at gmail.com 
> <mailto: axessofficetheo at gmail.com >] 
> *Sent:* Thursday, November 13, 2014 5:30 AM 
> *To:* Aziz Palmer 
> *Cc:* Andreas Granig; spce-user at lists.sipwise.com 
> <mailto: spce-user at lists.sipwise.com > 
> 
> *Subject:* Re: [Spce-user] sip registration stuck 3.5.16 
> 
> Sorry I accidentally replied just to you rather than to the 
> list.: 
> 
> Hi 
> 
> This looks like the stale nonce issue which is an asterisk 
> problem. The workaround has been posted a few times - search 
> for stale nonce and I am sure you'll find it easily 
> 
> Good luck 
> 
> 
> 
> 
> 
> 
> to Aziz 
> 
> It would be this one: 
> 
> http://lists.sipwise.com/pipermail/spce-user/2014-January/005712.html 
> 
> On Thu, Nov 13, 2014 at 3:00 PM, Aziz Palmer 
> < Aziz.Palmer at airfiber.com <mailto: Aziz.Palmer at airfiber.com >> 
> wrote: 
> 
> kamailio.location 
> what command do I need to execute to provide this 
> information? 
> 
> 
> kamailio-proxy.log 
> 
> Nov 13 06:49:36 tx-portal-2 proxy[22474]: NOTICE: 
> <script>: Authentication failed, stale nonce - 
> R=sip:sipwise.xyz..com 
> ID= 7233034d51f47c1a7ca56eb2353030ae at 104.34.42.10 
> <mailto: 7233034d51f47c1a7ca56eb2353030ae at 104.34.42.10 > 
> Nov 13 06:49:36 tx-portal-2 proxy[22473]: NOTICE: 
> <script>: New request on proxy - M=REGISTER 
> R=sip:sipwise.xyz..com 
> F=sip:master.sample.edge at sipwise.xyz..com 
> T=sip:master.sample.edge at sipwise.xyz..com 
> IP= 104.34.42.10:5060 < http://104.34.42.10:5060 > 
> ( 127.0.0.1:5060 < http://127.0.0.1:5060 >) 
> ID= 7233034d51f47c1a7ca56eb2353030ae at 104.34.42.10 
> <mailto: 7233034d51f47c1a7ca56eb2353030ae at 104.34.42.10 > 
> Nov 13 06:49:36 tx-portal-2 proxy[22473]: WARNING: 
> app_lua [app_lua_sr.c:100]: lua_sr_log(): lost database 
> connection. Reconnecting 
> Nov 13 06:49:36 tx-portal-2 proxy[22473]: NOTICE: 
> <script>: Contacts successfully updated, expires in 120s 
> - R=sip:sipwise.xyz..com 
> ID= 7233034d51f47c1a7ca56eb2353030ae at 104.34.42.10 
> <mailto: 7233034d51f47c1a7ca56eb2353030ae at 104.34.42.10 > 
> Nov 13 06:51:21 tx-portal-2 proxy[22480]: NOTICE: 
> <script>: New request on proxy - M=REGISTER 
> R=sip:sipwise.xyz..com 
> F=sip:master.sample.edge at sipwise.xyz..com 
> T=sip:master.sample.edge at sipwise.xyz..com 
> IP= 104.34.42.10:5060 < http://104.34.42.10:5060 > 
> ( 127.0.0.1:5060 < http://127.0.0.1:5060 >) 
> ID= 7233034d51f47c1a7ca56eb2353030ae at 104.34.42.10 
> <mailto: 7233034d51f47c1a7ca56eb2353030ae at 104.34.42.10 > 
> Nov 13 06:51:21 tx-portal-2 proxy[22480]: WARNING: 
> app_lua [app_lua_sr.c:100]: lua_sr_log(): lost database 
> connection. Reconnecting 
> Nov 13 06:51:21 tx-portal-2 proxy[22480]: NOTICE: 
> <script>: Contacts successfully updated, expires in 120s 
> - R=sip:sipwise.xyz..com 
> ID= 7233034d51f47c1a7ca56eb2353030ae at 104.34.42.10 
> <mailto: 7233034d51f47c1a7ca56eb2353030ae at 104.34.42.10 > 
> Nov 13 06:53:06 tx-portal-2 proxy[22475]: NOTICE: 
> <script>: New request on proxy - M=REGISTER 
> R=sip:sipwise.xyz..com 
> F=sip:master.sample.edge at sipwise.xyz..com 
> T=sip:master.sample.edge at sipwise.xyz..com 
> IP= 104.34.42.10:5060 < http://104.34.42.10:5060 > 
> ( 127.0.0.1:5060 < http://127.0.0.1:5060 >) 
> ID= 7233034d51f47c1a7ca56eb2353030ae at 104.34.42.10 
> <mailto: 7233034d51f47c1a7ca56eb2353030ae at 104.34.42.10 > 
> Nov 13 06:53:06 tx-portal-2 proxy[22475]: WARNING: 
> app_lua [app_lua_sr.c:100]: lua_sr_log(): lost database 
> connection. Reconnecting 
> Nov 13 06:53:06 tx-portal-2 proxy[22475]: NOTICE: 
> <script>: Contacts successfully updated, expires in 120s 
> - R=sip:sipwise.xyz..com 
> ID= 7233034d51f47c1a7ca56eb2353030ae at 104.34.42.10 
> <mailto: 7233034d51f47c1a7ca56eb2353030ae at 104.34.42.10 > 
> 
> 
> 
> 
> what does this mean "WARNING: app_lua 
> [app_lua_sr.c:100]: lua_sr_log(): lost database 
> connection. Reconnecting"? 
> 
> 
> 
> ________________________________________ 
> From: Spce-user [ spce-user-bounces at lists.sipwise.com 
> <mailto: spce-user-bounces at lists.sipwise.com >] on behalf 
> of Andreas Granig [ agranig at sipwise.com 
> <mailto: agranig at sipwise.com >] 
> Sent: Thursday, November 13, 2014 1:26 AM 
> To: spce-user at lists.sipwise.com 
> <mailto: spce-user at lists.sipwise.com > 
> Subject: Re: [Spce-user] sip registration stuck 3.5.16 
> 
> Hi, 
> 
> It's most likely not a bug of the system, rather than a 
> problem with 
> your clients registering on said system. I risk to say 
> that because the 
> registrar is based on Kamailio, which is serving dozens 
> of millions of 
> SIP devices out there, so it would be a huge surprise 
> for it not to work 
> correctly. 
> 
> The only way to find out what the REAL issue is for you 
> to send an 
> example entry in the kamailio.location table of the DB, 
> along with a 
> full network trace of this client trying to register on 
> the system, and 
> also the kamailio-proxy.log file. 
> 
> Andreas 
> 
> On 11/13/2014 06:47 AM, Aziz Palmer wrote: 
> > thats not the issue - the problem is when you 
> RE-register a sip trunk it 
> > will stop updating sipwise. in order to fix that you 
> need to manually go 
> > into the subscriber and delete the registration. it 
> will not delete but 
> > instead a new registration will appear. 
> > 
> > 
> > SPCE - are you aware of this bug (its a serous issue - 
> since it gives 
> > the admins a lot of manual work/tech support calls) I 
> would like to put 
> > this this into partial production but can't - please help? 
> > 
> > Thanks! 
> > 
> ------------------------------------------------------------------------ 
> > *From:* Raúl Alexis Betancor Santana 
> [ rabs at dimension-virtual.com 
> <mailto: rabs at dimension-virtual.com >] 
> > *Sent:* Wednesday, November 12, 2014 1:54 PM 
> > *To:* Aziz Palmer 
> > *Subject:* Re: sip registration stuck 3.5.16 
> > 
> > 3600 secs ... 1 h ... it's default time on the config.xml 
> > kamailio.proxy.default_expire option ... you could 
> change it ... or set 
> > it on the web interface at domain-level or user-level 
> if you prefer, and 
> > as far as you don't touched the .cfg's ... they expire. 
> > 
> > 
> ------------------------------------------------------------------------ 
> > *De: *"Aziz Palmer" <Aziz.Palmer at AirFIBER.com> 
> > *Para: *"Raúl Alexis Betancor Santana" 
> < rabs at dimension-virtual.com 
> <mailto: rabs at dimension-virtual.com >>, 
> > spce-user at lists.sipwise.com 
> <mailto: spce-user at lists.sipwise.com > 
> > *Enviados: *Miércoles, 12 de Noviembre 2014 21:17:53 
> > *Asunto: *RE: sip registration stuck 3.5.16 
> > 
> > they are not auto-expiring (I wish they were) 
> > 
> > if they are - taking a extremely long time 
> > 
> ------------------------------------------------------------------------ 
> > *From:* Spce-user [ spce-user-bounces at lists.sipwise.com 
> <mailto: spce-user-bounces at lists.sipwise.com >] on behalf of 
> > Raúl Alexis Betancor Santana 
> [ rabs at dimension-virtual.com 
> <mailto: rabs at dimension-virtual.com >] 
> > *Sent:* Wednesday, November 12, 2014 1:04 PM 
> > *To:* spce-user at lists.sipwise.com 
> <mailto: spce-user at lists.sipwise.com > 
> > *Subject:* Re: [Spce-user] sip registration stuck 3.5.16 
> > 
> > 
> > Why did you whant to remove the old register? ... 
> location entries 
> > auto-expire, so you don't have to worry about them. 
> > 
> > 
> ------------------------------------------------------------------------ 
> > *De: *"Aziz Palmer" <Aziz.Palmer at AirFIBER.com> 
> > *Para: * spce-user at lists.sipwise.com 
> <mailto: spce-user at lists.sipwise.com > 
> > *Enviados: *Miércoles, 12 de Noviembre 2014 20:47:13 
> > *Asunto: *[Spce-user] sip registration stuck 3.5.16 
> > 
> > I hate to ask again but this is turning into a real 
> issue - the 
> > "registered devices" are not automatically being 
> removed and when trying 
> > to manually delete them, nothing happens. 
> > 
> > Can i set a timer on how long these orphan registered 
> devices stay part 
> > of sipwise? 
> > 
> > how can i delete these devices without having to issue 
> the below command 
> > - I would like to delete specific devices and not all 
> of them 
> > 
> > mysql kamailio -e "delete from location" 
> > /etc/init.d/kamailio-proxy restart 
> > 
> > 
> > Thanks again, 
> > Aziz 
> > 
> > 
> ------------------------------------------------------------------------ 
> > *From:* Spce-user [ spce-user-bounces at lists.sipwise.com 
> <mailto: spce-user-bounces at lists.sipwise.com >] on behalf of 
> > Aziz Palmer [Aziz.Palmer at AirFIBER.com] 
> > *Sent:* Tuesday, November 11, 2014 12:15 PM 
> > *To:* spce-user at lists.sipwise.com 
> <mailto: spce-user at lists.sipwise.com > 
> > *Subject:* [Spce-user] sip registration stuck 5.3.16 
> > 
> > Hello SPCE Team, 
> > here is what appears to be a bug 
> > 
> > I enter sip registration settings into asterisk, 
> registers fine with 
> > sipwise. Then I make a change within the sip trunk 
> settings of asterisk 
> > and apply the new settings. Then previous sip 
> registration gets stuck; 
> > meaning within sipwise 2 registrations appear (old & 
> new). When i try to 
> > delete the old one, it stays stuck. The only way i 
> discovered to be able 
> > to remove the old registration is by issuing this 
> command, then i can 
> > delete the registration. 
> > 
> > /etc/init.d/kamailio-proxy restart 
> > 
> > what additional information can i provide to have your 
> team troubleshoot 
> > this issue? 
> > 
> > Thank you, 
> > Anees 
> > 
> > _______________________________________________ 
> > Spce-user mailing list 
> > Spce-user at lists.sipwise.com 
> <mailto: Spce-user at lists.sipwise.com > 
> > https://lists.sipwise.com/listinfo/spce-user 
> > 
> > 
> > 
> > 
> > _______________________________________________ 
> > Spce-user mailing list 
> > Spce-user at lists.sipwise.com 
> <mailto: Spce-user at lists.sipwise.com > 
> > https://lists.sipwise.com/listinfo/spce-user 
> > 
> _______________________________________________ 
> Spce-user mailing list 
> Spce-user at lists.sipwise.com 
> <mailto: Spce-user at lists.sipwise.com > 
> https://lists.sipwise.com/listinfo/spce-user 
> _______________________________________________ 
> Spce-user mailing list 
> Spce-user at lists.sipwise.com 
> <mailto: Spce-user at lists.sipwise.com > 
> https://lists.sipwise.com/listinfo/spce-user 
> 
> 
> 
> 
> 
> _______________________________________________ 
> Spce-user mailing list 
> Spce-user at lists.sipwise.com 
> https://lists.sipwise.com/listinfo/spce-user 
> 
> 
> 
> 
> _______________________________________________ 
> Spce-user mailing list 
> Spce-user at lists.sipwise.com 
> https://lists.sipwise.com/listinfo/spce-user 
> 
> 
> 
> 
> _______________________________________________ 
> Spce-user mailing list 
> Spce-user at lists.sipwise.com 
> https://lists.sipwise.com/listinfo/spce-user 
> 
_______________________________________________ 
Spce-user mailing list 
Spce-user at lists.sipwise.com 
https://lists.sipwise.com/listinfo/spce-user 





_______________________________________________ 
Spce-user mailing list 
Spce-user at lists.sipwise.com 
https://lists.sipwise.com/listinfo/spce-user 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sipwise.com/pipermail/spce-user_lists.sipwise.com/attachments/20141121/849aa22a/attachment-0001.html>


More information about the Spce-user mailing list