[Spce-user] sip registration stuck 3.5.16

Theo axessofficetheo at gmail.com
Fri Nov 21 10:25:55 EST 2014


 ngcp-sercmd proxy ul.dump
error: 500 - Internal error creating aor struct

Any ideas on that?

On Fri, Nov 21, 2014 at 5:10 PM, Andreas Granig <agranig at sipwise.com> wrote:

> Just to add my 2 cents:
>
> The proxy (kamailio) uses in-memory caching for registrations and writes
> them back to DB. The panel displays the registrations from the DB.
>
> Now if you delete something from the DB, it's not reflected in the
> proxy, because it still will be in the cache. So what we're doing is to
> delete a registration from the cache (by telling the proxy to purge it),
> and the proxy will propagate it down to the DB.
>
> The same happens if the proxy detects that a registration has expired.
> It will delete it from the cache and propagate the deletion down to the DB.
>
> Now here is where it's getting interesting: if the registration is no
> longer in the cache (for whatever reason) and it still is in the DB (for
> whatever else reason), you see it in the panel and can click on the
> delete button, but as the deletion request is sent to the proxy, it will
> just do nothing because it doesn't have it in the cache, so it will
> still persist in the DB.
>
> This scenario should never happen (famous last words), so to
> troubleshoot it, you'd need to dump the registrations from the cache
> using "ngcp-sercmd proxy ul.dump" and check the count (see
> "Stats"->"Records") and compare it to the db by doing "mysql kamailio -e
> "select id, username, domain, expires from location".
>
> Take into account that the cache count can/will potentially be lower
> than the db count because the proxy batches deletion in a separate
> process, which is executed every 60 sec. So to fetch the "real" expired
> registrations, do this: "mysql kamailio -e "select id, username, domain,
> expires from location where expires < date_sub(now(), interval 90 second)"
>
> So this will give you a representation of your current state. Note that
> if a registration is not in the cache but only in the DB, it's not a
> real operational issue, because the proxy will not know about it.
>
> It still doesn't answer the question how this can even happen, so you
> need to provide a full trace ("tcpdump -i any -s 0 -w /tmp/reg.pcap port
> 5060") of a complete cycle (fresh registration and subsequent
> registrations, which will then cause the issue you described), along
> with the full kamailio-proxy.log, so we can check for errors etc. Also
> provide the output of "ngcp-sercmd proxy ul.dump" and "mysql kamailio -e
> "select id, username, domain, expires from location" along with the
> timestamps when you executed those commands.
>
> Andreas
>
> On 11/21/2014 03:41 PM, Raúl Alexis Betancor Santana wrote:
> >
> > Helo Aziz ... let's talk about ...
> >
> > 1- Doesn't matter ... if you have more than one register for the same
> > user, the only thing that happens it's that on an outbound request to
> > that subscriber, SPCE will launch 2 branches ... one will fail and the
> > other one will sucess. No big deal at all.
> > 2- That should not happens ... as expired registration should be
> > auto-erased. Have you installed the system from 0 ? .. or did you
> > upgrade from older version?
> > 3- That sounds as an upgrades SPCE version that didn't complete the
> upgrade.
> >
> >
> > ------------------------------------------------------------------------
> > *De: *"Aziz Palmer" <Aziz.Palmer at AirFIBER.com>
> > *Para: *"Raúl Alexis Betancor Santana" <rabs at dimension-virtual.com>,
> > spce-user at lists.sipwise.com
> > *Enviados: *Viernes, 21 de Noviembre 2014 14:31:25
> > *Asunto: *RE: sip registration stuck 3.5.16
> >
> > Thank you Raul for your input and suggestions. Not to get into a long
> > discussion but I at first didn't think it was a big deal until the
> > following accours
> > 1. the network environment at subscriber changes (change of IP) -
> > asterisk registers and now you have 2 active registrations in the db.
> > 2. Customers logging into the user portal and see the expired time stamp
> > 3. Deleting the registration in the UI will not completely delete it in
> > the db but add a second and third and so on registration.
> >
> > all my test was with public IPs i've even tested it while both asterisk
> > and sipwise was connected to the same switch both in a virtual and hard
> > environment
> >
> > Thank you again Raul but before placing into production it's absolutely
> > imperative that the basic functionality work properly - register and
> > register.
> >
> > Warmest Regards,
> > Aziz
> >
> >
> > ------------------------------------------------------------------------
> > *From:* Spce-user [spce-user-bounces at lists.sipwise.com] on behalf of
> > Raúl Alexis Betancor Santana [rabs at dimension-virtual.com]
> > *Sent:* Friday, November 21, 2014 6:06 AM
> > *To:* spce-user at lists.sipwise.com
> > *Subject:* Re: [Spce-user] sip registration stuck 3.5.16
> >
> > Just to add ... that we are using SPCE since it's first versions ... and
> > don't have that issue. With lot of asterisk customers with versions from
> > Asterisk 1.2 to Asterisk 12 .
> >
> > As it seems that the register doesn't get auto-expired (I don't
> > undestand why it hurts you that they are on the location table ... but
> > that another thing), if you are experiencing 'registration lost' with
> > Asterisk and SPCE ... 99.9% of the times it a fault on Asterisk or in a
> > middle router that 'modify' the SIP request.
> >
> > Best and faster thing you could do to track the issue ... it to capture
> > the traffic JUST on the Asterisk ... and on the SPCE and compare them,
> > if you see any difference ... then you probably have a sip-alg-activated
> > device on the middle of the path.
> >
> > If you don't have access to the Asterisk boxes ... just install one
> > yourself for testing, it's easy and fast. ... Faster than trying that
> > someneone give you the solution on a comunity list
> >
> > ------------------------------------------------------------------------
> > *De: *"Daniel Grotti" <dgrotti at sipwise.com>
> > *Para: *"Aziz Palmer" <Aziz.Palmer at AirFIBER.com>, "Theo"
> > <axessofficetheo at gmail.com>
> > *CC: *spce-user at lists.sipwise.com
> > *Enviados: *Viernes, 21 de Noviembre 2014 14:08:31
> > *Asunto: *Re: [Spce-user] sip registration stuck 3.5.16
> >
> > Hi,
> > why are you so sure ?
> > Kamailio version is the same, so if there is an issue it's also in PRO.
> > I opened the internal ticket # 0010303.
> >
> > We will update you soon.
> >
> > Daniel
> >
> >
> >
> > On 11/21/2014 02:54 PM, Aziz Palmer wrote:
> >
> >     Thank you Daniel!
> >
> >     quick question - i'm sure the customers that are using the pro
> >     version aren't having this issue. Why is it happening on the CE
> >     version? (I've spent so much time on it and i'm trying to more
> >     understand it)
> >
> >     Thank you,
> >     Aziz
> >
>  ------------------------------------------------------------------------
> >     *From:* Daniel Grotti [dgrotti at sipwise.com]
> >     *Sent:* Friday, November 21, 2014 1:17 AM
> >     *To:* Aziz Palmer; Theo
> >     *Cc:* spce-user at lists.sipwise.com
> >     *Subject:* Re: [Spce-user] sip registration stuck 3.5.16
> >
> >     Aziz,
> >     I was able to have a look at the issue, and I opened an internal
> >     ticket about that.
> >     We will have a look at this and let you know.
> >
> >     Daniel
> >
> >
> >
> >     On 11/19/2014 12:41 AM, Aziz Palmer wrote:
> >
> >         Theo,
> >         can you send what you put in your sip trunk settings when you
> >         connect to sipwise? (this part seems pretty straight forward but
> >         would like to try someone else's settings.)
> >
> >         Thank you
> >
>  ------------------------------------------------------------------------
> >         *From:* Theo [axessofficetheo at gmail.com]
> >         *Sent:* Thursday, November 13, 2014 5:40 AM
> >         *To:* Aziz Palmer
> >         *Cc:* Andreas Granig; spce-user at lists.sipwise.com
> >         *Subject:* Re: [Spce-user] sip registration stuck 3.5.16
> >
> >         Hi
> >
> >         The problem is on the asterisk side so they are the ones to fix
> >         it (assuming that is indeed your problem). The "fix" on the spce
> >         is merely a work around to not ban the user and accept the
> >         registration.
> >
> >         On Thu, Nov 13, 2014 at 3:38 PM, Aziz Palmer
> >         <Aziz.Palmer at airfiber.com <mailto:Aziz.Palmer at airfiber.com>>
> wrote:
> >
> >             Thank you Theo!
> >
> >             but what i don't understand if this is the case why wasn't
> >             fixed ?
> >
> >             what should i be doing instead when having users register to
> >             avoid this issue?
> >
> >             Is this happening to you also? if so what did you do to fix
> >             it? (follow the steps below?)
> >
> >             Thank you!
> >
> >
> >
>  ------------------------------------------------------------------------
> >             *From:* Theo [axessofficetheo at gmail.com
> >             <mailto:axessofficetheo at gmail.com>]
> >             *Sent:* Thursday, November 13, 2014 5:30 AM
> >             *To:* Aziz Palmer
> >             *Cc:* Andreas Granig; spce-user at lists.sipwise.com
> >             <mailto:spce-user at lists.sipwise.com>
> >
> >             *Subject:* Re: [Spce-user] sip registration stuck 3.5.16
> >
> >             Sorry I accidentally replied just to you rather than to the
> >             list.:
> >
> >             Hi
> >
> >             This looks like the stale nonce issue which is an asterisk
> >             problem. The workaround has been posted a few times - search
> >             for stale nonce and I am sure you'll find it easily
> >
> >             Good luck
> >
> >
> >
> >
> >
> >
> >             to Aziz
> >
> >             It would be this one:
> >
> >
> http://lists.sipwise.com/pipermail/spce-user/2014-January/005712.html
> >
> >             On Thu, Nov 13, 2014 at 3:00 PM, Aziz Palmer
> >             <Aziz.Palmer at airfiber.com <mailto:Aziz.Palmer at airfiber.com>>
> >             wrote:
> >
> >                 kamailio.location
> >                 what command do I need to execute to provide this
> >                 information?
> >
> >
> >                 kamailio-proxy.log
> >
> >                 Nov 13 06:49:36 tx-portal-2 proxy[22474]: NOTICE:
> >                 <script>: Authentication failed, stale nonce -
> >                 R=sip:sipwise.xyz..com
> >                 ID=7233034d51f47c1a7ca56eb2353030ae at 104.34.42.10
> >                 <mailto:7233034d51f47c1a7ca56eb2353030ae at 104.34.42.10>
> >                 Nov 13 06:49:36 tx-portal-2 proxy[22473]: NOTICE:
> >                 <script>: New request on proxy - M=REGISTER
> >                 R=sip:sipwise.xyz..com
> >                 F=sip:master.sample.edge at sipwise.xyz..com
> >                 T=sip:master.sample.edge at sipwise.xyz..com
> >                 IP=104.34.42.10:5060 <http://104.34.42.10:5060>
> >                 (127.0.0.1:5060 <http://127.0.0.1:5060>)
> >                 ID=7233034d51f47c1a7ca56eb2353030ae at 104.34.42.10
> >                 <mailto:7233034d51f47c1a7ca56eb2353030ae at 104.34.42.10>
> >                 Nov 13 06:49:36 tx-portal-2 proxy[22473]: WARNING:
> >                 app_lua [app_lua_sr.c:100]: lua_sr_log(): lost database
> >                 connection. Reconnecting
> >                 Nov 13 06:49:36 tx-portal-2 proxy[22473]: NOTICE:
> >                 <script>: Contacts successfully updated, expires in 120s
> >                 - R=sip:sipwise.xyz..com
> >                 ID=7233034d51f47c1a7ca56eb2353030ae at 104.34.42.10
> >                 <mailto:7233034d51f47c1a7ca56eb2353030ae at 104.34.42.10>
> >                 Nov 13 06:51:21 tx-portal-2 proxy[22480]: NOTICE:
> >                 <script>: New request on proxy - M=REGISTER
> >                 R=sip:sipwise.xyz..com
> >                 F=sip:master.sample.edge at sipwise.xyz..com
> >                 T=sip:master.sample.edge at sipwise.xyz..com
> >                 IP=104.34.42.10:5060 <http://104.34.42.10:5060>
> >                 (127.0.0.1:5060 <http://127.0.0.1:5060>)
> >                 ID=7233034d51f47c1a7ca56eb2353030ae at 104.34.42.10
> >                 <mailto:7233034d51f47c1a7ca56eb2353030ae at 104.34.42.10>
> >                 Nov 13 06:51:21 tx-portal-2 proxy[22480]: WARNING:
> >                 app_lua [app_lua_sr.c:100]: lua_sr_log(): lost database
> >                 connection. Reconnecting
> >                 Nov 13 06:51:21 tx-portal-2 proxy[22480]: NOTICE:
> >                 <script>: Contacts successfully updated, expires in 120s
> >                 - R=sip:sipwise.xyz..com
> >                 ID=7233034d51f47c1a7ca56eb2353030ae at 104.34.42.10
> >                 <mailto:7233034d51f47c1a7ca56eb2353030ae at 104.34.42.10>
> >                 Nov 13 06:53:06 tx-portal-2 proxy[22475]: NOTICE:
> >                 <script>: New request on proxy - M=REGISTER
> >                 R=sip:sipwise.xyz..com
> >                 F=sip:master.sample.edge at sipwise.xyz..com
> >                 T=sip:master.sample.edge at sipwise.xyz..com
> >                 IP=104.34.42.10:5060 <http://104.34.42.10:5060>
> >                 (127.0.0.1:5060 <http://127.0.0.1:5060>)
> >                 ID=7233034d51f47c1a7ca56eb2353030ae at 104.34.42.10
> >                 <mailto:7233034d51f47c1a7ca56eb2353030ae at 104.34.42.10>
> >                 Nov 13 06:53:06 tx-portal-2 proxy[22475]: WARNING:
> >                 app_lua [app_lua_sr.c:100]: lua_sr_log(): lost database
> >                 connection. Reconnecting
> >                 Nov 13 06:53:06 tx-portal-2 proxy[22475]: NOTICE:
> >                 <script>: Contacts successfully updated, expires in 120s
> >                 - R=sip:sipwise.xyz..com
> >                 ID=7233034d51f47c1a7ca56eb2353030ae at 104.34.42.10
> >                 <mailto:7233034d51f47c1a7ca56eb2353030ae at 104.34.42.10>
> >
> >
> >
> >
> >                 what does this mean "WARNING: app_lua
> >                 [app_lua_sr.c:100]: lua_sr_log(): lost database
> >                 connection. Reconnecting"?
> >
> >
> >
> >                 ________________________________________
> >                 From: Spce-user [spce-user-bounces at lists.sipwise.com
> >                 <mailto:spce-user-bounces at lists.sipwise.com>] on behalf
> >                 of Andreas Granig [agranig at sipwise.com
> >                 <mailto:agranig at sipwise.com>]
> >                 Sent: Thursday, November 13, 2014 1:26 AM
> >                 To: spce-user at lists.sipwise.com
> >                 <mailto:spce-user at lists.sipwise.com>
> >                 Subject: Re: [Spce-user] sip registration stuck 3.5.16
> >
> >                 Hi,
> >
> >                 It's most likely not a bug of the system, rather than a
> >                 problem with
> >                 your clients registering on said system. I risk to say
> >                 that because the
> >                 registrar is based on Kamailio, which is serving dozens
> >                 of millions of
> >                 SIP devices out there, so it would be a huge surprise
> >                 for it not to work
> >                 correctly.
> >
> >                 The only way to find out what the REAL issue is for you
> >                 to send an
> >                 example entry in the kamailio.location table of the DB,
> >                 along with a
> >                 full network trace of this client trying to register on
> >                 the system, and
> >                 also the kamailio-proxy.log file.
> >
> >                 Andreas
> >
> >                 On 11/13/2014 06:47 AM, Aziz Palmer wrote:
> >                 > thats not the issue - the problem is when you
> >                 RE-register a sip trunk it
> >                 > will stop updating sipwise. in order to fix that you
> >                 need to manually go
> >                 > into the subscriber and delete the registration. it
> >                 will not delete but
> >                 > instead a new registration will appear.
> >                 >
> >                 >
> >                 > SPCE - are you aware of this bug (its a serous issue -
> >                 since it gives
> >                 > the admins a lot of manual work/tech support calls) I
> >                 would like to put
> >                 > this this into partial production but can't - please
> help?
> >                 >
> >                 > Thanks!
> >                 >
> >
>  ------------------------------------------------------------------------
> >                 > *From:* Raúl Alexis Betancor Santana
> >                 [rabs at dimension-virtual.com
> >                 <mailto:rabs at dimension-virtual.com>]
> >                 > *Sent:* Wednesday, November 12, 2014 1:54 PM
> >                 > *To:* Aziz Palmer
> >                 > *Subject:* Re: sip registration stuck 3.5.16
> >                 >
> >                 > 3600 secs ... 1 h ... it's default time on the
> config.xml
> >                 >  kamailio.proxy.default_expire option ... you could
> >                 change it ... or set
> >                 > it on the web interface at domain-level or user-level
> >                 if you prefer, and
> >                 > as far as you don't touched the .cfg's ... they expire.
> >                 >
> >                 >
> >
>  ------------------------------------------------------------------------
> >                 > *De: *"Aziz Palmer" <Aziz.Palmer at AirFIBER.com>
> >                 > *Para: *"Raúl Alexis Betancor Santana"
> >                 <rabs at dimension-virtual.com
> >                 <mailto:rabs at dimension-virtual.com>>,
> >                 > spce-user at lists.sipwise.com
> >                 <mailto:spce-user at lists.sipwise.com>
> >                 > *Enviados: *Miércoles, 12 de Noviembre 2014 21:17:53
> >                 > *Asunto: *RE: sip registration stuck 3.5.16
> >                 >
> >                 > they are not auto-expiring (I wish they were)
> >                 >
> >                 > if they are - taking a extremely long time
> >                 >
> >
>  ------------------------------------------------------------------------
> >                 > *From:* Spce-user [spce-user-bounces at lists.sipwise.com
> >                 <mailto:spce-user-bounces at lists.sipwise.com>] on behalf
> of
> >                 > Raúl Alexis Betancor Santana
> >                 [rabs at dimension-virtual.com
> >                 <mailto:rabs at dimension-virtual.com>]
> >                 > *Sent:* Wednesday, November 12, 2014 1:04 PM
> >                 > *To:* spce-user at lists.sipwise.com
> >                 <mailto:spce-user at lists.sipwise.com>
> >                 > *Subject:* Re: [Spce-user] sip registration stuck
> 3.5.16
> >                 >
> >                 >
> >                 > Why did you whant to remove the old register? ...
> >                 location entries
> >                 > auto-expire, so you don't have to worry about them.
> >                 >
> >                 >
> >
>  ------------------------------------------------------------------------
> >                 > *De: *"Aziz Palmer" <Aziz.Palmer at AirFIBER.com>
> >                 > *Para: *spce-user at lists.sipwise.com
> >                 <mailto:spce-user at lists.sipwise.com>
> >                 > *Enviados: *Miércoles, 12 de Noviembre 2014 20:47:13
> >                 > *Asunto: *[Spce-user] sip registration stuck 3.5.16
> >                 >
> >                 > I hate to ask again but this is turning into a real
> >                 issue - the
> >                 > "registered devices" are not automatically being
> >                 removed and when trying
> >                 > to manually delete them, nothing happens.
> >                 >
> >                 > Can i set a timer on how long these orphan registered
> >                 devices stay part
> >                 > of sipwise?
> >                 >
> >                 > how can i delete these devices without having to issue
> >                 the below command
> >                 > - I would like to delete specific devices and not all
> >                 of them
> >                 >
> >                 > mysql kamailio -e "delete from location"
> >                 > /etc/init.d/kamailio-proxy restart
> >                 >
> >                 >
> >                 > Thanks again,
> >                 > Aziz
> >                 >
> >                 >
> >
>  ------------------------------------------------------------------------
> >                 > *From:* Spce-user [spce-user-bounces at lists.sipwise.com
> >                 <mailto:spce-user-bounces at lists.sipwise.com>] on behalf
> of
> >                 > Aziz Palmer [Aziz.Palmer at AirFIBER.com]
> >                 > *Sent:* Tuesday, November 11, 2014 12:15 PM
> >                 > *To:* spce-user at lists.sipwise.com
> >                 <mailto:spce-user at lists.sipwise.com>
> >                 > *Subject:* [Spce-user] sip registration stuck 5.3.16
> >                 >
> >                 > Hello SPCE Team,
> >                 > here is what appears to be a bug
> >                 >
> >                 > I enter sip registration settings into asterisk,
> >                 registers fine with
> >                 > sipwise. Then I make a change within the sip trunk
> >                 settings of asterisk
> >                 > and apply the new settings. Then previous sip
> >                 registration gets stuck;
> >                 > meaning within sipwise 2 registrations appear (old &
> >                 new). When i try to
> >                 > delete the old one, it stays stuck. The only way i
> >                 discovered to be able
> >                 > to remove the old registration is by issuing this
> >                 command, then i can
> >                 > delete the registration.
> >                 >
> >                 > /etc/init.d/kamailio-proxy restart
> >                 >
> >                 > what additional information can i provide to have your
> >                 team troubleshoot
> >                 > this issue?
> >                 >
> >                 > Thank you,
> >                 > Anees
> >                 >
> >                 > _______________________________________________
> >                 > Spce-user mailing list
> >                 > Spce-user at lists.sipwise.com
> >                 <mailto:Spce-user at lists.sipwise.com>
> >                 > https://lists.sipwise.com/listinfo/spce-user
> >                 >
> >                 >
> >                 >
> >                 >
> >                 > _______________________________________________
> >                 > Spce-user mailing list
> >                 > Spce-user at lists.sipwise.com
> >                 <mailto:Spce-user at lists.sipwise.com>
> >                 > https://lists.sipwise.com/listinfo/spce-user
> >                 >
> >                 _______________________________________________
> >                 Spce-user mailing list
> >                 Spce-user at lists.sipwise.com
> >                 <mailto:Spce-user at lists.sipwise.com>
> >                 https://lists.sipwise.com/listinfo/spce-user
> >                 _______________________________________________
> >                 Spce-user mailing list
> >                 Spce-user at lists.sipwise.com
> >                 <mailto:Spce-user at lists.sipwise.com>
> >                 https://lists.sipwise.com/listinfo/spce-user
> >
> >
> >
> >
> >
> >         _______________________________________________
> >         Spce-user mailing list
> >         Spce-user at lists.sipwise.com
> >         https://lists.sipwise.com/listinfo/spce-user
> >
> >
> >
> >
> > _______________________________________________
> > Spce-user mailing list
> > Spce-user at lists.sipwise.com
> > https://lists.sipwise.com/listinfo/spce-user
> >
> >
> >
> >
> > _______________________________________________
> > Spce-user mailing list
> > Spce-user at lists.sipwise.com
> > https://lists.sipwise.com/listinfo/spce-user
> >
> _______________________________________________
> Spce-user mailing list
> Spce-user at lists.sipwise.com
> https://lists.sipwise.com/listinfo/spce-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sipwise.com/pipermail/spce-user_lists.sipwise.com/attachments/20141121/0241f22e/attachment-0001.html>


More information about the Spce-user mailing list