[Spce-user] sip registration stuck 3.5.16
Raúl Alexis Betancor Santana
rabs at dimension-virtual.com
Fri Nov 21 12:26:58 EST 2014
If you know a user that have the issue ... just check for it on the log statement of mysql, you will see the update statement, and could replicate it directly ... to check what it does.
----- Mensaje original -----
De: "Theo" <axessofficetheo at gmail.com>
Para: "Raúl Alexis Betancor Santana" <rabs at dimension-virtual.com>
CC: spce-user at lists.sipwise.com
Enviados: Viernes, 21 de Noviembre 2014 17:04:25
Asunto: Re: [Spce-user] sip registration stuck 3.5.16
Hi
I agree that it doesn't hurt and has no adverse effects, but now that we are onto this I would love to get it sorted. While we haven't use the customer portal extensively at all, I can see the scenario of receiving a call in the middle of the night "we have 15 users connected but I only have one phone"
I have enabled the logging in mysql but am not sure what to look for now. I see things being updated, and I see some being deleted. But because this is quite intermittent and only some seem to get stuck I am not sure how to replicate this. It mostly works fine and then every now and then one gets stuck and just remains there.
On Fri, Nov 21, 2014 at 5:43 PM, Raúl Alexis Betancor Santana < rabs at dimension-virtual.com > wrote:
Expired registers (or even duplicated ones) ... as opposite as Aziz thinks ... doesn't hurt the system ... it's only a matter of wasting DB space and loading time of kamailio-proxy, no more.
De: "Theo" < axessofficetheo at gmail.com >
Para: "Raúl Alexis Betancor Santana" < rabs at dimension-virtual.com >
CC: spce-user at lists.sipwise.com
Enviados: Viernes, 21 de Noviembre 2014 15:51:10
Asunto: Re: [Spce-user] sip registration stuck 3.5.16
I am very happy to accept there is something wrong with install(s) - just haven't really had any other issues.... I will do as you say and see if I can get any further with this. No IP address was changed on the box - and as I said all appears to be running fine.
Aziz - I have a feeling your thread is being hi-jacked a little here?
On Fri, Nov 21, 2014 at 5:28 PM, Raúl Alexis Betancor Santana < rabs at dimension-virtual.com > wrote:
<blockquote>
I'm sure ... 100% sure ... that you have DB issues between kamailio-proxy and mysql, or directly mysql issues ... I suggest you to activate sql statement log on mysql, and wait 60s ... you should see the statement from the kamailio-proxy thread that will erase the expired registrations, and then you will be able to trace the problem.
De: "Theo" < axessofficetheo at gmail.com >
Para: "Andreas Granig" < agranig at sipwise.com >
CC: spce-user at lists.sipwise.com
Enviados: Viernes, 21 de Noviembre 2014 15:23:06
Asunto: Re: [Spce-user] sip registration stuck 3.5.16
here is a nice example - you will see those are long expired:
*
User Agent:
A510IP/42.075.00.000.000
Contact
sip:tamasha1 at 192.168.10.205:5060
NAT
yes
Expires
2014-11-21 17:40:45
Expire Now
*
User Agent:
snom300/8.4.35
Contact
sip:tamasha1 at 192.168.10.224:2048;line=0g558d01
NAT
yes
Expires
2014-06-05 15:15:59
Expire Now
*
User Agent:
snom300/8.4.35
Contact
sip:tamasha1 at 192.168.10.224:2048;line=o7qm8al6
NAT
yes
Expires
2014-07-28 17:22:25
Expire Now
*
User Agent:
snom300/8.4.35
Contact
sip:tamasha1 at 192.168.10.224:2048;line=sjs2oc9h
NAT
yes
Expires
2014-11-21 18:24:43
Expire Now
*
User Agent:
snom300/8.4.35
Contact
sip:tamasha1 at 192.168.10.224:2056;line=zx2nckqz
NAT
yes
Expires
2014-10-10 14:01:27
Expire Now
*
User Agent:
snom300/8.4.35
Contact
sip:tamasha1 at 192.168.10.224:2068;line=zaul2pu8
NAT
yes
Expires
2014-09-23 09:28:59
Expire Now
*
User Agent:
snom300/8.4.35
Contact
sip:tamasha1 at 192.168.10.224:2075;line=q7mer4r4
NAT
yes
Expires
2014-11-19 11:04:58
Expire Now
*
User Agent:
snom300/8.4.35
Contact
sip:tamasha1 at 192.168.10.224:2094;line=y734quc2
NAT
yes
Expires
2014-10-02 14:41:07
Expire Now
*
User Agent:
snom300/8.4.35
Contact
sip:tamasha1 at 192.168.10.224:2106;line=bb7gge3q
NAT
yes
Expires
2014-09-10 09:18:30
Expire Now
*
User Agent:
snom300/ 8.7.3.19
Contact
sip:tamasha1 at 192.168.10.230:2048;line=degxi5oh
NAT
yes
Expires
2014-11-21 17:38:40
Expire Now
*
User Agent:
snom300/ 8.7.3.19
Contact
sip:tamasha1 at 192.168.10.230:2048;line=nuporyrn
NAT
yes
Expires
2014-10-16 10:48:20
On Fri, Nov 21, 2014 at 5:10 PM, Andreas Granig < agranig at sipwise.com > wrote:
<blockquote>
Just to add my 2 cents:
The proxy (kamailio) uses in-memory caching for registrations and writes
them back to DB. The panel displays the registrations from the DB.
Now if you delete something from the DB, it's not reflected in the
proxy, because it still will be in the cache. So what we're doing is to
delete a registration from the cache (by telling the proxy to purge it),
and the proxy will propagate it down to the DB.
The same happens if the proxy detects that a registration has expired.
It will delete it from the cache and propagate the deletion down to the DB.
Now here is where it's getting interesting: if the registration is no
longer in the cache (for whatever reason) and it still is in the DB (for
whatever else reason), you see it in the panel and can click on the
delete button, but as the deletion request is sent to the proxy, it will
just do nothing because it doesn't have it in the cache, so it will
still persist in the DB.
This scenario should never happen (famous last words), so to
troubleshoot it, you'd need to dump the registrations from the cache
using "ngcp-sercmd proxy ul.dump" and check the count (see
"Stats"->"Records") and compare it to the db by doing "mysql kamailio -e
"select id, username, domain, expires from location".
Take into account that the cache count can/will potentially be lower
than the db count because the proxy batches deletion in a separate
process, which is executed every 60 sec. So to fetch the "real" expired
registrations, do this: "mysql kamailio -e "select id, username, domain,
expires from location where expires < date_sub(now(), interval 90 second)"
So this will give you a representation of your current state. Note that
if a registration is not in the cache but only in the DB, it's not a
real operational issue, because the proxy will not know about it.
It still doesn't answer the question how this can even happen, so you
need to provide a full trace ("tcpdump -i any -s 0 -w /tmp/reg.pcap port
5060") of a complete cycle (fresh registration and subsequent
registrations, which will then cause the issue you described), along
with the full kamailio-proxy.log, so we can check for errors etc. Also
provide the output of "ngcp-sercmd proxy ul.dump" and "mysql kamailio -e
"select id, username, domain, expires from location" along with the
timestamps when you executed those commands.
Andreas
On 11/21/2014 03:41 PM, Raúl Alexis Betancor Santana wrote:
>
> Helo Aziz ... let's talk about ...
>
> 1- Doesn't matter ... if you have more than one register for the same
> user, the only thing that happens it's that on an outbound request to
> that subscriber, SPCE will launch 2 branches ... one will fail and the
> other one will sucess. No big deal at all.
> 2- That should not happens ... as expired registration should be
> auto-erased. Have you installed the system from 0 ? .. or did you
> upgrade from older version?
> 3- That sounds as an upgrades SPCE version that didn't complete the upgrade.
>
>
> ------------------------------------------------------------------------
> *De: *"Aziz Palmer" <Aziz.Palmer at AirFIBER.com>
> *Para: *"Raúl Alexis Betancor Santana" < rabs at dimension-virtual.com >,
> spce-user at lists.sipwise.com
> *Enviados: *Viernes, 21 de Noviembre 2014 14:31:25
> *Asunto: *RE: sip registration stuck 3.5.16
>
> Thank you Raul for your input and suggestions. Not to get into a long
> discussion but I at first didn't think it was a big deal until the
> following accours
> 1. the network environment at subscriber changes (change of IP) -
> asterisk registers and now you have 2 active registrations in the db.
> 2. Customers logging into the user portal and see the expired time stamp
> 3. Deleting the registration in the UI will not completely delete it in
> the db but add a second and third and so on registration.
>
> all my test was with public IPs i've even tested it while both asterisk
> and sipwise was connected to the same switch both in a virtual and hard
> environment
>
> Thank you again Raul but before placing into production it's absolutely
> imperative that the basic functionality work properly - register and
> register.
>
> Warmest Regards,
> Aziz
>
>
> ------------------------------------------------------------------------
> *From:* Spce-user [ spce-user-bounces at lists.sipwise.com ] on behalf of
> Raúl Alexis Betancor Santana [ rabs at dimension-virtual.com ]
> *Sent:* Friday, November 21, 2014 6:06 AM
> *To:* spce-user at lists.sipwise.com
> *Subject:* Re: [Spce-user] sip registration stuck 3.5.16
>
> Just to add ... that we are using SPCE since it's first versions ... and
> don't have that issue. With lot of asterisk customers with versions from
> Asterisk 1.2 to Asterisk 12 .
>
> As it seems that the register doesn't get auto-expired (I don't
> undestand why it hurts you that they are on the location table ... but
> that another thing), if you are experiencing 'registration lost' with
> Asterisk and SPCE ... 99.9% of the times it a fault on Asterisk or in a
> middle router that 'modify' the SIP request.
>
> Best and faster thing you could do to track the issue ... it to capture
> the traffic JUST on the Asterisk ... and on the SPCE and compare them,
> if you see any difference ... then you probably have a sip-alg-activated
> device on the middle of the path.
>
> If you don't have access to the Asterisk boxes ... just install one
> yourself for testing, it's easy and fast. ... Faster than trying that
> someneone give you the solution on a comunity list
>
> ------------------------------------------------------------------------
> *De: *"Daniel Grotti" < dgrotti at sipwise.com >
> *Para: *"Aziz Palmer" <Aziz.Palmer at AirFIBER.com>, "Theo"
> < axessofficetheo at gmail.com >
> *CC: * spce-user at lists.sipwise.com
> *Enviados: *Viernes, 21 de Noviembre 2014 14:08:31
> *Asunto: *Re: [Spce-user] sip registration stuck 3.5.16
>
> Hi,
> why are you so sure ?
> Kamailio version is the same, so if there is an issue it's also in PRO.
> I opened the internal ticket # 0010303.
>
> We will update you soon.
>
> Daniel
>
>
>
> On 11/21/2014 02:54 PM, Aziz Palmer wrote:
>
> Thank you Daniel!
>
> quick question - i'm sure the customers that are using the pro
> version aren't having this issue. Why is it happening on the CE
> version? (I've spent so much time on it and i'm trying to more
> understand it)
>
> Thank you,
> Aziz
> ------------------------------------------------------------------------
> *From:* Daniel Grotti [ dgrotti at sipwise.com ]
> *Sent:* Friday, November 21, 2014 1:17 AM
> *To:* Aziz Palmer; Theo
> *Cc:* spce-user at lists.sipwise.com
> *Subject:* Re: [Spce-user] sip registration stuck 3.5.16
>
> Aziz,
> I was able to have a look at the issue, and I opened an internal
> ticket about that.
> We will have a look at this and let you know.
>
> Daniel
>
>
>
> On 11/19/2014 12:41 AM, Aziz Palmer wrote:
>
> Theo,
> can you send what you put in your sip trunk settings when you
> connect to sipwise? (this part seems pretty straight forward but
> would like to try someone else's settings.)
>
> Thank you
> ------------------------------------------------------------------------
> *From:* Theo [ axessofficetheo at gmail.com ]
> *Sent:* Thursday, November 13, 2014 5:40 AM
> *To:* Aziz Palmer
> *Cc:* Andreas Granig; spce-user at lists.sipwise.com
> *Subject:* Re: [Spce-user] sip registration stuck 3.5.16
>
> Hi
>
> The problem is on the asterisk side so they are the ones to fix
> it (assuming that is indeed your problem). The "fix" on the spce
> is merely a work around to not ban the user and accept the
> registration.
>
> On Thu, Nov 13, 2014 at 3:38 PM, Aziz Palmer
> < Aziz.Palmer at airfiber.com <mailto: Aziz.Palmer at airfiber.com >> wrote:
>
> Thank you Theo!
>
> but what i don't understand if this is the case why wasn't
> fixed ?
>
> what should i be doing instead when having users register to
> avoid this issue?
>
> Is this happening to you also? if so what did you do to fix
> it? (follow the steps below?)
>
> Thank you!
>
>
> ------------------------------------------------------------------------
> *From:* Theo [ axessofficetheo at gmail.com
> <mailto: axessofficetheo at gmail.com >]
> *Sent:* Thursday, November 13, 2014 5:30 AM
> *To:* Aziz Palmer
> *Cc:* Andreas Granig; spce-user at lists.sipwise.com
> <mailto: spce-user at lists.sipwise.com >
>
> *Subject:* Re: [Spce-user] sip registration stuck 3.5.16
>
> Sorry I accidentally replied just to you rather than to the
> list.:
>
> Hi
>
> This looks like the stale nonce issue which is an asterisk
> problem. The workaround has been posted a few times - search
> for stale nonce and I am sure you'll find it easily
>
> Good luck
>
>
>
>
>
>
> to Aziz
>
> It would be this one:
>
> http://lists.sipwise.com/pipermail/spce-user/2014-January/005712.html
>
> On Thu, Nov 13, 2014 at 3:00 PM, Aziz Palmer
> < Aziz.Palmer at airfiber.com <mailto: Aziz.Palmer at airfiber.com >>
> wrote:
>
> kamailio.location
> what command do I need to execute to provide this
> information?
>
>
> kamailio-proxy.log
>
> Nov 13 06:49:36 tx-portal-2 proxy[22474]: NOTICE:
> <script>: Authentication failed, stale nonce -
> R=sip:sipwise.xyz..com
> ID= 7233034d51f47c1a7ca56eb2353030ae at 104.34.42.10
> <mailto: 7233034d51f47c1a7ca56eb2353030ae at 104.34.42.10 >
> Nov 13 06:49:36 tx-portal-2 proxy[22473]: NOTICE:
> <script>: New request on proxy - M=REGISTER
> R=sip:sipwise.xyz..com
> F=sip:master.sample.edge at sipwise.xyz..com
> T=sip:master.sample.edge at sipwise.xyz..com
> IP= 104.34.42.10:5060 < http://104.34.42.10:5060 >
> ( 127.0.0.1:5060 < http://127.0.0.1:5060 >)
> ID= 7233034d51f47c1a7ca56eb2353030ae at 104.34.42.10
> <mailto: 7233034d51f47c1a7ca56eb2353030ae at 104.34.42.10 >
> Nov 13 06:49:36 tx-portal-2 proxy[22473]: WARNING:
> app_lua [app_lua_sr.c:100]: lua_sr_log(): lost database
> connection. Reconnecting
> Nov 13 06:49:36 tx-portal-2 proxy[22473]: NOTICE:
> <script>: Contacts successfully updated, expires in 120s
> - R=sip:sipwise.xyz..com
> ID= 7233034d51f47c1a7ca56eb2353030ae at 104.34.42.10
> <mailto: 7233034d51f47c1a7ca56eb2353030ae at 104.34.42.10 >
> Nov 13 06:51:21 tx-portal-2 proxy[22480]: NOTICE:
> <script>: New request on proxy - M=REGISTER
> R=sip:sipwise.xyz..com
> F=sip:master.sample.edge at sipwise.xyz..com
> T=sip:master.sample.edge at sipwise.xyz..com
> IP= 104.34.42.10:5060 < http://104.34.42.10:5060 >
> ( 127.0.0.1:5060 < http://127.0.0.1:5060 >)
> ID= 7233034d51f47c1a7ca56eb2353030ae at 104.34.42.10
> <mailto: 7233034d51f47c1a7ca56eb2353030ae at 104.34.42.10 >
> Nov 13 06:51:21 tx-portal-2 proxy[22480]: WARNING:
> app_lua [app_lua_sr.c:100]: lua_sr_log(): lost database
> connection. Reconnecting
> Nov 13 06:51:21 tx-portal-2 proxy[22480]: NOTICE:
> <script>: Contacts successfully updated, expires in 120s
> - R=sip:sipwise.xyz..com
> ID= 7233034d51f47c1a7ca56eb2353030ae at 104.34.42.10
> <mailto: 7233034d51f47c1a7ca56eb2353030ae at 104.34.42.10 >
> Nov 13 06:53:06 tx-portal-2 proxy[22475]: NOTICE:
> <script>: New request on proxy - M=REGISTER
> R=sip:sipwise.xyz..com
> F=sip:master.sample.edge at sipwise.xyz..com
> T=sip:master.sample.edge at sipwise.xyz..com
> IP= 104.34.42.10:5060 < http://104.34.42.10:5060 >
> ( 127.0.0.1:5060 < http://127.0.0.1:5060 >)
> ID= 7233034d51f47c1a7ca56eb2353030ae at 104.34.42.10
> <mailto: 7233034d51f47c1a7ca56eb2353030ae at 104.34.42.10 >
> Nov 13 06:53:06 tx-portal-2 proxy[22475]: WARNING:
> app_lua [app_lua_sr.c:100]: lua_sr_log(): lost database
> connection. Reconnecting
> Nov 13 06:53:06 tx-portal-2 proxy[22475]: NOTICE:
> <script>: Contacts successfully updated, expires in 120s
> - R=sip:sipwise.xyz..com
> ID= 7233034d51f47c1a7ca56eb2353030ae at 104.34.42.10
> <mailto: 7233034d51f47c1a7ca56eb2353030ae at 104.34.42.10 >
>
>
>
>
> what does this mean "WARNING: app_lua
> [app_lua_sr.c:100]: lua_sr_log(): lost database
> connection. Reconnecting"?
>
>
>
> ________________________________________
> From: Spce-user [ spce-user-bounces at lists.sipwise.com
> <mailto: spce-user-bounces at lists.sipwise.com >] on behalf
> of Andreas Granig [ agranig at sipwise.com
> <mailto: agranig at sipwise.com >]
> Sent: Thursday, November 13, 2014 1:26 AM
> To: spce-user at lists.sipwise.com
> <mailto: spce-user at lists.sipwise.com >
> Subject: Re: [Spce-user] sip registration stuck 3.5.16
>
> Hi,
>
> It's most likely not a bug of the system, rather than a
> problem with
> your clients registering on said system. I risk to say
> that because the
> registrar is based on Kamailio, which is serving dozens
> of millions of
> SIP devices out there, so it would be a huge surprise
> for it not to work
> correctly.
>
> The only way to find out what the REAL issue is for you
> to send an
> example entry in the kamailio.location table of the DB,
> along with a
> full network trace of this client trying to register on
> the system, and
> also the kamailio-proxy.log file.
>
> Andreas
>
> On 11/13/2014 06:47 AM, Aziz Palmer wrote:
> > thats not the issue - the problem is when you
> RE-register a sip trunk it
> > will stop updating sipwise. in order to fix that you
> need to manually go
> > into the subscriber and delete the registration. it
> will not delete but
> > instead a new registration will appear.
> >
> >
> > SPCE - are you aware of this bug (its a serous issue -
> since it gives
> > the admins a lot of manual work/tech support calls) I
> would like to put
> > this this into partial production but can't - please help?
> >
> > Thanks!
> >
> ------------------------------------------------------------------------
> > *From:* Raúl Alexis Betancor Santana
> [ rabs at dimension-virtual.com
> <mailto: rabs at dimension-virtual.com >]
> > *Sent:* Wednesday, November 12, 2014 1:54 PM
> > *To:* Aziz Palmer
> > *Subject:* Re: sip registration stuck 3.5.16
> >
> > 3600 secs ... 1 h ... it's default time on the config.xml
> > kamailio.proxy.default_expire option ... you could
> change it ... or set
> > it on the web interface at domain-level or user-level
> if you prefer, and
> > as far as you don't touched the .cfg's ... they expire.
> >
> >
> ------------------------------------------------------------------------
> > *De: *"Aziz Palmer" <Aziz.Palmer at AirFIBER.com>
> > *Para: *"Raúl Alexis Betancor Santana"
> < rabs at dimension-virtual.com
> <mailto: rabs at dimension-virtual.com >>,
> > spce-user at lists.sipwise.com
> <mailto: spce-user at lists.sipwise.com >
> > *Enviados: *Miércoles, 12 de Noviembre 2014 21:17:53
> > *Asunto: *RE: sip registration stuck 3.5.16
> >
> > they are not auto-expiring (I wish they were)
> >
> > if they are - taking a extremely long time
> >
> ------------------------------------------------------------------------
> > *From:* Spce-user [ spce-user-bounces at lists.sipwise.com
> <mailto: spce-user-bounces at lists.sipwise.com >] on behalf of
> > Raúl Alexis Betancor Santana
> [ rabs at dimension-virtual.com
> <mailto: rabs at dimension-virtual.com >]
> > *Sent:* Wednesday, November 12, 2014 1:04 PM
> > *To:* spce-user at lists.sipwise.com
> <mailto: spce-user at lists.sipwise.com >
> > *Subject:* Re: [Spce-user] sip registration stuck 3.5.16
> >
> >
> > Why did you whant to remove the old register? ...
> location entries
> > auto-expire, so you don't have to worry about them.
> >
> >
> ------------------------------------------------------------------------
> > *De: *"Aziz Palmer" <Aziz.Palmer at AirFIBER.com>
> > *Para: * spce-user at lists.sipwise.com
> <mailto: spce-user at lists.sipwise.com >
> > *Enviados: *Miércoles, 12 de Noviembre 2014 20:47:13
> > *Asunto: *[Spce-user] sip registration stuck 3.5.16
> >
> > I hate to ask again but this is turning into a real
> issue - the
> > "registered devices" are not automatically being
> removed and when trying
> > to manually delete them, nothing happens.
> >
> > Can i set a timer on how long these orphan registered
> devices stay part
> > of sipwise?
> >
> > how can i delete these devices without having to issue
> the below command
> > - I would like to delete specific devices and not all
> of them
> >
> > mysql kamailio -e "delete from location"
> > /etc/init.d/kamailio-proxy restart
> >
> >
> > Thanks again,
> > Aziz
> >
> >
> ------------------------------------------------------------------------
> > *From:* Spce-user [ spce-user-bounces at lists.sipwise.com
> <mailto: spce-user-bounces at lists.sipwise.com >] on behalf of
> > Aziz Palmer [Aziz.Palmer at AirFIBER.com]
> > *Sent:* Tuesday, November 11, 2014 12:15 PM
> > *To:* spce-user at lists.sipwise.com
> <mailto: spce-user at lists.sipwise.com >
> > *Subject:* [Spce-user] sip registration stuck 5.3.16
> >
> > Hello SPCE Team,
> > here is what appears to be a bug
> >
> > I enter sip registration settings into asterisk,
> registers fine with
> > sipwise. Then I make a change within the sip trunk
> settings of asterisk
> > and apply the new settings. Then previous sip
> registration gets stuck;
> > meaning within sipwise 2 registrations appear (old &
> new). When i try to
> > delete the old one, it stays stuck. The only way i
> discovered to be able
> > to remove the old registration is by issuing this
> command, then i can
> > delete the registration.
> >
> > /etc/init.d/kamailio-proxy restart
> >
> > what additional information can i provide to have your
> team troubleshoot
> > this issue?
> >
> > Thank you,
> > Anees
> >
> > _______________________________________________
> > Spce-user mailing list
> > Spce-user at lists.sipwise.com
> <mailto: Spce-user at lists.sipwise.com >
> > https://lists.sipwise.com/listinfo/spce-user
> >
> >
> >
> >
> > _______________________________________________
> > Spce-user mailing list
> > Spce-user at lists.sipwise.com
> <mailto: Spce-user at lists.sipwise.com >
> > https://lists.sipwise.com/listinfo/spce-user
> >
> _______________________________________________
> Spce-user mailing list
> Spce-user at lists.sipwise.com
> <mailto: Spce-user at lists.sipwise.com >
> https://lists.sipwise.com/listinfo/spce-user
> _______________________________________________
> Spce-user mailing list
> Spce-user at lists.sipwise.com
> <mailto: Spce-user at lists.sipwise.com >
> https://lists.sipwise.com/listinfo/spce-user
>
>
>
>
>
> _______________________________________________
> Spce-user mailing list
> Spce-user at lists.sipwise.com
> https://lists.sipwise.com/listinfo/spce-user
>
>
>
>
> _______________________________________________
> Spce-user mailing list
> Spce-user at lists.sipwise.com
> https://lists.sipwise.com/listinfo/spce-user
>
>
>
>
> _______________________________________________
> Spce-user mailing list
> Spce-user at lists.sipwise.com
> https://lists.sipwise.com/listinfo/spce-user
>
_______________________________________________
Spce-user mailing list
Spce-user at lists.sipwise.com
https://lists.sipwise.com/listinfo/spce-user
_______________________________________________
Spce-user mailing list
Spce-user at lists.sipwise.com
https://lists.sipwise.com/listinfo/spce-user
_______________________________________________
Spce-user mailing list
Spce-user at lists.sipwise.com
https://lists.sipwise.com/listinfo/spce-user
</blockquote>
_______________________________________________
Spce-user mailing list
Spce-user at lists.sipwise.com
https://lists.sipwise.com/listinfo/spce-user
</blockquote>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sipwise.com/pipermail/spce-user_lists.sipwise.com/attachments/20141121/d1ff002f/attachment-0001.html>
More information about the Spce-user
mailing list