[Spce-user] mediator fails

[qabane me]

Hi Andreas

Thanks for the response.

To my knowledge I did nothing but apply hot-fixes to fix the other issue
with mediator (the one I provided backtraces for). The only other thing
that stands out from this is that I cloned this from another VM - changed
the MAC of the NIC and changed the IP address. I have no idea if that could
be related or not. Other than that I cannot think of anything.

I will give the cleanup a go and let you know how it goes.


On Mon, Sep 3, 2018 at 4:39 PM Andreas Granig <agranig at sipwise.com> wrote:

> Hi,
>
> We’ve pushed a code change to master for mediator to automatically trash
> and skip records with a mapping entry in acc:cid::$callid and
> acc:meth::$method, so starting at least with mr6.5 mediator won’t bail out
> on this anymore.
>
> For now, you can manually clean this up and let mediator continue by
> executing
>
> $ redis-cli -n 21 srem acc:meth::INVITE
> acc:entry::011cd8e3-220b-1237-d9a8-a6958035e26f:1535092647.008037
> $ redis-cli -n 21 srem acc:meth::INVITE
> acc:entry::011cd8e3-220b-1237-d9a8-a6958035e26f:1535092756.025455
> $ redis-cli -n 21 srem acc:meth::BYE
> acc:entry::011cd8e3-220b-1237-d9a8-a6958035e26f:1535092647.008037
> $ redis-cli -n 21 srem acc:meth::BYE
> acc:entry::011cd8e3-220b-1237-d9a8-a6958035e26f:1535092756.025455
> $ redis-cli -n 21 del acc:cid::011cd8e3-220b-1237-d9a8-a6958035e26f
>
> However, this and the scheduled mediator fix are actually just removing
> the sympthom by masking the root of the problem, namely how you ended up in
> the situation of having the acc:cid and acc:meth mappings pointing to
> non-existent hash keys in redis. So any insights into what happened to the
> system before that would be beneficial. In our case, there was one instance
> of that situation where it happened in the labs, namely when kamailio and
> sems crashed on some experimental code.
>
> If this actual problem persists and happens for others as well, we’ll
> consider back-porting the mediator trashing feature to mr6.3 and mr6.4,
> otherwise the advise is to remove the offending keys manually for now and
> report it on the list.
>
> Andreas
>
> > On 30.08.2018, at 13:26, qabane me <qabaneitsolutions at gmail.com> wrote:
> >
> > root at spce:~# redis-cli -n 21 hgetall
> acc:entry::011cd8e3-220b-1237-d9a8-a6958035e26f:1535092647.008037
> > (empty list or set)
> > root at spce:~# redis-cli -n 21 hgetall
> acc:entry::011cd8e3-220b-1237-d9a8-a6958035e26f:1535092756.025455
> > (empty list or set)
> >
> >
> > On Thu, Aug 30, 2018 at 1:26 PM qabane me <qabaneitsolutions at gmail.com>
> wrote:
> > Hi
> >
> >
> > 1) "acc:entry::011cd8e3-220b-1237-d9a8-a6958035e26f:1535092756.025455"
> > 2) "acc:entry::011cd8e3-220b-1237-d9a8-a6958035e26f:1535092647.008037"
> >
> > root at spce:~# redis-cli -n 21 hgetall
> acc:entry::011cd8e3-220b-1237-d9a8-a6958035e26f:1535092647.008037
> > (empty list or set)
> > root at spce:~# redis-cli -n 21 hgetall
> acc:entry::011cd8e3-220b-1237-d9a8-a6958035e26f:1535092756.025455
> >
> >
> > On Thu, Aug 30, 2018 at 1:09 PM Andreas Granig <agranig at sipwise.com>
> wrote:
> > Hi,
> >
> > Ah, right.
> >
> > So, please do
> >
> > redis-cli -n 21 smembers acc:cid::011cd8e3-220b-1237-d9a8-a6958035e26f
> >
> > and for each of the entries returned there, do
> >
> > redis-cli -n 21 hgetall $entry
> >
> > with $entry being the individual entries returned in the smembers
> command. Let’s see then what’s really in your database.
> >
> > Andreas
> >
> > > On 30.08.2018, at 12:03, qabane me <qabaneitsolutions at gmail.com>
> wrote:
> > >
> > > redis-cli -n 21 type acc:cid::011cd8e3-220b-1237-d9a8-a6958035e26f
> > > set
> > >
> > > Does that mean something to you?
> > >
> > > On Thu, Aug 30, 2018 at 11:56 AM Andreas Granig <agranig at sipwise.com>
> wrote:
> > > Hi,
> > >
> > > > On 30.08.2018, at 10:34, qabane me <qabaneitsolutions at gmail.com>
> wrote:
> > > >
> > > > root at spce:~# redis-cli -n 21 hgetall
> acc:cid::011cd8e3-220b-1237-d9a8-a6958035e26f
> > > > (error) WRONGTYPE Operation against a key holding the wrong kind of
> value
> > >
> > >
> > > What type is the data structure then? You can figure it out by
> > >
> > > redis-cli -n 21 type acc:cid::011cd8e3-220b-1237-d9a8-a6958035e26f
> > >
> > > Best,
> > > Andreas
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sipwise.com/mailman/private/spce-user_lists.sipwise.com/attachments/20180903/c4e5f8f6/attachment.html>