[Spce-user] mediator fails

[qabane me]

Hi

I ran the cleanup commands but that didn't work - the results were as
follows:

root at spce:~# redis-cli -n 21 srem acc:meth::INVITE
acc:entry::011cd8e3-220b-1237-d9a8-a6958035e26f:1535092647.008037
(integer) 1
root at spce:~# redis-cli -n 21 srem acc:meth::INVITE
acc:entry::011cd8e3-220b-1237-d9a8-a6958035e26f:1535092756.025455
(integer) 0
root at spce:~# redis-cli -n 21 srem acc:meth::BYE
acc:entry::011cd8e3-220b-1237-d9a8-a6958035e26f:1535092647.008037
(integer) 0
root at spce:~# redis-cli -n 21 srem acc:meth::BYE
acc:entry::011cd8e3-220b-1237-d9a8-a6958035e26f:1535092756.025455
(integer) 1
root at spce:~# redis-cli -n 21 del
acc:cid::011cd8e3-220b-1237-d9a8-a6958035e26f
(integer) 1

Mediator still fails with the same error though?

On Mon, Sep 3, 2018 at 8:25 PM qabane me <qabaneitsolutions at gmail.com>
wrote:

> Hi Andreas
>
> Thanks for the response.
>
> To my knowledge I did nothing but apply hot-fixes to fix the other issue
> with mediator (the one I provided backtraces for). The only other thing
> that stands out from this is that I cloned this from another VM - changed
> the MAC of the NIC and changed the IP address. I have no idea if that could
> be related or not. Other than that I cannot think of anything.
>
> I will give the cleanup a go and let you know how it goes.
>
>
> On Mon, Sep 3, 2018 at 4:39 PM Andreas Granig <agranig at sipwise.com> wrote:
>
>> Hi,
>>
>> We’ve pushed a code change to master for mediator to automatically trash
>> and skip records with a mapping entry in acc:cid::$callid and
>> acc:meth::$method, so starting at least with mr6.5 mediator won’t bail out
>> on this anymore.
>>
>> For now, you can manually clean this up and let mediator continue by
>> executing
>>
>> $ redis-cli -n 21 srem acc:meth::INVITE
>> acc:entry::011cd8e3-220b-1237-d9a8-a6958035e26f:1535092647.008037
>> $ redis-cli -n 21 srem acc:meth::INVITE
>> acc:entry::011cd8e3-220b-1237-d9a8-a6958035e26f:1535092756.025455
>> $ redis-cli -n 21 srem acc:meth::BYE
>> acc:entry::011cd8e3-220b-1237-d9a8-a6958035e26f:1535092647.008037
>> $ redis-cli -n 21 srem acc:meth::BYE
>> acc:entry::011cd8e3-220b-1237-d9a8-a6958035e26f:1535092756.025455
>> $ redis-cli -n 21 del acc:cid::011cd8e3-220b-1237-d9a8-a6958035e26f
>>
>> However, this and the scheduled mediator fix are actually just removing
>> the sympthom by masking the root of the problem, namely how you ended up in
>> the situation of having the acc:cid and acc:meth mappings pointing to
>> non-existent hash keys in redis. So any insights into what happened to the
>> system before that would be beneficial. In our case, there was one instance
>> of that situation where it happened in the labs, namely when kamailio and
>> sems crashed on some experimental code.
>>
>> If this actual problem persists and happens for others as well, we’ll
>> consider back-porting the mediator trashing feature to mr6.3 and mr6.4,
>> otherwise the advise is to remove the offending keys manually for now and
>> report it on the list.
>>
>> Andreas
>>
>> > On 30.08.2018, at 13:26, qabane me <qabaneitsolutions at gmail.com> wrote:
>> >
>> > root at spce:~# redis-cli -n 21 hgetall
>> acc:entry::011cd8e3-220b-1237-d9a8-a6958035e26f:1535092647.008037
>> > (empty list or set)
>> > root at spce:~# redis-cli -n 21 hgetall
>> acc:entry::011cd8e3-220b-1237-d9a8-a6958035e26f:1535092756.025455
>> > (empty list or set)
>> >
>> >
>> > On Thu, Aug 30, 2018 at 1:26 PM qabane me <qabaneitsolutions at gmail.com>
>> wrote:
>> > Hi
>> >
>> >
>> > 1) "acc:entry::011cd8e3-220b-1237-d9a8-a6958035e26f:1535092756.025455"
>> > 2) "acc:entry::011cd8e3-220b-1237-d9a8-a6958035e26f:1535092647.008037"
>> >
>> > root at spce:~# redis-cli -n 21 hgetall
>> acc:entry::011cd8e3-220b-1237-d9a8-a6958035e26f:1535092647.008037
>> > (empty list or set)
>> > root at spce:~# redis-cli -n 21 hgetall
>> acc:entry::011cd8e3-220b-1237-d9a8-a6958035e26f:1535092756.025455
>> >
>> >
>> > On Thu, Aug 30, 2018 at 1:09 PM Andreas Granig <agranig at sipwise.com>
>> wrote:
>> > Hi,
>> >
>> > Ah, right.
>> >
>> > So, please do
>> >
>> > redis-cli -n 21 smembers acc:cid::011cd8e3-220b-1237-d9a8-a6958035e26f
>> >
>> > and for each of the entries returned there, do
>> >
>> > redis-cli -n 21 hgetall $entry
>> >
>> > with $entry being the individual entries returned in the smembers
>> command. Let’s see then what’s really in your database.
>> >
>> > Andreas
>> >
>> > > On 30.08.2018, at 12:03, qabane me <qabaneitsolutions at gmail.com>
>> wrote:
>> > >
>> > > redis-cli -n 21 type acc:cid::011cd8e3-220b-1237-d9a8-a6958035e26f
>> > > set
>> > >
>> > > Does that mean something to you?
>> > >
>> > > On Thu, Aug 30, 2018 at 11:56 AM Andreas Granig <agranig at sipwise.com>
>> wrote:
>> > > Hi,
>> > >
>> > > > On 30.08.2018, at 10:34, qabane me <qabaneitsolutions at gmail.com>
>> wrote:
>> > > >
>> > > > root at spce:~# redis-cli -n 21 hgetall
>> acc:cid::011cd8e3-220b-1237-d9a8-a6958035e26f
>> > > > (error) WRONGTYPE Operation against a key holding the wrong kind of
>> value
>> > >
>> > >
>> > > What type is the data structure then? You can figure it out by
>> > >
>> > > redis-cli -n 21 type acc:cid::011cd8e3-220b-1237-d9a8-a6958035e26f
>> > >
>> > > Best,
>> > > Andreas
>> >
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sipwise.com/pipermail/spce-user_lists.sipwise.com/attachments/20180903/d1b238a1/attachment-0001.html>