[Spce-user] mediator fails in 8.5.1

Thu Oct 8 12:56:39 EDT 2020

Just for documentation, we solved the problem this way:

We start strace and and search out all call ids which “does not exist anywhere”.

Then we delete this call ids from redis db 21.

And reply this as long as no coredomp error comes again and mediator is up and running.

I really don’t know why this even happened.

In 5 years of Sipwise SPCE I never seen this error before.

Maybe a new bug in 8.5.1?

Von: Matthias Hohl <matthias.hohl at telematica.at> 
Gesendet: Donnerstag, 8. Oktober 2020 11:26
An: spce-user at lists.sipwise.com
Betreff: Re: [Spce-user] mediator fails in 8.5.1

root at spce:~/mediator_coredump_analysis_201008# gdb /usr/bin/ngcp-mediator core.ngcp-mediator.0.2bdbff08ba1d4a19b4d5abd3641456d2.22061.1602147893000000

GNU gdb (Debian 8.2.1-2+b3) 8.2.1

License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software: you are free to change and redistribute it.

There is NO WARRANTY, to the extent permitted by law.

Type "show copying" and "show warranty" for details.

This GDB was configured as "x86_64-linux-gnu".

Type "show configuration" for configuration details.

For bug reporting instructions, please see:

<http://www.gnu.org/software/gdb/bugs/>.

Find the GDB manual and other documentation resources online at:

    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".

Type "apropos word" to search for commands related to "word"...

Reading symbols from /usr/bin/ngcp-mediator...(no debugging symbols found)...done.

[New LWP 22061]

[Thread debugging using libthread_db enabled]

Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Core was generated by `/usr/bin/ngcp-mediator'.

Program terminated with signal SIGABRT, Aborted.

#0  0x00007f753fae07bb in raise () from /lib/x86_64-linux-gnu/libc.so.6

(gdb)

Von: Matthias Hohl <matthias.hohl at telematica.at <mailto:matthias.hohl at telematica.at> > 
Gesendet: Donnerstag, 8. Oktober 2020 11:20
An: spce-user at lists.sipwise.com <mailto:spce-user at lists.sipwise.com> 
Betreff: [Spce-user] mediator fails in 8.5.1

Hello.

Today mediator stops with this error:

Okt 08 11:03:03 spce mediator[21124]: mediator.c:191 [main]: Starting mediator

Okt 08 11:03:03 spce mediator[21124]: mediator.c:223 [main]: ACC acc database host='localhost', port='3306', user='kamailio', name='kamailio'

Okt 08 11:03:03 spce mediator[21124]: mediator.c:225 [main]: CDR acc database host='localhost', port='3306', user='mediator', name='accounting'

Okt 08 11:03:03 spce mediator[21124]: mediator.c:227 [main]: PROV database host='localhost', port='3306', user='mediator', name='provisioning'

Okt 08 11:03:03 spce mediator[21124]: mediator.c:229 [main]: STATS database host='localhost', port='3306', user='mediator', name='stats'

Okt 08 11:03:03 spce mediator[21124]: mediator.c:233 [main]: REDIS database host='localhost', port='6379', pass='<none>', id='21'

Okt 08 11:03:03 spce mediator[21124]: mediator.c:247 [main]: Up and running, daemonized=0, pid-path='/run/ngcp-mediator.pid', interval=10

Okt 08 11:03:03 spce systemd[1]: Started NGCP Mediator.

Okt 08 11:03:03 spce systemd[1]: ngcp-mediator.service: Main process exited, code=dumped, status=6/ABRT

Okt 08 11:03:03 spce systemd[1]: ngcp-mediator.service: Failed with result 'core-dump'.

I can’t find any logfile with any detailed info inside.

just something like this:

# journalctl -u ngcp-mediator

Oct 08 07:27:33 spce mediator[1989]: cdr.c:943 [cdr_parse_srcleg_list]: Call-Id '7ca858807ac29f3d41adbc8249405f31 at sip.telematica.at' has no separated group info, ''

Oct 08 07:27:33 spce mediator[1989]: cdr.c:943 [cdr_parse_srcleg_list]: Call-Id '7ba22c98-7d46-1239-9b95-ac1f6b222f98' has no separated group info, ''

And this error wit the same call-id repeating every minute.

It looks there are some ghost call-lds there he can’t proceed cause the didn’t exist anymore.

Also since mediator is down, my Grafana statistic for 

SELECT "kam_concurrent_calls" FROM "autogen"."kamailio" WHERE $timeFilter GROUP BY "host"

Shows me a flatline and no change in concurrent calls.

I personally believe this could be a problem if ghost calls in redisDB.

Any idea how to fix that?

Is it safe to do 

# redis-cli -n 3 FLUSHDB

Mit freundlichen Grüßen,

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sipwise.com/pipermail/spce-user_lists.sipwise.com/attachments/20201008/2e70207b/attachment-0002.html>