<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
span.E-MailFormatvorlage21
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 2.0cm 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=DE-AT link="#0563C1" vlink="#954F72" style='word-wrap:break-word'><div class=WordSection1><p class=MsoNormal><span lang=EN-US>Just for documentation, we solved the problem this way:<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>We start strace and and search out all call ids which “does not exist anywhere”.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>Then we delete this call ids from redis db 21.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>And reply this as long as no coredomp error comes again and mediator is up and running.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>I really don’t know why this even happened.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>In 5 years of Sipwise SPCE I never seen this error before.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>Maybe a new bug in 8.5.1?<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US> <o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><div><div style='border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm'><p class=MsoNormal><b><span lang=DE style='mso-fareast-language:DE-AT'>Von:</span></b><span lang=DE style='mso-fareast-language:DE-AT'> Matthias Hohl <matthias.hohl@telematica.at> <br><b>Gesendet:</b> Donnerstag, 8. Oktober 2020 11:26<br><b>An:</b> spce-user@lists.sipwise.com<br><b>Betreff:</b> Re: [Spce-user] mediator fails in 8.5.1<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:Consolas;color:#333333;background:#F8F8F8'>root@spce:~/mediator_coredump_analysis_201008# gdb /usr/bin/ngcp-mediator core.ngcp-mediator.0.2bdbff08ba1d4a19b4d5abd3641456d2.22061.1602147893000000<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:Consolas;color:#333333;background:#F8F8F8'>GNU gdb (Debian 8.2.1-2+b3) 8.2.1<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:Consolas;color:#333333;background:#F8F8F8'>Copyright (C) 2018 Free Software Foundation, Inc.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:Consolas;color:#333333;background:#F8F8F8'>License GPLv3+: GNU GPL version 3 or later <<a href="http://gnu.org/licenses/gpl.html">http://gnu.org/licenses/gpl.html</a>><o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:Consolas;color:#333333;background:#F8F8F8'>This is free software: you are free to change and redistribute it.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:Consolas;color:#333333;background:#F8F8F8'>There is NO WARRANTY, to the extent permitted by law.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:Consolas;color:#333333;background:#F8F8F8'>Type "show copying" and "show warranty" for details.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:Consolas;color:#333333;background:#F8F8F8'>This GDB was configured as "x86_64-linux-gnu".<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:Consolas;color:#333333;background:#F8F8F8'>Type "show configuration" for configuration details.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:Consolas;color:#333333;background:#F8F8F8'>For bug reporting instructions, please see:<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:Consolas;color:#333333;background:#F8F8F8'><<a href="http://www.gnu.org/software/gdb/bugs/">http://www.gnu.org/software/gdb/bugs/</a>>.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:Consolas;color:#333333;background:#F8F8F8'>Find the GDB manual and other documentation resources online at:<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:Consolas;color:#333333;background:#F8F8F8'> <<a href="http://www.gnu.org/software/gdb/documentation/">http://www.gnu.org/software/gdb/documentation/</a>>.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:Consolas;color:#333333;background:#F8F8F8'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:Consolas;color:#333333;background:#F8F8F8'>For help, type "help".<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:Consolas;color:#333333;background:#F8F8F8'>Type "apropos word" to search for commands related to "word"...<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:Consolas;color:#333333;background:#F8F8F8'>Reading symbols from /usr/bin/ngcp-mediator...(no debugging symbols found)...done.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:Consolas;color:#333333;background:#F8F8F8'>[New LWP 22061]<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:Consolas;color:#333333;background:#F8F8F8'>[Thread debugging using libthread_db enabled]<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:Consolas;color:#333333;background:#F8F8F8'>Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:Consolas;color:#333333;background:#F8F8F8'>Core was generated by `/usr/bin/ngcp-mediator'.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:Consolas;color:#333333;background:#F8F8F8'>Program terminated with signal SIGABRT, Aborted.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:Consolas;color:#333333;background:#F8F8F8'>#0 0x00007f753fae07bb in raise () from /lib/x86_64-linux-gnu/libc.so.6<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:Consolas;color:#333333;background:#F8F8F8'>(gdb)<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:Consolas;color:#333333;background:#F8F8F8'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><div><div style='border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm'><p class=MsoNormal><b><span lang=DE style='mso-fareast-language:DE-AT'>Von:</span></b><span lang=DE style='mso-fareast-language:DE-AT'> Matthias Hohl <<a href="mailto:matthias.hohl@telematica.at">matthias.hohl@telematica.at</a>> <br><b>Gesendet:</b> Donnerstag, 8. Oktober 2020 11:20<br><b>An:</b> <a href="mailto:spce-user@lists.sipwise.com">spce-user@lists.sipwise.com</a><br><b>Betreff:</b> [Spce-user] mediator fails in 8.5.1<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Hello.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><span lang=EN-US>Today mediator stops with this error:<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>Okt 08 11:03:03 spce mediator[21124]: mediator.c:191 [main]: Starting mediator<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>Okt 08 11:03:03 spce mediator[21124]: mediator.c:223 [main]: ACC acc database host='localhost', port='3306', user='kamailio', name='kamailio'<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>Okt 08 11:03:03 spce mediator[21124]: mediator.c:225 [main]: CDR acc database host='localhost', port='3306', user='mediator', name='accounting'<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>Okt 08 11:03:03 spce mediator[21124]: mediator.c:227 [main]: PROV database host='localhost', port='3306', user='mediator', name='provisioning'<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>Okt 08 11:03:03 spce mediator[21124]: mediator.c:229 [main]: STATS database host='localhost', port='3306', user='mediator', name='stats'<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>Okt 08 11:03:03 spce mediator[21124]: mediator.c:233 [main]: REDIS database host='localhost', port='6379', pass='<none>', id='21'<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>Okt 08 11:03:03 spce mediator[21124]: mediator.c:247 [main]: Up and running, daemonized=0, pid-path='/run/ngcp-mediator.pid', interval=10<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>Okt 08 11:03:03 spce systemd[1]: Started NGCP Mediator.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>Okt 08 11:03:03 spce systemd[1]: ngcp-mediator.service: Main process exited, code=dumped, status=6/ABRT<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>Okt 08 11:03:03 spce systemd[1]: ngcp-mediator.service: Failed with result 'core-dump'.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>I can’t find any logfile with any detailed info inside.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>just something like this:<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US># journalctl -u ngcp-mediator<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>Oct 08 07:27:33 spce mediator[1989]: cdr.c:943 [cdr_parse_srcleg_list]: Call-Id '7ca858807ac29f3d41adbc8249405f31@sip.telematica.at' has no separated group info, ''<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>Oct 08 07:27:33 spce mediator[1989]: cdr.c:943 [cdr_parse_srcleg_list]: Call-Id '7ba22c98-7d46-1239-9b95-ac1f6b222f98' has no separated group info, ''<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>And this error wit the same call-id repeating every minute.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>It looks there are some ghost call-lds there he can’t proceed cause the didn’t exist anymore.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>Also since mediator is down, my Grafana statistic for <o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>SELECT "kam_concurrent_calls" FROM "autogen"."kamailio" WHERE $timeFilter GROUP BY "host"<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>Shows me a flatline and no change in concurrent calls.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>I personally believe this could be a problem if ghost calls in redisDB.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>Any idea how to fix that?<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>Is it safe to do <o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US># redis-cli -n 3 FLUSHDB<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span style='mso-fareast-language:DE-AT'>Mit freundlichen Grüßen,<o:p></o:p></span></p><p class=MsoNormal><o:p> </o:p></p></div></body></html>