<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<font face="monospace">Hi Adam,<br>
<br>
This is something we never noticed in our tests, but it definitely
looks a problem related to proxy-redis communication.<br>
<br>
The first thing I would investigate is the number of opened
connection to redis. If redis reaches the max number of accepted
connections, then it starts dropping some of them or not accepting
new connections. <br>
<br>
You can use 'netstat' command to get how many connections redis
has active in a particular moment. Maybe you can find there some
of them stuck and you can try to identify who opened but didn't
close it properly.<br>
<br>
Thank you<br>
<br>
Best Regards<br>
Marco<br>
</font><font face="monospace"><br>
<br>
</font><br>
<div class="moz-cite-prefix">On 20/10/23 18:26, Adam Clark wrote:<br>
</div>
<blockquote type="cite"
cite="mid:LO2P265MB5658059FD2762FBBC265B987FCDBA@LO2P265MB5658.GBRP265.PROD.OUTLOOK.COM">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta name="Generator"
content="Microsoft Word 15 (filtered medium)">
<style>@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;
mso-ligatures:none;
mso-fareast-language:EN-US;}div.WordSection1
{page:WordSection1;}</style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="WordSection1">
<p class="MsoNormal">Afternoon all!<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Been a while since I last posted (back in
the v2.6 days)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Question…<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">We have a few boxes running mr9.5.4 that
will at times stop processing registrations. We have alerts
internally on systems that ping us on flapping SIP
registrations.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">On said boxes in the kamailio-lb.log we
see:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">lb[634146]: NOTICE: REPLY_FROM_INBOUND
<script>: Reply from Inbound - S=500 - I'm terribly
sorry, server error occurred (1/SL) M=REGISTER
IP=«udp»:«127.0.0.1»:«5062»
ID=«000001e4-27c6afde096c100099040080f0a581fc@IP-REMOVED»
UA='<null>' DESTIP=«127.0.0.1»:«5060»<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">In kamailio-options-lb.log we see:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Line 8307: Sep 1 09:05:56 box01
proxy[634207]: ERROR: ROUTE_REG_HANDLE db_redis
[redis_dbase.c:2043]: db_redis_perform_update(): Failed to get
reply for query: Connection reset by peer<o:p></o:p></p>
<p class="MsoNormal">Line 8308: Sep 1 09:05:56 box01
proxy[634207]: ERROR: ROUTE_REG_HANDLE db_redis
[redis_dbase.c:2298]: db_redis_perform_update(): failed to
perform the update<o:p></o:p></p>
<p class="MsoNormal">Line 8309: Sep 1 09:05:56 box01
proxy[634207]: ERROR: ROUTE_REG_HANDLE db_redis
[redis_dbase.c:2850]: db_redis_update(): failed to do the
query<o:p></o:p></p>
<p class="MsoNormal">Line 8310: Sep 1 09:05:56 box01
proxy[634207]: ERROR: ROUTE_REG_HANDLE usrloc
[ucontact.c:1147]: db_update_ucontact_ruid(): updating
database failed<o:p></o:p></p>
<p class="MsoNormal">Line 8311: Sep 1 09:05:56 box01
proxy[634207]: ERROR: ROUTE_REG_HANDLE usrloc
[ucontact.c:1663]: update_contact_db(): failed to update
database<o:p></o:p></p>
<p class="MsoNormal">Line 8312: Sep 1 09:05:56 box01
proxy[634207]: ERROR: ROUTE_REG_HANDLE registrar [save.c:796]:
update_contacts(): failed to update contact<o:p></o:p></p>
<p class="MsoNormal">Line 8313: Sep 1 09:05:56 box01
proxy[634207]: ERROR: ROUTE_REG_HANDLE <script>:
Contacts failed - R=«<a class="moz-txt-link-freetext" href="sip:box01.easyipt.co.uk;transport=udp»">sip:box01.easyipt.co.uk;transport=udp»</a>
ID=«<a class="moz-txt-link-abbreviated" href="mailto:0216ED4BE98140000000375C@92.207.169.202">0216ED4BE98140000000375C@92.207.169.202</a>» UA='NEC SL2100'<o:p></o:p></p>
<p class="MsoNormal">Line 8314: Sep 1 09:05:56 box01
proxy[634207]: NOTICE: ROUTE_REG_HANDLE <script>:
Sending reply S=500 Server Error fs='«127.0.0.1»:«5062»'
du='«127.0.0.1»:«5060»' - R=«<a class="moz-txt-link-freetext" href="sip:REMOVED;transport=udp»">sip:REMOVED;transport=udp»</a>
ID=«0216ED4BE98140000000375C@REMOVED» UA='NEC SL2100'<o:p></o:p></p>
<p class="MsoNormal">Line 8315: Sep 1 09:05:56 box01
proxy[634207]: ERROR: ROUTE_REG_HANDLE sl [sl_funcs.c:414]:
sl_reply_error(): stateless error reply used: I'm terribly
sorry, server error occurred (1/SL)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Redis.log does not show anything:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Sep 1 09:08:57 box01 (info)
redis[634122]: 634122:M 01 Sep 2023 09:08:57.481 - DB 3: 11
keys (0 volatile) in 64 slots HT.<o:p></o:p></p>
<p class="MsoNormal">Sep 1 09:08:57 box01 (info)
redis[634122]: 634122:M 01 Sep 2023 09:08:57.481 - DB 4: 2
keys (0 volatile) in 8 slots HT.<o:p></o:p></p>
<p class="MsoNormal">Sep 1 09:08:57 box01 (info)
redis[634122]: 634122:M 01 Sep 2023 09:08:57.481 - DB 19: 24
keys (24 volatile) in 32 slots HT.<o:p></o:p></p>
<p class="MsoNormal">Sep 1 09:08:57 box01 (info)
redis[634122]: 634122:M 01 Sep 2023 09:08:57.481 - DB 20: 57
keys (0 volatile) in 64 slots HT.<o:p></o:p></p>
<p class="MsoNormal">Sep 1 09:08:57 box01 (info)
redis[634122]: 634122:M 01 Sep 2023 09:08:57.481 - DB 21: 31
keys (0 volatile) in 64 slots HT.<o:p></o:p></p>
<p class="MsoNormal">Sep 1 09:08:57 box01 (info)
redis[634122]: 634122:M 01 Sep 2023 09:08:57.481 - DB 23: 32
keys (0 volatile) in 32 slots HT.<o:p></o:p></p>
<p class="MsoNormal">Sep 1 09:08:57 box01 (info)
redis[634122]: 634122:M 01 Sep 2023 09:08:57.481 - Closing
idle client<o:p></o:p></p>
<p class="MsoNormal">Sep 1 09:08:59 box01 (info)
redis[634122]: 634122:M 01 Sep 2023 09:08:59.290 - Closing
idle client<o:p></o:p></p>
<p class="MsoNormal">Sep 1 09:09:01 box01 (info)
redis[634122]: 634122:M 01 Sep 2023 09:09:01.252 - Accepted
127.0.0.1:51798<o:p></o:p></p>
<p class="MsoNormal">Sep 1 09:09:01 box01 (info)
redis[634122]: 634122:M 01 Sep 2023 09:09:01.258 - Client
closed connection<o:p></o:p></p>
<p class="MsoNormal">Sep 1 09:09:01 box01 (info)
redis[634122]: 634122:M 01 Sep 2023 09:09:01.517 - Accepted
127.0.0.1:51802<o:p></o:p></p>
<p class="MsoNormal">Sep 1 09:09:01 box01 (info)
redis[634122]: 634122:M 01 Sep 2023 09:09:01.518 - Accepted
127.0.0.1:51804<o:p></o:p></p>
<p class="MsoNormal">Sep 1 09:09:01 box01 (info)
redis[634122]: 634122:M 01 Sep 2023 09:09:01.519 - Accepted
127.0.0.1:51806<o:p></o:p></p>
<p class="MsoNormal">Sep 1 09:09:02 box01 (info)
redis[634122]: 634122:M 01 Sep 2023 09:09:02.312 - Accepted
127.0.0.1:51808<o:p></o:p></p>
<p class="MsoNormal">Sep 1 09:09:02 box01 (info)
redis[634122]: 634122:M 01 Sep 2023 09:09:02.314 - Accepted
127.0.0.1:51810<o:p></o:p></p>
<p class="MsoNormal">Sep 1 09:09:02 box01 (info)
redis[634122]: 634122:M 01 Sep 2023 09:09:02.506 - DB 3: 7
keys (0 volatile) in 64 slots HT.<o:p></o:p></p>
<p class="MsoNormal">Sep 1 09:09:02 box01 (info)
redis[634122]: 634122:M 01 Sep 2023 09:09:02.506 - DB 4: 1
keys (0 volatile) in 8 slots HT.<o:p></o:p></p>
<p class="MsoNormal">Sep 1 09:09:02 box01 (info)
redis[634122]: 634122:M 01 Sep 2023 09:09:02.506 - DB 19: 24
keys (24 volatile) in 32 slots HT.<o:p></o:p></p>
<p class="MsoNormal">Sep 1 09:09:02 box01 (info)
redis[634122]: 634122:M 01 Sep 2023 09:09:02.506 - DB 20: 57
keys (0 volatile) in 64 slots HT.<o:p></o:p></p>
<p class="MsoNormal">Sep 1 09:09:02 box01 (info)
redis[634122]: 634122:M 01 Sep 2023 09:09:02.506 - DB 21: 32
keys (0 volatile) in 64 slots HT.<o:p></o:p></p>
<p class="MsoNormal">Sep 1 09:09:02 box01 (info)
redis[634122]: 634122:M 01 Sep 2023 09:09:02.506 - DB 23: 32
keys (0 volatile) in 32 slots HT.<o:p></o:p></p>
<p class="MsoNormal">Sep 1 09:09:03 box01 (info)
redis[634122]: 634122:M 01 Sep 2023 09:09:03.553 - Client
closed connection<o:p></o:p></p>
<p class="MsoNormal">Sep 1 09:09:03 box01 (info)
redis[634122]: 634122:M 01 Sep 2023 09:09:03.553 - Client
closed connection<o:p></o:p></p>
<p class="MsoNormal">Sep 1 09:09:03 box01 (info)
redis[634122]: 634122:M 01 Sep 2023 09:09:03.553 - Client
closed connection<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Restarting REDIS does clear the problem for
a period of time however It does return. Typically, if the box
is physically restarted it goes away for a few months..<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Boxes are all VM on ESXi hosts.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Just wondering what I we can do to debug
this a little further, or if anyone knows what would be at
catalyst for this?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-GB">Thank
you in advance
</span><span
style="font-family:"Segoe UI Emoji",sans-serif;mso-fareast-language:EN-GB">😊</span><span
style="mso-fareast-language:EN-GB"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-GB"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-GB">Kind
Regards,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-GB"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-GB">Adam<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-GB"><o:p> </o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<br>
<fieldset class="moz-mime-attachment-header"></fieldset>
</blockquote>
<br>
<div class="moz-signature">-- <br>
<div
style="line-height:16px;
margin:6px 0;
padding:8px 8px 8px 8px;
border-top:1px #aeb1a6 dotted;
border-bottom:1px #aeb1a6 dotted;
font-family: 'Lucida Sans', Lucida Grande, Verdana, Arial, Sans-Serif;
font-size:11px;
color:#555555;"> <strong
style="color:#333333;
text-transform:uppercase;
font-size:10px;"> Marco Capetta </strong> <br>
R&D Manager
<p> <a href="http://www.sipwise.com"
style="color:rgb(0, 136, 204) !important;
text-decoration:none !important;
border-bottom:1px dotted #AAA;"> Sipwise GmbH | an ALE
Company </a><br>
c/o ALE International – Austria Branch<br>
Saturn Tower<br>
Leonard-Bernstein-Straße 10<br>
1220 Vienna, Austria </p>
<p> Email: <a href="mailto:mcapetta@sipwise.com"
style="color:rgb(0, 136, 204) !important;
text-decoration:none !important;
border-bottom:1px dotted #AAA;"
class="moz-txt-link-freetext"> mcapetta@sipwise.com </a> <br>
Website: <a href="http://www.sipwise.com"
style="color:rgb(0, 136, 204) !important;
text-decoration:none !important;
border-bottom:1px dotted #AAA;"> www.sipwise.com </a>
</p>
</div>
</div>
</body>
</html>