<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Segoe UI Emoji";
panose-1:2 11 5 2 4 2 4 2 2 3;}
@font-face
{font-family:"Lucida Sans";
panose-1:2 11 6 2 3 5 4 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle22
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
mso-ligatures:none;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-GB" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">Hey Marco,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thank you for coming back to me so quickly.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">We did have a look in the REDIS config and it did say that it would error in the event of maximum connections being reached noting it in the error log but maybe that’s only if REDIS is set to debug?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">We have 4 systems with this exact same problem, so I don’t know If its an underlying issue somewhere. Is there anything we can give you for debugging in your lab?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">We have 2 systems that we have not yet fully restarted so when one of them plays up again we will run the netstat command.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Is it also worth setting REDIS to debug?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Kind Regards,<o:p></o:p></p>
<p class="MsoNormal"><br>
Adam<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><span style="mso-fareast-language:EN-GB"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-GB"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-GB"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="mso-fareast-language:EN-GB">From:</span></b><span lang="EN-US" style="mso-fareast-language:EN-GB"> Marco Capetta via Spce-user <spce-user@lists.sipwise.com>
<br>
<b>Sent:</b> Monday, October 23, 2023 8:42 AM<br>
<b>To:</b> spce-user@lists.sipwise.com<br>
<b>Subject:</b> Re: [Spce-user] mr9.5.4 - REDIS - Failed to get reply for query: Connection reset by peer<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-family:"Courier New"">Hi Adam,<br>
<br>
This is something we never noticed in our tests, but it definitely looks a problem related to proxy-redis communication.<br>
<br>
The first thing I would investigate is the number of opened connection to redis. If redis reaches the max number of accepted connections, then it starts dropping some of them or not accepting new connections.
<br>
<br>
You can use 'netstat' command to get how many connections redis has active in a particular moment. Maybe you can find there some of them stuck and you can try to identify who opened but didn't close it properly.<br>
<br>
Thank you<br>
<br>
Best Regards<br>
Marco<br>
<br>
<br>
</span><span style="mso-fareast-language:EN-GB"><o:p></o:p></span></p>
<div>
<p class="MsoNormal">On 20/10/23 18:26, Adam Clark wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">Afternoon all!<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">Been a while since I last posted (back in the v2.6 days)<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">Question…<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">We have a few boxes running mr9.5.4 that will at times stop processing registrations. We have alerts internally on systems that ping us on flapping SIP registrations.<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">On said boxes in the kamailio-lb.log we see:<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">lb[634146]: NOTICE: REPLY_FROM_INBOUND <script>: Reply from Inbound - S=500 - I'm terribly sorry, server error occurred (1/SL) M=REGISTER IP=«udp»:«127.0.0.1»:«5062» ID=«000001e4-27c6afde096c100099040080f0a581fc@IP-REMOVED» UA='<null>'
DESTIP=«127.0.0.1»:«5060»<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">In kamailio-options-lb.log we see:<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">Line 8307: Sep 1 09:05:56 box01 proxy[634207]: ERROR: ROUTE_REG_HANDLE db_redis [redis_dbase.c:2043]: db_redis_perform_update(): Failed to get reply for query: Connection reset by peer<o:p></o:p></p>
<p class="MsoNormal">Line 8308: Sep 1 09:05:56 box01 proxy[634207]: ERROR: ROUTE_REG_HANDLE db_redis [redis_dbase.c:2298]: db_redis_perform_update(): failed to perform the update<o:p></o:p></p>
<p class="MsoNormal">Line 8309: Sep 1 09:05:56 box01 proxy[634207]: ERROR: ROUTE_REG_HANDLE db_redis [redis_dbase.c:2850]: db_redis_update(): failed to do the query<o:p></o:p></p>
<p class="MsoNormal">Line 8310: Sep 1 09:05:56 box01 proxy[634207]: ERROR: ROUTE_REG_HANDLE usrloc [ucontact.c:1147]: db_update_ucontact_ruid(): updating database failed<o:p></o:p></p>
<p class="MsoNormal">Line 8311: Sep 1 09:05:56 box01 proxy[634207]: ERROR: ROUTE_REG_HANDLE usrloc [ucontact.c:1663]: update_contact_db(): failed to update database<o:p></o:p></p>
<p class="MsoNormal">Line 8312: Sep 1 09:05:56 box01 proxy[634207]: ERROR: ROUTE_REG_HANDLE registrar [save.c:796]: update_contacts(): failed to update contact<o:p></o:p></p>
<p class="MsoNormal">Line 8313: Sep 1 09:05:56 box01 proxy[634207]: ERROR: ROUTE_REG_HANDLE <script>: Contacts failed - R=«<a href="sip:box01.easyipt.co.uk;transport=udp»">sip:box01.easyipt.co.uk;transport=udp»</a> ID=«<a href="mailto:0216ED4BE98140000000375C@92.207.169.202">0216ED4BE98140000000375C@92.207.169.202</a>»
UA='NEC SL2100'<o:p></o:p></p>
<p class="MsoNormal">Line 8314: Sep 1 09:05:56 box01 proxy[634207]: NOTICE: ROUTE_REG_HANDLE <script>: Sending reply S=500 Server Error fs='«127.0.0.1»:«5062»' du='«127.0.0.1»:«5060»' - R=«<a href="sip:REMOVED;transport=udp»">sip:REMOVED;transport=udp»</a>
ID=«0216ED4BE98140000000375C@REMOVED» UA='NEC SL2100'<o:p></o:p></p>
<p class="MsoNormal">Line 8315: Sep 1 09:05:56 box01 proxy[634207]: ERROR: ROUTE_REG_HANDLE sl [sl_funcs.c:414]: sl_reply_error(): stateless error reply used: I'm terribly sorry, server error occurred (1/SL)<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">Redis.log does not show anything:<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">Sep 1 09:08:57 box01 (info) redis[634122]: 634122:M 01 Sep 2023 09:08:57.481 - DB 3: 11 keys (0 volatile) in 64 slots HT.<o:p></o:p></p>
<p class="MsoNormal">Sep 1 09:08:57 box01 (info) redis[634122]: 634122:M 01 Sep 2023 09:08:57.481 - DB 4: 2 keys (0 volatile) in 8 slots HT.<o:p></o:p></p>
<p class="MsoNormal">Sep 1 09:08:57 box01 (info) redis[634122]: 634122:M 01 Sep 2023 09:08:57.481 - DB 19: 24 keys (24 volatile) in 32 slots HT.<o:p></o:p></p>
<p class="MsoNormal">Sep 1 09:08:57 box01 (info) redis[634122]: 634122:M 01 Sep 2023 09:08:57.481 - DB 20: 57 keys (0 volatile) in 64 slots HT.<o:p></o:p></p>
<p class="MsoNormal">Sep 1 09:08:57 box01 (info) redis[634122]: 634122:M 01 Sep 2023 09:08:57.481 - DB 21: 31 keys (0 volatile) in 64 slots HT.<o:p></o:p></p>
<p class="MsoNormal">Sep 1 09:08:57 box01 (info) redis[634122]: 634122:M 01 Sep 2023 09:08:57.481 - DB 23: 32 keys (0 volatile) in 32 slots HT.<o:p></o:p></p>
<p class="MsoNormal">Sep 1 09:08:57 box01 (info) redis[634122]: 634122:M 01 Sep 2023 09:08:57.481 - Closing idle client<o:p></o:p></p>
<p class="MsoNormal">Sep 1 09:08:59 box01 (info) redis[634122]: 634122:M 01 Sep 2023 09:08:59.290 - Closing idle client<o:p></o:p></p>
<p class="MsoNormal">Sep 1 09:09:01 box01 (info) redis[634122]: 634122:M 01 Sep 2023 09:09:01.252 - Accepted 127.0.0.1:51798<o:p></o:p></p>
<p class="MsoNormal">Sep 1 09:09:01 box01 (info) redis[634122]: 634122:M 01 Sep 2023 09:09:01.258 - Client closed connection<o:p></o:p></p>
<p class="MsoNormal">Sep 1 09:09:01 box01 (info) redis[634122]: 634122:M 01 Sep 2023 09:09:01.517 - Accepted 127.0.0.1:51802<o:p></o:p></p>
<p class="MsoNormal">Sep 1 09:09:01 box01 (info) redis[634122]: 634122:M 01 Sep 2023 09:09:01.518 - Accepted 127.0.0.1:51804<o:p></o:p></p>
<p class="MsoNormal">Sep 1 09:09:01 box01 (info) redis[634122]: 634122:M 01 Sep 2023 09:09:01.519 - Accepted 127.0.0.1:51806<o:p></o:p></p>
<p class="MsoNormal">Sep 1 09:09:02 box01 (info) redis[634122]: 634122:M 01 Sep 2023 09:09:02.312 - Accepted 127.0.0.1:51808<o:p></o:p></p>
<p class="MsoNormal">Sep 1 09:09:02 box01 (info) redis[634122]: 634122:M 01 Sep 2023 09:09:02.314 - Accepted 127.0.0.1:51810<o:p></o:p></p>
<p class="MsoNormal">Sep 1 09:09:02 box01 (info) redis[634122]: 634122:M 01 Sep 2023 09:09:02.506 - DB 3: 7 keys (0 volatile) in 64 slots HT.<o:p></o:p></p>
<p class="MsoNormal">Sep 1 09:09:02 box01 (info) redis[634122]: 634122:M 01 Sep 2023 09:09:02.506 - DB 4: 1 keys (0 volatile) in 8 slots HT.<o:p></o:p></p>
<p class="MsoNormal">Sep 1 09:09:02 box01 (info) redis[634122]: 634122:M 01 Sep 2023 09:09:02.506 - DB 19: 24 keys (24 volatile) in 32 slots HT.<o:p></o:p></p>
<p class="MsoNormal">Sep 1 09:09:02 box01 (info) redis[634122]: 634122:M 01 Sep 2023 09:09:02.506 - DB 20: 57 keys (0 volatile) in 64 slots HT.<o:p></o:p></p>
<p class="MsoNormal">Sep 1 09:09:02 box01 (info) redis[634122]: 634122:M 01 Sep 2023 09:09:02.506 - DB 21: 32 keys (0 volatile) in 64 slots HT.<o:p></o:p></p>
<p class="MsoNormal">Sep 1 09:09:02 box01 (info) redis[634122]: 634122:M 01 Sep 2023 09:09:02.506 - DB 23: 32 keys (0 volatile) in 32 slots HT.<o:p></o:p></p>
<p class="MsoNormal">Sep 1 09:09:03 box01 (info) redis[634122]: 634122:M 01 Sep 2023 09:09:03.553 - Client closed connection<o:p></o:p></p>
<p class="MsoNormal">Sep 1 09:09:03 box01 (info) redis[634122]: 634122:M 01 Sep 2023 09:09:03.553 - Client closed connection<o:p></o:p></p>
<p class="MsoNormal">Sep 1 09:09:03 box01 (info) redis[634122]: 634122:M 01 Sep 2023 09:09:03.553 - Client closed connection<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">Restarting REDIS does clear the problem for a period of time however It does return. Typically, if the box is physically restarted it goes away for a few months..<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">Boxes are all VM on ESXi hosts.<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">Just wondering what I we can do to debug this a little further, or if anyone knows what would be at catalyst for this?<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-GB">Thank you in advance
</span><span style="font-family:"Segoe UI Emoji",sans-serif;mso-fareast-language:EN-GB">😊</span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-GB"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-GB">Kind Regards,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-GB"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-GB">Adam</span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-GB"> </span><o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-GB"><br>
<br>
<o:p></o:p></span></p>
</blockquote>
<p class="MsoNormal"><span style="mso-fareast-language:EN-GB"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><span style="mso-fareast-language:EN-GB">-- <o:p></o:p></span></p>
<div style="border-top:dotted #AEB1A6 1.0pt;border-left:none;border-bottom:dotted #AEB1A6 1.0pt;border-right:none;padding:6.0pt 0cm 6.0pt 0cm;margin-top:4.5pt;margin-bottom:4.5pt">
<p class="MsoNormal" style="line-height:12.0pt"><strong><span style="font-size:7.5pt;font-family:"Lucida Sans",sans-serif;color:#333333;text-transform:uppercase;mso-fareast-language:EN-GB">Marco Capetta
</span></strong><span style="font-size:8.5pt;font-family:"Lucida Sans",sans-serif;color:#555555;mso-fareast-language:EN-GB"><br>
R&D Manager <o:p></o:p></span></p>
<p style="line-height:12.0pt"><span style="font-size:8.5pt;font-family:"Lucida Sans",sans-serif;color:#555555"><a href="http://www.sipwise.com/"><span style="color:#0088CC;text-decoration:none">Sipwise GmbH | an ALE Company
</span></a><br>
c/o ALE International – Austria Branch<br>
Saturn Tower<br>
Leonard-Bernstein-Straße 10<br>
1220 Vienna, Austria <o:p></o:p></span></p>
<p style="line-height:12.0pt"><span style="font-size:8.5pt;font-family:"Lucida Sans",sans-serif;color:#555555">Email:
<a href="mailto:mcapetta@sipwise.com"><span style="color:#0088CC;text-decoration:none">mcapetta@sipwise.com
</span></a><br>
Website: <a href="http://www.sipwise.com/"><span style="color:#0088CC;text-decoration:none">www.sipwise.com
</span></a><o:p></o:p></span></p>
</div>
</div>
</div>
</body>
</html>