[Spce-user] Kamailio-lb core dump

[Andreas Granig]

Hi John,

Thanks for the trace, we're going to check this. Can you by any chance 
also provide (also either anonymized, or sent to me in private) the ~100 
lines from the log file before the line "child process xxxxx exited by a 
signal 11"? I guess they will show more errors as well, which will help 
us pin the problem down in kamailio.

Thanks,
Andreas

On 01/08/2013 10:33 AM, John Murray wrote:
> Andreas hi,
>
>
> GNU gdb (GDB) 7.0.1-debian
> Copyright (C) 2009 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later
> <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-linux-gnu".
> For bug reporting instructions, please see:
> <http://www.gnu.org/software/gdb/bugs/>...
> Reading symbols from /usr/sbin/kamailio...(no debugging symbols
> found)...done.
>
> warning: Can't read pathname for load map: Input/output error.
> Reading symbols from /lib/libdl.so.2...(no debugging symbols found)...done.
> Loaded symbols for /lib/libdl.so.2
> Reading symbols from /lib/libresolv.so.2...(no debugging symbols
> found)...done.
> Loaded symbols for /lib/libresolv.so.2
> Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done.
> Loaded symbols for /lib/libc.so.6
> Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols
> found)...done.
> Loaded symbols for /lib64/ld-linux-x86-64.so.2
> Reading symbols from /usr/lib/kamailio/modules/corex.so...(no debugging
> symbols found)...done.
> Loaded symbols for /usr/lib/kamailio/modules/corex.so
> Reading symbols from /usr/lib/kamailio/modules/sl.so...(no debugging symbols
> found)...done.
> Loaded symbols for /usr/lib/kamailio/modules/sl.so
> Reading symbols from /usr/lib64/kamailio/libkcore.so.1...(no debugging
> symbols found)...done.
> Loaded symbols for /usr/lib64/kamailio/libkcore.so.1
> Reading symbols from /usr/lib/kamailio/modules_k/pv.so...(no debugging
> symbols found)...done.
> Loaded symbols for /usr/lib/kamailio/modules_k/pv.so
> Reading symbols from /usr/lib64/kamailio/libkmi.so.1...(no debugging symbols
> found)...done.
> Loaded symbols for /usr/lib64/kamailio/libkmi.so.1
> Reading symbols from /usr/lib/kamailio/modules_k/rr.so...(no debugging
> symbols found)...done.
> Loaded symbols for /usr/lib/kamailio/modules_k/rr.so
> Reading symbols from /usr/lib/kamailio/modules_k/maxfwd.so...(no debugging
> symbols found)...done.
> Loaded symbols for /usr/lib/kamailio/modules_k/maxfwd.so
> Reading symbols from /usr/lib/kamailio/modules_k/xlog.so...(no debugging
> symbols found)...done.
> Loaded symbols for /usr/lib/kamailio/modules_k/xlog.so
> Reading symbols from /usr/lib/kamailio/modules_k/usrloc.so...(no debugging
> symbols found)...done.
> Loaded symbols for /usr/lib/kamailio/modules_k/usrloc.so
> Reading symbols from /usr/lib64/kamailio/libsrdb1.so.1...(no debugging
> symbols found)...done.
> Loaded symbols for /usr/lib64/kamailio/libsrdb1.so.1
> Reading symbols from /usr/lib64/kamailio/libsrutils.so.1...(no debugging
> symbols found)...done.
> Loaded symbols for /usr/lib64/kamailio/libsrutils.so.1
> Reading symbols from /lib/libm.so.6...(no debugging symbols found)...done.
> Loaded symbols for /lib/libm.so.6
> Reading symbols from /usr/lib/kamailio/modules_k/textops.so...(no debugging
> symbols found)...done.
> Loaded symbols for /usr/lib/kamailio/modules_k/textops.so
> Reading symbols from /usr/lib/kamailio/modules_k/db_text.so...(no debugging
> symbols found)...done.
> Loaded symbols for /usr/lib/kamailio/modules_k/db_text.so
> Reading symbols from /usr/lib/kamailio/modules_k/uri_db.so...(no debugging
> symbols found)...done.
> Loaded symbols for /usr/lib/kamailio/modules_k/uri_db.so
> Reading symbols from /usr/lib/kamailio/modules_k/nathelper.so...(no
> debugging symbols found)...done.
> Loaded symbols for /usr/lib/kamailio/modules_k/nathelper.so
> Reading symbols from /usr/lib/kamailio/modules_k/path.so...(no debugging
> symbols found)...done.
> Loaded symbols for /usr/lib/kamailio/modules_k/path.so
> Reading symbols from /usr/lib/kamailio/modules_k/dispatcher.so...(no
> debugging symbols found)...done.
> Loaded symbols for /usr/lib/kamailio/modules_k/dispatcher.so
> Reading symbols from /usr/lib/kamailio/modules/avpops.so...(no debugging
> symbols found)...done.
> Loaded symbols for /usr/lib/kamailio/modules/avpops.so
> Reading symbols from /usr/lib/kamailio/modules_k/siputils.so...(no debugging
> symbols found)...done.
> Loaded symbols for /usr/lib/kamailio/modules_k/siputils.so
> Reading symbols from /usr/lib/kamailio/modules/sanity.so...(no debugging
> symbols found)...done.
> Loaded symbols for /usr/lib/kamailio/modules/sanity.so
> Reading symbols from /usr/lib/kamailio/modules/ctl.so...(no debugging
> symbols found)...done.
> Loaded symbols for /usr/lib/kamailio/modules/ctl.so
> Reading symbols from /usr/lib/kamailio/modules/xmlrpc.so...(no debugging
> symbols found)...done.
> Loaded symbols for /usr/lib/kamailio/modules/xmlrpc.so
> Reading symbols from /usr/lib/libxml2.so.2...(no debugging symbols
> found)...done.
> Loaded symbols for /usr/lib/libxml2.so.2
> Reading symbols from /usr/lib/libz.so.1...(no debugging symbols
> found)...done.
> Loaded symbols for /usr/lib/libz.so.1
> Reading symbols from /usr/lib/kamailio/modules/cfg_rpc.so...(no debugging
> symbols found)...done.
> Loaded symbols for /usr/lib/kamailio/modules/cfg_rpc.so
> Reading symbols from /usr/lib/kamailio/modules_k/mi_fifo.so...(no debugging
> symbols found)...done.
> Loaded symbols for /usr/lib/kamailio/modules_k/mi_fifo.so
> Reading symbols from /usr/lib/kamailio/modules_k/kex.so...(no debugging
> symbols found)...done.
> Loaded symbols for /usr/lib/kamailio/modules_k/kex.so
> Reading symbols from /usr/lib/kamailio/modules/textopsx.so...(no debugging
> symbols found)...done.
> Loaded symbols for /usr/lib/kamailio/modules/textopsx.so
> Reading symbols from /usr/lib/kamailio/modules_k/htable.so...(no debugging
> symbols found)...done.
> Loaded symbols for /usr/lib/kamailio/modules_k/htable.so
> Reading symbols from /usr/lib/kamailio/modules/tls.so...(no debugging
> symbols found)...done.
> Loaded symbols for /usr/lib/kamailio/modules/tls.so
> Reading symbols from /usr/lib/libssl.so.0.9.8...(no debugging symbols
> found)...done.
> Loaded symbols for /usr/lib/libssl.so.0.9.8
> Reading symbols from /usr/lib/libcrypto.so.0.9.8...(no debugging symbols
> found)...done.
> Loaded symbols for /usr/lib/libcrypto.so.0.9.8
> Reading symbols from /lib/libnss_compat.so.2...(no debugging symbols
> found)...done.
> Loaded symbols for /lib/libnss_compat.so.2
> Reading symbols from /lib/libnsl.so.1...(no debugging symbols found)...done.
> Loaded symbols for /lib/libnsl.so.1
> Reading symbols from /lib/libnss_nis.so.2...(no debugging symbols
> found)...done.
> Loaded symbols for /lib/libnss_nis.so.2
> Reading symbols from /lib/libnss_files.so.2...(no debugging symbols
> found)...done.
> Loaded symbols for /lib/libnss_files.so.2
> Reading symbols from /usr/lib/kamailio/modules_k/pike.so...(no debugging
> symbols found)...done.
> Loaded symbols for /usr/lib/kamailio/modules_k/pike.so
> Core was generated by `/usr/sbin/kamailio -f /etc/kamailio/lb/kamailio.cfg
> -P /var/run/kamailio/kamail'.
> Program terminated with signal 11, Segmentation fault.
> #0  0x00007f710d038bf4 in pv_set_ruri () from
> /usr/lib/kamailio/modules_k/pv.so
> (gdb)
> (gdb)
> (gdb)
> (gdb) bt
> #0  0x00007f710d038bf4 in pv_set_ruri () from
> /usr/lib/kamailio/modules_k/pv.so
> #1  0x000000000045ea29 in lval_assign ()
> #2  0x00000000004167b9 in do_action ()
> #3  0x000000000041dfd6 in run_actions ()
> #4  0x0000000000417993 in do_action ()
> #5  0x000000000041dfd6 in run_actions ()
> #6  0x0000000000417993 in do_action ()
> #7  0x000000000041dfd6 in run_actions ()
> #8  0x000000000041e580 in run_top_route ()
> #9  0x00000000004971ce in receive_msg ()
> #10 0x0000000000520387 in udp_rcv_loop ()
> #11 0x000000000046306d in main_loop ()
> #12 0x0000000000465c4c in main ()
>
>
> -----Original Message-----
> From: spce-user-bounces at lists.sipwise.com
> [mailto:spce-user-bounces at lists.sipwise.com] On Behalf Of Andreas Granig
> Sent: 08 January 2013 08:29
> To: spce-user at lists.sipwise.com
> Subject: Re: [Spce-user] Kamailio-lb core dump
>
> Hi,
>
> On 01/07/2013 07:27 PM, John Murray wrote:
>> Jan  7 18:04:43 skt-salem03 /usr/sbin/kamailio[10401]: ALERT: <core>
>> [main.c:785]: child process 10402 exited by a signal 11
>>
>> Jan  7 18:04:43 skt-salem03 /usr/sbin/kamailio[10432]: : <core>
>> [pass_fd.c:293]: ERROR: receive_fd: EOF on 12
>>
>> Jan  7 18:04:43 skt-salem03 /usr/sbin/kamailio[10401]: ALERT: <core>
>> [main.c:788]: core was generated
>>
>> Jan  7 18:04:43 skt-salem03 /usr/sbin/kamailio[10401]: INFO: <core>
>> [main.c:800]: INFO: terminating due to SIGCHLD
>
> That sounds bad. Could you please check if /var/cores/core.kamailio* exists,
> and if so, pick the latest one (something like
> /var/cores/core.kamailio.sig11.12345) and execute:
>
> gdb /usr/sbin/kamailio /var/cores/core.kamailio.sig11.12345
>
> and send us the output of the command "bt".
>
> Thanks!
> Andreas
>
> _______________________________________________
> Spce-user mailing list
> Spce-user at lists.sipwise.com
> http://lists.sipwise.com/listinfo/spce-user
>