[Spce-user] low mem?

[Matthew Ogden]

Hi All



Our box is pretty stable, today, kamailio proxy and lb were crashed. I
tried to restart them, but proxy would not start, with the error "could not
start"



The syslog show it killing the process, but I don't know enough about linux
memory inner workings... why all of a sudden, maybe the log is meaningful to
one of you out there?



Feb 19 23:59:16 spce rsyslogd: [origin software="rsyslogd"
swVersion="5.8.11" x-pid="945" x-info="http://www.rsyslog.com"] rsyslogd
was HUPed

Feb 20 07:09:02 spce kernel: [1455123.928553] sems invoked oom-killer:
gfp_mask=0x201da, order=0, oom_adj=0

Feb 20 07:09:02 spce kernel: [1455123.928560] sems cpuset=/ mems_allowed=0

Feb 20 07:09:02 spce kernel: [1455123.928565] Pid: 27688, comm: sems Not
tainted 2.6.32-5-amd64 #1

Feb 20 07:09:02 spce kernel: [1455123.928568] Call Trace:

Feb 20 07:09:02 spce kernel: [1455123.928580]  [<ffffffff810b66f0>] ?
oom_kill_process+0x7f/0x23f

Feb 20 07:09:02 spce kernel: [1455123.928587]  [<ffffffff8106bdca>] ?
timekeeping_get_ns+0xe/0x2e

Feb 20 07:09:02 spce kernel: [1455123.928592]  [<ffffffff810b6c14>] ?
__out_of_memory+0x12a/0x141

Feb 20 07:09:02 spce kernel: [1455123.928596]  [<ffffffff810b6d6b>] ?
out_of_memory+0x140/0x172

Feb 20 07:09:02 spce kernel: [1455123.928602]  [<ffffffff810baad0>] ?
__alloc_pages_nodemask+0x4ec/0x5fc

Feb 20 07:09:02 spce kernel: [1455123.928609]  [<ffffffff810bc039>] ?
__do_page_cache_readahead+0x9b/0x1b4

Feb 20 07:09:02 spce kernel: [1455123.928614]  [<ffffffff810bc16e>] ?
ra_submit+0x1c/0x20

Feb 20 07:09:02 spce kernel: [1455123.928618]  [<ffffffff810b4e3b>] ?
filemap_fault+0x17d/0x2f6

Feb 20 07:09:02 spce kernel: [1455123.928625]  [<ffffffff810cadf2>] ?
__do_fault+0x54/0x3c3

Feb 20 07:09:02 spce kernel: [1455123.928629]  [<ffffffff810cd146>] ?
handle_mm_fault+0x3b8/0x80f

Feb 20 07:09:02 spce kernel: [1455123.928636]  [<ffffffff812fd2fa>] ?
do_nanosleep+0x6d/0xa3

Feb 20 07:09:02 spce kernel: [1455123.928642]  [<ffffffff810682c3>] ?
hrtimer_nanosleep+0xee/0x10e

Feb 20 07:09:02 spce kernel: [1455123.928648]  [<ffffffff81300096>] ?
do_page_fault+0x2e0/0x2fc

Feb 20 07:09:02 spce kernel: [1455123.928653]  [<ffffffff812fdf35>] ?
page_fault+0x25/0x30

Feb 20 07:09:02 spce kernel: [1455123.928656] Mem-Info:

Feb 20 07:09:02 spce kernel: [1455123.928658] Node 0 DMA per-cpu:

Feb 20 07:09:02 spce kernel: [1455123.928662] CPU    0: hi:    0, btch:   1
usd:   0

Feb 20 07:09:02 spce kernel: [1455123.928665] CPU    1: hi:    0, btch:   1
usd:   0

Feb 20 07:09:02 spce kernel: [1455123.928668] CPU    2: hi:    0, btch:   1
usd:   0

Feb 20 07:09:02 spce kernel: [1455123.928670] CPU    3: hi:    0, btch:   1
usd:   0

Feb 20 07:09:02 spce kernel: [1455123.928673] Node 0 DMA32 per-cpu:

Feb 20 07:09:02 spce kernel: [1455123.928676] CPU    0: hi:  186, btch:  31
usd: 103

Feb 20 07:09:02 spce kernel: [1455123.928679] CPU    1: hi:  186, btch:  31
usd:   0

Feb 20 07:09:02 spce kernel: [1455123.928682] CPU    2: hi:  186, btch:  31
usd:  18

Feb 20 07:09:02 spce kernel: [1455123.928685] CPU    3: hi:  186, btch:  31
usd:   0

Feb 20 07:09:02 spce kernel: [1455123.928692] active_anon:182665
inactive_anon:61405 isolated_anon:6

Feb 20 07:09:02 spce kernel: [1455123.928694]  active_file:0
inactive_file:45 isolated_file:0

Feb 20 07:09:02 spce kernel: [1455123.928695]  unevictable:46048 dirty:0
writeback:0 unstable:0

Feb 20 07:09:02 spce kernel: [1455123.928696]  free:2372
slab_reclaimable:3439 slab_unreclaimable:4161

Feb 20 07:09:02 spce kernel: [1455123.928698]  mapped:40113 shmem:31864
pagetables:11263 bounce:0

Feb 20 07:09:02 spce kernel: [1455123.928701] Node 0 DMA free:5012kB
min:52kB low:64kB high:76kB active_anon:2680kB inactive_anon:3188kB
active_file:0kB inactive_file:4kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:15308kB mlocked:0kB dirty:0kB writeback:0kB
mapped:4104kB shmem:0kB slab_reclaimable:272kB slab_unreclaimable:552kB
kernel_stack:32kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB
pages_scanned:0 all_unreclaimable? no

Feb 20 07:09:02 spce kernel: [1455123.928717] lowmem_reserve[]: 0 1246 1246
1246

Feb 20 07:09:02 spce kernel: [1455123.928722] Node 0 DMA32 free:4476kB
min:4488kB low:5608kB high:6732kB active_anon:727980kB
inactive_anon:242432kB active_file:0kB inactive_file:176kB
unevictable:184192kB isolated(anon):24kB isolated(file):0kB
present:1276512kB mlocked:184192kB dirty:0kB writeback:0kB mapped:156348kB
shmem:127456kB slab_reclaimable:13484kB slab_unreclaimable:16092kB
kernel_stack:2872kB pagetables:45052kB unstable:0kB bounce:0kB
writeback_tmp:0kB pages_scanned:133 all_unreclaimable? no

Feb 20 07:09:02 spce kernel: [1455123.928738] lowmem_reserve[]: 0 0 0 0

Feb 20 07:09:02 spce kernel: [1455123.928743] Node 0 DMA: 54*4kB 39*8kB
39*16kB 29*32kB 20*64kB 5*128kB 2*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB
= 5024kB

Feb 20 07:09:02 spce kernel: [1455123.928755] Node 0 DMA32: 1010*4kB 28*8kB
3*16kB 0*32kB 2*64kB 0*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB =
4696kB

Feb 20 07:09:02 spce kernel: [1455123.928767] 64100 total pagecache pages

Feb 20 07:09:02 spce kernel: [1455123.928769] 27549 pages in swap cache

Feb 20 07:09:02 spce kernel: [1455123.928772] Swap cache stats: add
2969403, delete 2941854, find 17745333/18198177

Feb 20 07:09:02 spce kernel: [1455123.928775] Free swap  = 0kB

Feb 20 07:09:02 spce kernel: [1455123.928777] Total swap = 838648kB

Feb 20 07:09:02 spce kernel: [1455123.934710] 327664 pages RAM

Feb 20 07:09:02 spce kernel: [1455123.934713] 7450 pages reserved

Feb 20 07:09:02 spce kernel: [1455123.934716] 525286 pages shared

Feb 20 07:09:02 spce kernel: [1455123.934717] 261262 pages non-shared

Feb 20 07:09:02 spce kernel: [1455123.934722] Out of memory: kill process
27551 (kamailio) score 187690 or a child

Feb 20 07:09:02 spce kernel: [1455123.934928] Killed process 27552
(kamailio)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sipwise.com/mailman/private/spce-user_lists.sipwise.com/attachments/20140221/09eaa285/attachment.html>