CentOS

Hi,

We are using CENTOS 5.8 in our product.
We get the following errors in var logs
kernel: httpd.worker invoked oom-killer: gfp_mask=0xd0, order=0, oomkilladj=0
kernel: [<c04463d2>] out_of_memory+0x72/0x178
kernel: [<c0447607>] __alloc_pages+0x1ff/0x280
kernel: [<c045a677>] cache_alloc_refill+0x217/0x3b5
kernel: [<c045a85a>] __kmalloc+0x45/0x51
kernel: [<c05915c0>] __alloc_skb+0x49/0xf4
kernel: [<c05c176d>] tcp_send_fin+0x55/0x138
kernel: [<c05b797e>] tcp_close+0x1ee/0x49e
kernel: [<c05cdc36>] inet_release+0x43/0x48
kernel: [<c058c88c>] sock_release+0x11/0x63
kernel: [<c058c904>] sock_close+0x26/0x2a
kernel: [<c045e4c2>] __fput+0x8a/0x13f
kernel: [<c045bf5e>] filp_close+0x4e/0x54
kernel: [<c041ca0c>] put_files_struct+0x64/0xa6
kernel: [<c041d934>] do_exit+0x240/0x6d2
kernel: [<c041de22>] sys_exit_group+0x0/0xd
kernel: [<c042406b>] get_signal_to_deliver+0x301/0x311
kernel: [<c04033f6>] do_notify_resume+0x76/0x642
kernel: [<c045dba5>] do_readv_writev+0x259/0x277
kernel: [<c05eb9a8>] do_page_fault+0x2d2/0x600
kernel: [<c045dbfa>] vfs_writev+0x37/0x43
kernel: [<c0403d2a>] work_notifysig+0x13/0x19
kernel: =======================
kernel: Mem-info:
kernel: DMA per-cpu:
kernel: cpu 0 hot: high 0, batch 1 used:0
kernel: cpu 0 cold: high 0, batch 1 used:0
kernel: DMA32 per-cpu: empty
kernel: Normal per-cpu:
kernel: cpu 0 hot: high 186, batch 31 used:144
kernel: cpu 0 cold: high 62, batch 15 used:51
kernel: HighMem per-cpu:
kernel: cpu 0 hot: high 186, batch 31 used:169
kernel: cpu 0 cold: high 62, batch 15 used:6
kernel: Free pages: 2939112kB (2927928kB HighMem)
kernel: Active:71597 inactive:9304 dirty:4 writeback:0 unstable:0 free:734778 slab:178814 mapped-file:4284 mapped-anon:46446 pagetables:768
kernel: DMA free:3664kB min:144kB low:180kB high:216kB active:16kB inactive:0kB present:16384kB pages_scanned:1034839 all_unreclaimable? yes
kernel: lowmem_reserve[]: 0 0 880 6128
kernel: DMA32 free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
kernel: lowmem_reserve[]: 0 0 880 6128
kernel: Normal free:7520kB min:8044kB low:10052kB high:12064kB active:132kB inactive:0kB present:901120kB pages_scanned:592 all_unreclaimable? no
kernel: lowmem_reserve[]: 0 0 0 41984
kernel: HighMem free:2927928kB min:512kB low:12504kB high:24500kB active:286240kB inactive:37216kB present:5373952kB pages_scanned:0 all_unreclaimable? no
kernel: lowmem_reserve[]: 0 0 0 0
kernel: DMA: 0*4kB 0*8kB 1*16kB 0*32kB 1*64kB 0*128kB 0*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 3664kB
kernel: DMA32: empty
kernel: Normal: 72*4kB 20*8kB 2*16kB 0*32kB 0*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 1*2048kB 1*4096kB = 7520kB
kernel: HighMem: 2316*4kB 2293*8kB 3690*16kB 3764*32kB 2315*64kB 1081*128kB 413*256kB 156*512kB 82*1024kB 45*2048kB 506*4096kB = 2927928kB
kernel: 34092 pagecache pages
kernel: Swap cache: add 0, delete 0, find 0/0, race 0+0
kernel: Free swap = 4192956kB
kernel: Total swap = 4192956kB
kernel: Free swap: 4192956kB
kernel: 1572864 pages of RAM
kernel: 1343488 pages of HIGHMEM
kernel: 573086 reserved pages
kernel: 39978 pages shared
kernel: 0 pages swap cached
kernel: 4 pages dirty
kernel: 0 pages writeback
kernel: 4284 pages mapped
kernel: 178827 pages slab
kernel: 768 pages pagetables
kernel: Out of memory: Killed process 22174 (httpd.worker).

Eventhough we have enough memory, httpd.worked process is killed saying that there was out of memory.
The server gets hung and we need to reboot the server to make it work again.

Kindly share your thoughts on this and if there is any workaround/solution related to this?

Thanks,
Tharini.

The solution is to run yum update to get the new kernel (among other packages), then reboot to start using the new kernel.

We are using the below kernel version
2.6.18-308.4.1.el5PAE

Can you please tell me to which kernel version i can upgrade?

Whether is it a known issue?
Is there any bug filed for this issue?

Thanks,
Tharini.

Sorry, I'm unable to locate a bug report for this issue. I would recommend using the 2.6.18-371.11.1 kernel.

What makes you say you are having enough RAM?

With a 32-bit PAE kernel, a single process is still limited to less than 4 GB, regardless of how much you have put into your machine.

The best solution would be to use a 64 bit distribution since it would appear at a quick read that you have plenty of RAM available (~2.9GB free) but the RAM is in the high memory region which is not usable for the purposes that are required. In a 32 bit system some functions require memory in the low region and your system seems to have run out of this - that's not something you can address by adding more RAM.