I discovered yesterday that for the past few days, my R5F27 MBE has been experiencing random out of memory errors that are forcing a shutdown of the mythtv-backend daemon.

a sample from syslog on the latest event:

Nov 1 19:51:16 AVBox3 kernel: oom-killer: gfp_mask=0x280d2, order=0
Nov 1 19:51:16 AVBox3 kernel: [<c0153c66>] out_of_memory+0x136/0x170
Nov 1 19:51:16 AVBox3 kernel: [<c01556fa>] __alloc_pages+0x2da/0x2f0
Nov 1 19:51:16 AVBox3 kernel: [<c015e309>] __handle_mm_fault+0x7c9/0x950
Nov 1 19:51:16 AVBox3 kernel: [<c013a855>] hrtimer_run_queues+0xf5/0x150
Nov 1 19:51:16 AVBox3 kernel: [<c0135218>] __rcu_process_callbacks+0x198/0x1a0
Nov 1 19:51:16 AVBox3 kernel: [<c0128dde>] tasklet_action+0x5e/0xd0
Nov 1 19:51:16 AVBox3 kernel: [<c0119c6b>] do_page_fault+0xdb/0x5a8
Nov 1 19:51:16 AVBox3 kernel: [<c0119b90>] do_page_fault+0x0/0x5a8
Nov 1 19:51:16 AVBox3 kernel: [<c01042f9>] error_code+0x39/0x40
Nov 1 19:51:16 AVBox3 kernel: Mem-info:
Nov 1 19:51:16 AVBox3 kernel: DMA per-cpu:
Nov 1 19:51:16 AVBox3 kernel: cpu 0 hot: high 0, batch 1 used:0
Nov 1 19:51:16 AVBox3 kernel: cpu 0 cold: high 0, batch 1 used:0
Nov 1 19:51:16 AVBox3 kernel: DMA32 per-cpu: empty
Nov 1 19:51:16 AVBox3 kernel: Normal per-cpu:
Nov 1 19:51:16 AVBox3 kernel: cpu 0 hot: high 186, batch 31 used:28
Nov 1 19:51:16 AVBox3 kernel: cpu 0 cold: high 62, batch 15 used:54
Nov 1 19:51:16 AVBox3 kernel: HighMem per-cpu: empty
Nov 1 19:51:16 AVBox3 kernel: Free pages: 6024kB (0kB HighMem)
Nov 1 19:51:16 AVBox3 kernel: Active:19856 inactive:5725 dirty:0 writeback:0 unstable:0 free:1506 slab:96562 mapped:3765 pagetables:329
Nov 1 19:51:16 AVBox3 kernel: DMA free:2068kB min:88kB low:108kB high:132kB lots:176kB active:7272kB inactive:0kB present:16384kB pages_scanned:8270 all_unreclaimable? yes
Nov 1 19:51:16 AVBox3 kernel: lowmem_reserve[]: 0 0 495 495
Nov 1 19:51:16 AVBox3 kernel: DMA32 free:0kB min:0kB low:0kB high:0kB lots:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
Nov 1 19:51:16 AVBox3 kernel: lowmem_reserve[]: 0 0 495 495
Nov 1 19:51:16 AVBox3 kernel: Normal free:3956kB min:2804kB low:3504kB high:4204kB lots:5608kB active:72152kB inactive:22900kB present:507840kB pages_scanned:5408 all_unreclaimable? no
Nov 1 19:51:16 AVBox3 kernel: lowmem_reserve[]: 0 0 0 0
Nov 1 19:51:16 AVBox3 kernel: HighMem free:0kB min:128kB low:128kB high:128kB lots:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
Nov 1 19:51:16 AVBox3 kernel: lowmem_reserve[]: 0 0 0 0
Nov 1 19:51:16 AVBox3 kernel: DMA: 3*4kB 1*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 2068kB
Nov 1 19:51:16 AVBox3 kernel: DMA32: empty
Nov 1 19:51:16 AVBox3 kernel: Normal: 295*4kB 9*8kB 1*16kB 2*32kB 1*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 3956kB
Nov 1 19:51:16 AVBox3 kernel: HighMem: empty
Nov 1 19:51:16 AVBox3 kernel: Swap cache: add 143811, delete 136465, find 104450/116859, race 0+0
Nov 1 19:51:16 AVBox3 kernel: Free swap = 577048kB
Nov 1 19:51:16 AVBox3 kernel: Total swap = 779144kB
Nov 1 19:51:16 AVBox3 kernel: Free swap: 577048kB
Nov 1 19:51:16 AVBox3 kernel: 131056 pages of RAM
Nov 1 19:51:16 AVBox3 kernel: 0 pages of HIGHMEM
Nov 1 19:51:16 AVBox3 kernel: 2774 reserved pages
Nov 1 19:51:16 AVBox3 kernel: 11419 pages shared
Nov 1 19:51:16 AVBox3 kernel: 7348 pages swap cached

the kern.log and other logs post identical entries concerning this event.

I can resart the mythtv-backend manually, but obviously this is not an ideal solution since I can't be standing by to do this at any time it happens.

I've always used the babysit_backend.sh script on the MBE, ever since setting it up with R5E50.

A few (perhaps pertinent) historical events.

1. Machine as originally configured worked absolutely flawlessly w/ R5E50.
2. Upgraded to R5F27 when it was released and the upgrade went fine.
3. Machine under R5F27 worked just as flawlessly as R5E50.
4. Power outage trashed the machine, forcing a new upgrade from an old backup.
5. All orphaned recordings were restored, and database checked for errors.
6. Installed a USB connected APC UPS, following the directions found here in the forums. This did involve turning on USB devices in the BIOS, which had previously been disabled.
7. Logs show a couple of low voltage events from the UPS, lasting only a few seconds, which seems to indicate that the acpupsd modules are functioning properly.
8. Ever since this problem surfaced, there have also been error messages regarding transcode errors. "transcode failed with status:0"

Technical specs on this MBE:
K7VTA3 v6.0c Mainboard
Sempron 2200 CPU
512Mb Ram
500Gb Maxtor STM3500630A
4 PVR-150s
(3 connected to cable coax)
(1 connected via composite inputs, with serial control of cable box)
nVidia TNT2 vidcard connected to 15inch monitor.

A few other details:
1.This machine has successfully encoded four simultaneous recordings from all four PVRs with no problems in the past, using R5F27
2. Although the full frontend is active on this MBE, it is used only for maintenance and/or testing, etc. , never used for actual watching.
3. 2 Slave frontends are in use for watching recorded programs. Watching Live TV is not used on the entire system. Both frontends function fine, except of course when this recently developed problem prevents them from connecting to the MBE.

Troubleshooting this problem goes WAY BEYOND my Linux skills, which are rudimentary at best, although not completely clueless.

Searching the forums has resulted in nothing that seems to shed any light on this particular problem.

So....

What should I do?

DaveJ45

_________________
LinHES - The only SANE Multimedia Option!

dump the output of "ps uaxww" to file when it happens.. maybe mythbackend is not the cause.

you could also do "lsof -p <pid>" where pid is the pid of the process you want to take a closer look at.. it might give you an idea of what is going on..

you might also play around with the boot time options relating to apci.

Since I couldn't seem to diagnose the source of the problem, and even though this MBE had been working just fine with 512MB of ram, I went ahead and yanked out the memory and replaced it with a 1GB instead.

Recording one show, commercial flagging one show, and transcoding one show all at the same time, I get the following stats on memory and CPU:


CPU load averages 1.14 (1 min) 1.18 (5 mins) 1.00 (15 mins)
Real memory 1008.95 MB total, 733.77 MB used
Virtual memory 760.88 MB total, 48 kB used

Judging from the 733.77MB used, I can only conclude that, for whatever reason, I was indeed running out of memory.

_________________
LinHES - The only SANE Multimedia Option!