View unanswered posts    View active topics

All times are UTC - 6 hours





Post new topic Reply to topic  [ 12 posts ] 
Print view Previous topic   Next topic  
Author Message
Search for:
PostPosted: Mon Feb 18, 2008 9:16 pm 
Offline
Joined: Fri Sep 15, 2006 12:16 pm
Posts: 292
Solved:
Apparently my /myth partition had failed to mount one time and
it had started recording shows in the unmounted /myth directory.
Of course these files are invisible once /myth is mounted properly.

Cliff

...and it is not the normal "log files ate up the space" or the
slightly less normal "open files have been deleted but not closed"

I found that my root partition was out of space. I have
temporarily rescued it by booting Knoppix and tune2fs'ing
it to reduce the space reserved for root and thus get
back some root free space to run in.

"du -xhs /" shows that there is
2.3 gig of the 4.7 gig root partition used.
"df -h /" shows that there is
4.2 gig of the 4.7 gig root partition used
(334 meg free at this point. I am a little
disappointed in df in that it is self inconsistent in
that 4.2 + (334/1024) is not close to 4.7).

I have done my homework as detailed below, but I haven't
found the problem yet. Sorry for the long post, but better
that then bouncing Q&A for days.

--- DETAILS FOLLOW ---

Searching the forums, wiki, and Googling around I found
that the most frequent causes of this are:
1 - corrupt file system (I have never seen file system
problems on this system and "shutdown -Fr" proved
that is not the problem, there really are that few
free blocks left, so it is not an incorrect free block
count in the superblock)
2 - deleted files left open (reboot would have fixed
this so that is not the problem)
3 - huge log files (I verified that the log file sizes are
the same on my good system as on my bas system
and there are no huge files to be found as detailed
below and this would not explain the discrepancy
between df results and du results)

I have tried running "lsof | grep deleted" and there are
no large files hanging out there. I suppose I could boot
it up with Knoppix again to prove that the Knoppmyth
boot up sequence isn't creating huge files and deleting
them while keeping them open, but the "lsof | grep deleted"
seems enough proof to me.

"find / -xdev -size +100M" (find all files bigger than
100 meg) finds no files.
"find / -xdev -size +10M" (bigger than 10 meg) finds 18
reasonable files that I can vouch for.

The last page of "du -xk / | sort -n" (the one page list
of the directories using the most space) on the good
system is basically the same as on the bad system.

Basically, any research with find or du adds up to
there being 2.3 gig of files while df says it is out
of space and the OS thinks it is out of space too.
fsck says that the free block count is correct, so
it really is out of disk space.

There should not be 1.9 gig of fs overhead in a
4.7 gig file system.

I noticed some articles about snapshots on ext3. I
suppose that could be it, but I haven't taken any
snapshots that I know of.

I verified that the journal file has not somehow gotten
larger than it should be with
'debugfs -R "stat <8>" /dev/sda1'

I verified that the Mythtv, etc. log file sizes are
reasonable and I can delete some logs to keep it
running, but I want to find the cause and fix it right.
It may be getting worse and worse and get to the
point that deleting logs won't keep it running.

System Details:
This system is R5F27 with Dual core Intel, 1 gig RAM,
root on sda1, 1TB LVM on sda3 and hda1, HD5500,
and GF4-MX440. The nvidia driver in this system
has been down graded to R5F1 due to known interlace
issues that started in R5F27 on Geforce MX-440's. I
have delete-files-slowly turned on and I frequently
delete multiple recordings in rapid succession.

Another system does not have this issue:
My girlfriends R5F27 with single core AMD 1400,
512 meg RAM, 1TB LVM on hda3 and hdb1, and
PVR-350 does not exhibit this problem.

Both of these systems are used heavily and are
around 95% full.

I suspect that I could get the 1.9 gig back temporarily
by copying the whole root file system off, reformatting
it, and copying it back on with something like Knoppix
cp -r --preserve=all or GPartEd/Clonezilla, but I want
to know it won't sneak up on me or anyone else whose
system I maintain. (I know I will have to reinstall LILO)

Anyone out there with a better angle on this than what
I have done so far?

If you have read this far you might want to check
"du -xhs /" (the only number it prints, mine says 2.3G)
versus
"df -h /" (the Used column, mine says 4.2G)
and reply to this post if they are out of whack like mine.
These numbers should be close to the same. On my good
system they are 2.4G and 2.5G respectively.

By the way, all the phrases in double quotes above are
the commands exactly the way I typed them (as root.
without the double quotes). There is one command that
has embedded double quotes so I used single quotes
around it. Remove the single quotes and leave the double
quotes for it.

Cliff

_________________
R5F27 using R5F1 Nvidia drivers
HD-5500 analog from NTSC Sat Rx, with OTA DVB too
GeForce MX-440 SVideo tvout to a TV
Older dual core 3.4ghz Intel CPU
Asus P5PE-VM Motherboard
2 GB RAM
1 TB LVM2
VirtualBox
Samba


Last edited by cliffsjunk on Sat Mar 01, 2008 3:16 pm, edited 2 times in total.


Top
 Profile  
 
PostPosted: Mon Feb 18, 2008 9:38 pm 
Offline
Joined: Wed Nov 16, 2005 8:55 pm
Posts: 1381
Location: Farmington, MI USA
cliffsjunk wrote:
...
"du -xhs /" shows that there is
2.3 gig of the 4.7 gig root partition used.
"df -h /" shows that there is
4.2 gig of the 4.7 gig root partition used...


I don't know what the discrepancies are between the -xhs and -h options in du vs. df, but what does df -i show you for the filesystem in question? I'm guessing you are out of inodes, not space.


Top
 Profile  
 
 Post subject:
PostPosted: Tue Feb 19, 2008 12:02 am 
Offline
Joined: Fri Sep 15, 2006 12:16 pm
Posts: 292
"df -i /" shows about 395000 out of 500000 inodes free, so that's not it.

... and recall that simply unreserving some of the reserved (for root) data
blocks got the system working when it was down.

I examined the root file system offline while it was down and indeed
there were no free data blocks over those reserved for the root user.
That is why I chose to unreserve some of the reserved data blocks.
This was all on a clean file system as proven by a forced fsck.

Cliff

_________________
R5F27 using R5F1 Nvidia drivers
HD-5500 analog from NTSC Sat Rx, with OTA DVB too
GeForce MX-440 SVideo tvout to a TV
Older dual core 3.4ghz Intel CPU
Asus P5PE-VM Motherboard
2 GB RAM
1 TB LVM2
VirtualBox
Samba


Top
 Profile  
 
PostPosted: Tue Feb 19, 2008 8:30 pm 
Offline
Joined: Thu Mar 25, 2004 11:00 am
Posts: 9551
Location: Arlington, MA
slowtolearn wrote:
cliffsjunk wrote:
...
"du -xhs /" shows that there is
2.3 gig of the 4.7 gig root partition used.
"df -h /" shows that there is
4.2 gig of the 4.7 gig root partition used...


I don't know what the discrepancies are between the -xhs and -h options in du vs. df, ...

The du command shows what the directory tree contains when you sum up all the files sizes. This can include duplicates when there are files which are hard linked, and it doesn't take into account space wasted by partially used blocks or invisible files. The df command shows how many blocks (at the minimal allocation unit) are marked as in-used versus free.

For example:
Code:
root@black2:~# df /
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/hda1             20243048   2367112  16850592  13% /
root@black2:~# echo >foo
root@black2:~# df /
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/hda1             20243048   2367116  16850588  13% /

Note that writing 1 byte to the file foo used 4k of space from the df point of view. Lots of little files means lots of waste... This can get even worse if you have your root partitioned for large files like is normally done for /myth.


Top
 Profile  
 
PostPosted: Sat Mar 01, 2008 2:20 pm 
Offline
Joined: Fri Sep 15, 2006 12:16 pm
Posts: 292
tjc wrote:
The du command shows what the directory tree contains when you sum up all the files sizes. This can include duplicates when there are files which are hard linked, and it doesn't take into account space wasted by partially used blocks or invisible files. The df command shows how many blocks (at the minimal allocation unit) are marked as in-used versus free.

For example:
Code:
root@black2:~# df /
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/hda1             20243048   2367112  16850592  13% /
root@black2:~# echo >foo
root@black2:~# df /
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/hda1             20243048   2367116  16850588  13% /

Note that writing 1 byte to the file foo used 4k of space from the df point of view. Lots of little files means lots of waste... This can get even worse if you have your root partitioned for large files like is normally done for /myth.


du does not count hard links multiplte times unless you ask it to.

du (like df) does show the amount of disk space allocated,
unless you ask it to show the file size (which I did not).

...and as I said, my girlfriend's R5F27 system produces very similar
numbers from du and df for the amount of disk actually used.

Cliff

_________________
R5F27 using R5F1 Nvidia drivers
HD-5500 analog from NTSC Sat Rx, with OTA DVB too
GeForce MX-440 SVideo tvout to a TV
Older dual core 3.4ghz Intel CPU
Asus P5PE-VM Motherboard
2 GB RAM
1 TB LVM2
VirtualBox
Samba


Top
 Profile  
 
 Post subject:
PostPosted: Sat Mar 01, 2008 3:17 pm 
Offline
Joined: Fri Sep 15, 2006 12:16 pm
Posts: 292
Solved:
Apparently my /myth partition had failed to mount one time and
it had started recording shows in the unmounted /myth directory.
Of course these files are invisible once /myth is mounted properly.

Cliff

_________________
R5F27 using R5F1 Nvidia drivers
HD-5500 analog from NTSC Sat Rx, with OTA DVB too
GeForce MX-440 SVideo tvout to a TV
Older dual core 3.4ghz Intel CPU
Asus P5PE-VM Motherboard
2 GB RAM
1 TB LVM2
VirtualBox
Samba


Top
 Profile  
 
PostPosted: Sun Mar 02, 2008 12:43 am 
Offline
Joined: Thu Mar 25, 2004 11:00 am
Posts: 9551
Location: Arlington, MA
cliffsjunk wrote:
du does not count hard links multiplte times unless you ask it to.

du (like df) does show the amount of disk space allocated,
unless you ask it to show the file size (which I did not).

Possibly true now, not necessarily true historically... Checking the man page as you apparently did is always a good idea.


Top
 Profile  
 
 Post subject:
PostPosted: Mon Nov 24, 2008 4:16 am 
Offline
Joined: Tue Mar 22, 2005 9:18 pm
Posts: 1422
Location: Brisbane, Queensland, Australia
This is exactly the problem that I have, but I don't know how to fix it.

When I do a du -xhs / I get 2.3G, when I do df -h I get 4.7G.

I did have some disk problems that I have now corrected but I can't get the system back up till I fix the space problem. Please tell me how?

_________________
Girkers


Top
 Profile  
 
 Post subject:
PostPosted: Sat Jan 17, 2009 4:25 am 
Offline
Joined: Fri Sep 15, 2006 12:16 pm
Posts: 292
Sorry. Been away for a while.

If you or anyone else reading this would like to know, I wrote up a "what to do" that covers
all the bases that I could think of. It is on knoppmythwiki at:
http://www.knoppmythwiki.org/index.php?page=DiskFullError

Cliff


Top
 Profile  
 
 Post subject:
PostPosted: Sat Jan 17, 2009 7:12 pm 
Offline
Joined: Tue Mar 22, 2005 9:18 pm
Posts: 1422
Location: Brisbane, Queensland, Australia
Thanks Cliff, I have since reinstalled and changed hardware since then. Whilst I haven't read your wiki article in detail, it looks very good, well done.

_________________
Girkers


Top
 Profile  
 
 Post subject:
PostPosted: Tue Feb 24, 2009 9:52 pm 
Offline
Joined: Tue Aug 15, 2006 11:14 am
Posts: 1343
Location: Orlando FL
thanks cliff


Top
 Profile  
 
 Post subject:
PostPosted: Fri Mar 06, 2009 4:45 pm 
Offline
Joined: Tue Aug 15, 2006 11:14 am
Posts: 1343
Location: Orlando FL
I also fixed this by using Gparted to expand the / partition so that I don't have this problem again.


Top
 Profile  
 

Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 12 posts ] 


All times are UTC - 6 hours




Who is online

Users browsing this forum: No registered users and 28 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Jump to:  
cron
Powered by phpBB® Forum Software © phpBB Group

Theme Created By ceyhansuyu