View unanswered posts    View active topics

All times are UTC - 6 hours





Post new topic Reply to topic  [ 41 posts ] 
Go to page 1, 2, 3  Next

Print view Previous topic   Next topic  
Author Message
Search for:
PostPosted: Sun Jul 20, 2008 4:27 am 
Offline
Joined: Wed Dec 10, 2003 8:31 pm
Posts: 1996
Location: /dev/null
First off, here are two related threads for those of us new to XFS:

How and why - Create an XFS Partition
XFS Maintenance - Read if You're New to XFS (teaches you how to use xfs_repair which is like fsck for XFS)

In short, XFS is much more efficient dealing with large files such as captured videos. I recently converted my /myth over to XFS and wanted to learn more about it through casual googling. I came across some defrag utilities that I wanted to share with the community. The below is from my R5.5 box.

1. Check the Fragmentation of Your XFS Partition

Use the supplied xfs_db util as follows:

Code:
$ xfs_db -c frag -r /dev/hda3


Enter the command to measure the fragmentation on that partition which is simple:

Code:
actual 5342, ideal 2568, fragmentation factor 51.93%


The above is my /myth after I used rsync to transfer my old drive's /myth to the new drive and then installed R5.5. From what I've read, 50 % is pretty heavily fragmented!

2. Defrag Your XFS Partition

Use xfs_fsr to defrag your partition. When started, it reorganizes all regular files in all mounted XFS filesystems. It works differently than Micro$oft Windows based defrag utils: xfs_fsr runs in many cycles each time making a single pass over each XFS filesystem (one in my case, just /myth). It's smart and will select the files that are most fragmented, attempting to defrag the top 10 % of them on each pass.

You can specify how long you want it to run (default is 2 hours) and it will remember where it left off if more defragmenting is required. This information is stored in /var/tmp/.fsrlast_xfs.

Unlike some other disk utils, you actually leave the filesystem mounted when you use this one! Finally, it isn't required, but it is recommended that you enter single user mode before running xfs_fsr so programs/users don't interfere with the optimization process by read/writing to the partition. You can do this via the telinit command:

Code:
# telinit 1


Issue the following command to begin and know that unless you specify otherwise, default time it'll run is 7200 sec (2 h) as mentioned above:
Code:
# xfs_fsr -v


If 2 h seems excessive (and it very well could be depending on your level of fragmentation, size of drive partition, etc.), you can specify the time in seconds for it to run. For example, try it for 30 minutes:

Code:
# xfs_fsr -v -t 1800


The -v switch isn't needed really, it just prints out a lot of cryptic info about what it's doing, example:
Code:
Found 1 mounted, writable, XFS filesystems
xfs_fsr -m /etc/mtab -t 300 -f /var/tmp/.fsrlast_xfs ...
START: pass=0 ino=1989755660 /dev/hda3 /myth
/myth start inode=1989755660
/myth start inode=0
ino=62438287
extents before:23 after:1 DONE ino=62438287
ino=62438285
extents before:14 after:1 DONE ino=62438285
ino=62438288
extents before:11 after:1 DONE ino=62438288
ino=134210286
extents before:18 after:1 DONE ino=134210286
ino=696484542
extents before:5 after:1 DONE ino=696484542
ino=696484537
extents before:4 after:1 DONE ino=696484537
ino=1074384332
extents before:2 after:1 DONE ino=1074384332
ino=1202781670


I like it because it lets me know it's still running. Omit it if you wish.

Anyway, watch your HD light pretty much stay solid as fragmented files are moved to unfragmented files in that time period as the app does its thing.

I let mine run for the default of 2 h and I went to the store. You can see the CPU usage for my little Athlon 1900+ increase over that 2 h time period in the rrd_CPU data:
Image

2 h later the defragmentation is complete. If I query the FS again, that huge 52 % fragments dropped to <1 %:

Code:
$ xfs_db -c frag -r /dev/hda3
actual 2552, ideal 2546, fragmentation factor 0.24%


Remember to switch back to multiuser mode if you dropped down to single user mode:

Code:
# telinit 5


3. Avoiding Future Fragmentation

See the optimizing performance link below wherein it is suggested that you add allocsize to your /etc/fstab to help minimize future fragmentation.

From that wikipage:
Quote:
The xfs filesystem has a mount option which can help combat this fragmentation: allocsize.

This can be added to /etc/fstab, for example:
Code:
/dev/hdd1  /video     xfs     defaults,allocsize=512m 0 0


This essentially causes xfs to speculatively preallocate 512m of space at a time for a file when writing, and can greatly reduce fragmentation. For example on my box with HD streams that typically take about 3Gb for an hour of video, I used to get thousands of extents. This is largely due to the fsync loop in the file writer, which un-does any benefit that xfs's delayed allocation would otherwise provide. With the allocsize mount option as above, now I get at most 30 or so extents, because the periodic fsync now flushes to pre-allocated blocks.

For files which are already heavily fragmented, the xfs_fsr command (from the xfsdump package) can be used to defragment individual files, or an entire filesystem.


Here is my R5.5 auto install etc/fstab that I modded for XFS using the allocsize suggested above
Code:
# /etc/fstab: filesystem table.
#
# filesystem  mountpoint  type  options  dump  pass
/dev/hda1  /  ext3 defaults,errors=remount-ro  0  1

#original line for /myth
#/dev/hda3  /myth  xfs  defaults,auto  0  2

/dev/hda3  /myth  xfs  defaults,allocsize=512m  0 0

proc  /proc  proc  defaults  0  0
/dev/fd0  /floppy  vfat  defaults,user,noauto,showexec,umask=022  0  0
usbfs  /proc/bus/usb  usbfs  devmode=0666  0  0
sysfs  /sys  sysfs  defaults  0  0
tmpfs  /dev/shm  tmpfs defaults  0  0
/dev/cdrom /cdrom  auto  defaults,ro,user,noexec,noauto  0  0
# Added by KNOPPIX
/dev/hda2 none swap defaults 0 0


References:
debian-administration.org - Filesystems (ext3, reiser, xfs, jfs) comparison on Debian Etch
linux.com - Use xfs_fsr to keep your XFS filesystem optimal
mythtv.org/wiki - Optimizing Performance page on mythtvwiki
tuxfiles.org - How to edit and understand /etc/fstab

_________________
Retired KM user (R4 - R6.04); friend to LH users.


Last edited by graysky on Thu Aug 14, 2008 4:03 pm, edited 9 times in total.


Top
 Profile  
 
 Post subject:
PostPosted: Sun Jul 20, 2008 3:04 pm 
Offline
Joined: Sun Sep 25, 2005 3:50 pm
Posts: 1013
Location: Los Angeles
Re: Point #2

If your rrd database is written to your /myth partition, you may want to stop that as well. Also to be clear, xfs_fsr is supposed to be run on a mounted filesystem. Nonetheless, it is a good idea to stop any process which may be using it.

Good consolidation of info, graysky. :)

_________________
Mike
My Hardware Profile


Top
 Profile  
 
PostPosted: Mon Jul 21, 2008 2:29 am 
Offline
Joined: Sun Aug 28, 2005 7:07 pm
Posts: 821
Location: Melbourne, Australia
Great work Graysky. I admit I had no idea xfs had a problem with fragmentation. A 512MB block allocation sounds like just what most of /myth needs.

Now how do we script this? ;-)

Mike

_________________
*********************
LinHES 7.4
Australian Dragon
*********************


Top
 Profile  
 
 Post subject:
PostPosted: Mon Jul 21, 2008 6:58 am 
Offline
Joined: Thu Mar 02, 2006 5:42 pm
Posts: 410
Location: middleton wi usa atsc
mihanson wrote:
it is a good idea to stop any process which may be using it.
Is making sure there are no recordings scheduled sufficient? Or does one have to do something more drastic to fullfill this requirement?
TIA


Top
 Profile  
 
 Post subject:
PostPosted: Mon Jul 21, 2008 6:59 am 
Offline
Joined: Thu Mar 25, 2004 11:00 am
Posts: 9551
Location: Arlington, MA
Dropping to single user mode before doing this kind of maintenance is strongly recommended.


Top
 Profile  
 
 Post subject:
PostPosted: Mon Jul 21, 2008 7:07 am 
Offline
Joined: Thu Mar 02, 2006 5:42 pm
Posts: 410
Location: middleton wi usa atsc
tjc wrote:
Dropping to single user mode before doing this kind of maintenance is strongly recommended.
Ok. Just saw this thread which seems to cover that aspect of things. Thanks.
http://knoppmyth.net/phpBB2/viewtopic.p ... highlight=


Top
 Profile  
 
PostPosted: Mon Jul 21, 2008 9:13 am 
Offline
Joined: Tue Sep 12, 2006 6:03 am
Posts: 210
Location: Roseville, MI
manicmike wrote:
Now how do we script this?


My thoughts exactly. Set it to run once a month or so (maybe twice a month) in the wee hours of the morning. Use the idle.sh to make sure nothing is running first.

_________________
-Roseville, Michigan USA
LinHES R8: FE/BE, FE (x2)


Top
 Profile  
 
 Post subject:
PostPosted: Mon Jul 21, 2008 1:47 pm 
Offline
Joined: Wed Dec 10, 2003 8:31 pm
Posts: 1996
Location: /dev/null
tjc wrote:
Dropping to single user mode before doing this kind of maintenance is strongly recommended.


True... I edited the first post of this thread accordingly.

_________________
Retired KM user (R4 - R6.04); friend to LH users.


Top
 Profile  
 
 Post subject:
PostPosted: Tue Aug 19, 2008 9:14 am 
Offline
Joined: Thu Jan 13, 2005 9:55 am
Posts: 135
Location: Decatur, Ga., USA
Just for a point of reference regarding this thread, I could not get my mythbackend to drop to single user mode...it would hang.

I never figured out why, and even then, I couldn't umount my SATA drive that holds /myth. Kept getting a busy message.

Didn't figure that one out, either.

My /myth partition was over 97% fragged...! Egads! It must be all the HD TV shows I've been recording.

Didn't matter in the end, because I booted with the latest Knoppix DVD and ran the xfs utils from there. Worked like a charm.

Thanks, graysky, for the excellent writeup!!

-sTv

_________________
R7.3 Backend
CPU: 1.8 GHz
RAM: 2Gb
Drives: 500Gb IDE; IDE DVD-RW
Mobo: eMachines
Tuners (Analog): PVR-250
Tuners (Digital): DCT-6200 Motorola Cable Box, HDHomeRun (OTA + cable)


Top
 Profile  
 
 Post subject:
PostPosted: Tue Aug 19, 2008 1:21 pm 
Offline
Joined: Wed Dec 10, 2003 8:31 pm
Posts: 1996
Location: /dev/null
steeve wrote:
Just for a point of reference regarding this thread, I could not get my mythbackend to drop to single user mode...it would hang.

I never figured out why, and even then, I couldn't umount my SATA drive that holds /myth. Kept getting a busy message.

Didn't figure that one out, either.


That's bizarre. Do you have other machines on your LAN that have your /myth mounted (i.e. Windows machines, other KM machines, etc.)?

As to the inability to telinit 1 on the box.... did you do it from the local box or via ssh? If you did it though the local box, did you open a new virtual console via ctrl+alt+F1 for example, or did you do it from X? I dunno if it matters or not as I've never tried from X or from the local box, but I'm wondering if you attempted it from your X server if it would allow it or not?

steeve wrote:
My /myth partition was over 97% fragged...! Egads! It must be all the HD TV shows I've been recording.


Did you add the allocsize=512m to your /etc/fstab? It made a difference for my machine capturing SDTV. Also, what percent of your file system is free space ($ df -h will tell). I've read that if it's nearly full, writing fragmented files is much more likely.

_________________
Retired KM user (R4 - R6.04); friend to LH users.


Top
 Profile  
 
 Post subject:
PostPosted: Tue Aug 19, 2008 9:16 pm 
Offline
Joined: Thu Jan 13, 2005 9:55 am
Posts: 135
Location: Decatur, Ga., USA
graysky wrote:
steeve wrote:
Just for a point of reference regarding this thread, I could not get my mythbackend to drop to single user mode...it would hang.

I never figured out why, and even then, I couldn't umount my SATA drive that holds /myth. Kept getting a busy message.

Didn't figure that one out, either.


That's bizarre. Do you have other machines on your LAN that have your /myth mounted (i.e. Windows machines, other KM machines, etc.)?


Good questions. Yes, other machines mount /myth, but I figured dropping to single-user mode would kill off any daemons/servers.

graysky wrote:
As to the inability to telinit 1 on the box.... did you do it from the local box or via ssh? If you did it though the local box, did you open a new virtual console via ctrl+alt+F1 for example, or did you do it from X? I dunno if it matters or not as I've never tried from X or from the local box, but I'm wondering if you attempted it from your X server if it would allow it or not?


I did it from the local box in a new virtual console. It would just hang after it stopped some processes and daemons, rolling text infinitely up the screen until I manually forced it to stop.

graysky wrote:
steeve wrote:
My /myth partition was over 97% fragged...! Egads! It must be all the HD TV shows I've been recording.


Did you add the allocsize=512m to your /etc/fstab? It made a difference for my machine capturing SDTV. Also, what percent of your file system is free space ($ df -h will tell). I've read that if it's nearly full, writing fragmented files is much more likely.


I'm about to edit my fstab at this moment. The file system is 68% full. I'll be doing an upgrade to R5.5 very soon, making some major changes to the hardware in the backend in terms of storage space. Since we're moving to HD for almost everything I record, it just makes sense to use XFS and a great big drive for /myth.

Thanks again!

-sTv

_________________
R7.3 Backend
CPU: 1.8 GHz
RAM: 2Gb
Drives: 500Gb IDE; IDE DVD-RW
Mobo: eMachines
Tuners (Analog): PVR-250
Tuners (Digital): DCT-6200 Motorola Cable Box, HDHomeRun (OTA + cable)


Top
 Profile  
 
 Post subject:
PostPosted: Wed Aug 20, 2008 9:27 am 
Offline
Joined: Tue Sep 13, 2005 10:48 am
Posts: 852
Location: London, UK
Great HOW-TO Graysky,

I couldn't believe how much fragmentation there was on my hard drives.

Now all I need to do is add the allocsize=512m to my fstab.

Thanks.

_________________
Version:R8
Intel C2D 7400, Nvidia 5600 via HDMI to Samsung B37B650TW (PAL), Asus P5QL-E mobo, 4Gb PC6400 DDR2 ram, Samsung Spinpoint 500 Gb & 1Tb drive, Nova-HD-S2 (x2)


Top
 Profile  
 
 Post subject:
PostPosted: Tue Sep 23, 2008 1:50 am 
Offline
Joined: Wed Dec 10, 2003 8:31 pm
Posts: 1996
Location: /dev/null
@tophee - glad to hear it! How has your frag load been using using the 512 switch?

_________________
Retired KM user (R4 - R6.04); friend to LH users.


Top
 Profile  
 
 Post subject:
PostPosted: Mon Sep 29, 2008 8:14 am 
Offline
Joined: Tue Jan 18, 2005 12:11 pm
Posts: 100
Location: MA, USA
mihanson wrote:
Re: Point #2

If your rrd database is written to your /myth partition, you may want to stop that as well.


So to keep it running, I assume moving /myth/rrd to the / partition would be sufficient, putting a symbolic link in... As in something like:

mv /myth/rrd /var/rrd
ln -s /var/rrd /myth/rrd


Top
 Profile  
 
 Post subject:
PostPosted: Mon Sep 29, 2008 10:03 am 
Offline
Joined: Tue Sep 13, 2005 10:48 am
Posts: 852
Location: London, UK
@ Graysky,
In a completely unscientific way, it seems good. My impression is that it has retained the quickness of a fresh install. When I reran xfs_db -c frag -r /dev/sda3 and then xfs_db -c frag -r /dev/sdb1 the other week, things seemed in pretty good shape.

No helpful information at all then from me :lol:

_________________
Version:R8
Intel C2D 7400, Nvidia 5600 via HDMI to Samsung B37B650TW (PAL), Asus P5QL-E mobo, 4Gb PC6400 DDR2 ram, Samsung Spinpoint 500 Gb & 1Tb drive, Nova-HD-S2 (x2)


Top
 Profile  
 

Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 41 posts ] 
Go to page 1, 2, 3  Next



All times are UTC - 6 hours




Who is online

Users browsing this forum: Google [Bot] and 16 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Jump to:  
cron
Powered by phpBB® Forum Software © phpBB Group

Theme Created By ceyhansuyu