FC6 and disk larger than 2 terabytes

Place to discuss Fedora and/or Red Hat
User avatar
Basher52
guru
guru
Posts: 919
Joined: Wed Oct 22, 2003 5:57 am
Location: .SE

Post by Basher52 » Wed Apr 04, 2007 2:14 pm

hmm ok, I'll try do the delete of the partition and see what result I get from the ending cyl.

And no, the data is just copied into these disks as a test to see if the result will not damage it, as I have CRC's of all files I put in there.

and last... you should NOT say you're sorry, you're DA MAN :P I have not got this explicit help anywhere, not even on the "big" Linux forums, so beware, lol there's more to come and that's a big one(I think).

I will try this and see what'll happen.

User avatar
Void Main
Site Admin
Site Admin
Posts: 5716
Joined: Wed Jan 08, 2003 5:24 am
Location: Tuxville, USA
Contact:

Post by Void Main » Wed Apr 04, 2007 2:25 pm

When I get home I'll throw a spare drive in my machine and play around with parted and see what kind of carnage I can create.

User avatar
Basher52
guru
guru
Posts: 919
Joined: Wed Oct 22, 2003 5:57 am
Location: .SE

Post by Basher52 » Wed Apr 04, 2007 2:48 pm

And as always, you're right, lol

FDISK

Code: Select all

Partition number (1-4): 1
First cylinder (1-72915, default 1):

and after I tried the parted on it:

Code: Select all

[root@ftp ~]# parted /dev/sdb
GNU Parted 1.8.2
Using /dev/sdb
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) p                                                                
Warning: Not all of the space available to /dev/sdb appears to be used, you can fix the GPT to use all of the space (an extra
390463488 blocks) or continue with the current setting? 
Fix/Ignore? I                                                             
Model: HPT DISK_2_0 (scsi)
Disk /dev/sdb: 600GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start   End    Size   File system  Name     Flags
 1      17.4kB  400GB  400GB  ext3         primary       

(parted)
I'm gonna try the FIX and see what'll happen:

Code: Select all

[root@ftp ~]# parted /dev/sdb
GNU Parted 1.8.2
Using /dev/sdb
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) p                                                                
Warning: Not all of the space available to /dev/sdb appears to be used, you can fix the GPT to use all of the space (an extra
390463488 blocks) or continue with the current setting? 
Fix/Ignore? Fix                                                           
Model: HPT DISK_2_0 (scsi)
Disk /dev/sdb: 600GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start   End    Size   File system  Name     Flags
 1      17.4kB  400GB  400GB  ext3         primary       

(parted) p                                                                
Model: HPT DISK_2_0 (scsi)
Disk /dev/sdb: 600GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start   End    Size   File system  Name     Flags
 1      17.4kB  400GB  400GB  ext3         primary       

(parted) q                                                                
Information: Don't forget to update /etc/fstab, if necessary.             

[root@ftp ~]# parted /dev/sdb
GNU Parted 1.8.2
Using /dev/sdb
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) p                                                                
Model: HPT DISK_2_0 (scsi)
Disk /dev/sdb: 600GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start   End    Size   File system  Name     Flags
 1      17.4kB  400GB  400GB  ext3         primary       

(parted)

didnt seem to do any changes really... but I'll continue...

OH... WAIT...
I gotta hold here :P
I know that in fdisk you need to 'w' to make the changes be written to disk, but not in parted... and its looks a bit different in parted:

Code: Select all

[root@ftp ~]# parted /dev/sdb
GNU Parted 1.8.2
Using /dev/sdb
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) p                                                                
Model: HPT DISK_2_0 (scsi)
Disk /dev/sdb: 600GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start   End    Size   File system  Name     Flags
 1      17.4kB  400GB  400GB  ext3         primary       

(parted)
I can't see the value(cyl=1) as in fdisk, just the start... uh... kilobyte-something.
I do think that this is the value that always appears, but I can't be sure.
I'm waiting for your input on this before I'll do anything else

User avatar
Void Main
Site Admin
Site Admin
Posts: 5716
Joined: Wed Jan 08, 2003 5:24 am
Location: Tuxville, USA
Contact:

Post by Void Main » Wed Apr 04, 2007 5:15 pm

I don't see where you told parted to resize anything. It recognizes that your disk has a capacity of 600GB but your partition is only 400GB. Type "parted /dev/sdb" and then "resize 1" and it should prompt you for a start and an end. Unfortunately the spare disk I have at home is dead so I can't test on it. Everything else is in use and I would have to move stuff around to free up but the above should work.

http://www.gnu.org/software/parted/manu ... ing-Parted

EDIT: Heh heh, I'm an idiot. I have a 1GB thumb drive that I can test everything on, could have done that at work. :)

So I created a GPT partition on my thumb drive and then created a partition in parted that was 100MB. Formated it with "mkfs.ext3 /dev/sda1", mounted it, copied a directory and a few files to it and now I have this:

(parted) p

Code: Select all

Model: SanDisk U3 Cruzer Micro (scsi)
Disk /dev/sda: 1027MB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start   End    Size   File system  Name  Flags
 1      17.4kB  100MB  100MB  ext3                    
# fdisk -l /dev/sda

Code: Select all


WARNING: GPT (GUID Partition Table) detected on '/dev/sda'! The util fdisk doesn't support GPT. Use GNU Parted.


Disk /dev/sda: 1027 MB, 1027416576 bytes
255 heads, 63 sectors/track, 124 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1               1         125     1003336   ee  EFI GPT
Partition 1 has different physical/logical beginnings (non-Linux?):
     phys=(0, 0, 1) logical=(0, 0, 2)
Partition 1 has different physical/logical endings:
     phys=(1023, 254, 63) logical=(124, 231, 60)
# df -h /dev/sda1

Code: Select all

Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1              93M  5.6M   83M   7% /mnt
Notice my fdisk doesn't support GPT type partitions but also notice that the ending cylinder is 1 greater than the number of cylinders I actually have just like in your case but fdisk notices this in my case. I think the reason that is is because what fdisk is telling us is the GPT partition isn't the actual partition that contains our file system. If it was then the ending cylinder would be much lower than what mine shows. This GPT type partition table seems to be a much different animal. It's almost like a partition table within a partition table with GPT but I need to do some more research on this.

No problem though, parted sees the partition properly as only 100MB and we're going to use parted for partitioning from here on out anyway. Now pretend my thumb drive was only 100MB, which is why I created the partition that size. Now pretend I have physically increased the size of it (added disks to my RAID set) to 1GB and now I need to resize the partition. parted doesn't seem to want to do the resize with ext3 for some reason (I found many posts on the net backing this up) but that's ok, we'll just delete the partition and create a new one like I would have done if I were using fdisk:

# parted /dev/sda

Code: Select all

(parted) p                                                                
Model: SanDisk U3 Cruzer Micro (scsi)
Disk /dev/sda: 1027MB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start   End    Size   File system  Name  Flags
 1      17.4kB  100MB  100MB  ext3                    

(parted) rm 1
(parted) mkpart
Partition name?  []?                                                      
File system type?  [ext2]? ext3                                           
Start? 0                                                                  
End? 1027MB
And now you can see the new partition size with the beginning in the same spot but the end is at 1027MB instead of 100MB:

Code: Select all

(parted) p                                                                
Model: SanDisk U3 Cruzer Micro (scsi)
Disk /dev/sda: 1027MB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start   End     Size    File system  Name  Flags
 1      17.4kB  1027MB  1027MB  ext3                    

I could still mount my drive and the file system was in tact:

# mount /dev/sda1 /mnt
# df -h /dev/sda1

Code: Select all

Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1              93M  5.6M   83M   7% /mnt
It's still only 100MB file system but now that the partition is the full size of the drive we can run resize2fs (I am doing ths without unmounting the drive):

# resize2fs /dev/sda1

Code: Select all

resize2fs 1.39 (29-May-2006)
Filesystem at /dev/sda1 is mounted on /media/disk; on-line resizing required
Performing an on-line resize of /dev/sda1 to 1003300 (1k) blocks.

The filesystem on /dev/sda1 is now 1003300 blocks long.
Now I have the full 1GB:

# df -h /dev/sda1

Code: Select all

Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1             949M  6.6M  894M   1% /mnt
My files and directories are still in tact. What's cool is the resize2fs took a minute to run and while it was running I kept doing a "df -h /dev/sda1" in another window and I could see the "Size" column increase each time until it reached the full size of the drive at which time resize2fs finished.

So, in your case you would:
# parted /dev/sdb

Code: Select all

(parted) rm 1
(parted) mkpart
Partition name?  []?                                                     
File system type?  [ext2]? ext3                                           
Start? 0                                                                 
End? 600GB 
(parted) p
(parted) q
#
Then resize the file system to the full 600GB:

Code: Select all

# resize2fs /dev/sdb1
That should give you what you need. You could also break it up into more smaller partitions if you want but you'll have to use parted and not fdisk because of the GPT partition and the size limit.

User avatar
Basher52
guru
guru
Posts: 919
Joined: Wed Oct 22, 2003 5:57 am
Location: .SE

Post by Basher52 » Wed Apr 04, 2007 10:13 pm

I'll try this :)

It sure does EXACTLY what you said, I see the increase in space but it take way more than one minute but that gotta be the bigger size.
btw... you do NOT have to test things just for me, if you dont know the answer, just say that, I appreciate the answer anyways :D

after resize:

Code: Select all

/dev/sdb1             550G  1.6G  549G   1%
I know(think) that it show the size in 1.0 k-units and not 1.024 but ONLY 550G?
how do I know that this size is the FULL size I can get? since I vrote 600G in parted, is there no... write-nothing-and-you'll-get -maximum-size-thing here?
550 in 1.0k units is ONLY 562G in 1.024 units or am I totally wrong here?


anyways, the size has grown into the next 200G disk :)
and hopefully the ENTIRE disk too :P

Do I lose extra space when doing a resize?
cos I KNOW!! I got a bigger one than 550G when I initialized the whole 600G in one job.
Last edited by Basher52 on Thu Apr 05, 2007 10:46 am, edited 1 time in total.

User avatar
Void Main
Site Admin
Site Admin
Posts: 5716
Joined: Wed Jan 08, 2003 5:24 am
Location: Tuxville, USA
Contact:

Post by Void Main » Wed Apr 04, 2007 10:47 pm

http://www.linux.com/howtos/Large-Disk-HOWTO-14.shtml

Code: Select all

14.4 Nonproblem: fdisk sees much more room than df?

fdisk will tell you how many blocks there are on the disk. If you make a filesystem on the disk, say with mke2fs, then this filesystem needs some space for bookkeeping - typically something like 4% of the filesystem size, more if you ask for a lot of inodes during mke2fs. For example:

    # sfdisk -s /dev/hda9
    4095976
    # mke2fs -i 1024 /dev/hda9
    mke2fs 1.12, 9-Jul-98 for EXT2 FS 0.5b, 95/08/09
    ...
    204798 blocks (5.00%) reserved for the super user
    ...
    # mount /dev/hda9 /somewhere
    # df /somewhere
    Filesystem         1024-blocks  Used Available Capacity Mounted on
    /dev/hda9            3574475      13  3369664      0%   /mnt
    # df -i /somewhere
    Filesystem           Inodes   IUsed   IFree  %IUsed Mounted on
    /dev/hda9            4096000      11 4095989     0%  /mnt
    #

We have a partition with 4095976 blocks, make an ext2 filesystem on it, mount it somewhere and find that it only has 3574475 blocks - 521501 blocks (12%) was lost to inodes and other bookkeeping. Note that the difference between the total 3574475 and the 3369664 available to the user are the 13 blocks in use plus the 204798 blocks reserved for root. This latter number can be changed by tune2fs. This `-i 1024' is only reasonable for news spools and the like, with lots and lots of small files. The default would be:

    # mke2fs /dev/hda9
    # mount /dev/hda9 /somewhere
    # df /somewhere
    Filesystem         1024-blocks  Used Available Capacity Mounted on
    /dev/hda9            3958475      13  3753664      0%   /mnt
    # df -i /somewhere
    Filesystem           Inodes   IUsed   IFree  %IUsed Mounted on
    /dev/hda9            1024000      11 1023989     0%  /mnt
    #

Now only 137501 blocks (3.3%) are used for inodes, so that we have 384 MB more than before. (Apparently, each inode takes 128 bytes.) On the other hand, this filesystem can have at most 1024000 files (more than enough), against 4096000 (too much) earlier.
For a partition as large as yours if you plan on having a smaller number of large files than a normal system has you might consider tuning your file system so your inode table is smaller and the amount of reserved space is smaller. It just doesn't make sense taking up 50GB of the drive for this. To see a lot more details on your file system run this:

Code: Select all

# tune2fs -l /dev/sdb1
I don't believe df shows the number of reserved blocks in it's total size count since normal users can't write data when the system only has that much space left. It also doesn't show the amount of space the inode table takes up. You should be able to use tune2fs to reduce that reserved space to a smaller amount which should help a lot. I am pretty sure if you want to decrease the size of your inode table you would have to reformat your file system and use the proper inode parameters. You should be able to reduce the amount of reserved space on the fly with tune2fs as I said though.

See:
$ man tune2fs

Do a "df -i" to see how many inodes you have total and how many are free. Every file and directory requires 1 inode. If it says you have 200 million inodes available that's probably a crazy amount if you don't plan on filling the drive up with tiny files. If you create the file system with a much smaller inode table you'll have more free space for data.

Also, no biggy regarding me doing this along with you. I learn, or at least get a refresher in the process.

User avatar
Basher52
guru
guru
Posts: 919
Joined: Wed Oct 22, 2003 5:57 am
Location: .SE

Post by Basher52 » Thu Apr 05, 2007 10:45 am

Code: Select all

[root@ftp ~]# tune2fs -l /dev/sdb1 
tune2fs 1.39 (29-May-2006)
Filesystem volume name:   <none>
Last mounted on:          <not available>
Filesystem UUID:          7799ea73-ae17-4a09-9abb-953869be7e26
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal resize_inode dir_index filetype needs_recovery sparse_super large_file
Default mount options:    (none)
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              73220096
Block count:              146423799
Reserved block count:     0
Free blocks:              143718725
Free inodes:              73219977
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      989
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         16384
Inode blocks per group:   512
Filesystem created:       Tue Apr  3 13:59:32 2007
Last mount time:          Thu Apr  5 06:14:29 2007
Last write time:          Thu Apr  5 06:14:29 2007
Mount count:              3
Maximum mount count:      26
Last checked:             Tue Apr  3 13:59:32 2007
Check interval:           15552000 (6 months)
Next check after:         Sun Sep 30 13:59:32 2007
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               128
Journal inode:            8
Default directory hash:   tea
Directory Hash Seed:      5ed7e8b1-d9e9-4c8b-8ff3-0ea2f753d0dd
Journal backup:           inode blocks

Code: Select all

[root@ftp ~]# df -i
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/sdb1            73220096     119 73219977    1% /root/test

User avatar
Void Main
Site Admin
Site Admin
Posts: 5716
Joined: Wed Jan 08, 2003 5:24 am
Location: Tuxville, USA
Contact:

Post by Void Main » Thu Apr 05, 2007 11:39 am

So you have no reserved space. I actually believe you should have enough reserved space so if the file system gets full critical file system operations (writing to the journal, etc) can still take place. Without researching it I am not sure how you would come up with the exact minimum amount. tune2fs shows you have 73 million inodes which would take close to 9GB at 128 bytes per inode. If you don't think you'll ever get close to 73 million files and directories on that file system then you might consider recreating the file system with a smaller inode table. If you dropped it to 1 mllion you could get back over 8GB of space.

Now your tune2fs reports the total size of your file system is 146,423,799 blocks * 4,096 bytes = 558.6 gigabytes. That is quite a bit less than parted's reported 600GB. I just did a tune2fs on one of my laptop file sytems and when I add it up I come up with 19.6GB which is exactly the amount I come up with when I check the number of blocks and multiply it out. parted says it's 20.1GB so I don't know where parted comes up with it's numbers but it's about 1% off on my system. Yours seems to be a lot farther off than that. Time to do some more digging. :)

EDIT: I don't know how parted figures Gigabytes but it is apparently incorrect. If I change the "units" to "Bytes" and look at my partition table I get this:

Code: Select all

(parted) unit B
 5      16860888576B  37917573119B  21056684544B  logical   ext3              
So you can see it tells me the total size is 21056684544 bytes but in Gigabytes it tells me I have 21.1GB.

Code: Select all

 5      16.9GB  37.9GB  21.1GB  logical   ext3    
Using Google to do the conversion from bytes to Gigabytes I get 19.6105657GB which is exactly what I get when I multiply the total number of blocks times the blocks size reported by tune2fs. So parted uses some funky way to calculate GB. I have no idea what that is but I would say it's wrong.
Last edited by Void Main on Thu Apr 05, 2007 12:08 pm, edited 1 time in total.

User avatar
Basher52
guru
guru
Posts: 919
Joined: Wed Oct 22, 2003 5:57 am
Location: .SE

Post by Basher52 » Thu Apr 05, 2007 11:58 am

when you say "recreating the file system with a smaller inode table"
is this the same thing as blocksize in winblows?
The bigger size I set the less space I lose onbigger files, but each file takes more space on disk. eg. 1 small text file that has 1 character in it could take 64k bytes on disk if you go for hte maximum amount on the blocksize.

The only switch I use when I create the filesystem is: mke2fs -m0 -j
and the 'm0' a buddy told me to use to get the extra space available and I've been using that since.
The only files that are gonna be hosted on these disks are big.
I would believe that files are goint to be in the range of 100Mbytes and up, but there could be some small files too, but that would be very rare.
The files don't have to be accessed very often and the access time has not that big impact.

So know my question is, what switches of the mke2fs should I use to get the best possible filesystem.


UPDATE: I just took a peek at the 'man' for mke2fs and found that there is a blocksize option there too.

User avatar
Void Main
Site Admin
Site Admin
Posts: 5716
Joined: Wed Jan 08, 2003 5:24 am
Location: Tuxville, USA
Contact:

Post by Void Main » Thu Apr 05, 2007 12:13 pm

No, the inode table is not the same as blocksize. Every file and every directory requires exactly 1 inode. The inode table would be somewhat akin to the file allocation table in a FAT file system. NTFS has something similar to inodes. I wouldn't mess with block size unless you plan on running millions of tiny files. Also, reread my previous message as I added something about parted and that the 600GB number it reports is probably wrong. Believe what the number of blocks * blocksize number is as reported by tune2fs. Also, the defaults used by mkfs.ext3 will create the best "average" file system. If you want to tune it for non-average use you have that capability.

User avatar
Basher52
guru
guru
Posts: 919
Joined: Wed Oct 22, 2003 5:57 am
Location: .SE

Post by Basher52 » Thu Apr 05, 2007 12:40 pm

I saw this when looking more into the 'man' of mkfs.ext3(mke2fs)

Code: Select all

-T fs-type
    Specify how the filesystem is going to be used, so that mke2fs can chose optimal filesystem parameters for that use. The supported filesystem types are:

        news
            one inode per 4kb block 
        largefile
            one inode per megabyte 
        largefile4
            one inode per 4 megabytes 
I could use this instead of the value of the inode you said, right?
but I couldn't find anything about what happens if the inode value are out, but I saw that it can't be changed after the filesystem is created.

Thinking of using: -T largefile maybe even largefile4
the 'one inode per megabyte' is that 'megabyte' almost the same as the size of files?[/b]



UPDATE: I just recreated the filesystem using mkfs.ext3 with no parameters and I got this:

Code: Select all

Filesystem            Size  Used Avail Use% Mounted on
/dev/sdb1             550G  198M  522G   1% /root/test
before I had this:

Code: Select all

Filesystem            Size  Used Avail Use% Mounted on
/dev/sdb1             550G  1.6G  549G   1% /root/test
This smaller size is due to the reserved space, right? which I removed through the -m0 option.
That is 27G that I "lose" and that's ALOT! of space for 'critical file system operations' :P

User avatar
Void Main
Site Admin
Site Admin
Posts: 5716
Joined: Wed Jan 08, 2003 5:24 am
Location: Tuxville, USA
Contact:

Post by Void Main » Thu Apr 05, 2007 1:12 pm

Basically what you did is say that your files will average about 4MB so mkfs created enough inodes for the number of 4MB files it would take to fill your drive. As I said, every file and directory uses exactly 1 inode regardless of the size. And no, the -m0 is not where you gained the space. You gained the space from creating fewer inodes (the inode table was smaller). As I said in my previous message, on an ext3 file system I personally would not use the "-m0" because if you ever do fill the file system up you might destroy your journal. I need to look into that a little more to be sure but I think that is one of the reasons for the rerserved space. In fact ext2 might even be a better choice for you. ext2 doesn't even use a journal. I am not sure about that and need to do a little more digging to find out pros/cons.

EDIT: Actually I don't think there is any need to reduce the amount of reserved blocks. It appears this probably has nothing to do with the journal getting corrupted or not getting corrupted but more for performance. I don't think reducing the amount of reserved blocks in your file system will actually change anything regarding the amount of space that is used whether physical or reported. Read more about it here:

http://www.redhat.com/magazine/011sep05 ... ps_tricks/

Reducing the number of inodes like you did will certainly reclaim some space as you saw. The only con you have by reducing the number of inodes is you can't put as many files on the file system (your current number of inodes is still probably way more than you'll ever use).

Also I would stick with ext3 because of the recovery advantages on a crashed system. I don't think there are really any big advantages to going back to ext2.

Tux
guru
guru
Posts: 689
Joined: Wed Jan 08, 2003 10:40 am

Post by Tux » Thu Apr 05, 2007 1:24 pm

Void and I briefly discussed ext3 reserved space over on this thread:
http://voidmain.is-a-geek.net/forums/vi ... php?t=1491

I find it pretty ridiculous that ext3fs uses(loses) so much space, but i've played it safe due to the links from the thread I linked.
Also in a filesystem review I read recently it says basically that ext3 wastes huge amount of space compared to the other major FS out there. Please read for yourself

User avatar
Void Main
Site Admin
Site Admin
Posts: 5716
Joined: Wed Jan 08, 2003 5:24 am
Location: Tuxville, USA
Contact:

Post by Void Main » Thu Apr 05, 2007 1:53 pm

Thanks for that Tux. It looks like that article concludes that XFS leads most categories and is probably the best all around file system. It's interesting that we already mentioned that as being a possible solution to the >2TB issue because it can be set up on a whole disk without using a partition table. The *only* reason I don't personally use that is because it's not including in the system I currently use as I mentioned. ext2/ext3 are the evils I am most familiar with. I wonder why Fedora doesn't include xfs support?

EDIT: I take that back. It appears you can set up Fedora with xfs, jfs, or reiserfs when installing just by passing a parameter to the installer:

http://www.fedorafaq.org/#reiserjfs

I might have to use that on the next install.

User avatar
Basher52
guru
guru
Posts: 919
Joined: Wed Oct 22, 2003 5:57 am
Location: .SE

Post by Basher52 » Thu Apr 05, 2007 3:04 pm

Then I think I'll setup the smaller RAID with XFS and if anything happens I'll come crawling for help :P

Post Reply