FC6 and disk larger than 2 terabytes

[Basher52]

I have sent a question to their support and I'll get back when they've answered.
With the smaller array set the one with 5 200Gb drives, I fisrt tested to create a RAID 5 array of three drives then after that was done I added another 200Gb disk into that array.
It took a while for it to initialize(and this can be done in production as the manual says) and after it was done the same device (/dev/sdb) became 600Gb instead of 400Gb, but I didnt have any data on it so I can't tell if the problem you said could occur would have happened, the problem with the drive geomery.

[Void Main]

It took a while for it to initialize(and this can be done in production as the manual says) and after it was done the same device (/dev/sdb) became 600Gb instead of 400Gb, but I didnt have any data on it so I can't tell if the problem you said could occur would have happened, the problem with the drive geomery.

When you say the 400GB became 600GB do you mean fdisk recognized the increase in size? Did you have any partitions on the drive? It would be intesting to create a partition and a file system on it when it is 400GB, add the drives and increase the size, modify the partition with fdisk to change the ending cylinder (I assume the drive will just show up with more cylinders after adding the other disks). Then resize2fs the file system to expand it and if everything works then we're golden.

It will be interesting to hear what the manufacturer suggests. You probably would have been better off getting more smaller disks so you could create 2 1.5TB RAID sets and lose less space to the two parity drives.

[Void Main]

Just found another alternative workaround that might be your best solution. If you go with XFS you don't have to partition the drive, xfs can apparently be set up to run directly on the drive without partitioning which means you don't run into the 2TB fdisk limit:

http://www.hardforum.com/showthread.php?t=1074277

If you use XFS and have not already put data on the drives, follow the steps to setup xfs but SKIP the partitioning. XFS can run directly on the drive without a partition.

so instead of accessing /dev/sdb1, you would use /dev/sdb

This means that when you expand, all you have to do is xfs_growfs. No mucking with partitions. Besides, fdisk caps out at 2TB.

I didn't find out about XFS being able to run directly on the drive till it was too late, now it would be a royal pain to redo my stuff, so I settled with the 2TB limit(and lost 40gigs off my array). When I build my second array, I'll do things properly on that array, move all my data over, and setup my current array the right way.

It looks like I'll be building that second array in a few months, at the rate I'm going.

Here are the steps if you do use a partition:
Delete partition
n for new partition
p for primary
1 for number one partition
enter twice to make is use all space
w to write changes
reboot your system
xfs_growfs /your/mount/point

Here is the process if you put XFS directly on the drive:
xfs_growfs /your/mount/point

[Basher52]

When you say the 400GB became 600GB do you mean fdisk recognized the increase in size?

yes that's correct but no sorry I didn't have any data on it, but since I'm just testing things now, I'll nuke the setup of the 2nd array set and do a test.
first create a 400G drive, partition it, format it and make a small file on it,
when add another 200G to see whats happening.
As the manual says, I can do this "on the fly" so I wonder if it's supposed to grow automatically, but I can't believe that from what you've describing, using resize2fs, but we'll see

Initializing a "normal" array takes about 6-10 hours and adding another disk takes even longer, just to tell you the time spam here

[Void Main]

As the manual says, I can do this "on the fly" so I wonder if it's supposed to grow automatically, but I can't believe that from what you've describing, using resize2fs, but we'll see :)

No, the "disk" will grow automatically but not the "partition" or the "file system". You would have to modify the partition by changing the ending cylinder for the partition to the new ending cylinder number of the drive and then resize the file system with a "resize2fs /dev/sdaX". But you will have the same 2TB problem doing this with the big drives so I would suggest testing "xfs" on a partitionless disk like I said in my previous post. That would solve your problem. I would create one smaller array with your smaller disks for your OS formatted with ext3 and a 2.5TB RAID set of your large drives with an xfs filesystem on the raw disk for a data file system. You can add more drives to it and use the xfs_growfs command to increase the file system size after running your RAID utility to increase the size of the RAID set and not mess with fdisk at all.

EDIT: Here is a post claiming you should be able to partition your disk using parted:

http://www.linuxquestions.org/questions ... p?t=446046

Is anyone successfully using a single > 2TB fs?

edit: finally found the info...

you have to use parted & create a GPT partition....

> parted
(parted) mklabel gpt
(parted) mkpart primary 0 3490000
(parted) quit

and you'll have a 3.49TB partition (actually about 3.2TB with reiserfs)

If this is true your problem should be completely solved and you should be able to use ext3 if you want on a partition >2TB.

[Basher52]

So know I just have to ask the expert
So what would you choose?
To create a GPT partition or to use XFS filesystem

If I go for the GPT partition I can use ext3
but in the XFS filesystem, well that IS the filesystem, but is this a better filesystem than ext3?

[Basher52]

This is the layout of the 400Gb disk:
as you may see I'm trying the GPT approach I also copied some data into the partition and I have CRC files on it for later check.

FDISK

Code: Select all

Command (m for help): p
Disk /dev/sdb: 399.8 GB, 399834611712 bytes
255 heads, 63 sectors/track, 48610 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1       48611   390463487+  ee  EFI GPT


PARTED

Code: Select all

(parted) p
Model: HPT DISK_2_0 (scsi)
Disk /dev/sdb: 400GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start   End    Size   File system  Name     Flags
 1      17.4kB  400GB  400GB  ext3         primary       


Adding 200G


After the add is done the RAID set RAID_5_3 will be named RAID 5_4 and 5_3 is gone

[Void Main]

After the add is done the RAID set RAID_5_3 will be named RAID 5_4 and 5_3 is gone

Was the drive still accessible to the OS under the same device name? Could you resize the partition and increase the size of the file system? Ragarding xfs vs ext3 I "personally" would go with creating the partition table and using ext3. Not because I think ext3 is better, in fact it just might be a worse choice than xfs depending on what you will be using this partition for but it's what I know. I have no personal experience with xfs and I would probably want to use it on a smaller less critical installation first to become familiar with it. So I would say that's completely up to you which way you go. If you go with xfs I would certainly test it on the smaller array first and add a drive to see if it will expand without any problem. I have resized ext3 file systems many times without any problem.

[Basher52]

The adding of disk is not done yet, it has done about 11,87% and the 'time left' is very sick, veries from 5 mins to 19 hours, so I can't trust that.
Maybe I can give the result tonight(GMT+1) but I doubt it.

I'll get back...

[Void Main]

Actually that will probably take many hours. Some controllers allow you to use the new size in a performance degraded state while it rebuilds the RAID set.

[Basher52]

yep 32% done now, dang, it's not gonna be done until tomorrow, lol

[Basher52]

It's done adding another 200G to it


FDISK:

Code: Select all

Disk /dev/sdb: 599.7 GB, 599751917568 bytes
255 heads, 63 sectors/track, 72915 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1       48611   390463487+  ee  EFI GPT


$df -h:

Code: Select all

Filesystem            Size  Used Avail Use% Mounted on
/dev/sdb1             367G  1.6G  365G   1% /root/test


So now I'm gonna use resize2fs to get it bigger but this is,
as I think fdisk and parted is too, kinda wierd programs,
there is no parameter to use ALL left space available.
I have to know the size in either k, M or G or maybe sectors,
but how am I supposed to know this
I usually go with big numbers and make them smaller and smaller
until it works, but if you know something to help out...

I'll ask before I use resize2fs so I don't muck it up

[Void Main]

Before you can increase the size of your file system with resize2fs you have to increase the size of your partition. I usually use fdisk for this but in your case you will not be able to use fdisk because of the 2TB limit so you might as well use parted to do this now. You can delete and recreate a partition without damaging your file system as long as you don't change the address of the beginning of the partition and don't make the address of the ending of the partition less than it is.

What I normally do is note the number of cylinders before you added the disks (48610) and note the starting and ending cylinder number of the partition before adding the disk (1/48611). In fdisk I would delete that partition and then tell it to add a new partition with a starting cylinder at "1" and it will default to the last cylinder for the end which in your case should now be 72916.

A paritition is much like a "fence" around a field. The file system is a field. If you have a 1 acre corn field with a fence around it and you want to make a 2 acre corn field you first have to move the fence to make the area larger. The existing corn field is still only 1 acre until you plow up the other acre and plant more corn. So changing the partition does not automatically make your file system larger so you use resize2fs to make the file system larger. All you should have to do after increasing the size of the partition is

Code: Select all

# resize2fs /dev/sdb1


By default it should increase the size of the file system to the size of the partition.

Having said all of that you will not be able to use fdisk for this operation when you get above 2TB so you better use parted here to change the size of the partition instead of fdisk so you are familiar with the parted way of doing it. Then use resize2fs once you get the parition boundries changed.

Here's the parted resize section in the documentation:
http://www.gnu.org/software/parted/manu ... html#SEC25

Or if you have gparted:
http://gparted.sourceforge.net/larry/re ... sizing.htm



gparted may do the resize2fs for you. I'm not sure on that because I have actually never used it.

[Basher52]

First I gotta ask, but I guess you dont make any mistakes

You say that its a total of 48610 cylinders of the disk, before doing any change..
As I can see in 'fdisk' it starts at cyl=1 and ends at cyl=48611
for me that is 48611 cylinders, maybe even 48612 if you count the cylinder=0(if this is 0-based, donno). Do you mean that the end cylinder is up to 48611 but NOT INCLUDING the cylinder 48611? if so, I can understand why you say only 48610 cyls otherwise I dont get it

I like the comparision of a partition to the "fence" and I get that part and now I also know why the data is still there(well i figured that anyways, its never gone until you overwrite the tracks) but I gotta ask and I hope you know this... why is the data nuked when you do this in DOS? shouldn't the data be handled the same way? Does the bulldozers have to wipe all "corns" when they plant the new fence, lol?

I won't do anything, although I know the data will be intact if I go along what you say, but I wanna understand the number of cylcs first.

The server with these disks is standing a bit away and I only has an SSH session to get into it, unless using the crappy 14" monitor at only 1024*768 and I'm trying to get to know the console command for all this incase I have to use it elsewhere so I have no graphic interface to it and thereby no 'gparted'

[Void Main]

You say that its a total of 48610 cylinders of the disk, before doing any change..
As I can see in 'fdisk' it starts at cyl=1 and ends at cyl=48611
for me that is 48611 cylinders, maybe even 48612 if you count the cylinder=0(if this is 0-based, donno). Do you mean that the end cylinder is up to 48611 but NOT INCLUDING the cylinder 48611? if so, I can understand why you say only 48610 cyls otherwise I dont get it :(

To tell you the truth that struck me as odd as well. I was just looking at how you said yours was currently configured and I assumed that it must set the ending cylinder to 1 greater than the ending cylinder. Looking at my disk partition layouts that is not the case. My ending cylinder number = my total cyliners on my last partition on a few machines I have checked. That's also how the fdisk documentation shows it:

http://tldp.org/HOWTO/Partition/fdisk_partitioning.html

It would be interesting if you ran "fdisk" and deleted the existing partition and then did a "n" for new partition and see what it defaults to for ending cylinder (you can do this without actually w)riting the new partition table to disk). I have to assume it will default to 72915 which is the total number of cylinders you currently have. I have to assume the GPT type partitions must be different than normal Linux partitions. Either that or parted screwed up when you created the partition. I assume that is not the case but I would have to do some testing of my own before I am sure. On a side note here is another article talking about what we are doing here:

http://www.coraid.com/support/linux/con ... w/gpt.html

I like the comparision of a partition to the "fence" and I get that part and now I also know why the data is still there(well i figured that anyways, its never gone until you overwrite the tracks) but I gotta ask and I hope you know this... why is the data nuked when you do this in DOS? shouldn't the data be handled the same way? Does the bulldozers have to wipe all "corns" when they plant the new fence, lol?

I don't think FDISK in DOS behaves any differently. It doesn't wipe out the data just by deleting a partition as far as I know. Of course you can't access the data until you create the partition exactly as it was before you deleted it. I could be wrong of course, it's been years and years since I have used a DOS FDISK. Back then you still had to do a "FORMAT" to create a file system after you created a partition. It's no different than having to do a mkfs/newfs after creating a partition using Linux fdisk. Again, there may be more to it than I remember.

I won't do anything, although I know the data will be intact if I go along what you say, but I wanna understand the number of cylcs first.

Me too. :) I was under the impression that this ARRAY of small drives didn't have any data on it (if it got trashed it wouldn't be a big deal). Is this not the case? Really, fdisk and parted should not let you create a partition bigger than the size of the drive so I wouldn't be too concerned. I try and find an exact answer on that though.

The server with these disks is standing a bit away and I only has an SSH session to get into it, unless using the crappy 14" monitor at only 1024*768 and I'm trying to get to know the console command for all this incase I have to use it elsewhere so I have no graphic interface to it and thereby no 'gparted'

Sounds like fun. :) I have always preferred fdisk/resize2fs over parted just because I feel that I know exactly what is happening. Basically I haven't used parted enough to have a good feeling for what it's actually doing. Maybe when I get home I'll have to do some more playing with it so I can feel more confident in giving advice on it. Sorry about that.