Void's Forums

by **Basher52** » Wed Apr 04, 2007 2:14 pm

hmm ok, I'll try do the delete of the partition and see what result I get from the ending cyl.

And no, the data is just copied into these disks as a test to see if the result will not damage it, as I have CRC's of all files I put in there.

and last... you should NOT say you're sorry, you're DA MAN

I have not got this explicit help anywhere, not even on the "big" Linux forums, so beware, lol there's more to come and that's a big one(I think).

I will try this and see what'll happen.

by **Void Main** » Wed Apr 04, 2007 2:25 pm

When I get home I'll throw a spare drive in my machine and play around with parted and see what kind of carnage I can create.

by **Basher52** » Wed Apr 04, 2007 2:48 pm

And as always, you're right, lol

FDISK

Code: Select all: Partition number (1-4): 1 First cylinder (1-72915, default 1):

and after I tried the parted on it:

Code: Select all: [root@ftp ~]# parted /dev/sdb GNU Parted 1.8.2 Using /dev/sdb Welcome to GNU Parted! Type 'help' to view a list of commands. (parted) p Warning: Not all of the space available to /dev/sdb appears to be used, you can fix the GPT to use all of the space (an extra 390463488 blocks) or continue with the current setting? Fix/Ignore? I Model: HPT DISK_2_0 (scsi) Disk /dev/sdb: 600GB Sector size (logical/physical): 512B/512B Partition Table: gpt Number Start End Size File system Name Flags 1 17.4kB 400GB 400GB ext3 primary (parted)

I'm gonna try the FIX and see what'll happen:

Code: Select all: [root@ftp ~]# parted /dev/sdb GNU Parted 1.8.2 Using /dev/sdb Welcome to GNU Parted! Type 'help' to view a list of commands. (parted) p Warning: Not all of the space available to /dev/sdb appears to be used, you can fix the GPT to use all of the space (an extra 390463488 blocks) or continue with the current setting? Fix/Ignore? Fix Model: HPT DISK_2_0 (scsi) Disk /dev/sdb: 600GB Sector size (logical/physical): 512B/512B Partition Table: gpt Number Start End Size File system Name Flags 1 17.4kB 400GB 400GB ext3 primary (parted) p Model: HPT DISK_2_0 (scsi) Disk /dev/sdb: 600GB Sector size (logical/physical): 512B/512B Partition Table: gpt Number Start End Size File system Name Flags 1 17.4kB 400GB 400GB ext3 primary (parted) q Information: Don't forget to update /etc/fstab, if necessary. [root@ftp ~]# parted /dev/sdb GNU Parted 1.8.2 Using /dev/sdb Welcome to GNU Parted! Type 'help' to view a list of commands. (parted) p Model: HPT DISK_2_0 (scsi) Disk /dev/sdb: 600GB Sector size (logical/physical): 512B/512B Partition Table: gpt Number Start End Size File system Name Flags 1 17.4kB 400GB 400GB ext3 primary (parted)

didnt seem to do any changes really... but I'll continue...

OH... WAIT...
I gotta hold here

I know that in fdisk you need to 'w' to make the changes be written to disk, but not in parted... and its looks a bit different in parted:

Code: Select all: [root@ftp ~]# parted /dev/sdb GNU Parted 1.8.2 Using /dev/sdb Welcome to GNU Parted! Type 'help' to view a list of commands. (parted) p Model: HPT DISK_2_0 (scsi) Disk /dev/sdb: 600GB Sector size (logical/physical): 512B/512B Partition Table: gpt Number Start End Size File system Name Flags 1 17.4kB 400GB 400GB ext3 primary (parted)

I can't see the value(cyl=1) as in fdisk, just the start... uh... kilobyte-something.
I do think that this is the value that always appears, but I can't be sure.
I'm waiting for your input on this before I'll do anything else

by **Void Main** » Wed Apr 04, 2007 5:15 pm

I don't see where you told parted to resize anything. It recognizes that your disk has a capacity of 600GB but your partition is only 400GB. Type "parted /dev/sdb" and then "resize 1" and it should prompt you for a start and an end. Unfortunately the spare disk I have at home is dead so I can't test on it. Everything else is in use and I would have to move stuff around to free up but the above should work.

http://www.gnu.org/software/parted/manu ... ing-Parted

EDIT: Heh heh, I'm an idiot. I have a 1GB thumb drive that I can test everything on, could have done that at work. :)

So I created a GPT partition on my thumb drive and then created a partition in parted that was 100MB. Formated it with "mkfs.ext3 /dev/sda1", mounted it, copied a directory and a few files to it and now I have this:

(parted) p

Code: Select all: Model: SanDisk U3 Cruzer Micro (scsi) Disk /dev/sda: 1027MB Sector size (logical/physical): 512B/512B Partition Table: gpt Number Start End Size File system Name Flags 1 17.4kB 100MB 100MB ext3

# fdisk -l /dev/sda

Code: Select all: WARNING: GPT (GUID Partition Table) detected on '/dev/sda'! The util fdisk doesn't support GPT. Use GNU Parted. Disk /dev/sda: 1027 MB, 1027416576 bytes 255 heads, 63 sectors/track, 124 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/sda1 1 125 1003336 ee EFI GPT Partition 1 has different physical/logical beginnings (non-Linux?): phys=(0, 0, 1) logical=(0, 0, 2) Partition 1 has different physical/logical endings: phys=(1023, 254, 63) logical=(124, 231, 60)

# df -h /dev/sda1

Code: Select all: Filesystem Size Used Avail Use% Mounted on /dev/sda1 93M 5.6M 83M 7% /mnt

Notice my fdisk doesn't support GPT type partitions but also notice that the ending cylinder is 1 greater than the number of cylinders I actually have just like in your case but fdisk notices this in my case. I think the reason that is is because what fdisk is telling us is the GPT partition isn't the actual partition that contains our file system. If it was then the ending cylinder would be much lower than what mine shows. This GPT type partition table seems to be a much different animal. It's almost like a partition table within a partition table with GPT but I need to do some more research on this.

No problem though, parted sees the partition properly as only 100MB and we're going to use parted for partitioning from here on out anyway. Now pretend my thumb drive was only 100MB, which is why I created the partition that size. Now pretend I have physically increased the size of it (added disks to my RAID set) to 1GB and now I need to resize the partition. parted doesn't seem to want to do the resize with ext3 for some reason (I found many posts on the net backing this up) but that's ok, we'll just delete the partition and create a new one like I would have done if I were using fdisk:

# parted /dev/sda

Code: Select all: (parted) p Model: SanDisk U3 Cruzer Micro (scsi) Disk /dev/sda: 1027MB Sector size (logical/physical): 512B/512B Partition Table: gpt Number Start End Size File system Name Flags 1 17.4kB 100MB 100MB ext3 (parted) rm 1 (parted) mkpart Partition name? []? File system type? [ext2]? ext3 Start? 0 End? 1027MB

And now you can see the new partition size with the beginning in the same spot but the end is at 1027MB instead of 100MB:

Code: Select all: (parted) p Model: SanDisk U3 Cruzer Micro (scsi) Disk /dev/sda: 1027MB Sector size (logical/physical): 512B/512B Partition Table: gpt Number Start End Size File system Name Flags 1 17.4kB 1027MB 1027MB ext3

I could still mount my drive and the file system was in tact:

# mount /dev/sda1 /mnt
# df -h /dev/sda1

Code: Select all: Filesystem Size Used Avail Use% Mounted on /dev/sda1 93M 5.6M 83M 7% /mnt

It's still only 100MB file system but now that the partition is the full size of the drive we can run resize2fs (I am doing ths without unmounting the drive):

# resize2fs /dev/sda1

Code: Select all: resize2fs 1.39 (29-May-2006) Filesystem at /dev/sda1 is mounted on /media/disk; on-line resizing required Performing an on-line resize of /dev/sda1 to 1003300 (1k) blocks. The filesystem on /dev/sda1 is now 1003300 blocks long.

Now I have the full 1GB:

# df -h /dev/sda1

Code: Select all: Filesystem Size Used Avail Use% Mounted on /dev/sda1 949M 6.6M 894M 1% /mnt

My files and directories are still in tact. What's cool is the resize2fs took a minute to run and while it was running I kept doing a "df -h /dev/sda1" in another window and I could see the "Size" column increase each time until it reached the full size of the drive at which time resize2fs finished.

So, in your case you would:
# parted /dev/sdb

Code: Select all: (parted) rm 1 (parted) mkpart Partition name? []? File system type? [ext2]? ext3 Start? 0 End? 600GB (parted) p (parted) q #

Then resize the file system to the full 600GB:

Code: Select all: # resize2fs /dev/sdb1

That should give you what you need. You could also break it up into more smaller partitions if you want but you'll have to use parted and not fdisk because of the GPT partition and the size limit.

by **Basher52** » Wed Apr 04, 2007 10:13 pm

I'll try this

It sure does EXACTLY what you said, I see the increase in space but it take way more than one minute but that gotta be the bigger size.
btw... you do NOT have to test things just for me, if you dont know the answer, just say that, I appreciate the answer anyways

after resize:

Code: Select all: /dev/sdb1 550G 1.6G 549G 1%

I know(think) that it show the size in 1.0 k-units and not 1.024 but ONLY 550G?
how do I know that this size is the FULL size I can get? since I vrote 600G in parted, is there no... write-nothing-and-you'll-get -maximum-size-thing here?
550 in 1.0k units is ONLY 562G in 1.024 units or am I totally wrong here?

anyways, the size has grown into the next 200G disk

and hopefully the ENTIRE disk too

Do I lose extra space when doing a resize?
cos I KNOW!! I got a bigger one than 550G when I initialized the whole 600G in one job.

by **Void Main** » Wed Apr 04, 2007 10:47 pm

http://www.linux.com/howtos/Large-Disk-HOWTO-14.shtml

Code: Select all: 14.4 Nonproblem: fdisk sees much more room than df? fdisk will tell you how many blocks there are on the disk. If you make a filesystem on the disk, say with mke2fs, then this filesystem needs some space for bookkeeping - typically something like 4% of the filesystem size, more if you ask for a lot of inodes during mke2fs. For example: # sfdisk -s /dev/hda9 4095976 # mke2fs -i 1024 /dev/hda9 mke2fs 1.12, 9-Jul-98 for EXT2 FS 0.5b, 95/08/09 ... 204798 blocks (5.00%) reserved for the super user ... # mount /dev/hda9 /somewhere # df /somewhere Filesystem 1024-blocks Used Available Capacity Mounted on /dev/hda9 3574475 13 3369664 0% /mnt # df -i /somewhere Filesystem Inodes IUsed IFree %IUsed Mounted on /dev/hda9 4096000 11 4095989 0% /mnt # We have a partition with 4095976 blocks, make an ext2 filesystem on it, mount it somewhere and find that it only has 3574475 blocks - 521501 blocks (12%) was lost to inodes and other bookkeeping. Note that the difference between the total 3574475 and the 3369664 available to the user are the 13 blocks in use plus the 204798 blocks reserved for root. This latter number can be changed by tune2fs. This `-i 1024' is only reasonable for news spools and the like, with lots and lots of small files. The default would be: # mke2fs /dev/hda9 # mount /dev/hda9 /somewhere # df /somewhere Filesystem 1024-blocks Used Available Capacity Mounted on /dev/hda9 3958475 13 3753664 0% /mnt # df -i /somewhere Filesystem Inodes IUsed IFree %IUsed Mounted on /dev/hda9 1024000 11 1023989 0% /mnt # Now only 137501 blocks (3.3%) are used for inodes, so that we have 384 MB more than before. (Apparently, each inode takes 128 bytes.) On the other hand, this filesystem can have at most 1024000 files (more than enough), against 4096000 (too much) earlier.

For a partition as large as yours if you plan on having a smaller number of large files than a normal system has you might consider tuning your file system so your inode table is smaller and the amount of reserved space is smaller. It just doesn't make sense taking up 50GB of the drive for this. To see a lot more details on your file system run this:

Code: Select all: # tune2fs -l /dev/sdb1

I don't believe df shows the number of reserved blocks in it's total size count since normal users can't write data when the system only has that much space left. It also doesn't show the amount of space the inode table takes up. You should be able to use tune2fs to reduce that reserved space to a smaller amount which should help a lot. I am pretty sure if you want to decrease the size of your inode table you would have to reformat your file system and use the proper inode parameters. You should be able to reduce the amount of reserved space on the fly with tune2fs as I said though.

See:
$ man tune2fs

Do a "df -i" to see how many inodes you have total and how many are free. Every file and directory requires 1 inode. If it says you have 200 million inodes available that's probably a crazy amount if you don't plan on filling the drive up with tiny files. If you create the file system with a much smaller inode table you'll have more free space for data.

Also, no biggy regarding me doing this along with you. I learn, or at least get a refresher in the process.

by **Basher52** » Thu Apr 05, 2007 10:45 am

Code: Select all: [root@ftp ~]# tune2fs -l /dev/sdb1 tune2fs 1.39 (29-May-2006) Filesystem volume name: <none> Last mounted on: <not available> Filesystem UUID: 7799ea73-ae17-4a09-9abb-953869be7e26 Filesystem magic number: 0xEF53 Filesystem revision #: 1 (dynamic) Filesystem features: has_journal resize_inode dir_index filetype needs_recovery sparse_super large_file Default mount options: (none) Filesystem state: clean Errors behavior: Continue Filesystem OS type: Linux Inode count: 73220096 Block count: 146423799 Reserved block count: 0 Free blocks: 143718725 Free inodes: 73219977 First block: 0 Block size: 4096 Fragment size: 4096 Reserved GDT blocks: 989 Blocks per group: 32768 Fragments per group: 32768 Inodes per group: 16384 Inode blocks per group: 512 Filesystem created: Tue Apr 3 13:59:32 2007 Last mount time: Thu Apr 5 06:14:29 2007 Last write time: Thu Apr 5 06:14:29 2007 Mount count: 3 Maximum mount count: 26 Last checked: Tue Apr 3 13:59:32 2007 Check interval: 15552000 (6 months) Next check after: Sun Sep 30 13:59:32 2007 Reserved blocks uid: 0 (user root) Reserved blocks gid: 0 (group root) First inode: 11 Inode size: 128 Journal inode: 8 Default directory hash: tea Directory Hash Seed: 5ed7e8b1-d9e9-4c8b-8ff3-0ea2f753d0dd Journal backup: inode blocks

Code: Select all: [root@ftp ~]# df -i Filesystem Inodes IUsed IFree IUse% Mounted on /dev/sdb1 73220096 119 73219977 1% /root/test

by **Void Main** » Thu Apr 05, 2007 11:39 am

So you have no reserved space. I actually believe you should have enough reserved space so if the file system gets full critical file system operations (writing to the journal, etc) can still take place. Without researching it I am not sure how you would come up with the exact minimum amount. tune2fs shows you have 73 million inodes which would take close to 9GB at 128 bytes per inode. If you don't think you'll ever get close to 73 million files and directories on that file system then you might consider recreating the file system with a smaller inode table. If you dropped it to 1 mllion you could get back over 8GB of space.

Now your tune2fs reports the total size of your file system is 146,423,799 blocks * 4,096 bytes = 558.6 gigabytes. That is quite a bit less than parted's reported 600GB. I just did a tune2fs on one of my laptop file sytems and when I add it up I come up with 19.6GB which is exactly the amount I come up with when I check the number of blocks and multiply it out. parted says it's 20.1GB so I don't know where parted comes up with it's numbers but it's about 1% off on my system. Yours seems to be a lot farther off than that. Time to do some more digging. :)

EDIT: I don't know how parted figures Gigabytes but it is apparently incorrect. If I change the "units" to "Bytes" and look at my partition table I get this:

Code: Select all: (parted) unit B 5 16860888576B 37917573119B 21056684544B logical ext3

So you can see it tells me the total size is 21056684544 bytes but in Gigabytes it tells me I have 21.1GB.

Code: Select all: 5 16.9GB 37.9GB 21.1GB logical ext3

Using Google to do the conversion from bytes to Gigabytes I get 19.6105657GB which is exactly what I get when I multiply the total number of blocks times the blocks size reported by tune2fs. So parted uses some funky way to calculate GB. I have no idea what that is but I would say it's wrong.

by **Basher52** » Thu Apr 05, 2007 11:58 am

when you say "recreating the file system with a smaller inode table"
is this the same thing as blocksize in winblows?
The bigger size I set the less space I lose onbigger files, but each file takes more space on disk. eg. 1 small text file that has 1 character in it could take 64k bytes on disk if you go for hte maximum amount on the blocksize.

The only switch I use when I create the filesystem is: mke2fs -m0 -j
and the 'm0' a buddy told me to use to get the extra space available and I've been using that since.
The only files that are gonna be hosted on these disks are big.
I would believe that files are goint to be in the range of 100Mbytes and up, but there could be some small files too, but that would be very rare.
The files don't have to be accessed very often and the access time has not that big impact.

So know my question is, what switches of the mke2fs should I use to get the best possible filesystem.

UPDATE: I just took a peek at the 'man' for mke2fs and found that there is a blocksize option there too.

by **Void Main** » Thu Apr 05, 2007 12:13 pm

No, the inode table is not the same as blocksize. Every file and every directory requires exactly 1 inode. The inode table would be somewhat akin to the file allocation table in a FAT file system. NTFS has something similar to inodes. I wouldn't mess with block size unless you plan on running millions of tiny files. Also, reread my previous message as I added something about parted and that the 600GB number it reports is probably wrong. Believe what the number of blocks * blocksize number is as reported by tune2fs. Also, the defaults used by mkfs.ext3 will create the best "average" file system. If you want to tune it for non-average use you have that capability.

by **Basher52** » Thu Apr 05, 2007 12:40 pm

I saw this when looking more into the 'man' of mkfs.ext3(mke2fs)

Code: Select all: -T fs-type Specify how the filesystem is going to be used, so that mke2fs can chose optimal filesystem parameters for that use. The supported filesystem types are: news one inode per 4kb block largefile one inode per megabyte largefile4 one inode per 4 megabytes

I could use this instead of the value of the inode you said, right?
but I couldn't find anything about what happens if the inode value are out, but I saw that it can't be changed after the filesystem is created.

Thinking of using: -T largefile maybe even largefile4
the 'one inode per megabyte' is that 'megabyte' almost the same as the size of files?[/b]

UPDATE: I just recreated the filesystem using mkfs.ext3 with no parameters and I got this:

Code: Select all: Filesystem Size Used Avail Use% Mounted on /dev/sdb1 550G 198M 522G 1% /root/test

before I had this:

Code: Select all: Filesystem Size Used Avail Use% Mounted on /dev/sdb1 550G 1.6G 549G 1% /root/test

This smaller size is due to the reserved space, right? which I removed through the -m0 option.
That is 27G that I "lose" and that's ALOT! of space for 'critical file system operations'

by **Void Main** » Thu Apr 05, 2007 1:12 pm

Basically what you did is say that your files will average about 4MB so mkfs created enough inodes for the number of 4MB files it would take to fill your drive. As I said, every file and directory uses exactly 1 inode regardless of the size. And no, the -m0 is not where you gained the space. You gained the space from creating fewer inodes (the inode table was smaller). As I said in my previous message, on an ext3 file system I personally would not use the "-m0" because if you ever do fill the file system up you might destroy your journal. I need to look into that a little more to be sure but I think that is one of the reasons for the rerserved space. In fact ext2 might even be a better choice for you. ext2 doesn't even use a journal. I am not sure about that and need to do a little more digging to find out pros/cons.

EDIT: Actually I don't think there is any need to reduce the amount of reserved blocks. It appears this probably has nothing to do with the journal getting corrupted or not getting corrupted but more for performance. I don't think reducing the amount of reserved blocks in your file system will actually change anything regarding the amount of space that is used whether physical or reported. Read more about it here:

http://www.redhat.com/magazine/011sep05 ... ps_tricks/

Reducing the number of inodes like you did will certainly reclaim some space as you saw. The only con you have by reducing the number of inodes is you can't put as many files on the file system (your current number of inodes is still probably way more than you'll ever use).

Also I would stick with ext3 because of the recovery advantages on a crashed system. I don't think there are really any big advantages to going back to ext2.

by **Tux** » Thu Apr 05, 2007 1:24 pm

Void and I briefly discussed ext3 reserved space over on this thread:
viewtopic.php?t=1491

I find it pretty ridiculous that ext3fs uses(loses) so much space, but i've played it safe due to the links from the thread I linked.
Also in a filesystem review I read recently it says basically that ext3 wastes huge amount of space compared to the other major FS out there. Please read for yourself

by **Void Main** » Thu Apr 05, 2007 1:53 pm

Thanks for that Tux. It looks like that article concludes that XFS leads most categories and is probably the best all around file system. It's interesting that we already mentioned that as being a possible solution to the >2TB issue because it can be set up on a whole disk without using a partition table. The *only* reason I don't personally use that is because it's not including in the system I currently use as I mentioned. ext2/ext3 are the evils I am most familiar with. I wonder why Fedora doesn't include xfs support?

EDIT: I take that back. It appears you can set up Fedora with xfs, jfs, or reiserfs when installing just by passing a parameter to the installer:

http://www.fedorafaq.org/#reiserjfs

I might have to use that on the next install.

by **Basher52** » Thu Apr 05, 2007 3:04 pm

Then I think I'll setup the smaller RAID with XFS and if anything happens I'll come crawling for help

Void's Forums

FC6 and disk larger than 2 terabytes

Who is online