Where's GRUB?! Ubuntu 8.04 LTS Server & RAID1

Where to begin? (I guess check out my previous post?)

First, my release to release upgrade to Ubuntu 8.04 LTS server was going along fairly well. The upgrade from Edgy (6.10) to Feisty (7.04) went fairly well, except that it dropped one of my drives from the RAID arrays—easily remedied. Just added the partitions back into their respective RAID devices.

Next up, time to move from Feisty (7.04) to Gutsy (7.10) and if all went well, the final move to Hardy Heron (8.04 LTS).

All did not go well in the upgrade from 7.04 to 7.10—although, I must admit here that most of it was my fault. This time I did the upgrade the "official" way.

Network upgrade for Ubuntu servers (recommended)

If you run an Ubuntu server, you should use the new server upgrade system.

1. enable the "dapper-updates" repository
2. install the new "update-manager-core" package - dependencies include python-apt, python-gnupginterface and python2.4-apt.
3. run "sudo do-release-upgrade" in a terminal window
4. follow the steps on the terminal window

This approach seemed to work just fine, and since my box is headless I even ran it over SSH without any issue (even with the warning that doing the upgrade over SSH is probably not ideal).

When I rebooted however the box had no network connectivity. ifconfig revealed only the lo interface. eth0 was gone. sudo lshw showed that the NIC was disabled for some reason. I finally tracked the problem down to /etc/udev/rules.d/70-persistent-net.rules that had "updated" eth0 to eth1 for whatever bizarre reason. I simply change eth0 to eth1 in /etc/network/interfaces and ran sudo /etc/init.d/network restart and all was well again.

Next (again), one of my drives was missing from the RAID array devices. It should be a simple matter of adding them back in via the Webmin Linux RAID module. However, I wasn't paying enough attention and it appears that I tried to add a partition already in use to it's own array (why Webmin would even list the partition to add when it is already in use is questionable). It is possible that I am totally confused on this point but when I did a cat /proc/mdstat it showed the array "rebuilding" so slowly that I was sure something was definitely wrong.

Here my brilliance really shines through. Since the other partitions were delaying sync until the first one finished, I thought I would stave off as much damage as I could by shutting down the box. I can't recall if I tried this via telinit 0 or if I simply powered off the box in my haste. At any rate, I really wreaked havoc on my /home & /data RAID5 partitions. / on md0 (RAID1) was fine. The important partitions did not fair so well. reiserfsck --rebuild-tree did it's best to salvage the carnage but a lot of damage was done. I quickly determined that a restore of /home & /data from my external backup drive would be necessary. [big sad sigh]

Well, if I was going to go through that grief I figured I might as just rebuild the whole box with Hardy Heron from a fresh CD install using the ext3 file system instead of reiserfs since any further development of reiserfs is almost certainly at an end.

And thus it began.

Installing from the 8.04 LTS is really quick and fairly painless, aside from manually setting up the RAID partitions--even that goes pretty fast though (once you've got through it about a dozen times). This is where most of my troubles began. I would set up the RAID arrays & partitions during the install but it would go crazy. The arrays would start rebuilding before the process was completed. RAID devices would show up that weren't even added during the partitioning process. On & on the troubles went.

I can't tell you how many times I tried getting things to work and how many different approaches I took to the problem. I will save you the gory details. The fix is rather arcane and it took forever to figure out. Google was not my friend on this matter. Am I the only one to have these issues? Lucky me...

Here is the problem: Even though I would completely delete the partitions and format them with ext3 instead of reiserfs it didn't fix anything. What was happening is that the install program was seeing the old RAID superblocks from my original setup and using them to rebuild arrays during the install process. This had dreadful effects. I had to get rid of those old RAID superblocks and start fresh. Enter Knoppix.

Initially, I thought I would completely wipe the three drives with the following from Knoppix CLI command:

dd if=/dev/zero of=/dev/hda (hde & hdg)

As each drive is 320GB, this would have taken FOR-EV-ER. Forget it. Next please...

Note: If you ever need to use the following procedure, don't delete your partitions before running this command (because you won't be able to).

You can probably do the same thing from the install CD by exiting to a shell, but Knoppix booted to init 2 was fine for my purposes...

If you are doing this from the install CD use the following first:
make sure the RAID devices are not mounted (i.e. umount /dev/md0 etc.)
sudo mdadm –stop /dev/md0 (repeat until all RAID arrays are stopped, i.e. md1, md2, etc.)

Using either Knoppix or install CD, kill the RAID super-blocks:
mdadm –-misc –-zero-superblock /dev/hda1 (or sda1 if the distro installed shows your IDE drives as SCSI.)

Repeat for each RAID partitions on each of the drives! For example:

mdadm –-misc –-zero-superblock /dev/sda1
mdadm –-misc –-zero-superblock /dev/sda2
mdadm –-misc –-zero-superblock /dev/sdb1
mdadm –-misc –-zero-superblock /dev/sdb2

You get the idea...

OK. The partitioning problem is solved. Your back to installing via CD and partitioning is working just as you want it to. The rest of the installation process runs smooth as glass.

Enter problem two...

Upon completion of the installation , the system WILL NOT BOOT?!?!

And when I say it won't boot, I mean not at all. Grub doesn't even try to load. I was stuck at "Booting CD" and it just hung there!

Unbelievable. I tried reinstalling Grub from the install CD in Rescue Mode to no avail. I tried nuking the MBR on each drive via dd if=/dev/zero of=/dev/hda bs=512 count=1 (same command for the other two drives, hde & hdg) and then attempted to install Grub again... Nothing I did mattered. It would not boot!

At this point I gave up on installing Ubuntu 8.04 LTS Hardy Heron server edition.

I was beaten.

Since Dapper Drake 6.06 LTS is still supported (until June 2011) and I had the install disc. I decided to give it a go--what the heck, after all the time I'd wasted already, why not give it a go?

It installed perfectly. It booted perfectly. It updated via apt-get perfectly.

After I determined I wasn't dreaming and I had not as yet spent the hours it would take to restore the backup files to the server drives, I figured I would try a network upgrade from 6.06 to 8.04 LTS (since you can skip all the intermediate releases when going from one LTS version to the next).

It worked!

Everything appears to be in order. It booted up just fine. The network started properly. The RAID arrays are running. cat /proc/mdstat indicates no problems with them. All is well so far.

I just had to make a slight change in the (official) upgrade process:

sudo apt-get install update-manager-core

sudo do-release-upgrade --mode=server

Without --mode=server it didn't think there was an upgrade available.

Time to install Webmin (it just makes life easier). Don't use apt-get to install Webmin, get it from the main site. This is what I did to get it rolling:

sudo nano /etc/apt/sources.list

Add the following lines:

deb http://us.archive.ubuntu.com/ubuntu/ hardy universe
deb-src http://us.archive.ubuntu.com/ubuntu/ hardy universe
deb http://security.ubuntu.com/ubuntu hardy-security universe
deb-src http://security.ubuntu.com/ubuntu hardy-security universe
# deb http://us.archive.ubuntu.com/ubuntu/ hardy-backports main restricted universe multiverse
# deb-src http://us.archive.ubuntu.com/ubuntu/ hardy-backports main restricted universe multiverse

Next run the following:

wget -v http://some-mirror/sourceforge/webadmin/webmin_some-version_all.deb

md5sum webmin_some-version_all.deb and check it against the hash listed at webmin.com

Follow these instructions when ready:

sudo apt-get update

sudo apt-get install perl libnet-ssleay-perl openssl libauthen-pam-perl libpam-runtime libio-pty-perl libmd5-perl

sudo dpkg --install webmin_some-version_all.deb

And that's it. I restored my files to the /data & /home partitions, configured my Samba shares and all is right with the world.

For now...

Ubuntu Upgrade & Software RAID

You never know what you are going to get when you do a distribution upgrade.

Somewhere along the way I "upgraded" my little fileserver from Ubuntu 6.06 LTS (Long Term Support) to 6.10 (NOT Long Term Support). Why? I do not know. Support for 6.06 LTS will end June 2011 but support for 6.10 ended April 2008!! Unfortunately, I only realized this yesterday. No wonder I hadn't seen any package updates for some time. :-(

Time to upgrade, NOW!

Ubuntu's latest is another LTS release, 8.04. So my big plan is go to somehow get from 6.10 to this latest LTS release. Supposedly you can do it if you upgrade a release at a time. Away we went using Method 2 here...

Changed my sources in /etc/apt/sources.list replacing all instances of edgy to feisty.

sudo apt-get update

sudo apt-get dist-upgrade

sudo apt-get -f install

sudo dpkg --configure -a

sudo telinit 6

It rebooted, so I thought I was probably safe.
sudo lsb_release -a showed me that I was indeed upgraded to 7.04. All was right with the world...

Not so.

My RAID5 & RAID1 devices were missing a drive! What happened?!

It turns out that upgrading turned two of my IDE drives into SCSI devices and left one as a regular IDE device. Bizarre. The missing RAID drives were the partitions from the one IDE device that was left. I used Webmin to simply add the appropriate partions from the IDE drive to the SCSI RAID devices and that was it. It worked without sending my data to /dev/null.

The RAID5 devices rebuilt and the RAID1 was back to mirroring with a spare. All was right with the world.

My question is, what's going to happen during my upgrade from 7.04 - 7.10 - 8.04? Will I be so lucky. I was pretty freaked out when my IDE drives "turned" into SCSI devices. Just weird, and why the heck did it leave one of the IDE drives as such instead of making it a SCSI device? I do not know...

Ubuntu Edgy Server Troubles

One thing leads to another but finally it all worked out (for now).

First, I found that my VIA C3 Linux Software RAID server (scroll down in this link for details) couldn't take a simple cp backup without taking a dive. Unreal.

I got a reasonable deal on a 500GB external drive. USB 2 of course. That was my first problem. The junk board didn't have USB 2 connections so I had to upgrade it with a two port USB 2 PCI card. Great.

My first attempt at backing up my /data & /home partitions with a simple tar command (no compression) ended in a kernel crash. Next I tried an arcane cp | tar type command (to maintain file permissions). I will have to see if I can find it again. Anyway, it ended the same way. What's going on here?

Time for some custom kernel action. I built a couple on my VMWare build server (basically the same OS sans the RAID setup, and the kernel source & build tools installed.) I have the VMWare build box on the AMD64 box--much faster kernel builds. It's very easy to do. Customize you configuration and use these instructions to build your packages. Nothing to it. I got rid of extemporaneous fluff that wasn't needed for my C3 box, configured it so that Reiserfs and RAID 1 & 5 were compiled into the kernel (as opposed to modules), and hoped that would do the trick. No dice.

I was even reduce to running simple cp -av commands for the back up, I mean really, what OS is going to choke on a copy command? Still no luck. It still broke.

Ok, one more shot with software and then we are trying something drastic. I grabbed the latest stable linux source from kernel.org and again compiled yet another kernel. Surely the bugs were gone and I would see success. Nope. It was apparent (and I had a hunch all along that this was the case) that the processor just could not deal with the load. Sure, there was enough memory on the board but it just wasn't going to happen. The C3 is just too weak.

Drastic Measures

I pulled my Windows 2000 drive out of the AMD64 box (with all my neato PCI cards in it: RME Hammerfall; NTSC/HDTV tuner card; etc.) and put my RAID disks into the case. I used a standard (none C3 specific) kernel to boot and everything was running smoothly--except no ethernet. Oh brother...

The kernel modules for the NIC was loaded but ifconfig showed nothing but the loopback. I was forced to track down the configs to make it work. You would think it would all be in /etc/network but not quite. I will spare you the details and get to the point: you make changes in two places--or perhaps one really (if you have the right NIC driver loaded). In my case I made changes to /etc/network/interfaces & /etc/iftab.

Here is what interfaces looks like (I have a static IP address for it, makes like easier on my network & it is a server--if you believe it or not):

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo eth1
iface lo inet loopback
# This is a list of hotpluggable network interfaces.
# They will be activated automatically by the hotplug subsystem.
# The primary network interface
iface eth1 inet static
address 192.168.0.5
netmask 255.255.255.0
broadcast 192.168.0.255
network 192.168.0.0
gateway 192.168.0.1


The only real change I made here was replacing eth0 with eth1. Even without modifying the next file, it would come up after a lengthy wait while the system autoconfigured the NIC. The next file /etc/iftab could have remedied this issue without even touching the previous file I believe--you'll see why shortly:


# This file assigns persistent names to network interfaces. See iftab(5).
# eth0 mac 00:00:0d:33:50:83
eth1 mac 02:R2:D2:C3:P0:07


I could have just changed the MAC address of the NIC for eth0 and I'll bet everything would have come up just fine. Instead, in my case, eth0 disappears and eth1 takes over. No big deal. It comes up just as it should. I may go back and "fix" it for semantics I suppose but I doubt it since if it ain't broke, don't fix it.

Ok, so the network is now working, but I forgot to mention during all of the crashes when the drives were in the C3 box I was doing reiserfsck check on the md devices all the time--even on the external drive. I will have to say ReiserFS is pretty dang solid. When it did come up with errors reiserfsck --fix-fixable /dev/mdx would generally do the job. Unfortunately, since you cannot unmout your root filesystem that one was a bit of a trick. A couple of times though --fix-fixable didn't work and I had to use the more drastic --rebuild-tree (absolutely back up your partition before running this command). Since I was now able to actually get a good backup onto the external HDD without the OS dying, it was a good time for --rebuild-tree where needed. It was nice and fast, and it worked without any issues on the partition that needed it.

So now you would think all was right with the world. The server running smoothly on nice fast hardware--life is good. However, I had this nagging feeling that / needed to be checked--since it had been crashed several times. You can't unmount root when it's running of course, so an extensive test required outside intervention. Enter everyone's favorite LiveCD, Knoppix.

I wanted to get Knoppix to assemble /dev/md0 (my RAID1 root device) with mdadm but I was never able to get it to work for me. Instead I ran a simple reiserfsck check on each unmounted partition that was part of the array. It's RAID1 (mirroring) so each disk is simply a copy of the others. The first two disks came out fine, no corruptions. The third, hdg1 was not so fortunate. reiserfsck wanted a --rebuild-tree on that one. Uh, no thanks... since I had no idea what that would do to the other two drives when it was rebooted without Knoppix. Enter the aforementioned mdadm, an extremely useful & powerful tool for RAID administration even if it is not well documented (at least I should say, I didn't find everything on it that I was looking for even though "google is [my] friend.") I decided that the simplest fix would be to "fail " the drive with the corrupted partition, reformat it, and then put it back into the array. I don't know about you kids but this was what I came up with (via this article) and it worked perfectly well for me:

# Fail the corrupted partition
mdadm /dev/md0 --fail /dev/hdg1 --remove /dev/hdg1

# Reformat the partition
mkreiserfs /dev/hdg1

# Add the partition back into the RAID device
mdadm /dev/md0 --add /dev/hdg1

The OS knew right what to do and added hdg1 right back in as the spare drive for the RAID1 device (which was the initial configuration of this particular configuration--two drives mirroring with one spare).

Ok, so now I am confident that all my partitions are clean, even the external drive, and my RAID1 & 5 devices are working properly. What else could possibly go wrong now?

I really cannot say what change(s) I made that caused the following disaster but the recovery was really quite simple in the end--even if it did take almost all day to finally stumble upon it. Here's the list of things that went terribly wrong in no particular order:

• Samba ceased to start automatically
• Webmin ceased to start automatically or not at all.
  
• /etc/mtab & mount would only display the root filesystem as being mounted (even though nothing had been changed in /etc/fstab). mount -a would not resolve the issue either, but even though they were not listed /home & /data were still available. Very, very strange. Totally disconcerting.
• Swap would not automatically start.
  

I would manually start Webmin and proceeded to change various boot time options. One option that caused me terrible grief was the checkfs.sh and/or checkroot.sh options at boot time. Do yourself a favor and don't enable these in Webmin. The box would lock up hard and keep the root filesystem in a read-only state. This causes a world of grief because the OS needs writing capabilities. In this state you couldn't even manually start some of the services you wanted. Fortunately I found a command that would undo this particular disaster:

mount -w -n -o remount /

I've seen this before, but in the midst of my sorrows, I couldn't remember it. Once / was back in rw mode, I could then start Webmin again. Play around with boot time settings, and still have problems. Something was wrong with the init scripts. Something terribly wrong, and I hadn't a clue how to fix it.

I decided that I would try to find a package through aptitude search that would allow me to reset the init scripts to "factory default" since I had obviously blown mine up somehow. Wonderful tool aptitude, highly recommended. This is what I did:

sudo -s
apt-get update
apt-get upgrade

# After finding what I was looking for
# via aptitude search I issued the
# following command

aptitude remove initscripts

# Ugly messages follow about breaking
# things and uninstalling other important
# packages but I went ahead with the removal
# anyway (making note of the other packages
# that were getting whacked as well).
#
# Time to put everything back based on my
# notes of the packages removed

aptitude install initscripts ubuntu-minimal system-services upstart-compat-sysv
reboot

Reboot... The moment of truth... And there it was! Everything, was back to normal. All the filesystems mounted--showing up in /etc/mtab & when the mount command was executed. Samba & Webmin started automatically. "No runs, no drips, no errors." "All was right with the world."

And there you have it. Many disasters and many (recovery) lessons learned. Using Knoppix to run reiserfsck on the root partition RAID1 drives was nice. I wish I could have figured out how to make it assemble the RAID device under Knoppix though. That would have been even better, and a necessity to checking a RAID5 device if your OS blows up completely.

Now, I have to find a permanent home for the drives (an the OS that lives on them). A box with good airflow & a CPU /MoBo /RAM with enough capacity to meet my seemingly simple needs. I'm not leaving them in the AMD64 box. I need to put that back together so we can use it as the big workstation with all the toys in it.

Well, when all is said and done, at least when I move the drives again I should be better prepared for some of the weirdness that may occur...