Ubuntu Edgy Server Troubles

One thing leads to another but finally it all worked out (for now).

First, I found that my VIA C3 Linux Software RAID server (scroll down in this link for details) couldn't take a simple cp backup without taking a dive. Unreal.

I got a reasonable deal on a 500GB external drive. USB 2 of course. That was my first problem. The junk board didn't have USB 2 connections so I had to upgrade it with a two port USB 2 PCI card. Great.

My first attempt at backing up my /data & /home partitions with a simple tar command (no compression) ended in a kernel crash. Next I tried an arcane cp | tar type command (to maintain file permissions). I will have to see if I can find it again. Anyway, it ended the same way. What's going on here?

Time for some custom kernel action. I built a couple on my VMWare build server (basically the same OS sans the RAID setup, and the kernel source & build tools installed.) I have the VMWare build box on the AMD64 box--much faster kernel builds. It's very easy to do. Customize you configuration and use these instructions to build your packages. Nothing to it. I got rid of extemporaneous fluff that wasn't needed for my C3 box, configured it so that Reiserfs and RAID 1 & 5 were compiled into the kernel (as opposed to modules), and hoped that would do the trick. No dice.

I was even reduce to running simple cp -av commands for the back up, I mean really, what OS is going to choke on a copy command? Still no luck. It still broke.

Ok, one more shot with software and then we are trying something drastic. I grabbed the latest stable linux source from kernel.org and again compiled yet another kernel. Surely the bugs were gone and I would see success. Nope. It was apparent (and I had a hunch all along that this was the case) that the processor just could not deal with the load. Sure, there was enough memory on the board but it just wasn't going to happen. The C3 is just too weak.

Drastic Measures

I pulled my Windows 2000 drive out of the AMD64 box (with all my neato PCI cards in it: RME Hammerfall; NTSC/HDTV tuner card; etc.) and put my RAID disks into the case. I used a standard (none C3 specific) kernel to boot and everything was running smoothly--except no ethernet. Oh brother...

The kernel modules for the NIC was loaded but ifconfig showed nothing but the loopback. I was forced to track down the configs to make it work. You would think it would all be in /etc/network but not quite. I will spare you the details and get to the point: you make changes in two places--or perhaps one really (if you have the right NIC driver loaded). In my case I made changes to /etc/network/interfaces & /etc/iftab.

Here is what interfaces looks like (I have a static IP address for it, makes like easier on my network & it is a server--if you believe it or not):

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo eth1
iface lo inet loopback
# This is a list of hotpluggable network interfaces.
# They will be activated automatically by the hotplug subsystem.
# The primary network interface
iface eth1 inet static
address 192.168.0.5
netmask 255.255.255.0
broadcast 192.168.0.255
network 192.168.0.0
gateway 192.168.0.1


The only real change I made here was replacing eth0 with eth1. Even without modifying the next file, it would come up after a lengthy wait while the system autoconfigured the NIC. The next file /etc/iftab could have remedied this issue without even touching the previous file I believe--you'll see why shortly:


# This file assigns persistent names to network interfaces. See iftab(5).
# eth0 mac 00:00:0d:33:50:83
eth1 mac 02:R2:D2:C3:P0:07


I could have just changed the MAC address of the NIC for eth0 and I'll bet everything would have come up just fine. Instead, in my case, eth0 disappears and eth1 takes over. No big deal. It comes up just as it should. I may go back and "fix" it for semantics I suppose but I doubt it since if it ain't broke, don't fix it.

Ok, so the network is now working, but I forgot to mention during all of the crashes when the drives were in the C3 box I was doing reiserfsck check on the md devices all the time--even on the external drive. I will have to say ReiserFS is pretty dang solid. When it did come up with errors reiserfsck --fix-fixable /dev/mdx would generally do the job. Unfortunately, since you cannot unmout your root filesystem that one was a bit of a trick. A couple of times though --fix-fixable didn't work and I had to use the more drastic --rebuild-tree (absolutely back up your partition before running this command). Since I was now able to actually get a good backup onto the external HDD without the OS dying, it was a good time for --rebuild-tree where needed. It was nice and fast, and it worked without any issues on the partition that needed it.

So now you would think all was right with the world. The server running smoothly on nice fast hardware--life is good. However, I had this nagging feeling that / needed to be checked--since it had been crashed several times. You can't unmount root when it's running of course, so an extensive test required outside intervention. Enter everyone's favorite LiveCD, Knoppix.

I wanted to get Knoppix to assemble /dev/md0 (my RAID1 root device) with mdadm but I was never able to get it to work for me. Instead I ran a simple reiserfsck check on each unmounted partition that was part of the array. It's RAID1 (mirroring) so each disk is simply a copy of the others. The first two disks came out fine, no corruptions. The third, hdg1 was not so fortunate. reiserfsck wanted a --rebuild-tree on that one. Uh, no thanks... since I had no idea what that would do to the other two drives when it was rebooted without Knoppix. Enter the aforementioned mdadm, an extremely useful & powerful tool for RAID administration even if it is not well documented (at least I should say, I didn't find everything on it that I was looking for even though "google is [my] friend.") I decided that the simplest fix would be to "fail " the drive with the corrupted partition, reformat it, and then put it back into the array. I don't know about you kids but this was what I came up with (via this article) and it worked perfectly well for me:

# Fail the corrupted partition
mdadm /dev/md0 --fail /dev/hdg1 --remove /dev/hdg1

# Reformat the partition
mkreiserfs /dev/hdg1

# Add the partition back into the RAID device
mdadm /dev/md0 --add /dev/hdg1

The OS knew right what to do and added hdg1 right back in as the spare drive for the RAID1 device (which was the initial configuration of this particular configuration--two drives mirroring with one spare).

Ok, so now I am confident that all my partitions are clean, even the external drive, and my RAID1 & 5 devices are working properly. What else could possibly go wrong now?

I really cannot say what change(s) I made that caused the following disaster but the recovery was really quite simple in the end--even if it did take almost all day to finally stumble upon it. Here's the list of things that went terribly wrong in no particular order:

• Samba ceased to start automatically
• Webmin ceased to start automatically or not at all.
  
• /etc/mtab & mount would only display the root filesystem as being mounted (even though nothing had been changed in /etc/fstab). mount -a would not resolve the issue either, but even though they were not listed /home & /data were still available. Very, very strange. Totally disconcerting.
• Swap would not automatically start.
  

I would manually start Webmin and proceeded to change various boot time options. One option that caused me terrible grief was the checkfs.sh and/or checkroot.sh options at boot time. Do yourself a favor and don't enable these in Webmin. The box would lock up hard and keep the root filesystem in a read-only state. This causes a world of grief because the OS needs writing capabilities. In this state you couldn't even manually start some of the services you wanted. Fortunately I found a command that would undo this particular disaster:

mount -w -n -o remount /

I've seen this before, but in the midst of my sorrows, I couldn't remember it. Once / was back in rw mode, I could then start Webmin again. Play around with boot time settings, and still have problems. Something was wrong with the init scripts. Something terribly wrong, and I hadn't a clue how to fix it.

I decided that I would try to find a package through aptitude search that would allow me to reset the init scripts to "factory default" since I had obviously blown mine up somehow. Wonderful tool aptitude, highly recommended. This is what I did:

sudo -s
apt-get update
apt-get upgrade

# After finding what I was looking for
# via aptitude search I issued the
# following command

aptitude remove initscripts

# Ugly messages follow about breaking
# things and uninstalling other important
# packages but I went ahead with the removal
# anyway (making note of the other packages
# that were getting whacked as well).
#
# Time to put everything back based on my
# notes of the packages removed

aptitude install initscripts ubuntu-minimal system-services upstart-compat-sysv
reboot

Reboot... The moment of truth... And there it was! Everything, was back to normal. All the filesystems mounted--showing up in /etc/mtab & when the mount command was executed. Samba & Webmin started automatically. "No runs, no drips, no errors." "All was right with the world."

And there you have it. Many disasters and many (recovery) lessons learned. Using Knoppix to run reiserfsck on the root partition RAID1 drives was nice. I wish I could have figured out how to make it assemble the RAID device under Knoppix though. That would have been even better, and a necessity to checking a RAID5 device if your OS blows up completely.

Now, I have to find a permanent home for the drives (an the OS that lives on them). A box with good airflow & a CPU /MoBo /RAM with enough capacity to meet my seemingly simple needs. I'm not leaving them in the AMD64 box. I need to put that back together so we can use it as the big workstation with all the toys in it.

Well, when all is said and done, at least when I move the drives again I should be better prepared for some of the weirdness that may occur...

Tube Screamer TS-9

Updated June 15, 2007: Check out the Ibanez Tube Screamer History page. It's clear(er) now from checking out this site that it is in fact the TS-808 that would be the ideal pedal in this series. Analogman offers custom mods to the TS-9 that give you exactly what you need. I want one of these!!


I have been interested in getting this pedal for a long time. A vintage model is probably prohibitively expensive but I have also read the the TS-9 reissues do not use the same circuitry as the originals. (Another Hammett reference for those who read my last post where he is quoted in Guitar World Jan. 2006) I've read that it is much less expensive to get the reissued model and have it modified to get the vintage sound then it is to buy an original. So what difference is the modified circuitry making to the reissues that is missing in the first place?

Everybody's favorite, Wikipedia says, "...The classic Tube Screamer sound includes a "mid-hump," which means that the circuit accentuates freqencies [sic] between the bass and treble ranges (mid-frequencies). Many guitarists prefer this sort of equalization, as it helps to keep their sound from getting lost in the overall mix of the band."

A couple of articles for electronics folk delve into the "secrets" of the Tube Screamer and Tube Screamer modifications. Rather then modifying a stock reissue model, it may be simpler to obtain that classic tone by going to another source: Maxon. A company who apparently built Tube Screamers for Ibanez from 1971 -2002. The Maxon OD-9 looks like it is just what I am after, and the OD-9+Pro looks interesting as well with quite a number of options for, "...effectively function[ing] as a 4-voiced monster, offering everything from the classic, bluesy tones of the original 808 to the searing saturated chunk demanded by today’s extreme metal music."

Kirk Hammett: Why the taped hand?

See that tape around Kirk's picking hand? I have seen a lot of bizarre (or sometimes reasonable) reasons that people thinks he tapes his hand. Someone e-mailed me asking about it and I dug this bit out of 20th Anniversary of Master of Puppets edition of Guitar World (January 2006). The definitive answer from the man himself:

[Question] There have been many theories spread on the Internet about why you tape your hands. So once and for all, why do you do it? --Jerry Glass

If you look at the side of your hand, right below your pinkie, you'll see there are some wrinkle lines. At least there are on my hand... [laughs] Because of my playing style, I'm constantly hitting the strings and palm muting and, generally, beating up my right hand. Over the course of a tour, those wrinkles start busting open and bleeding. I put tape over the wrinkles so they don't break open and bleed all over my guitar, 'cause if a string gets in there once your hand is cut --wow!--that stings! Plus, it lets me play faster since it reduces drag between my hand and the strings.

Once, when we were on tour, I looked out into the audience and saw a kid with both his hands taped up like mine. I was cracking up so much I had to give him a backstage pass. --Kirk Hammett, Guitar World, January 2006, 112.

Cool Rails

I have had Hot Stacks in the mid & neck positions on my Warmoth since it was put together. They were never the ideal match for the JB Trembucker in the bridge and too bright for an intrinsically bright guitar from the start. The output mismatch between the Hot Stacks and the JB was just plain annoying, but in my defense I will say I think they were (at the time) my best option for a stacked humbucker for the mid & neck slots.

Times have long since changed, and a change in tone (particularly in the neck position) was well overdue.

I used the Duncan Tone Wizard to find out what they would recommend for overall balance with the JB & the general characteristics of the guitar. In my case it suggested the following:

Neck 1	SCR-1 Cool Rails
Neck 2	SL59-1 Little '59
Middle 1	SCR-1 Cool Rails
Middle 2	SSL-5 Custom Staggered

I even e-mailed them with the current config of the guitar with the Hot Stacks to get a custom recommendation from them, and the response was: go with the Cool Rails.

So I did.

I ordered two of them from an on-line vendor. Their site said they had them in stock, and they had the best pricing. Turned out that they only sent one, and the other went to back-order. When I called to find out when they might be getting the other PUP I was told it would be a month or so before they got them in stock. I e-mailed them and they really had no idea when they would get them so I canceled the order for the second one.

Eventually, I got the pickup via US Postal mail in a crappy padded envelope--how lame is that?! First of all, who ships something like that through USPS with no tracking number, and a padded envelope? Whatever... A return customer? Not me.

I removed the Hot Stack from the neck position and started to install the Cool Rails. It was an extremely tight fit, but I managed to get it in there at the sacrifice of a couple of paint chips (not happy about that one bit). This PUP is obviously designed with Strat pickguard mounting in mind, but a couple of longer wood screws and I was good to go. I wired it right up with no problems and the moment of truth had arrived. Would the output balance with the JB? Would it even out the ultra bright tone that I had grown to despise? The verdict? This is what I e-mailed to a friend:

...it is so nice to have a neck PUP in there that has [output] balance with the JB!!! Although I was really taken by surprise at how much darker it sounded vs. the Hot Stack that was in there previously. As you know, I couldn't stand how bright that PUP was, it was so bright it sounded harsh and of course it didn't balance with the JB at all. Way too weak. Anyway, it initially took me a while to get used to playing a darker PUP in the neck position. Here is the really cool thing though, it sounds totally awesome in the 2 position (mid/neck)!!! I mean it is just great! My favorite Strat sound without any hum! Tone-wise, those two pickups balance so well, it's incredible!

...The guitar looks a little, uh, unique now with the side by side rails in the neck, the stack blade in the mid and the big JB in the bridge as you can see in this picture. Hey, as long as it sounds good and plays well, it's all good to me...

So I ended up with the right combination of pickups even though I didn't get that second Cool Rails for the mid slot. I never play the mid PUP by itself anyway, so the output balance isn't an issue with it. Tone-wise it is exactly what I need in the 2 & 4 positions with the other two pickups.

It has taken me a bit to get used to a humbucker tone in the neck position but now I am using it constantly. It sounds really good clean, and just gets better as you turn up the gain. The sustain is just exceptional with this one. The Hot Stack couldn't even begin to come close.

Initially I thought that perhaps I should have gone with the Little '59 but I have no doubts now. The Cool Rails has provided exactly the output & tone I needed in this guitar.

Cool Rails^TM image
provided by seymourduncan.com