Actually it was more like I was losing my mind, not my Ubuntu…
But let’s start from the beginning… I have two identical 250GB hard disks so I’ve decided to create a RAID array out of them. Not a system (bootable) one as I had too much trouble setting it up (I’ve set it up but dist-upgrade broke it all too nicely; kernel panic, etc.). I’ve set up a separate 5GB system partition on the first drive, leaving the rest for RAID. This left me with 5GB of free space to spare on the second drive. Smart as I was, I decided to clone the system partition from first drive to the second one, using dd, so I’d still be able to boot if either of the drives crashed. I called it semi-RAID built-by-hand and, well, I was quite proud of it. All seemed fine as months passed (and remember, that this was a server and as such almost did not require any reboots). But time passed and suddenly there was the new Ubuntu out, the Feisty one, so I decided it was time to upgrade. As I had some minor troubles during the upgrade (obsolete packages, invalid config files that I ordered to keep, etc.), I was rebooting every few minutes. And this is where fun comes in…
After a successful upgrade to 7.04 the screen greeted me with a 6.04 prompt. Hmm… strange. Let’s see what’s going on. Okey, so this upgrade actually did go so well. No problem, let’s do it again. This time I did not reboot, but kept making other changes. At some point I had to reboot, though. Now I was scratching my head really hard. Some packages, I knew I had uninstalled previously, kept coming back. I was making changes to various config files only to see those changes not being written to the disk after rebooting. Like, WTF? Now I was rebooting like crazy… losing my mind more with every reboot. I was making directories like THIS-IS-FIRST-HARD-DISK-FOR-SURE only to see them disappear and reappear a couple of reboots later. I was almost crying with despair. I’ve came up with the idea to compare /dev/sda1 with /dev/sdb1. Funny thing, they turned out to be the same. Who knew, maybe my RAID-by-hand automatically turned into a real one?
I had dark thoughts. I was thinking about giving up on having two identical hard disks inside one PC and maybe about downgrading to Edgy, not even knowing whether that was possible. I was even thinking about giving up on those two 250GB disks. I was really desperate. I knew, I needed a break.
10 minutes and one glass of cold water later I was on the mission to find out what is exactly wrong with my Ubuntu. Or my PC. Or my hard disks. Or the world around me.
It wasn’t easy. The df command reported my system being on /dev/sda1. Mounting /dev/sdb1 did not help as it has been showing me the same partition. But then came the bright idea to try and mount /dev/sda1, despite it being already mounted. To my surprise it turned out to be a completely another partition! The lost one! The one I missed so much. I was in heaven, so I started googling, because by that time I just knew it had something to do with those weird UUIDs. And I’d found out that I was not alone. I was so happy…
Now I know that my mistake was to make the exact clone of the system partition and have those two partitions (with the same UUIDs, yeah, unique ids my ass) available at the same time. No wonder my Ubuntu felt schizophrenic, but it still does not justify all of this weird behavior I was greeted with. Some error, some syslog entry, anything would be helpful… is that too much to ask?
What I was left with after I’ve figured it all out was this nice free disk space report (notice the double /dev/sda1 entry):
$ df
Filesystem 1K-blocks Used Available Use% Monuted on
/dev/sda1 5162796 1650512 3250028 34% /
(..)
/dev/sda1 5162796 1558632 3341908 33% /mnt/disk-aThe root of the problem is that I base most of my core linux knowledge on the RedHat from the 90s when /dev/hda1 was saint and meant exactly what it represented, namely the first partition of the first hard disk (presumably connected using the first cable and set as master). With UUIDs all this has changed. Apparently for the better, but leaving some folks like me scratching their heads with disbelief.
Yes, Ubuntu is Linux for human beings. Apparently not for all…
PS: For future reference, remember to set the UUID after doing any partition duplication using dd. You do it like this:
tune2fs -U random /dev/sdb1
Oh man. I would probably give up and try to install Feisty from scratch on one of the 5GB partitions.
Btw, isn’t generating a random UUID like playing Russian roulette a bit. There might be like a one in a million chance that the randomly generated value will be the same one as your current partition. After going though all this, are you still willing to risk it. ;)
Murphy’s law – one in a million chance will usually happen 9 out of 10 times – but if and only if it means massive damage to the system and/or hours of frustration for you.
I was very close to installing Edgy (not Feisty as after the upgrade all hell broke loose) from scratch on one of those 5GB partitions, I just did not want to make this step. I’d rather fix and learn than reinstall and have the same problems in the future (in this case I would… as I’d probably do a
ddcopy again…). But yea, it was Murphy’s law to the fullest.Apparently UUIDs are pretty unique (lots of bits involved as they’re quite long), except, of course, the situation where you (un)intentionally copy them over :)
What I gained was a good piece of invaluable knowledge… What I paid with were hours of frustration…
It all sounds as if taken straight from an IT thriller directed by Darren Aronofsky on black and white tape – okey digital camera enhanced by Mac machine (but not Jackson’s Weta Workshop type – just much smaller, and humbler – independent kind of thing)
But it is all just one step from the scene in “Pi” in which this young character stands in front of a mirror with a drill in his hand… Passion, maths, nerves, looking for THE answer…
User error.
You should have done some research and known that your hap-hazard system administration wasn’t going to survive the upgrade…
Stop bad-mouthing Ubuntu.
It’s hard to do any “research” when you’re sure your knowledge is sufficient. It’s the underlying mechanism that changed (OS being more dependable on UUID), API stayed the same (“dd” command for one). In such cases you just don’t know that you need to study in the first place.
I had the exact same problem with cloning my master hda1 to hdb1 with G4U. Ubuntu should create an easy to use script or gui interface to correct the problem. After all Ubuntu is beginning to able to users without technical skills. Your critism and blog is justified. Spread the word and maybe Ubuntu will listen and make life easier for all of us.
I just now experienced this after cloning (via ‘dd’) a freshly installed Ubuntu 7.04 LTSP server, and started seeing remnants of the clone (I had just switched to the ‘server’ kernel). My knowledge has always been of the static “THIS is /dev/XXX”, not the new UUID stuff. I guess I had to learn sooner or later… I just didn’t know it would cause this kind of issue.
Thank you for the ‘tune2fs’ command. This will have saved my ass since it was going to go into production next week!
Thanks for writing this up. I ran into a similar problem where I have a boot disk that I want to copy and use in case of failure. One problem is the boot disk will change (OS updates etc), so I want the backup disk connected 24×7 so I can keep it fresh. It’d also be nice to pull out the OS disk and machine automatically boot from backup too. But I found the UUID issue mixed the disks’ partitions up big time.
Instead of regenerating UUIDs, I found that setting labels and referencing drives by label is much easier. You can do this to ext3 (tune2fs -L ), swap (mkswap -L ) .. and reiserfs too (i believe). You can view the label and uuid etc using “vol_id “.
In any case, you replace the “UUID=…” with “LABEL=…” in /etc/fstab. You ALSO need to do the same for grub’s /boot/mnt.lst like this:
kernel /vmlinuz-2.6.20-15-generic root=LABEL=os_root ro quiet splash
(where label is “os_root”). So in my case, after I make a backup copy of my OS disk, I change the labels of each partition on the backup, and mount and edit the /boot/mnt.lst and /etc/fstab files to reference the backup-labels.
You could do exactly the same with UUIDs, but user-defined labels are a lot easier.
BTW, I have no idea why – but dd is damn slow. Nobody seems to realize this. Even setting big block-sizes .. it is damn slow. Instead, use ‘cp’ .. like “cp /dev/sdb /dev/sdc”. It is a lot faster.
Same here! …but ..is it me or this UUID sound to much like the Microsoft black_magick in the registry , …Please dont get me wrong …Im not in the position to criticize. but, .. what was wrong with the old human readable unix format …?
I’ve done this three times. The first two times I figured out what the problem was, and then forgot about it, so I had to reconstruct my understanding each time. The last time was last night, so I haven’t forgotten it yet.
I agree that this should not be so hard to do.
>>what was wrong with the old human readable unix format …?
USB drives just don’t stay in the same order, so what was drives 1 & 2 yesterday might be 2 & 1 today. IDE drives never had this problem because they have a fixed order; but then again people never hot-plugged them and expected it all to magically work anyway.
A “better way” was required, but I’m not sure UUIDs are quite right.
Last morning I also experienced the same thing after I dd(ed:) to a new partition and reboot. I hadn’t realized that time it happened because of the UUID. Thanks the tune2fs, I ignored Murphy’s law and I got back my precious data!! :)
Hi folks, just to add a comment on what Staz said. IDE drives DO NOT have a fixed order on machine with new BIOS. I got crazy with my 3 HD machines when I upgrade my motherboard. And UUID was my salvation!!!
Dude! Found this after trying to move my Ubuntu installation to a new hard disc, leaving the old one with just Windows on it. Thanks so much for keeping the page alive and reminding me that I’m not likely to be the only person losing my mind over this stuff.
coming from using solaris for eons, and cloning with dd for backups (no issues other than changing vfstab) , Just ran into the exact same problem with uuid’s and dd, keep this thread alive for other converts to ubuntu