convert existing live btrfs root filesystem to raid1

Fuck it, we're doing it live!

All commands below make these assumptions:

• sda = EXISTING boot drive
• sda1 = EXISTING root fs
• sdz = NEW boot drive

Modify the dev entries in the commands below according to your setup or bad things will happen!

1. Copy the partition table from the old disk to the new one.  You probably don't want to get this backwards.
   
   yo@mama:# sfdisk -d /dev/sda | sfdisk /dev/sdz
   After doing this you may want to pull and then reinsert the drive, to refresh the entries in /dev.  Linux can be weird about this, partprobe doesn't always work.
2. The disks I'm using are not the same size, the new one is a bit larger than the current one.  If the new one were smaller that would be a bit trickier, but since it isn't I won't really get into that.  (Basically you'd either need to recreate the partitions by hand on the new drive, or send the sfdisk output to file and modify that by hand before throwing on the new drive.)
   
   Instead, we'll just let that extra space be.
3. Add the new disk.  Assuming your existing disk is mounted as the root filesystem, the command would go like this:
   
   yo@mama:# btrfs device add /dev/sdz1 /
   yo@mama:# btrfs balance start -dconvert=raid1 -mconvert=raid1 /
   
   If you've read the docs, you know there is also an -s flag for system chunks that can be passed to btrfs balance start.  You don't need to use -s here, according to various people on the internet, and I even found this comment that verifies it in the btrfs-progs sources:
   
   

   /* 
    * allow -s only under --force, otherwise do with system chunks
    * the same thing we were ordered to do with meta chunks
    */

   
   Also, for some reason balances happen in the foreground by default and output nothing.  Watch the progress by opening another terminal and running:
   
   yo@mama:# while btrfs balance status / ; do sleep 10; clear; done
4. And that should be it.  You can make sure it went down as it was supposed to by running:
   
   yo@mama:# btrfs fi show /
   
   Which should give a listing with two devices.

Then, you can give it the balls-to-the-wall test by disconnecting your old boot drive. Make sure that the balance has finished before you do this!  If you're a sissy, run sync before pulling the drive.

on rtfm, from a manual reader

Sometimes you're reading and researching online and you get to a post where someone is asking something very basic, or is very misinformed, and a peanut gallery lurker will inevitably step forward and suggest that the poor sap "read the fucking manual", or rtfm.  When this happens, I cheer - manpages are the bedrock of unix; once you've consumed one, you should be 100% competent in the use of that tool.

But sometimes it doesn't work out like it should.  The ip and sudoers manpages are notorious disasters.  (At one point I remember reading an in-depth writeup about what went wrong with the creation of iproute2, but I can't find it now.  If anyone else has the link, please send it to me!)

Despite some gripes I have with the command interface - the primary operations are too dangerous, so there should be no short options - mdadm has a decent manpage.  But today I came across an insane, inscrutable gem:

These same layouts are available for RAID6.  There are also  4  layouts  that  will provide  an  intermediate stage for converting between RAID5 and RAID6.  These provide a layout which is identical to the corresponding RAID5 layout on the first N-1 devices,  and has the 'Q' syndrome (the second 'parity' block used by RAID6) on the last device.  These layouts are: left-symmetric-6, right-symmetric-6, left-asymmetric-6, right-asymmetric-6, and parity-first-6.

What is the "Q" syndrome??  What does it all mean??  Google came up empty.  The world may never know.

btrfs raid1 as root file system - the immortal life of lil turbo

Sometimes, you just need to reformat.  Instead of trying to convert my existing system from extX to btrfs raid, I reinstalled.  And then converted my brand-new system from ext to btrfs.  Because why do things the easy way when you could do them the hard way?  (If you want to actually convert an existing system, just skip to step 3.  It should work, but this is all cowboy-style, so don't blame me if everything explodes.)  Here are the basic steps.  Be warned, this is from a few weeks old memory:

(If you want to read about how I got here, check this out.  If you just want a guide to do this, the backstory doesn't matter, so just keep reading this page.)

1. Install Wheezy.  However you normally do; all the defaults are fine.  If you feel like it, you can halve the size of swap and add that back in to the system partition.  Or you can do this later with gparted, or you can leave it and have twice as much disk devoted to swap as the Debian installer thinks you'll need.
2. Upgrade to Jessie.  Also in the normal way.

root@serv$ vi /etc/apt/sources.list
:%s/wheezy/jessie/g
:%s/stable/testing/g
:%s/^deb-src/#deb-src/g
:wq
root@serv$ apt-get update
root@serv$ apt-get dist-upgrade -y

3. Boot into an alternate Jessie environment.  Or at least something with recent btrfs-tools.  Ubuntu may work, but I made a custom Jessie iso on the Debian live-systems build interface.  This tool is really cool, someone's dedicating a lot of server time to make this thing happen and I think it's awesome.  On the downside, you'll probably have to wait a few days before you make it to the top of the queue, and once your "build finished" email is sent out, you'll have to download the iso in the 24 hours before they delete it.
4. Install btrfs-tools in the live boot.  Once booted into the new environment, we'll need btrfs-tools, of course.  btrfs --version to make sure you've done stuff right - if you're using the ancient Wheezy 0.19 version this stuff may not work right.  The correct version should sound like a kernel version number, mine is currently π:

:)

5. Convert the just-installed ext root to btrfs.  I got most of my instructions on this step from the occasionally wonderful btrfs wiki.  It doesn't matter if your root is ext3 or ext4 - in fact, these steps may even work with ext2, how should I know.  The steps go something like this:

   root@serv$ # run fdisk -l as root to make sure you're using the hard disk's root filesystem partition for these next steps
   root@serv$ fsck -f /dev/sdX1
   root@serv$ btrfs-convert /dev/sdX1
   

   Use the following optional but prudent steps to make sure your data survived:
   

   root@serv$ mkdir /btrfs && mount -t btrfs /dev/sdX1 /mnt/btrfs
   root@serv$ btrfs subvol list
   root@serv$ # find the name of the saved subvolume, something like extX_saved
   root@serv$ mkdir /ext_saved && mount -t btrfs -o subvol=extX_saved /dev/sdX1 /ext_saved
   root@serv$ mkdir /orig && mount -o loop,ro /ext_saved/image /orig
   

   Yep, that's a triple mount.  The contents of the last mount should be the same as the contents of your root filesystem.  Check anything important or customized, and, if you're satisfied and want to set everything in stone:

root@serv$ btrfs subvol delete extX_saved
root@serv$ umount /orig
root@serv$ rm /ext_saved/image
root@serv$ umount /btrfs /ext_saved
root@serv$ rmdir /orig /ext_saved /btrfs

6. Modify fstab. Make sure you change fstab or your system isn't going to boot, fool.  Use blkid to get the UUID of the boot partition and make sure this matches the entry for your / in fstab (I don't think the UUID will change but I can't remember).  Then make sure the line looks something like this:

UUID=deadbeef-beef-dead-beef-deadbeefbeef    /    btrfs    noatime,ssd,discard,space_cache    0    0

Yes, it is correct that btrfs roots get a 0 for passno, the last number - this means don't worry about running fsck, since fsck.btrfs is just a feel-good utility anyway.  They only released it to fit in, the whole story's in the manpage, which is a pretty good read, btw.

root@serv$ man fsck.btrfs

Anyway, back to stuff that matters - don't just blindly use those mount options in my fstab line - if you use ssd on a drive that isn't an ssd, you'll probably have a bad time.  I didn't turn on certain options like autodefrag and compress=lzo becuase this is intended to be a VM server, and also probably because I don't know what I'm doing.  Check this out, the corresponding page on the ever-helpful wiki.  The whole thing is worth a read, make some damn decisions of your own!
7. Pop out the alternate boot media and reboot.  Sometimes, when emerging from deeply nested sessions, chroots, or alternate boot environments, don't you feel like Cobb waking at the end of Inception?  Anyway, you should be booted into the newly buttery root of your recently installed system now.
8. Verify integrity and clean up.  I know that shit's boring, yo, but we're gonna do it anyway.

root@serv$ btrfs subvol delete ext_saved
root@serv$ # allegedly you can verify with btrfs subvol list -d /, but the manpage for the btrfs-tools version pi on Jessie didn't have this documented
root@serv$ btrfs fi defrag -r /
root@serv$ btrfs balance start /

9. Add the secondary drive and partition.  To get the second drive partitioned properly, I simply popped in the second drive and dd'ed the existing disk to the second one.

root@serv$ dd if=/dev/sdSETUPDRIVE of=/dev/sdNEWDRIVE bs=32M # don't fuck this up, mmk?

This will clone our boot, system and swap partitions to the new drive.  For general applications, I recommend halving each swap.  Even though I never had you modify the swap part of fstab, Linux is smart and will find and use all swap partitions attached to the computer.
10. Convert to raid1 live!  "Fuck it, we're doing it live."  Yeah, computers are pretty cool I guess.  From here.

root@serv$ btrfs fi show # to see which device is mounted as root
root@serv$ fdisk -l # to see which device will be added to form our raid1 (aka, which one is NOT root)
root@serv$ btrfs device add /dev/sdNOTBOOT1 /
root@serv$ btrfs balance start -dconvert=raid1 -mconvert=raid1 -sconvert=raid1 -f /

The last command complains if you try to convert system blocks to raid1 as well (-sconvert=raid1), which is why we use the -f flag, which has the ominous manpage description "force reducing of metadata integrity".  But there is no information I could find out there regarding this, and I want to support complete failover, so this is what I'm using and it's working ok for now.
11. And we're done!  Isn't it great?  Hypothetically, one of our drives can fail and we'll still be able to boot!  I think we might be screwed if the boot partition gives us trouble, but I'm not realy sure yet.

As always, the Arch wiki docs are unparalleled, peruse related info here.  I hope it all worked, drop a line below if something didn't, or if something did!

keeping that data forever (with btrfs)

So we like Linux, that is established.  And we must have our data forever, that is also established.  But the ability to do this is a new thing.  btrfs is a very new thing, allegedly not supported well before kernel 3.9, but with the ability to do some pretty awesome stuff.

If that interests you, read this article.  I've read it a few times, and it is absolutely making me drool over the possibilites of btrfs.  Failing-ish drives will no longer corrupt files in the mp3 collection you've been curating since high school - an errant cosmic ray can no longer bork that video file from senior week.  With anything but zfs and btrfs (even mdadm or hardware raid) this can happen.

So, even though it's probably ill-advised and guaranteed to lose all the data because it's so new, I built a nice shiny new NAS box to test btrfs out on.  I'll just be doing backups from other personal machines with it, so if it crashes and burns, the worst that will happen is another device fails catastrophically at the same time and my decades of personal data are all lost.  If you use this at your job based solely on these instructions, you're insane.  And I like your style.

Start off by installing Debian jessie.  Or use wheezy and get the newer kernel from backports, but the newer btrfs-tools in jessie isn't backported.  Some of this stuff probably won't work with the older wheezy btrfs-tools.

I bought a couple 3T drives, and I had a couple 1.5T drives laying around, so I figured, why not make it interesting and get another 3T by turning the three spare 1.5T drives into a 3T mdadm raid5?  One of the drives was busy elsewhere today, so we're making the array degraded.

Make the filesystem on all three disks, like this:

$ fdisk /dev/sdd
Command (m for help): n
Partition type:
   p   primary (0 primary, 0 extended, 4 free)
   e   extended
Select (default p): 
Using default response p
Partition number (1-4, default 1): 
Using default value 1
First sector (2048-2930277167, default 2048): 
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-2930277167, default 2930277167): 
Using default value 2930277167

Command (m for help): t
Selected partition 1
Hex code (type L to list codes): fd
Changed system type of partition 1 to fd (Linux raid autodetect)

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

Make the mdadm array, make sure it exists and write the mdadm conf file:

$ mdadm --create /dev/md0 --level=5 --raid-devices=3 /dev/sdd1 /dev/sde1 missing
$ cat /proc/mdstat 
Personalities : [raid6] [raid5] [raid4] 
md0 : active raid5 sde1[1] sdd1[0]
      2930012160 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/2] [UU_]
      bitmap: 0/11 pages [0KB], 65536KB chunk

unused devices: <none>

$ mdadm --examine --scan >> /etc/mdadm/mdadm.conf

Now we make the raid6 btrfs.  I used force because some of these disks had already been formatted, but be careful with that option - it doesn't fail mkfs if there is already an fs on the disk.

$ mkfs.btrfs --data raid6 --metadata raid6 --label thevault --format /dev/sd[abfgh] /dev/md0

I stupidly ran the command on the actual physical console so I can't give output, but it looks like the optional features "extref" and "raid56" are turned on by default when you use mkfs.btrfs this way.

Check for your new fs:

$ btrfs filesystem show

Label: 'root'  uuid: 810ff954-180f-4997-a487-30c561ff3820

Total devices 1 FS bytes used 1.28GiB

devid    1 size 28.35GiB used 5.04GiB path /dev/sdc1

Label: 'thevault'  uuid: 7f407975-6c24-42ec-a9a5-4d1967d89cbe

Total devices 6 FS bytes used 112.00KiB

devid    1 size 2.73TiB used 2.02GiB path /dev/sda

devid    2 size 2.73TiB used 2.00GiB path /dev/sdb

devid    3 size 2.73TiB used 2.00GiB path /dev/sdf

devid    4 size 2.73TiB used 2.00GiB path /dev/sdg

devid    5 size 2.73TiB used 2.00GiB path /dev/sdh

devid    6 size 2.73TiB used 2.00GiB path /dev/md0

Btrfs v3.14.1

There's my boot disk and new storage array!  I'm guessing the already-used space is the disk metadata?

Now mount it so we can df it.  You can use any component drive as the source of the mount command and btrfs mount will automatically find the other members of the filesystem.  I'm using the md block device becuase it's easiest to remember here:

$ mkdir /mnt/thevault && mount /dev/md0 /mnt/thevault

$ btrfs filesystem df /mnt/thevault

Data, single: total=8.00MiB, used=0.00

Data, RAID6: total=4.00GiB, used=2.00MiB

System, single: total=4.00MiB, used=0.00

System, RAID6: total=10.50MiB, used=16.00KiB

Metadata, single: total=8.00MiB, used=0.00

Metadata, RAID6: total=4.00GiB, used=112.00KiB

We have metadata, outstanding.  Regular df appears to work too:

$ df

Filesystem       1K-blocks    Used   Available Use% Mounted on

/dev/sdc1         29729792 1424256    26350720   6% /

udev                 10240       0       10240   0% /dev

tmpfs               828964     416      828548   1% /run

tmpfs                 5120       0        5120   0% /run/lock

tmpfs              1964900       0     1964900   0% /run/shm

/dev/sda       17581345080    2176 17572912576   1% /mnt/thevault

So it looks like it works.  Time to make some samba shares and copy in a ton of stuff.  I'll post back later if there are any problems, or just to mention how it goes after some time has elapsed.

upgrade debian squeeze to wheezy, cowboy style

DISCLAIMER: If your system is important, i.e. if destroying it would cause you to lose money or sleep, stop reading this and go follow the complete instructions.

Recently, the folks over at the Debian Project decided it was time to move the testing channel, Wheezy, up to become the new stable release.  Making the move requires making a bunch of changes to the OS, so I figured it would be a good time to make a bunch more.  (Full instructions from the Debian Project, which I used as a starting place for this guide.  Also, this guy's blog post helped me wade through the extraneous instruction a bit.)

Initial steps were simple.  Open sources.list and replace occurrences of squeeze with wheezy:

vi /etc/apt/sources.list
:%s/squeeze/wheezy/g

If, like me, you had backports added to squeeze, you'll need to take note of the new URL format for wheezy backports.  A simple S&R won't cut it here.  The new format looks like this:


deb http://ftp.debian.org/debian/ wheezy-backports main contrib non-free

Of course, remove contrib and non-free if you don't want to use any packages that aren't free software (or require non-free software to function).

I didn't back up anything, because the system I'm upgrading isn't important, but if your system is important, back some stuff up.  This is a pretty big update, and all sorts of things could go wrong, so, once again, if you care about the contents of your hard disks, go do it right.

Typically I would mentally :%s/apt-[a-zA-Z]/aptitude/g, but the big guide seemed a bit preferential toward apt-get, so I went with that for the instructions here.

apt-get update
apt-get upgrade

Not too bad.

apt-get dist-upgrade

Oh god, this thing wants to install so many packages.  At this point I considered backing out and just running off the wheezy repos, leaving the system as is.  But this system was newly installed in February, so I decided to take the plunge.  I'll just uninstall all that junk later, assuming I don't trash the install completely.

This took a long time, probably because the Debian servers are getting slammed.

After an age, I got a fullscreen prompt relating to a change in mdadm.  It wants to know if you need a filesystem on the RAID to boot, which seems like it would usually be a bad idea, but I don't know what kind of constraints people work under.  My box does have a RAID array, but it boots from a filesystem on a different disk, so I changed the value in the text field here from "all" to "none".  I selected "yes" to the next question, on whether to start all arrays after boot.

Towards the end of installation, I was seeing output for initramfs, and my desktop background turned this scary error-shade of green.  May have had to do with using the background slideshow, I'm not sure.

My bash.rc had been edited, and the maintained version had changed, so I was asked to do something about it.  I took the "open a shell" option and copied out my modified version so I could merge it back with the new version later.

After the dist-upgrade completed, as you might expect, Gnome went nuts.  The shut down prompt that opened had Unicode Squares of Confusion instead of text; I gambled on the farthest left button and was greeted with a black screen telling me I had won - the system was rebooting.

It came back with a new and exciting Grub background.  I chose the older 2.6 kernel and booted.  A couple of warnings, nothing that looked too serious, and then an empty black screen.  Waited here for 10 minutes, and then rebooted into recovery mode.

Looking around a bit, I found about what I had expected - X didn't boot properly.  /var/log/Xorg.0.log told me what I needed to know - the kernel I was using was too old.  Time to uninstall that old kernel anyway.  Farewell, Linux 2.6.32, you've treated me well.

aptitude remove linux-image-2.6-686 linux-image-2.6.32-5-686

Next installment: I think I accidentally ran an aptitude install ubuntu.