Monday, October 27, 2014

getting inside a VM disk image

Have a VM disk image that you need to pull a few files from, but you don't want to spin it up into a VM?  Here's the simplest way I've found to get at the files inside it:

First, convert whatever format it's in to .raw using qemu-img - you'll need a good bit of disk space for this to work.  If your source virtual disk is a qcow2, for example, the command might look like this:

me@box$ qemu-img convert -f qcow2 -O raw var/lib/libvirt/images/own.qcow2 /mnt/thevault/everything/backups/own.raw

Then find the offset of the primary partition inside the image with fdisk:

me@box$ fdisk -l /mnt/thevault/everything/backups/own.raw

Disk /mnt/thevault/everything/backups/own.raw: 21.5 GB, 21474836480 bytes
255 heads, 63 sectors/track, 2610 cylinders, total 41943040 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0008fa87

                                      Device Boot      Start         End      Blocks   Id  System
/mnt/thevault/everything/backups/own.raw1   *        2048    40136703    20067328   83  Linux
/mnt/thevault/everything/backups/own.raw2        40138750    41940991      901121    5  Extended
/mnt/thevault/everything/backups/own.raw5        40138752    41940991      901120   82  Linux swap / Solaris

The partition we are interested in is raw1, the Linux (type 83) partition.  If you installed Debian on the disk and went with standard default partitioning options, putting everything under / in the same partition, yours will be the same.

Pull your unit size from the output as well, which in this case is 512 bytes, and then multiply that by the start sector (2048) to get 1048576.

Now we mount the .img file as a loopback device, starting with our calculated offset:

me@box$ mount -o loop,offset=1048576 /mnt/thevault/everything/backups/own.raw /mnt/own

And now you can access your files inside the old VM disk!  Don't forget to clean up, now.

Wednesday, October 1, 2014

systemd hate - console autologin on Jessie

Quick tip - systemd sucks.

Just kidding, I actually have no opinion on the matter.  I haven't noticed much of a difference yet, except that all the old snippets I find on the internet seem to apply mostly to sysv, and from what I hear, grepping over all of /var/log won't return systemd log results, as they're a binary format.

But I'm a fan of progress, and if systemd fixes some obscure problems deep inside Linux that will eventually bubble up and result in better stuff in my world, then I'm all for it.

The one particular problem I was looking to solve was how to autologin at the console.  Arch's wiki was helpful in this regard.  Unfortunately for me, I failed to realize that Arch and Debian Jessie are different distros, and so when I rebooted to test my console autologin, I instead got a blinking cursor on an empty, unresponsive screen shortly after the kernel messages began their scroll.

After some live booting and poking around, I came to the realization that while agetty is at /usr/bin/ on Arch, it's at /sbin on Debian.  Quick fix, problem solved.

Here's all you have to do to get autologin on Debian Jessie:

root@serv$ mkdir -pv /etc/systemd/system/getty@tty1.service.d/
root@serv$ vi /etc/systemd/system/getty@tty1.service.d/autologin.conf

Then, inside the file type this stuff, substituting $username for the name of the user you want to be automatically logged in:

ExecStart=-/sbin/agetty --autologin $username --noclear %I 38400 linux

If you're actually typing, note the tricky little dash in front of the agetty path, that matters.  Enjoy not having to type two dozen characters the once a year you actually use that physical console on your server!

Tuesday, September 30, 2014

btrfs raid1 as root file system - the immortal life of lil turbo

Sometimes, you just need to reformat.  Instead of trying to convert my existing system from extX to btrfs raid, I reinstalled.  And then converted my brand-new system from ext to btrfs.  Because why do things the easy way when you could do them the hard way?  (If you want to actually convert an existing system, just skip to step 3.  It should work, but this is all cowboy-style, so don't blame me if everything explodes.)  Here are the basic steps.  Be warned, this is from a few weeks old memory:

(If you want to read about how I got here, check this out.  If you just want a guide to do this, the backstory doesn't matter, so just keep reading this page.)
  1. Install Wheezy.  However you normally do; all the defaults are fine.  If you feel like it, you can halve the size of swap and add that back in to the system partition.  Or you can do this later with gparted, or you can leave it and have twice as much disk devoted to swap as the Debian installer thinks you'll need.
  2. Upgrade to Jessie.  Also in the normal way.
  3. root@serv$ vi /etc/apt/sources.list
    root@serv$ apt-get update
    root@serv$ apt-get dist-upgrade -y
  4. Boot into an alternate Jessie environment.  Or at least something with recent btrfs-tools.  Ubuntu may work, but I made a custom Jessie iso on the Debian live-systems build interface.  This tool is really cool, someone's dedicating a lot of server time to make this thing happen and I think it's awesome.  On the downside, you'll probably have to wait a few days before you make it to the top of the queue, and once your "build finished" email is sent out, you'll have to download the iso in the 24 hours before they delete it.
  5. Install btrfs-tools in the live boot.  Once booted into the new environment, we'll need btrfs-tools, of course.  btrfs --version to make sure you've done stuff right - if you're using the ancient Wheezy 0.19 version this stuff may not work right.  The correct version should sound like a kernel version number, mine is currently π:

  6. :)
  7. Convert the just-installed ext root to btrfs.  I got most of my instructions on this step from the occasionally wonderful btrfs wiki.  It doesn't matter if your root is ext3 or ext4 - in fact, these steps may even work with ext2, how should I know.  The steps go something like this:
    root@serv$ # run fdisk -l as root to make sure you're using the hard disk's root filesystem partition for these next steps
    root@serv$ fsck -f /dev/sdX1
    root@serv$ btrfs-convert /dev/sdX1
    Use the following optional but prudent steps to make sure your data survived:
    root@serv$ mkdir /btrfs && mount -t btrfs /dev/sdX1 /mnt/btrfs
    root@serv$ btrfs subvol list
    root@serv$ # find the name of the saved subvolume, something like extX_saved
    root@serv$ mkdir /ext_saved && mount -t btrfs -o subvol=extX_saved /dev/sdX1 /ext_saved
    root@serv$ mkdir /orig && mount -o loop,ro /ext_saved/image /orig
    Yep, that's a triple mount.  The contents of the last mount should be the same as the contents of your root filesystem.  Check anything important or customized, and, if you're satisfied and want to set everything in stone:
  8. root@serv$ btrfs subvol delete extX_saved
    root@serv$ umount /orig
    root@serv$ rm /ext_saved/image
    root@serv$ umount /btrfs /ext_saved
    root@serv$ rmdir /orig /ext_saved /btrfs
  9. Modify fstab. Make sure you change fstab or your system isn't going to boot, fool.  Use blkid to get the UUID of the boot partition and make sure this matches the entry for your / in fstab (I don't think the UUID will change but I can't remember).  Then make sure the line looks something like this:
  10. UUID=deadbeef-beef-dead-beef-deadbeefbeef    /    btrfs    noatime,ssd,discard,space_cache    0    0
    Yes, it is correct that btrfs roots get a 0 for passno, the last number - this means don't worry about running fsck, since fsck.btrfs is just a feel-good utility anyway.  They only released it to fit in, the whole story's in the manpage, which is a pretty good read, btw.
    root@serv$ man fsck.btrfs
    Anyway, back to stuff that matters - don't just blindly use those mount options in my fstab line - if you use ssd on a drive that isn't an ssd, you'll probably have a bad time.  I didn't turn on certain options like autodefrag and compress=lzo becuase this is intended to be a VM server, and also probably because I don't know what I'm doing.  Check this out, the corresponding page on the ever-helpful wiki.  The whole thing is worth a read, make some damn decisions of your own!
  11. Pop out the alternate boot media and reboot.  Sometimes, when emerging from deeply nested sessions, chroots, or alternate boot environments, don't you feel like Cobb waking at the end of Inception?  Anyway, you should be booted into the newly buttery root of your recently installed system now.
  12. Verify integrity and clean up.  I know that shit's boring, yo, but we're gonna do it anyway.
  13. root@serv$ btrfs subvol delete ext_saved
    root@serv$ # allegedly you can verify with btrfs subvol list -d /, but the manpage for the btrfs-tools version pi on Jessie didn't have this documented
    root@serv$ btrfs fi defrag -r /
    root@serv$ btrfs balance start /
  14. Add the secondary drive and partition.  To get the second drive partitioned properly, I simply popped in the second drive and dd'ed the existing disk to the second one.
  15. root@serv$ dd if=/dev/sdSETUPDRIVE of=/dev/sdNEWDRIVE bs=32M # don't fuck this up, mmk?
    This will clone our boot, system and swap partitions to the new drive.  For general applications, I recommend halving each swap.  Even though I never had you modify the swap part of fstab, Linux is smart and will find and use all swap partitions attached to the computer.
  16. Convert to raid1 live!  "Fuck it, we're doing it live."  Yeah, computers are pretty cool I guess.  From here.
  17. root@serv$ btrfs fi show # to see which device is mounted as root
    root@serv$ fdisk -l # to see which device will be added to form our raid1 (aka, which one is NOT root)
    root@serv$ btrfs device add /dev/sdNOTBOOT1 /
    root@serv$ btrfs balance start -dconvert=raid1 -mconvert=raid1 -sconvert=raid1 -f /
    The last command complains if you try to convert system blocks to raid1 as well (-sconvert=raid1), which is why we use the -f flag, which has the ominous manpage description "force reducing of metadata integrity".  But there is no information I could find out there regarding this, and I want to support complete failover, so this is what I'm using and it's working ok for now.
  18. And we're done!  Isn't it great?  Hypothetically, one of our drives can fail and we'll still be able to boot!  I think we might be screwed if the boot partition gives us trouble, but I'm not realy sure yet.
As always, the Arch wiki docs are unparalleled, peruse related info here.  I hope it all worked, drop a line below if something didn't, or if something did!

Tuesday, September 9, 2014

on the mortality of SSDs

One of my servers, lil turbo, was booting from one of those bottom-of-the-barrel ADATA 32GB SSDs.  There are tons of reviews out there saying that these things are little turds, but I was feeling ballsy.  Then, one day, the server wasn't on the network any more.  I went into the closet, where lil turbo lives, to see what was the matter.

One of the non-boot drives was locked in a death grip on the sector it had been reading when it was interrupted, and fractured, seemingly non-Latin characters were bleeding all over the display.  Fuck.

Rebooted, and no dice.  Neither SSD was even seen in POST, not the boot drive and not the one I bought a year ago to mirror the boot drive with.

That was three months ago.

Last week, I decided to take a crack at reviving the comatose lil turbo.  Thinking either the SSD hot swap module or the SATA controller had died, I tried replacing both parts.  Still no dice.

So I started working on something else, and needed a spare 3.5" HDD to test a bus on a different server (vault 101).  So I pulled one of the RAID drives from lil turbo to use.  Then, forgetting that lil turbo was missing a drive, I booted it again, and the SSDs showed up!  However, they didn't boot - the screen came up with "Missing boot drive" or some shit.

I was thinking that the hot swap enclosure must be loose, and the drive was making connection and then loosing it.  But several subsequent boots failed the same way.

Then it hit me.  I grabbed the RAID disk back from vault 101 and inserted it in lil turbo's yawning, empty bay, but not all the way.  Then I went down the front and opened all the hot swap bays for the RAID disks, nine in all, so none of the would be seen or spun up when I next booted lil turbo.

When lil turbo booted, both SSDs were seen, and once it got to grub, I slowly began closing all the RAID drive bays.  Once the system had booted, I issued an mdadm --assemble --verbose /dev/md0 /dev/sd[abcdehijk] and a mount /dev/md0 /mnt/store, and watched the drive lights flicker as my data, marooned for three months, finally came back to me.

* * *

Later I learned that the ADATA was a turd after all - the smart log showed two critical-looking errors from around the time that the server would have crashed.

Next step: turn the root into a btrfs RAID1 and mirror it across both drives, finally!

(Edit: So I ended up trying various things live and borked the install.  Rather than fixing it or restoring from backup, I decided it was time for a fresh start.  Read about how I reinstalled lil turbo to boot from a raid1 btrfs root here.)

Tuesday, May 13, 2014

ipv6 is a pain in the ass: fixing openvpn for tmobile

For those of you following along at home, we recently set up an openvpn server and connected our android phone to it.  Unfortunately, Big Red doesn't offer unlimited data, and while ovpn doesn't really have too bad of a bandwidth overhead, I decided I'd stop playing chicken with the limited data plan and go to tmobile.

The new phone is a goofy-ass samsung note 3, but I think I like it.  Everything went smoothly until I reimported my profile and certificates and tried to connect again - it wouldn't work.  Oh shit, does tmobile block VPN?  A bit of googling led me to this post.  Summarizing: go to Settings -> More Networks -> Mobile Networks -> Access Point Names and change a field in the only apn in the list.  But all the fields were greyed out.  Do I have to root already?

Nope.  The solution is to create another APN, copying in all fields from the first.  For your/my convenience, the relevant fields and the values were:

Name: big bob's fun time apn - suck it tmobile!
MCC: 310
MNC: 260
APN type: default,mms,supl
APN protocol: IPv4/IPv6
Roaming APN protocol: IPv4/IPv6

The last two fields are the important ones - "APN protocol" and "Roaming APN protocol".  If you don't change the roaming field as well, you won't be able to use vpn when roaming.  I don't know what the hell those other fields do, you should use whatever values are in the preset APN on your phone.

The default values are "IPv6 only", but we set up ovpn in tun mode, which does not support IPv6.  Change these APN protocol fields to "IPv4/IPv6" and switch to your new  and you're golden.  Fortunately, the server config, client profile and all certs are still valid.

Yeah, I know IPv4 needs to die, but sometimes we just want our shit to work right and I can't remember a damn IPv6 address other than fe80:: and if you do away with NAT your shit will kind of all be exposed but not really and ...whatever.

Friday, May 2, 2014

keeping that data forever (with btrfs)

So we like Linux, that is established.  And we must have our data forever, that is also established.  But the ability to do this is a new thing.  btrfs is a very new thing, allegedly not supported well before kernel 3.9, but with the ability to do some pretty awesome stuff.

If that interests you, read this article.  I've read it a few times, and it is absolutely making me drool over the possibilites of btrfs.  Failing-ish drives will no longer corrupt files in the mp3 collection you've been curating since high school - an errant cosmic ray can no longer bork that video file from senior week.  With anything but zfs and btrfs (even mdadm or hardware raid) this can happen.

So, even though it's probably ill-advised and guaranteed to lose all the data because it's so new, I built a nice shiny new NAS box to test btrfs out on.  I'll just be doing backups from other personal machines with it, so if it crashes and burns, the worst that will happen is another device fails catastrophically at the same time and my decades of personal data are all lost.  If you use this at your job based solely on these instructions, you're insane.  And I like your style.

Start off by installing Debian jessie.  Or use wheezy and get the newer kernel from backports, but the newer btrfs-tools in jessie isn't backported.  Some of this stuff probably won't work with the older wheezy btrfs-tools.

I bought a couple 3T drives, and I had a couple 1.5T drives laying around, so I figured, why not make it interesting and get another 3T by turning the three spare 1.5T drives into a 3T mdadm raid5?  One of the drives was busy elsewhere today, so we're making the array degraded.

Make the filesystem on all three disks, like this:

$ fdisk /dev/sdd
Command (m for help): n
Partition type:
   p   primary (0 primary, 0 extended, 4 free)
   e   extended
Select (default p): 
Using default response p
Partition number (1-4, default 1): 
Using default value 1
First sector (2048-2930277167, default 2048): 
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-2930277167, default 2930277167): 
Using default value 2930277167

Command (m for help): t
Selected partition 1
Hex code (type L to list codes): fd
Changed system type of partition 1 to fd (Linux raid autodetect)

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

Make the mdadm array, make sure it exists and write the mdadm conf file:

$ mdadm --create /dev/md0 --level=5 --raid-devices=3 /dev/sdd1 /dev/sde1 missing
$ cat /proc/mdstat 
Personalities : [raid6] [raid5] [raid4] 
md0 : active raid5 sde1[1] sdd1[0]
      2930012160 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/2] [UU_]
      bitmap: 0/11 pages [0KB], 65536KB chunk

unused devices: <none>
$ mdadm --examine --scan >> /etc/mdadm/mdadm.conf

Now we make the raid6 btrfs.  I used force because some of these disks had already been formatted, but be careful with that option - it doesn't fail mkfs if there is already an fs on the disk.

$ mkfs.btrfs --data raid6 --metadata raid6 --label thevault --format /dev/sd[abfgh] /dev/md0

I stupidly ran the command on the actual physical console so I can't give output, but it looks like the optional features "extref" and "raid56" are turned on by default when you use mkfs.btrfs this way.

Check for your new fs:

$ btrfs filesystem show
Label: 'root'  uuid: 810ff954-180f-4997-a487-30c561ff3820
Total devices 1 FS bytes used 1.28GiB
devid    1 size 28.35GiB used 5.04GiB path /dev/sdc1

Label: 'thevault'  uuid: 7f407975-6c24-42ec-a9a5-4d1967d89cbe
Total devices 6 FS bytes used 112.00KiB
devid    1 size 2.73TiB used 2.02GiB path /dev/sda
devid    2 size 2.73TiB used 2.00GiB path /dev/sdb
devid    3 size 2.73TiB used 2.00GiB path /dev/sdf
devid    4 size 2.73TiB used 2.00GiB path /dev/sdg
devid    5 size 2.73TiB used 2.00GiB path /dev/sdh
devid    6 size 2.73TiB used 2.00GiB path /dev/md0

Btrfs v3.14.1

There's my boot disk and new storage array!  I'm guessing the already-used space is the disk metadata?

Now mount it so we can df it.  You can use any component drive as the source of the mount command and btrfs mount will automatically find the other members of the filesystem.  I'm using the md block device becuase it's easiest to remember here:

$ mkdir /mnt/thevault && mount /dev/md0 /mnt/thevault
$ btrfs filesystem df /mnt/thevault
Data, single: total=8.00MiB, used=0.00
Data, RAID6: total=4.00GiB, used=2.00MiB
System, single: total=4.00MiB, used=0.00
System, RAID6: total=10.50MiB, used=16.00KiB
Metadata, single: total=8.00MiB, used=0.00
Metadata, RAID6: total=4.00GiB, used=112.00KiB

We have metadata, outstanding.  Regular df appears to work too:

$ df
Filesystem       1K-blocks    Used   Available Use% Mounted on
/dev/sdc1         29729792 1424256    26350720   6% /
udev                 10240       0       10240   0% /dev
tmpfs               828964     416      828548   1% /run
tmpfs                 5120       0        5120   0% /run/lock
tmpfs              1964900       0     1964900   0% /run/shm
/dev/sda       17581345080    2176 17572912576   1% /mnt/thevault

So it looks like it works.  Time to make some samba shares and copy in a ton of stuff.  I'll post back later if there are any problems, or just to mention how it goes after some time has elapsed.

Sunday, April 27, 2014

owncloud 5->6 migration

You had some old owncloud VM kicking around for awhile.  You want to save the data but trash the VM and start over - you're not sure how you installed owncloud any more and you just don't want to deal with old VM baggage.  Or maybe your VM took a shit and all you have are the data and config files from the old release.

Either way, here's how to take some old owncloud 5 data files and get them operational with a new owncloud 6 install on debian jessie:

(From here on out our backups are assumed to be in the directory in the $ocbak variable.  Before running these commands just export that to be your actual backup location:

slice@own $ export ocbak='/mnt/ocbak'

Note also that you may be able to omit some of these steps, but this is the way I did it and it worked.

Also note that this guide doesn't cover apache config at all; a standard owncloud config should work fine, and make sure everything under /var/www can be served.)

Installation the first

First you need to find out what old owncloud version you were running.  Try running this command if you don't care:

slice@own $ wget`cat $ocbak/config/config.php | grep "'version'" | cut -d'=' -f2 | sed "s/.*'\\(.*\\)'.*/\\1/g"`.tar.bz2

That may fail - if so, take a look at the config file to get your version number.

slice@own $ vi $ocbak/config/config.php

Somewhere you'll see a line that looks like this:

'version' => '5.0.14',

Hit google for the download link for that tar file, or just do this:

slice@own $ wget
slice@own $ tar xjf owncloud-5.0.14a.tar.bz2
slice@own $ mv owncloud /var/www/owncloud-5
slice@own $ cp -rfv $ocbak/* /var/www/owncloud-5
slice@own $ chown -Rc www-data:www-data owncloud-5

Edit the config file (/var/www/owncloud-5/config/config.php) to point to new data dir  (/var/www/owncloud-5/data).  Then restart apache and log in to the site and make sure everything is good.

slice@own $ /etc/init.d/apache2 restart

Installation the second

Now we basically do the same thing over again with owncloud 6.  (Right now this is the latest - in the far future you might want to go to version 6 before going to whatever the current version in your unimaginable future time is.)

slice@own $ wget
slice@own $ tar xjf owncloud-latest.tar.bz2
slice@own $ mv owncloud /var/www/owncloud-6
slice@own $ cp -rfv /var/www/owncloud-5/data /var/www/owncloud-5/config /var/www/owncloud-6/
slice@own $ chown -Rc www-data:www-data owncloud-6

Edit the config file (/var/www/owncloud-6/config/config.php) to point to new data dir  (/var/www/owncloud-6/data).  Then restart apache and log in to the site and make sure everything is good.

slice@own $ /etc/init.d/apache2 restart

Installation the third (?)

You now have a working owncloud 6 setup, so you could just stop here.  However, a little extra work now should make future upgrades as simple as a aptitude update && aptitude upgrade.

Just to make sure some old configs don't get in the way, we'll try to purge owncloud before we install:

slice@own $ aptitude update
slice@own $ aptitude purge owncloud
slice@own $ aptitude install owncloud
slice@own $ cp -fv /var/www/owncloud-6/config/* /etc/owncloud/
slice@own $ cp -rfv /var/www/owncloud-6/data /usr/share/owncloud/
slice@own $ chown -Rc www-data:www-data /usr/share/owncloud/data /etc/owncloud/

Edit the config file (/etc/owncloud/config.php) to point to new data dir  (/usr/share/owncloud/data).  Then restart apache and log in to the site and make sure everything is good.

slice@own $ /etc/init.d/apache2 restart


If everything worked ok, we can get rid of the intermediate installations and the tar files:

slice@own $ rm -rfv owncloud-5.0.14a.tar.bz2 owncloud-latest.tar.bz2 /var/www/owncloud-5 /var/www/owncloud-6
slice@own $ /etc/init.d/apache2 restart

I'm pretty sure some steps could be skipped - you may be able to skip both manual installations and just drop your old data files in the debian install directories.  If you're brave, try it and leave a comment.  But this way certainly works.

Friday, April 25, 2014

what exchange server version is my company using?

You have to use an Exchange server someone else administers.  You want to configure some other piece of software to talk to it, but all you have is your outlook web access page.  The other piece of software depends on the Exchange version.  Read on:

Pull up OWA.  In the relatively recent version I'm using, which does not list a year, the menu choices are Options->About, and then you look at the value for the "Version" field.  How to get to "About" may differ for you, but just click around a bit - OWA doesn't usually have a lot of menus, so it shouldn't be too hard to find.

Once you have your version string (mine was 14.1.438.0, but yours may differ), compare it to the following table to find the actual product version:

Product nameBuild numberDate
 Microsoft Exchange Server 2003 6.5.6944233 03
 Microsoft Exchange Server 2003 SP1 6.5.7226 5/25/2004
 Microsoft Exchange Server 2003 SP2 6.5.7638 10/19/2005
 Microsoft Exchange Server 2007 8.0.685.24 12/9/2006
 Microsoft Exchange Server 2007 8.0.685.25 12/9/2006
 Microsoft Exchange Server 2007 SP1 11/29/2007
 Microsoft Exchange Server 2007 SP2 8/24/2009
 Microsoft Exchange Server 2007 SP3 6/20/2010
 Microsoft Exchange Server 2010 14.0.639.21 11/9/2009
 Microsoft Exchange Server 2010 SP1 8/24/2010
 Microsoft Exchange Server 2010 SP2 12/4/2011
 Microsoft Exchange Server 2010 SP3 2/12/2013
 Microsoft Exchange Server 2013 15.0.516.32 10/11/2012

(table from Technet)

So the version I'm using is 2010 SP1 with some patches applied.

Good luck; with Exchange interoperability, you'll need it.

Wednesday, April 23, 2014

running an openvpn server from home

After beating down my sysadmin ego by pouring days into alternate setups, I finally got an OpenVPN server running exactly as I wanted it on my home network.  If you need complete(-ish) local network access from remote devices, want to run it in a virtual environment, and only want to expose the single VPN port, here's how to do it.  Note that some of these steps assume you're using a VM server on the LAN running qemu-kvm, and your local machine connects to it with virt-manager.

    1. Download the i386 LiveCD w/ Installer pfsense image to your server.

root@server $ wget
root@server $ mv download.php? /var/lib/libvirt/images/pfSense-LiveCD-2.1.2-RELEASE-i386.iso.gz

    2. Spin up the VM with pfsense.

lol@workstation $ virt-manager &

Connect to the server and make a new VM.  Choose local install media -> iso image -> choose the pfsense image.  OS type=UNIX, Version should autodetect to FreeBSD 8.x.  512MB RAM and a single core has been plenty for me, if my host gets strapped I'll probably drop the pfsense guest down to 256 and see how it does.  5GB hard disk is more than enough also.

Once the VM comes up, choosing all defaults is fine for installation.  Once the ncurses install is complete, the configurator will ask for the primary network interface, or try to do some kind of autodetection.  We created the VM with the default single interface because there is upstream DHCP coming from the home router and we're only using pfsense for the OpenVPN features, so the answer to the interface question is probably "em0".  Choose sensible options for everything else.

Note that after you choose what hostname pfsense will tell the upstream DHCP server, you will likely get a new IP after reboot, so check your DHCP logs for what the new IP is.

    3. Use this youtube video to configure the OpenVPN server.

I think I did everything exactly the same as the guy in the video, except I chose max key lengths, a longer AES cipher, and 9999 days for everything.  Stop when he gets to the client export/configuration part.

Make sure the "Local network" option is set to your local subnet, and the IPv4 tunnel network" option is NOT set to your local subnet.  Not sure why the guy in the video used; I chose since it's also designated for private networks and it's way more visible at a glance.

    4. Open the port on your router.

My router (a D-Link DIR-655) has a "virtual servers" option, which is like port forwarding on steroids.  Either way, this part is simple - make sure UDP port 1194 is open and points to your pfsense server.  It's probably a good idea to make sure the pfsense VM has a static IP at this point too.

    5. Set up dynamic DNS pointing at your house.

pfsense comes with clients for many free dynamic DNS services - I'm using  Make an account and grab some trashy domain.  In pfsense, go to services -> dynamic DNS and add your account.  Super simple, now your no-ip subdomain will always point at your house no matter how often your ISP changes your WAN IP.

    6. Make the Android client work.

The OpenVPN Connect app worked on the first try for me, and it's free, small, has few permissions and has a clean interface.  Plug your phone into the computer you're checking out pfsense on, then in the pfsense web UI go to VPN -> OpenVPN -> Client Export.  This will only be there if you installed the "export" package from the youtube video.

Make sure to choose the DynDNS option for your domain under Host name resolution.  If it isn't there for some reason, or if you don't want to run the dynamic DNS client on pfsense or whatever, just choose "other" and enter the domain manually there.

At the bottom, under the "Client Install Packages" section, click the "OpenVPN Connect (iOS/Android)" link.  The downloaded .ovpn file has "iOS" in the name, but don't worry about that.  Copy that file to your phone's storage (SD or onboard).  Pull up the OpenVPN app, choose Menu (the vertical ellipsis (...) button) ->Import Profile from SD card -> enter your username and password, and boom!  It should just work.

EDIT FOR T-MOBILE USERS: It just worked when I was on Verizon, because they're a lumbering behemoth and are still using IPv4 everyplace - if you're on T-Mobile, you'll need to follow a substep that I wrote up here.

    7. Make the Linux (NetworkManager) client work.

From the same client export page that we used in the last step, under the "Client Install Packages" section, click the "Standard Configurations: Archive" link.  Unzip the downloaded file somewhere in your home directory - default permissions were not cool, so take away everything but rx on the dir for owner, and just r for the owner on the files inside:

lol@vpnclient $ mv downloads/ vpn-stuff/
lol@vpnclient $ unzip
lol@vpnclient $ sudo chmod -Rc a-rwx vpn-files
lol@vpnclient $ sudo chmod -Rc u+rx vpn-files
lol@vpnclient $ sudo chmod -c u-x vpn-files/*

(The chmod steps in the above are optional.  I know it could have been done more succinctly, but I don't typically use masks for chmod - of course I'm familiar with 0777, 0655 etc, but for non-standard permissions sets they seem obfuscating.  In ten years they probably won't, but I'm just not there yet.)

Click Network Manager icon -> VPN connections -> Configure VPN -> Import -> Choose the unzipped .ovpn file.  Everything except username and the two password boxes should be filled in, enter the user credentials you chose on pfsense and enter the same password again for the "private key password".  The three file pickers were all pointed at the same .p12 file - this is ok.

Remember that weird stuff might happen if you try to connect to the VPN while the client is already attached to the network you're attaching to.  To test, I just shut down the Android VPN client, created a wifi hotspot with my phone, attached to that with my laptop, and then connected just the laptop to the home VPN.

Hope it works for you!

Thursday, April 17, 2014

fun times with the ASRock Rack C2550D4I

An Intel Atom board with 4 cores, passively cooled, ~20 watt TDP, with 12 onboard SATA, one PCI-E 8x, and serious management features?  What's not to like?  This is a perfect board for a NAS build.

So, I got one.  Everything looked cool - it posted with no hassle, I went through and tweaked the BIOS for the heck of it, and then hit a brick wall when trying to install an operating system.

The thing would not boot from USB.  The motherboard only has one USB controller but can only do three ports - they probably used up the other one internally for something or other, maybe the DRAC stuff, this board has a lot of connectivity.  A jumper controls whether you only get one port on the rear or one port through the header.  But I tried all the ports - the devices showed up in the boot menu, but would not boot.  USB DVD drive, multiple flash drives, a USB Micro SD card adapter.  All of these things showed up in the boot menu, but when I chose them there was no read indicator on the device and the screen showed just a lonely, infinitely blinking underscore in the top left corner.

So I went back to read the newegg reviews to see if anyone could help.  The first reviewer was talking all "console redirection" and "remote media" and "you have to install Java".  So I realized I had to use the weird dedicated third Ethernet port to install the OS.  Seems convenient.

Buuuuut... it wasn't.  Here's what you have to do to install an OS ISO via the web management interface (also known as a "DRAC"):

1. Find it

Basically trivial - the C255 Drac uses DHCP so it'll be on your subnet no matter how it's configured.  You can grab the IP from the BIOS (Server Mgmt -> BMC network configuration) or from your DHCP log.  Conveniently, Asrock decided not to give this interface a name, but it has Asrock's OUI (first half of the MAC), which is bc:5f:f4, so you should be able to narrow it down from there.  Or set a reservation from the BIOS, or any other fucking thing you'd like to do.

2. Access it

Go to the IP for the Drac page with a web browser.  This may be the worst embedded webpage I've ever seen.  It almost has it all - you have to accept a self-signed (or otherwise invalid) certificate for the page, install a recent Java (v7), add this certificate as an exception in Java, allow popups, and either allow all downloads automatically or widen popups and accept executable downloads (JNLP) any time you want to do something.  Let's go over that in more detail:
  • Accept the certificate:
This was much easier in Firefox - you just had to access the site via HTTPS and click the correct buttons a couple times.  In Chrome, this is what I had to do: go to the site via https by prepending "https://" to the IP, click the red broken evil lock icon to the left of the URL, then click Certificate Information -> Details tab -> Copy to File button -> Next -> choose option "Cryptographic Message Syntax Standard - PKCS #7 Certificates (.P7B)" -> choose a filename.

Then you have to import this certificate file: Chrome Menu -> Settings -> Show Advanced -> Manage certificates button under the HTTPS/SSL heading -> Import -> Next -> Choose the file we just exported.
  • Install Java:
I'm sure you can figure this out., make sure it's at least Java 7.  Don't forget to untick the "turn my computer into a turd" box(es).
  • Allow Java to accept the site's bullshit:
Trying to use most any feature of the site will cause some supposedly friendly application to start shouting boxes at you, not identifying itself and talking all this trash about the Megarac SP and the folks that vouched for it.  Eventually I realized this is Java itself.  Fix it (on Windows) by going to Start Menu -> Java -> Configure Java -> Security tab -> Edit Site List button -> type your Megarac IP prefixed with "https://" in the weird box under the word "Location" -> click Add.
  • Allow popups:
When you go to the management page in Chrome, to the right of the address bar there's a little angry icon.  Click this and tick all the little things that you would never tick if you weren't forced to.

What the fuck, right?

3. Do it

Now you're in and the site will allegedly work for some features some of the time.  There's a gazillion options, but if you're trying to install an OS to that mother(board), you want Remote Control -> Console Redirection -> Java Console button.  This pops up a popup that you'll need to expand so you can click that you want to execute the binary file from unknown internet sources, and then you have to manually execute that binary file yourself by clicking it.  That spawns a popup which may itself have had a litter of popups - a grandpopup - and the whole family comes at you.   Just tell them all that you're fine, you don't need your windows washed, don't look them in the eyes, give them a fiver to go away or whatever.  Eventually, you get to something pretty cool, which for me is a Java app displaying the BIOS screen over my network.

Five seconds later, this effect wears off.  You realize that this feature would be cool if the server itself were downstairs, or in Pakistan, but when you have the board sitting in an open case right next to you, why do you have to use some of the shittiest software in the world to do it?  Why did those poor developers have to go to all the effort to build this?  It took them so much work they didn't even have time to replace the default Java Swing/AWT/IDGAF skin.  Furthermore, if I wanted my server in Pakistan, I would have gone there and paid like five dollars for it.

Now you click the Media menu option and the only suboption, Virtual Media Wizard.  It isn't a wizard, but that's ok because this is just about the simplest thing we've had to do this whole time.  Browse for your install ISO, connect the virtual DVD drive if you need to, and close the box.  Choose the Power menu option and reboot the server.

When it comes back, hit F11 to choose the boot device, and lo and behold, there's the virtual CDROM.  Choose it.  Install your operating system, and enjoy.

It would be worth it if your server were located in Pakistan.

P.S.: It turned out my failure to boot issue stopped happening when I pulled out the HighPoint RocketRaid 2680SGL that had been in the PCI slot.  Still working on that one, but it's cool that I'll be able to reinstall my operating system or reboot my server later without going downstairs.

P.S.S.: The RocketRaid needed to be flashed to the latest BIOS.  This card claims to have support on Windows, Linux, and Mac, but the flash utility only worked on Windows - extensive searching revealed a Linux version, but this never seemed to see the card.  Fortunately I had a Windows machine, otherwise that motherfucker would have been returned.  When flashing, you need to turn off the "INT13" option, but this didn't really do the trick either.  Finally, I found some sort of "disable option ROM" option in the motherboard BIOS.  Now I don't get to see the status of my attached drives when booting, but that's ok because I'm not using this thing for hardware RAID anyway.  But if you want to do hardware RAID for some reason, watch out - the RocketRaid was damn finnicky, and if I had to do it over again I would have shelled the extra $50 out for the low-tier eight-port LSI card.