Finally Gave RouterBoard a Try

I’ve been wanting to try out a RouterBoard for a long while. I ran into them many moons ago in Linux Journal and just could never quite justify it until now. RouterBoards are, as the name implies, routers which run a variant of Linux (RouterOS which is also available on its own). You can either build your own router by assembling their semi-modular components or grab an off the shelf solution. You can sort of think of them as a router running DD-WRT, but on steroids. In any case, I finally took the plunge and bought a CRS-109. I’m still tweaking things, but wow it’s a glorious piece of hardware!

At least for my use-case. The 109 has plenty of power to keep up with my U-Verse FTTP 45/5 connection. It would not be the best choice for trying to keep up with something like Google Fiber, though RouterBoard has solutions for those cases. I didn’t need that sort of horsepower, and the 109 was very affordable as a result. The 109 is powered by a MIPS-based Atheros AR9344, which is a platform that is used by other consumer routers. It adds to this a dedicated switch type which is capable of some QoS, VLANs, and other fun things. In my setup, it replaced my Apple Time Capsule and D-Link GigE router and, in doing so, greatly improved by network setup.

RouterOS supports IPv6 in various incarnations. It took a lot of work, but I do have IPv6 via ATT (using their tunnel, but I don’t have to fuss with that) and setup my 109 to handle most of my IPv6 delegation. I have a basic set of iptables rules for both IPv4 and IPv6 along with the switch-based QoS and software QoS running on the WAN port. The rest of the switch ports are all part of my (currently) single segment LAN and also died to the WiFi. Since the 109 has a dedicated switch chip, communicating on the LAN between the ports is at wire speed and doesn’t burden the main CPU of the router. Instead, the CPU is focused on the firewall, QoS, and some other periphery services like DNS, DHCP, NTP, etc.. I also have SNMP setup, which I’m using via Cacti running on my file-server to graph switch port and CPU usage.

I have to say, QoS is glorious. For outbound from the WAN I’m using the ‘default-small’ queue, and for incoming I’m using ‘RED Default’. The latter is a bit more difficult to test at the moment, though so far, so good. Uploads, however, are dramatically improved. I can upload a file via SSH while also using SSH for interactive sessions and hardly notice any impact. Even when I rolled my own QoS directly via iptables I couldn’t get SSH to work under that use case. Likewise, the impact to gaming is minimal when uploading a file as well.

Long story short, I love this box! I’m a believer! As a Linux sysadmin and somewhat of a network admin, I love this thing. It does what I need, gives me plenty of info to make informed decisions, runs well, wasn’t expensive. This thing isn’t for your average consumer that just wants to plug in a magic box and have it work. For me, though? It’s just about perfect!

Posted in Computers | Tagged | Comments Off on Finally Gave RouterBoard a Try

Fully online RAID1 to RAID10 migration using Btrfs

I have actually been running Btrfs on our home NAS for a while. At first I was sort of turned off by the name and a bit by the folks behind if. But a nice article over at Ars Technica convinced me to give it a go. Previously, I was using MDRAID + LVM + XFS, a setup that is still solid. Btrfs can replace that entire stack but really only if you use Btrfs exclusively (a least on that particular set of drives). In some cases, that is not ideal – I would say MySQL on XFS is probably still the best fit there. In multi-filesystem cases, Btrfs can live happily on MDRAID and/or LVM, but one will miss some of the really neat features that way.

Regardless of those cases, Btrfs seemed like soemthing worthy of trying out on my NAS where data integrity is rather important. Setting it up is definitely different. I won’t cover that here (the Btrfs wiki linked above should get most folks going), but it is easy if a tad bit alien. Instead, I thought I’d share my experience with going from a RAID1 of 2x 1TB drives to a monster (for me) RAID10 of 4x 4TB drives. I had been wanting to do that for a while, particularly since the new line of consumer NAS drives came out. I opted to go with Seagate’s NAS drives, having been a long time Seagate fan. Other hard drive vendors make similar products these days.

As a bit of an aside, while I opted for a RAID10, Btrfs has a pretty neat approach to mirroring – Btrfs will generally figure out the most optimum way to store the data while keeping two copies of the data across different drives. That means configurations of more than 2 drives, or even drives of different sizes, are possible while having a balance between data integrity and space usage. I didn’t go with this option because I had 4 of the exact same drive, and my hunch is that RAID10 will still be faster (it usually is on classic RAID).

So how did I make all this happen? I started on a healthy RAID1 setup already so the first step was to add 2 4TB drives and setup a mirror. I only have 4 bays available so I wasn’t able to just chuck all 4 of the new drives in, although conceivably that may have removed a few steps if I could have. My system supports SATA hotplug devices, so it was just a matter of putting the drives in caddies and shoving them in, making sure the system saw the new drives, and then adding them to btrfs. In brief that was, more or less the following:

# echo "- - -" > /sys/class/scsi_host/host0/scan
# partprobe
# mount /mnt/btrfs
# btrfs device add /dev/sdd /mnt/btrfs
# btrfs device add /dev/sde /mnt/btrfs
# btrfs filesystem df /mnt/btrfs/

Note that host0 may be different depending on your configuration (I actually had two buses in my case). You can figure that out by doing an ls -l /sys/class/scsi_host/. For those not familiar with Btrfs subvoluming, I’m using that extensively but normally do not keep the root btrfs file-system mounted (as there is no reason to during normal operations). Also note that I’m not partitioning the drives, though that certainly can be done if preferred. I got in the habit of not doing that when dealing with LVM and SSDs, where cluster alignment was a headache when dealing with old style partitions. It just seemed easier to use the entire drive unless I have a need to partition. Btrfs makes that need less since it supports subvolumes anyway. Anyways, after doing the above, the next step was to remove the old drives.

That is where I ran into a minor snag. Turns out, removing drives requires mounting the file-system in degraded mode. Otherwise you get a somewhat cryptic error message. So removing the drives required:

# mount -oremount,degraded /mnt/btrfs
# btrfs device delete /dev/sdb
# btrfs filesystem df /mnt/btrfs/
(wait for migration to complete)
# btrfs device delete /dev/sdc

According to the documentation, you can remove more than one device at once, so I could have removed both 1TB drives. In either case, btrfs will migrate the data over to the other drives for you. In my case, I opted to do it one at a time. Once that was done, I yanked the old drives out, added the new 4TB drives in and then converted btrfs over to a RAID10:

# echo "- - -" > /sys/class/scsi_host/host0/scan
# partprobe
# btrfs device add /dev/sdb /dev/sdc
# btrfs balance start -dconvert=raid10 -mconvert=raid10 /mnt/btrfs

I didn’t think to grab some screenshots of some of the btrfs output of the previous steps, but I do have what the conversion to RAID10 looks like:

# btrfs filesystem df /mnt/btrfs/
Data, RAID10: total=220.00GiB, used=109.49GiB
Data, RAID1: total=818.00GiB, used=817.99GiB
System, RAID10: total=64.00MiB, used=148.00KiB
Metadata, RAID10: total=1.00GiB, used=294.68MiB
Metadata, RAID1: total=2.00GiB, used=1.54GiB

Once that is done, I’ll have around 8TB (not accounting for compression) of usable space! A minor quirk with Btrfs is that conventional tools like df don’t give you the whole story, so df actually looks more impressive than it is:

[email protected]:~# df -hT /mnt/btrfs/
Filesystem     Type   Size  Used Avail Use% Mounted on
/dev/sdc       btrfs   15T  1.9T   12T  14% /mnt/btrfs

This is both a good and bad thing. Given the way Btrfs works, alongside compression and de-duplication, providing a simple answer to how much space is available is no longer quite so trivial. In my opinion, that, plus the other minor quirks of Btrfs, are totally worth it given how easy it was to convert, without any downtime, from a RAID1 to RAID10. I could have done the same thing using MDRAID + LVM, but it would have involved more steps. I don’t think Btrfs is a one-size-fits-all sort of file-system – at least not yet, but for my NAS it has done a pretty good job and I’ve been quite happy with it!

All that said, it is still experimental, though, and I take backups of my important data (in my case to an XFS file-system running on an ioSafe), but *knock on wood*, it has served me very well so far!

Posted in Computers | Tagged | Comments Off on Fully online RAID1 to RAID10 migration using Btrfs

Watched the New Cosmos

And it was absolutely glorious! Really hope it was a ratings success as well as this show deserves to be on TV and on prime-time. So good!

Posted in Ramblings | Comments Off on Watched the New Cosmos

Merry Christmas!

Christmas was lovely this year. The usual material possessions were exchanged with some nice surprises. My mom gave me a dive knife, for instance, which was a total surprise! Likewise, Camden got some neat things.

But I think the thing I will remember the most is what Camden said when we finally buried Chaucer’s ashes. Yeah, I know, doing it on Christmas might seem odd, but that’s not important. When Paul asked if anyone wanted to say anything, Camden spoke up. He said “I love you, Chaucer…”

That was very special.

Posted in Uncategorized | Tagged , | Comments Off on Merry Christmas!

Chaucer, 1997-2013


I can’t imagine there ever being a dog quite like Chaucer. He’s had a good life and certainly made mine ever so special. I’m gonna miss you good buddy. See you again someday.

Posted in Personal | Tagged | Comments Off on Chaucer, 1997-2013

Love at First Sight, Live at Artslam!

A song about the hero, early in his leveling up, falling in love with the princess. Performed live at Artslam!

Posted in artslam!, Music, Songs | Tagged | Comments Off on Love at First Sight, Live at Artslam!

Hurt, Live at Artslam!

Our cover of Hurt performed live at Artslam! with a cleaned up mix. The mix isn’t perfect but it certainly has a live sound to it! Perfection is boring, anyway.

Posted in artslam!, Music, Songs | Tagged | Comments Off on Hurt, Live at Artslam!

Killin’ Slimes, Live at Artslam!

Well, the live mix wasn’t so great, but we were able to get the multitrack at least and fix things up a bit. So if you missed Artslam! (you missed out!), or just wanna hear it again, here you go! Regardless of the not so great mix, we had a blast and seems like most everyone else did too!

Posted in artslam!, Live, Music, Songs, White Rabbit | Tagged | Comments Off on Killin’ Slimes, Live at Artslam!

Cassette Culture

Hero Underground On Cassette

I didn’t understand it quite at first. In fact, I still don’t think I have it fully figured out. But cassettes are making a bit of a comeback these days. I call it the “poor man’s analog.” Cassettes are cheap to make, durable, and actually sound pretty damn good with a good deck. They are aiming for making music something more than just an under-appreciated modern commodity. I initially wrote them off as the next hipster thing. And that may be the case still, actually, but it won’t stop us from trying it anyway.

If you have been keeping up with our Facebook page, you know we have been playing around with the idea of doing some limited releases on cassette. Things like demos, live shows, practices, among other things. Some of that stuff we may release digitally, but some we may not, or at least do so in very limited capacity.

Why? A few reasons, but mostly we wanted something fun to give away at live shows that was more fun than just a download code or a CD. A CD is arguably more irrelevant than cassette because, as a digital format, it is superior perhaps to an MP3, but inferior to the high quality options we can provide via Bandcamp and other means. Most folks rip ‘em to MP3s and stow them away anyway.

A cassette? Well, it does have an obvious analog sound and, it, unlike a CD, is something you basically have to listen to from start to finish. Given that our next album is set to a story-driven landscape, that is something we really like. It is rather unique too, especially if we put limited release content on it, and is rather inexpensive to make.

Obviously, we want our fans to be able to listen to our music in as many ways as possible. Our future digital and vinyl release plans haven’t changed. We have simply added more avenues for content that our more devoted fans might appreciate as both a unique collectable and a different musical experience. If it is even remotely successful, we may start offering our official albums on tape too.

In fact, I noticed recently that our latest Chipsurf single, Space Hatchet (which we hope to release later this week), actually sounds better when bouncing out to chrome tape and back. As a result, we thought about creating a special edition of Chipsurf Pipeline on tape that includes that song. Tape-haters will still be able to grab Space Hatchet as a single in the usual places, though. If we think we can recoup at least some costs, we may look at professional duplication too, though for now, we’re are mostly doing these types of things “DIY style.”

Our next album, likewise, I hope to be able to record to reel-to-reel if we can make that at all possible. That means vinyl, and if we decide to, cassette, releases will be fully analog. I find that highly cool and also a bit ironic (given the lo-fi digital synths we use).

As a listener, I have been exploring the whole cassette culture thing and not only have I found some neat music, the culture has also been both friendly and rewarding. It is not for everyone but, hey, if you have a tape deck that still works, or can find one easily, it might be worth giving it a go.

Posted in cassette, Music, Ramblings | Tagged | Comments Off on Cassette Culture

First Show Complete, Next Steps

Last Saturday we performed to an audience the first time ever! A huge thanks to 10bitWorks and everyone involved. A special shout-out to Nick who was up incredibly late working around a last minute issue for the chiptune workshop. The workshop was a fantastic success! Likewise, we feel pretty good about our show. We hoped folks enjoyed it as much as we did!

Missed our first show and feel sad? There will be more! In the meantime, we are trying to figure out the best way to put the recording up while keeping it a bit limited. We want people to come out to the shows, after-all! Listening at home is boring! The current plan is to throw the show up for download for an extremely limited time – maybe even less than a day. So fans that really want it can get it, but slackers will just have to wait or come out to a show!

Now that we have a show under our belts, it’s time for the next steps! We are planning on being at Classic Gaming Fest in Austin with TX Chip, although details for that haven’t quite been ironed out. In the interim, if anyone has suggestions on a place to play, let us know on our Facebook page!

Posted in hero, Music, News | Tagged | Comments Off on First Show Complete, Next Steps