File server, using Raspberry Pi as the controller

This weekend’s task is to convert an ageing and awkward file server into something more manageable. I’ll do this by basically replacing the current controller (an old laptop) with a Raspberry Pi, and shoving everything into a box.

Firstly, an apology: I’m a hacker, in that when I create something, it will work, but it may not be pretty. I’ll leave the works of art to others, and focus on what I’m good at: solving the current problem with what I currently have at hand.

Here is the old system:

IMAG1051

Four hard-drives, a laptop, a USB hub, lots of cables, lots of power supplies.

The goal is to cut it all down into just one box, one power cable, one ethernet cable.

IMAG1148

So, first, I got an old ATX power supply from a machine I had lying around, and converted it so it didn’t require a motherboard to run.

IMAG1038

To do this, open up the power supply box, remove any cables (leading to outside the box) that are not red, orange, black or yellow, and connect the green cable to one of the blacks (remove some of the insulation from the black cable first, obviously). The green cable tells the power supply that it’s okay to turn on.

IMAG1048

While in there, I cleaned out some of the dust and fluff, and neatened the cables a little. There are four voltages that come from the power supply unit: 0v (black), 3.3v (orange), 5v (red), 12v (yellow). I wasn’t sure how many of each of these cables are actually needed to supply enough current, so when I was pruning cables from the ATX box, I just left these all intact.

Next, I laid out everything in a way that I thought made sense:

IMAG1056

If the cables are removed, then the above will fit into a box that’s 27cm x 27cm x 18cm, with a few millimetres spare on sides and top, and a few centimetres at the back. We need the space at the back so we can re-organise the cables.

So, I cut out the pieces for the box walls. The wood I had (which I scavenged from the attic “floor”) is 1.2cm in width, so the panels I cut are:

  • two 27cm x 27cm, for bottom and top of the box
  • two 27cm x 15.6cm, for front and back
  • two 24.6cm x 15.6cm, for side walls

The last two, I left a little shy of 15.6. Partly because of ventilation, but mostly because I forgot the old adage to measure twice before cutting.

I put the floors and walls of the box together, with wood glue and screws.

IMAG1060

Next, I needed to make the power rails. After a little thought, I decided to use two coat hangers (yes, seriously!), straightened then cut into four lengths of 27cm each. The resistance of each length was between .5 and .7 ohm, so I was pretty sure they’d do fine.

IMAG1059

I drilled holes at the back end of the side walls and put the rails in. I’ve been told since building the thing that this setup may cause electro-magnetic interference – especially as the rails look a bit like aerials. If it causes a problem, I think I’ll need to coat the box with metal.

IMAG1061

Next, I started wiring up the PSU. First, I placed the PSU in the box and measured roughly how long the cables should be, then I cut them about 3cm longer than that, and stripped the extra 3cm of each wire.

IMAG1062

I then tied each wire to a rail making sure the stripped area of the wire was in full contact with the rail. Orange (3v) at the bottom as it’s unlikely to be used, then red (5v), black (earth) and yellow (12v) at the top. In retrospect, the 12v and 5v should be the other way around for neatness, but I didn’t think of that at the time.

IMAG1063

Most hard drive enclosures use a 12v input, so what I did next was to take the hard drive power supplies, snip the cables so they were about 15cm long from the hard-drive end, then strip the ends and wire them up. The cables were wrapped in black plastic, the “live” wire having a broken white line printed along it. I connected the live wire to the 12v rail and the other to the 0v.

IMAG1064

Next, I took the case off the USB hub, and screwed it and the Raspberry Pi to the underside of the case’s box.

IMAG1066

The power supply for the USB hub is 5v, and when I stripped it, I found this was indicated by the live wire (5v) being red. I left this cable longish (about 30cm), so I can hinge the top of the case and open it without disconnecting anything.

Power for the Raspberry Pi is supplied through a micro-USB cable. I scrounged one from somewhere in the house and stripped it down to about 30cm in length. The cable has four wires – black, red, green, white. We only need the red (5v) and black (0v).

Finally, I inserted all the hard-drives with USB cables connected, and hooked it all up. I’ll need to get shorter USB cables to neaten it further, but the original cables will do for now.

IMAG1148

Software-wise, I used Raspbian for the operating system, and ZFS for connecting the drives together and serving to the network as one large hard-drive.

networking a raspberry pi through your laptop

I finally got my own Raspberry Pi; a credit-card-sized computer that’s very cheap and low-power.

It didn’t come with any of the niceties that you would expect from another computer, such as a power supply or a case, or keyboard, monitor, or anything else. Basically, it’s like being given the motherboard of a desktop computer and you need to do the rest yourself.

So first thing was to install the operating system on it. This was easy. Just buy an SD card, and download the ISO of the OS that you want and copy it onto the card.

I already had a micro-SD card from an old Bada phone, so I just stuck that in an adaptor to bring it up to SD card size, then installed the Fedora remix using the Fedora Arm Installer. Painless.

Raspberry Pi connected to laptop using cross-over cable

Next, we need to connect the machine to the network.

My network is mostly WiFi-based, so I chose to hook the RaspPi to the network by piping its network through my laptop.

We need to set up the laptop so it can hand out IP addresses. Install a DHCP server on your laptop. I’m using Fedora, so installed with “yum install dhcp”, then edited /etc/dhcp/dhcpd.conf:

[code]
option domain-name "example.org";
option domain-name-servers ns1.example.org, ns2.example.org;
default-lease-time 600;
max-lease-time 7200;
log-facility local7;
subnet 10.5.5.0 netmask 255.255.255.224 {
range 10.5.5.26 10.5.5.30;
option domain-name-servers ns1.internal.example.org;
option domain-name "internal.example.org";
option routers 10.5.5.1;
option broadcast-address 10.5.5.31;
default-lease-time 600;
max-lease-time 7200;
}
[/code]

Then start the DHCP server with “service dhcpd start”.

Next, we need to connect the laptop to the RaspPi. To do this, I made a cross-over cable, and plugged it into the RaspPi.

Before plugging it into the laptop, we need to tell the laptop’s network manager not to set up DHCP over eth0 (as we’re the server, not the client, as far as the cable is concerned). To do this in Gnome, right-click your netowkr icon on the top-right, click Network Settings, and in Wired, click Options, then change the type to “Shared connection” (or whatever sounds like that).

Now plugin the ethernet cable into the laptop, then plug a USB cable into the laptop and the RaspPi.

If you “tail -f /var/log/messages”, you should get something like the below after a minute:

[code]
Sep 28 21:12:30 iga dnsmasq-dhcp[30147]: DHCPDISCOVER(em1) 192.168.1.19 b8:27:eb:87:1d:86
Sep 28 21:12:30 iga dnsmasq-dhcp[30147]: DHCPOFFER(em1) 10.42.0.62 b8:27:eb:87:1d:86
Sep 28 21:12:30 iga dnsmasq-dhcp[30147]: DHCPREQUEST(em1) 10.42.0.62 b8:27:eb:87:1d:86
Sep 28 21:12:30 iga dnsmasq-dhcp[30147]: DHCPACK(em1) 10.42.0.62 b8:27:eb:87:1d:86 raspi
[/code]

That 10.42.0.62 is the address of the RaspPi. You can ssh into it (username root, password fedoraarm), and do stuff!

installing kv-WebME in CentOS 6

I’m setting up a new server and need to install kv-WebME in it. The previous instructions (for Fedora 16) were fine for an older version of WebME, but the most recent requires more uptodate packages.

So, first, we need to tell CentOS 6 to use more uptodate packages than are provided by default.

[bash]
su –
rpm –import http://apt.sw.be/RPM-GPG-KEY.dag.txt
rpm -ivh http://packages.sw.be/rpmforge-release/rpmforge-release-0.5.2-2.el6.rf.x86_64.rpm
[/bash]

Now we need to install PHP (at /least/ 5.3), Apache, MySQL, etc.

[bash]
yum install httpd mysql-server php php-gd php-mysql php-xml git zip unzip
[/bash]

Add a user account for the website, and download the kv-webme repository

[bash]
adduser webme
su – webme
git clone https://github.com/kaeverens/kvwebme webme
chmod 755 /home/webme
exit
[/bash]

Now add the web configuration to /etc/httpd/conf/httpd.conf

[xml]
NameVirtualHost *:80

AllowOverride All


DocumentRoot /home/webme/kv-webme

[/xml]

And finally, turn it all on.

[bash]
service httpd start
service mysqld start
chkconfig –level 35 mysqld on
chkconfig –level 35 httpd on
[/bash]

That’s it!

FB and G+, the new OS-wars

Back when Linux was a gangly youth, there was a great excitement every time you did an update. Unlike Windows and Mac with their expensive three-year-nothing-then-a-service-pack-with-cool-stuff routines, in Linux, there were so many cool free packages around, and /every one/ of them would have something new almost every week.

So, I spent quite a lot of time compiling and recompiling, everything from wget up to the KDE behemoth. And I’d be reading the weekly release notes as well to see if there was any new trick out that week.

I can’t tell you how excited I was when I first installed a copy of Linux where I didn’t have to configure X11 using a text-mode Xconfigureateur!

Or when I recompiled my kernel the first time, or when I recovered from a bad update straight from the GRUB command-line. Or the first time I ejected a CD from across the room (logged in remotely), or freaking out the wife by playing some music on her laptop from a different room.

In the last few years, Linux development has matured, though, so there’s not the same edge-of-the-seat excitement that there used to be, but I think there’s still hope for the techadrenaline junkies out there, because Facebook and Google+ are the new cool, and there’s a /ton/ of stuff that can be done with that!

My eyes are on how FB and G+ evolve. now that FB has competition, I expect some /really/ cool stuff is going to come out of the woodwork.

multi-tenanted CMS architecture

Last week, I did a talk at the Dublin Google buildings titled “Multi-tenanted CMS Architecture using PHP”.

Here are the slides that I used:

While talking with Google’s Brian Brazil, he explained that it is actually more efficient to use one database and many separate tables, than to separate each installation into a separate database, so one point I made (that KV-WebME uses separate databases per site) will change in the future.

I think the talk went down well, by the number of questions afterwards.

Last year, I gave a similar talk, and made the mistake of including way too much PHP in it – I had assumed that the audience would be composed of PHP developers. This year, there is just one slide of PHP, and that’s just to illustrate one possible way to build a proxy config.

Lesson’s learned for this time:

  • Talk slower. When I’m explaining something, I tend to try to get as much in as possible, so speak very fast. This makes it hard to hear what I’m saying.
  • More pictures, less words!
  • Stats. Some of the questions were around how efficient certain parts of the method were – particularly on the overhead of piping a file through a script as opposed to simply delivering it via Apache. I need to come up with numbers for that.

Overall, I was happy with this presentation.

script for shrinking videos

For this tip, you’ll need mencoder, ffmpeg and php installed. Only tested on Linux.

I’ve a load of videos, because I don’t like deleting anything in case I want to watch it again later (hoarding!).

I’ve three 1TB drives stuffed full of what I’ve collected over the last 20 years, but still I keep running out of room.

So, choice – buy more drives, or shrink what I have? I chose to shrink what I have, because I’m totally happy with VHS-quality video in stereo – I don’t need to watch fan-subbed Bleach in HD in 5.1 surround sound!

So, I wrote a script which will search the current directory (and its sub-directories) for all videos, will figure out their bit-rates, and will shrink them if they’re too large.

The script also reduces the videos so they match a maximum width and height.

It’s written in such a way that it will start with the most wasteful files (those that are taking up the most room), and work from that to the least wasteful.

It’s also written to only do 100 videos in a run. There’s no point shrinking absolutely every video you have, as most will probably save only a few kilobytes. The script does 100 of the most wasteful ones.

Of course, you could always run it again and again afterwards, and it should choose different videos each time to shrink.

How to use: copy the file into the root directory of your video repository. Go to that directory. Type “php shrink.php” into the console. Now go do something for a few days. Simple as that!

[php]
<?php

$maxwidth=624;
$maxheight=352;

function list_files($dir, $list) {
$files=new DirectoryIterator($dir);
foreach ($files as $f) {
if ($f->isDot()) {
continue;
}
$fname=$f->getPathname();
if (is_dir($fname)) {
$list=list_files($fname, $list);
continue;
}
if (in_array(
strtolower(preg_replace(‘/.*\./’, ”, $fname)),
array(‘avi’, ‘mpg’, ‘mp4’)
)) {
if (preg_match(‘/-shrink.avi$/’, $fname)) {
unlink($fname);
continue;
}
$arr2=array();
// { get duration of video
exec(‘ffmpeg -i "’.$fname.’" 2>&1 | grep Duration’, $arr2);
$bits=explode(‘:’, preg_replace(‘/.*Duration: ([^,]*),.*/’, ‘\1’, $arr2[0]));
if (count($bits)!=3) {
echo ‘failed to get duration of "’.$fname.’"’."\n";
}
$duration=(int)preg_replace(‘/^0/’, ”, $bits[2])
+((int)preg_replace(‘/^0/’, ”, $bits[1]))*60
+((int)preg_replace(‘/^0/’, ”, $bits[0]))*3600;
// }
// { get width/height of video
exec(‘ffmpeg -i "’.$fname.’" 2>&1 | grep "Video:"’, $arr2);
$bits=explode(‘x’, preg_replace(‘/.*\s([0-9]+x[0-9]+)[\s,].*/’, ‘\1’, $arr2[1]));
if (count($bits)!=2) {
echo ‘failed to get dimensions of "’.$fname.’"’."\n";
continue;
}
$width=$bits[0];
$height=$bits[1];
// }
$fsize=filesize($fname);
$bps=$fsize/$duration;
echo ‘.’;
$list[]=array($bps, $fsize, $width, $height, $duration, $fname);
}
}
return $list;
}

echo "\ngetting file details\n";
$list=list_files(‘.’, array());

echo "\nsorting by bits per second\n";
for ($i=0; $i<count($list)-1; ++$i) {
echo ‘.’;
for ($j=$i+1; $j<count($list); ++$j) {
if ($list[$j][0]>$list[$i][0]) {
$tmp=$list[$j];
$list[$j]=$list[$i];
$list[$i]=$tmp;
}
}
}

echo "\nshrinking\n";
for ($i=0; $i<100&&$i<count($list); ++$i) {
$fname=$list[$i][5];
echo $fname."\nold size: ".$list[$i][1]."\n";
@unlink(‘divx2pass.log’);
$resize=”;
if ($list[$i][2]>$maxwidth || $list[$i][3]>$maxheight) {
$rx=$maxwidth/$list[$i][2];
$ry=$maxheight/$list[$i][3];
$ratio=$rx<$ry?$rx:$ry;
$width=(int)($list[$i][2]*$ratio);
$height=(int)($list[$i][3]*$ratio);
$resize=’-vf scale=’.$width.’:’.$height;
}
exec(‘mencoder "’.$fname.’" -ovc xvid -xvidencopts bvhq=1:chroma_opt:quan’
.’t_type=mpeg:bitrate=400:pass=1 ‘.$resize.’ -alang en -oac copy -o "’
.$fname.’-shrink.avi" 2>&1′);
exec(‘mencoder "’.$fname.’" -ovc xvid -xvidencopts bvhq=1:chroma_opt:quan’
.’t_type=mpeg:bitrate=400:pass=2 ‘.$resize.’ -alang en -oac mp3lame ‘
.’ -o "’.$fname.’-shrink.avi" 2>&1′);
echo ‘new size: ‘.filesize($fname.’-shrink.avi’)."\n\n";
if (filesize($fname.’-shrink.avi’)<$list[$i][1]) {
unlink($fname);
rename($fname.’-shrink.avi’, preg_replace(‘/\.[^\.]*$/’, ‘.avi’, $fname));
}
else {
unlink($fname.’-shrink.avi’);
}
}
[/php]

SaorFM, first few commits

Last Sunday, I announced that Conor and myself were starting a new file manager called SaorFM.

We started out slow, because we want to have absolutely everything in this to be checked by PHPUnit (for console-style testing) or Selenium (for GUI testing).

We’ve got enough written that you can do the following:

  • Write and save a config file.
  • Rename/Move files.
  • Delete files/directories.
  • List the contents of a directory.

All of the above have automated tests, so we’re fairly sure they’re correct. The main class’s source is readable here, and the test’s source is readable here.

Conor is currently adding support for SaorFM into his own CMS, Furasta. I’ll be working on adding it to my own tomorrow (got relatives coming over today – busy busy!).

We need to add some more functionality to make this basically complete:

  • Move directories.
  • Upload files.

After that, things get interesting, as we’ll be adding a plugin architecture.

Tomorrow, I’ll be finishing up the basics, and will write some simple jQuery plugins to select files, select directories, upload files, etc.

Soon, we’ll be catching up on KFM in functionality ;-).

SaorFM

Last night, I spent four hours chatting with Conor MacAoidh.

We’re both the authors of CMSes, and both need file managers.

I’m the original creator of KFM, but recently, I’ve been getting annoyed at it. The project has grown too large to be easily managed, and it’s slow to start up because of the amount of database configuration involved.

We discussed this, and came up with a plan, which coincides with what I wanted to do for KFM2, but is probably much better.

We are going to reboot the whole thing – write a complete new file manager from scratch. It will only use code from the original KFM2 if the code is demonstrably better than any alternative we come up with.

The project will be properly documented, will have 100% test coverage, and will be completely free.

It will come in a number of separate parts, but only one, the core, will be absolutely needed.

The core of the engine is the bit which handles the actual file management. It will be designed to load in only two or three files for the most part, and as fast as possible.

Communication with the core will be done by either including the core as part of your own CMS, or by interacting with it via RPC.

The RPC will be very important – you send a command such as /rpc.php?action=move&from=/my-files/test1.jpg&to=/images/me.jpg, and results will be returned as JSON.

We decided on the name SaorFM. While this may be slightly confusing for non-Irish-speakers (“Saor”, pronounced half-way between “sair” and “seer”, means “Free”), we feel this is not very important. After all, Ubuntu is a household name, and that’s Bantu.

The main site will be SaorFM.org, the blog will be here, and downloads, issue tracker and SVN can be found here.

We’re still deciding on how to go about things, so there are no downloads yet. The decision to do this was made literally last night.

Having a co-developer on board from the absolute start will encourage me to get my arse in gear on this – if Conor does something cool, I have to beat that. And vice-versa, hopefully!

I’m starting the project off at the moment by working on a description of what it’s all about, and then will start writing some starter tests. This will use “test-driven development”, so every single line of code in this project will be repeatedly tested throughout the development.

We’re planning a load of features, such as desktop-/system- integration for Linux, Mac and Windows, and having totally external UI systems. We even considered going mad and creating a bump-top-like UI for it.

It’s taken me almost a year since planning KFM2 and getting to this.

Part of the reason for the delay is that this is so far removed from the current KFM, that I really didn’t know how to bring KFM1 up to the specification I wanted to reach.

Starting from absolute scratch with a brand new name is the right thing to do, I think.

audio alerts for php errors

I tend to keep a log tailing in one console while working in another.

tail is a Linux program that displays the last few lines of a file. If you run tail -f /var/log/httpd/error_log as root, or as your normal user if you’ve set the right permissions, then you will see any errors as they are added to the log.

However, I also tend to get immersed in my coding and not notice any errors until they cause a visual problem.

What I needed was a program that would watch my log, and beep to get my attention if an error was added.

This article explains how to set that up in Fedora.

First, install the program swatch:

[bash]
[root@ryuk ~]# yum install swatch
[/bash]

Now create /etc/init.d/swatch, which is the startup/shutdown script for the logger:

[bash]
#!/bin/sh

case "$1" in
‘start’)
/usr/bin/swatch –daemon –config-file=/etc/swatch-httpd-errors –tail-file=/var/log/httpd/error_log –pid-file=/var/run/swatch-httpd-errors.pid
;;
‘stop’)
PID=`cat /var/run/swatch-httpd-errors.pid`
kill -9 $PID
;;
*)
echo "Usage: $0 { start | stop }"
;;
esac
exit 0
[/bash]

Notice the two file locations in the /usr/bin/swatch command – the configuration file location, and the log file to be tailed.

Make the file executable:

[bash]
[root@ryuk ~]# chmod +x /etc/init.d/swatch
[/bash]

Now create the configuration file, /etc/swatch-httpd-errors:

[bash]
watchfor /PHP Parse error|PHP Fatal error/
exec mplayer /home/kae/sounds/beep-8.wav > /dev/null 2> /dev/null
[/bash]

You can have swatch do basically anything, from sending an email, to flashing lights in your face if you have them connected to the computer. All I wanted was a little beep.

Change the exec command to whatever your want.

I started experimenting by having Arnold Schwarzenegger yell “What the hell are you doing??” at me, but that could get annoying for other people. In the end, I changed it to a beep – a very /short/ beep. More of a tick than a beep.

Now start up the daemon:

[bash]
[root@ryuk ~]# /etc/init.d/swatch start
[/bash]

Create a php script with an error in it, and view it in your browser. You should get a satisfying beep.

If you want to have this run automatically when the laptop boots, add the above command to the /etc/rc.local file.

php.ie slowly upgrading

It’s been a while since I wrote anything vaguely technical. I guess it’s because I like to write only when there’s something new to say, and usually only if I have some new code to give away.

No new code today, but I can describe the recent work on php.ie (I’m the secretary of the Irish PHP Users’ Group).

So firstly, it was basically a static/brochure site for about a year, until we installed WebME (written by me!) as the CMS and created a skin for it so there’s only a tiny design difference. If you want to try out WebME, then download the SVN version from the google code site, or create a test site here (uses a really old version of WebME – you’re better off using the SVN version).

Then, I started rewriting the right panel. Beforehand, it displayed recent twitter messages, but they’re not often put out so it was a bit of a wasted space.

The panel now uses a WebME widget which displays recent Twitter messages, emails from the mailing list, and posts from the forum.

Over the next few days, I’ll be adding a new News section to the site, and the message widget will be able to show new articles from planet php.ie and new jobs from the jobs page.

I’m currently reading through Ken’s linux.ie todo list to see what I can appropriate for php.ie for its ongoing development.

Big thanks go to Michele and the team at blacknight for hosting the site.

Oh! Just a reminder, buy my book! JQuery 1.3 with PHP – hasn’t been reviewed by anyone yet, as far as I know, but my own opinion is that it is worth having on your shelf if you are a PHP developer that wants to step into jQuery.