SaorFM

Last night, I spent four hours chatting with Conor MacAoidh.

We’re both the authors of CMSes, and both need file managers.

I’m the original creator of KFM, but recently, I’ve been getting annoyed at it. The project has grown too large to be easily managed, and it’s slow to start up because of the amount of database configuration involved.

We discussed this, and came up with a plan, which coincides with what I wanted to do for KFM2, but is probably much better.

We are going to reboot the whole thing – write a complete new file manager from scratch. It will only use code from the original KFM2 if the code is demonstrably better than any alternative we come up with.

The project will be properly documented, will have 100% test coverage, and will be completely free.

It will come in a number of separate parts, but only one, the core, will be absolutely needed.

The core of the engine is the bit which handles the actual file management. It will be designed to load in only two or three files for the most part, and as fast as possible.

Communication with the core will be done by either including the core as part of your own CMS, or by interacting with it via RPC.

The RPC will be very important – you send a command such as /rpc.php?action=move&from=/my-files/test1.jpg&to=/images/me.jpg, and results will be returned as JSON.

We decided on the name SaorFM. While this may be slightly confusing for non-Irish-speakers (“Saor”, pronounced half-way between “sair” and “seer”, means “Free”), we feel this is not very important. After all, Ubuntu is a household name, and that’s Bantu.

The main site will be SaorFM.org, the blog will be here, and downloads, issue tracker and SVN can be found here.

We’re still deciding on how to go about things, so there are no downloads yet. The decision to do this was made literally last night.

Having a co-developer on board from the absolute start will encourage me to get my arse in gear on this – if Conor does something cool, I have to beat that. And vice-versa, hopefully!

I’m starting the project off at the moment by working on a description of what it’s all about, and then will start writing some starter tests. This will use “test-driven development”, so every single line of code in this project will be repeatedly tested throughout the development.

We’re planning a load of features, such as desktop-/system- integration for Linux, Mac and Windows, and having totally external UI systems. We even considered going mad and creating a bump-top-like UI for it.

It’s taken me almost a year since planning KFM2 and getting to this.

Part of the reason for the delay is that this is so far removed from the current KFM, that I really didn’t know how to bring KFM1 up to the specification I wanted to reach.

Starting from absolute scratch with a brand new name is the right thing to do, I think.

what’s up!

Short run-down of what I’m doing lately: nothing.

Less short: I’m trying to get work out the door, get a good run at some personal projects, pass grade 2 piano, get organised, and generally improve my lot.

None of this is working. I think the “get organised” bit is the most important, as it will help the rest of it fall into place.

I usually only post about web-development-related topics here, as that’s the only subject where I feel I can contribute something new and interesting, so I tend to not talk about other stuff. But sometimes, rattling off the current state of the head is good for clearing it.

In work, I can’t really complain – we have a number of largish projects which are slowly creeping towards completion. The hardest thing about them is getting information from the clients, and then a week or two later being told that half the information is not required. I guess my main complaint at work is the inexorably slow completion rate.

On the personal projects side:

There are still a number of small bugs in KFM 1.4, and either I don’t have the time to get to them, or there is no enough information to recreate the bug and the submitter doesn’t give me access to their copy so I can’t see it from their side.

KFM 2 has been halted for a while – the idea is huge, but I simply don’t have the time, and no-one is clambering for it. I’ll get to it when I have time, but I might have to approach it by evolving KFM 1.x into meeting what I wanted, instead of the original goal of building KFM 2 from scratch.

I started a new project, OddJobs4Locals two weeks back, and got a good two-day run at it, then time got ahead of me again. I think this will be a good one, when I can complete it. Useful for students, people with a little spare time, or simply people that just want to make a little extra cash. Not yet working, but it will be soon, I hope… This is doubly interesting to me, as it is done purely through AJAX, so it will be easy to do a smart-phone client or a desktop client when the time comes.

I’m in the back/forth stage of working with Packt publishing to see if they want me to do a second book (the first one has no bad reviews at all). We’ve mostly agreed on a table of contents, and I’m just trying to get the time to combine a few of the smaller chapters together.

On the piano, I’ve been ready for the grade 2 exam since November, and am still waiting to see if there will be an exam near me any time soon – I hate the effort that goes into travelling (I have a family, and no car). I was hoping to do a grade every 6 months. It looks like this might not be possible, despite me being ready for it… The tunes I’m doing for it are Beethoven’s Sonatina in G Major, a waltz by Bela Bartok, and Boys And Girls Come Out To Samba, by Terence Greaves – by the way, I don’t like those videos; there are no dynamics in any of them, and I can hear a number of mistakes as well. No video apparently of the Terence Greaves one.

As for organisation… well I guess I’d better start working with Mantis again.

My lot will have to wait – I’ve a load of work to get done before it can improve.

Meh. Depression taking hold again.

multiple file uploads using HTML5

As a response to a reported bug where Chrome was taking ages to load up a flash multiple-file uploader, I’ve updated KFM to use HTML5’s multiple-file input box where possible.

To do this, first create the element:

[javascript]
var input=document.createElement(‘input’);
input.type=’file’; // use old-style JavaScript method to make sure all browsers respect it
input.name=’kfm_file’;
[/javascript]

Notice that we’re not using setAttribute to set the type and name – that’s a DOM method which works in most browsers but not (of course…) in Internet Explorer 6, where it has bugs.

And now, we tell the input to use the multiple-upload method. We use .setAttribute in this case because we only expect newer browsers to succeed with it.

[javascript]
input.setAttribute(‘multiple’,’multiple’);
if(input.multiple)input.name=’kfm_file[]’;
[/javascript]

In the second line, we check to see if the element is now marked as a multiple-uploader (most current browsers will not succeed in this), and if it does, then rename the input element by adding a [] to the end. If this is not done, then the server will only see the first file which is uploaded.

That’s the client-side done. This will only be visible in newer browsers such as Chrome, Safari 4, Firefox 3.6. I expect Internet Explorer will eventually catch up by 2020 or so.

If you’re doing this in pure HTML, then I suppose this would be good enough for you:

[html]
<input type="file" multiple="multiple" name="kfm_file[]"/>
[/html]

In this case, you must put the [] in the name in all cases.

On the server-side, you need to write your upload receiver to expect either a single element, or an array.

For some really goddamned stupid reason, when multiple files are uploaded to PHP, the results are interlaced in a really crappy and awkward manner (I don’t like it).

Instead of something logical and easy to use, like this:

[php]
array(
[0] => array(
‘name’ => ‘file1.txt’,
‘tmp_name’ => ‘/tmp/abcdef’
….
),
[1] => array(
‘name’ => ‘file2.txt’,
‘tmp_name’ => ‘/tmp/ghijkl’
….
)
);
[/php]

You get this…

[php]
array(
‘name’ => array(
[0] => ‘file1.txt’,
[1] => ‘file2.txt’
),
‘tmp_name’ => array(
[0] => ‘/tmp/abcdef’,
[1] => ‘/tmp/ghijkl’
),

);
[/php]

While that looks at first glance to be easy to use, it’s not. You can’t do a simple “foreach($_FILES['kfm_file'] as $file)” and expect the above to be usable at all…

So, the first thing I do, is to check for the $_FILES[‘kfm_file’], and convert it into the first form above, which is very easy to work with:

[php]
$files=array();
$fdata=$_FILES[‘kfm_file’];
if(is_array($fdata[‘name’])){
for($i=0;$i<count($fdata[‘name’]);++$i){
$files[]=array(
‘name’ => $fdata[‘name’][$i],
‘tmp_name’ => $fdata[‘tmp_name’][$i],
);
}
}
else $files[]=$fdata;
[/php]

In my own case, I’m only interested in the name and tmp_name variables, so that’s all I set up.

Now you can do a foreach on $files and treat them all individually.

[php]
foreach ($files as $file) {
// uploaded location of file is $file[‘tmp_name’]
// original filename of file is $file[‘name’]
}
[/php]

If you want to see this in KFM, have a look at the nightly-updated demo tomorrow, or download from SVN right now.

oh – and buy my book!

kfm2: example widget

As an example of a front-end widget, I’ve whipped up a demo of file-tree navigation using the excellent jsTree widget.

here is the demo

You can see from the source how short it is – all it does is load up the jsTree library (and jQuery), then it connects directly to KFM2 to get data about the files on the server.

in fact… it’s so short I’ll just paste the demo into this page as well:

$(document).ready(function () {
$(“#tree_div”).tree({
data : {
type : “json”,
async : true,
opts : {
async : true,
method : “POST”,
url : “http://demo.verens.com/kfm2/trunk/?p=get_file_listing” // KFM LOCATION
}
},
callback : {
‘beforedata’:function(node){
var id=$(node).attr(‘id’);
if(!id)return {‘d’:’/’};
return {‘d’:$(node).attr(‘id’).replace(/^kfm2_node_/,”)};
},
‘ondata’:function(res){
var data=[],i;
for(i in res.files)data.push({‘data’:i,’attributes’:{‘id’:’kfm2_node_’+res.d+’/’+i}});
for(i in res.directories)data.push({‘data’:i,’state’:’closed’,’attributes’:{‘id’:’kfm2_node_’+res.d+’/’+i}});
return data;
}
}
});
});

You should see a list of files just above this paragraph.

No modification of jsTree was necessary, as it handily gives two event triggers which can be used to translate the request data to KFM’s request format, and the received data back into jsTree’s data format.

Here is the code used to attach the tree to the #tree_div element in this page:

$("#tree_div").tree({
	data : {
		type : "json",
		async : true,
		opts : {
			async : true,
			method : "POST",
			url : "http://demo.verens.com/kfm2/trunk/?p=get_file_listing" // KFM LOCATION
		}
	},
	callback : {
		'beforedata':function(node){
			var id=$(node).attr('id');
			if(!id)return {'d':'/'};
			return {'d':$(node).attr('id').replace(/^kfm2_node_/,'')};
		},
		'ondata':function(res){
			var data=[],i;
			for(i in res.files)data.push({'data':i,'attributes':{'id':'kfm2_node_'+res.d+'/'+i}});
			for(i in res.directories)data.push({'data':i,'state':'closed','attributes':{'id':'kfm2_node_'+res.d+'/'+i}});
			return data;
		}
	}
});

This is just an example of what can be built quickly with the new RPC-based KFM.

Further and more useful examples will come soon.

kfm2: plugin management

I know, I promised this last week – I’m afraid I’ve been up to my eyes in work and just exhausted at home, so all I can do is apologise.

Anyway, I’ve put together the basics enough that there is one single plugin, which gets a directory listing, and echoes it to the screen in JSON.

You can get a copy of the basics using the following SVN command:

svn checkout http://kfm.googlecode.com/svn/kfm2/ kfm2

In that, there are two directories – trunk and plugins. The trunk is what will become the new KFM core, and the items in plugins will be server-side plugins that can be used by copying them into trunk/plugins/ (or do as I do and ln -s ../../plugins/* from the trunk/plugins/ to create symlinks).

When you load up KFM2 initially, it will do an extremely basic install, with no plugins enabled. Follow the instructions on-screen to get set up.

The config.php contains just two items at the moment. I want to keep this as absolutely clean as possible, so the items in there are the only essentials – all other configuration will be done using KFM’s admin section.

  • KFM_USERDIR – this is the directory that is to be managed by KFM. Just like KFM1, a directory called .files will be created in there that holds config stuff, such as the list of enabled plugins, etc.
  • KFM_ADMIN_PASSWORD – this password allows access to KFM’s admin section, where you can currently just enable/disable plugins. By default, it is blank, and you will need to change it before you can access the admin section.

There is no JavaScript yet in this, so it’s tricky to see exactly what it’s supposed to do.

I’ve set up a basic demo here, but loading that directly will just get you a login screen.

KFM2 is designed to be interacted with purely through RPC (except for configuration which requires browser access). The demo has the first plugin, “get_file_listing” enabled, which we can use to see what happens.

/?p=get_file_listing – this example calls KFM2 and tells it to run the get_file_listing plugin. by default, it will return the root directory of your KFM_USERDIR in JSON format:

{
  "d":"",
  "files":{
    "file2":{
      "size":0,
      "mtime":1255875795
    },
    "file1":{
      "size":0,
      "mtime":1255875794
    },
    "file3":{
      "size":0,
      "mtime":1255875796
    }
  },
  "directories":{
    "dir1":{
      "mtime":1255875809
    }
  }
}

The d parameter there is the directory that was requested (by default, it’s blank), which is internally added to the KFM_USERDIR to get the actual local directory.

The results returned are separated into two arrays, of files and directories. My guess is that most required uses on the client-side will be specific to either directories or files, so this saves you having to separate them yourself.

Changing the requested directory is simple – just add a d parameter to the request URL:

/?p=get_file_listing – this example requests the contents of /dir1.

How to write your own plugin

The system as it is will let you write plugins that handle most of the things that you might want to do – upload files, create/move/delete files or directories, download files, etc.

I’ll get around to writing these anyway, but if you feel like writing them, then feel free! Just copy how the get_file_listing plugin works.

So, here’s how a basic plugin works.

In KFM2’s plugin’s directory, create a directory named after the plugin. Please use just lowercase letters and underscores. Examples: “upload”, “get_file”, “rename_file”, “zip_directory”, etc.

In that directory, create a plugin.php file, which describes the plugin:

<?php
$plugin=array(
  'name'=>'Get File Listing',
  'description'=>'Retrieves a list of files and directories within a given directory',
  'version'=>0
);

Use an integer for the version number. It will probably never need to be used, but if it is, it is easier to compare two integers than two of the usual x.y.z version numbers.

When you log into your admin area, you’ll see the details above shown in your plugin management module.

For most plugin cases, you will also need a file named after the plugin itself. Until the events code is written, this is all cases.

In our example, we create a file named plugins/get_file_listing/get_file_listing.php. It should contain a function also named after the plugin:

function get_file_listing(){
  // place plugin code here
}

In the URL above, I had /?p=get_file_listing – what KFM then did was to load up the get_file_listing/get_file_listing.php file and run get_file_listing(). That function then used whatever other things were mentioned in the $_REQUEST array to carry out its work.

That’s pretty much it for now. That should be enough to create most basic plugins.

For the next KFM article, I’ll write up another server-side plugin, and will add in some client-side code so you can see how it would actually be used in a real project. It will demonstrate how JavaScript and KFM will talk to each other.

If anyone writes up any plugins that they want included in the SVN download, please email them to me (kae@verens.com).

Requested plugins

Here’s the list of plugins that have already been identified as needing to be written.

Server-side plugins that can be written right now:

  • file upload
  • delete file / directory
  • rename file / directory (also counts as “move”)
  • download file
  • create zip of directory
  • unzip a zip file
  • manipulate an online image

Plugins that require more work in the core:

  • connect to remote file-systems (FTP, NFS, other KFMs, etc)
  • user-based security
  • different KFM_USERDIR based on the logged-in user, or even the site’s hostname
  • plugin to resize images that were uploaded so they’re kept to a certain max size
  • quotas (can /probably/ be done at the moment… but not sure)
  • databases

KFM2 has begun. requesting ideas of sample plugins to develop

As ye all know now, KFM1.x is finished. I’m not interested in struggling with it anymore. I’ve started a new project which will exceed KFM1.x’s capabilities.

The main problem with 1.x was that it was monolithic – there was one single codebase, which encompassed both the server-side and the client-side. The client-side code was useless without the server-side, and the server-side code had hooks specifically designed to be used by the client-side.

While it was possible to build a different client-side GUI that would hook into the server-side, it would be a large undertaking, and would involve changing a lot of the PHP – especially as the PHP wrote the “boot” JavaScript in the first place…

KFM2 is designed from the beginning to have a different architecture based specifically on plugins. The core code is a small plugin manager (mostly written already), written in PHP, but it will not be necessary that the client-side is in JavaScript – or even that there is any interaction with a browser at all!

The idea is this – if you want to do something incredibly simple, such as select an existing directory that will be used in your CMS to create an image gallery out-of (let’s say you need to choose from a number of existing directories full of images), then it is a total waste of bandwidth and CPU to load up a full-on file-manager.

In this case, and it’s the first I’ll be building to show how all this works, what should happen is that the browser will show something simple like this:

Choose the image directory

…and when that’s clicked, a small popup will appear which lets you choose the directory you want, in a tree-like manner – you open the branches until you find the one you want, then you click it. Afterwards, the form will change:

Choose the image directory /images/the-lads-2009/ selected

From the technical point of view, there’re a few things to explain about how this would be done.

jQuery would be used to attach an event to the button which would popup the selection window. This selection window will use Ajax to speak with the KFM2 server and retrieve the list of directories, one level at a time as the branches are expanded. When one is selected, a hidden input will be set to the name of the directory.

This sounds incredibly simple, and it is. What’s important, though, is that no-one is doing this at the moment!. Right now, if you go search for existing plugins to do this kind of thing, you’ll find either small widgets with both the server and client-side code ready for you, or full-on applications (such as KFM1.x) which can do the job, but which are overkill.

The problem with the applications is obvious – too much stuff going on, making it slow and cumbersome.

The problem with the small widgets is that each one is complete in itself, and therefore a lot of the functionality is repeated. This is a bad idea – what if you need to make a site-wide change? You’d need to go through every widget and change it!

What I’m proposing to make of KFM2 is that it will mostly be a server-side application, but which has a nice easy-to-use API which can be hooked into by lots of different jQuery widgets (or Mootools, or whatever you want, really – write it in Python or Delphi if you want…).

So, my first widget will be the one described above – a directory selector.

I’d like to know what other things people need to do with files online, which are simple and they’d prefer not to have to load a whole file-manager to do.

I’ll release the first widget and a demo of it some time over the next week. I intend to write a whole load of widgets, culminating in a full-on GUI similar to the present KFM1.x one, all hooking into the new engine.

KFM 1.4.2 released. now with CKeditor support

kfm

KFM 1.4.2 has been released. It’s a minor maintenance release, but it adds support for CKeditor. TinyMCE and FCKeditor have been checked as well.

demo using CKeditor

This release also adds a configurable restriction, where images larger than a certain resolution will simply not be allowed in the repository.

Very large images can cause servers to crash when the system tries to generate thumbnails, because of the large amount of RAM needed. For example, a 3000×2000 image will use up at least 24MB just to hold the thing in memory, and much more to handle the manipulation. 24MB is a large amount of RAM to a web app.

There are also some code additions which are not used by the KFM GUI, but can be used by tools that hook into the KFM engine. These allow uploaded files to be placed in directories specified at the time of upload, or to replace existing files, or even to take an uploaded image and apply HSL transformations to it (when ImageMagick is installed). These are not documented, but anyone that needs this functionality will be a good enough programmer to read the source and find it (or email me 😉 ).

Download KFM 1.4.2 here

KFM2: the beginning

KFM 1.x has reached the end of its development. It has fulfilled its original purpose; to improve on the default file-manager for FCKeditor, and well-exceeded it.

From my records, I can see that KFM has been installed on over 9000 separate domains. That is quite a lot of sites, and says quite loudly how important it is that KFM2 is at least as good as KFM1 (we all know about the KDE4.0 fiasco…).

KFM2 will blow KFM1 well out of the water. It will do everything that KFM1 does, but much quicker, and with a much less monolithic architecture, allowing it to be hugely flexible. I am planning on creating a download section similar to jQuery UI’s, where you can choose the components that you want in the system, and it will be built up and configured for you, ready for download.

See, KFM 2 will be modular. If you just want a simple upload/download facility, then you will only need the core, and plugins to handle uploads and downloads. If you want the whole shebang, you will find plugins for various ways of selecting files and directories, full-on graphical UIs such as the existing KFM1 forces on the user, plugins for hooking into external file systems using FTP or whatever other method people can imagine, plugins for multi-media, file editing, databases, user authentication, search, RSS, and so on.

Basically everything that KFM1 already does, but with the option to easily remove/add bits that you want. Make it as fast or as “bling” as you want.

KFM2 will have a number of new things which are not available in any other online file-systems, that I know of.

new file-system things

The file system will be modular. You will be able to attach symbolic links to any part of the file system, linking to other external systems.

As an example, let’s look at a URL: http://example.com/kfm/get/images/site1/logo.jpg.

One of the new ideas is that this will not necessarily be located on the server named example.com.

If you are running a very busy website, you might prefer to delegate image management to a different machine deeper in your network, so you add a symbolic link /images/ linking to the image-management machine. This causes the server to issue a redirect to the browser, redirecting it to http://images.example.com/kfm/get/site1/logo.jpg.

Now, imagine that the image management server doesn’t host the logo.jpg file on itself. Maybe it’s one of a number of images.example.com servers in a load-balancing cluster, and the file is actually located on a file-server accessible only by FTP. So, we add a symbolic link in the server’s KFM, telling it how to connect to the FTP server. The image server does this, gets the required file, and sends it to the browser.

This sounds a bit complex, but at least it’s possible with the new system. The old system simply would not be able to handle that at all.

API

At the moment, if you want to select a file or directory, you fire up KFM1, wait 15 seconds for it to finish booting itself up, and select the file or directory. That’s way too long. Really, you should only need to load up a tiny widget that’s designed specifically for picking a file.

The new system will have a simple core with an API that can be connected to by little JavaScript widgets. As an example, let’s say you just want to get the file-listing for the foo directory.

At the moment, you need to go through the whole business of booting up KFM’s GUI, navigating to the right directory, then waiting for the server to connect to the database, generate thumbnails and other unnecessary stuff.

You should just have to do something simple like load up http://example.com/kfm/rpc.php?a=get_file_listing&v=/foo through jQuery’s $.get() function. The RPC script does exactly what was asked, and nothing more.

This should allow us to write some incredibly fast and tiny scripts for all the file manipulation things you could possibly want.

Think of it as the “gnu” way of doing KFM – many small tools, each of which is designed to do one task well.

Of course, you will still be able to use the original GUI with this, after it’s been re-engineered to hook into the new RPC system.

Or, you can write your own – the RPC’s API will be well-documented, and will use simple HTTP parameters, so you can write your widgets in whatever language you want – JavaScript is what I will be writing in, but there’s no reason why you’re confined to that – write a GUI in Python, C, Flash – whatever you’re comfortable with.

plugins

I think the new plugin architecture will make this profoundly extensible. I’m writing the core to be as flexible and stable as possible. You will be able to write a plugin to just about anything:

  • authenticate a user before serving a file.
  • change a user’s root directory based on the server-name or the user’s authentication.
  • the entire system is held on a different machine accessible only by FTP? fine – let’s change how files are accessed.
  • record all files and directories in a database, allowing searches to be done.
  • manipulate files online – edit text files, rotate or crop or resize images, move or rename or delete, etc. standard KFM1 stuff, these.
  • create a log plugin, so all access is recorded.
  • create a quota plugin, for denying uploads or downloads when the quota is exceeded.

It was possible for us (Benjamin and myself) to write all of these into KFM1, but as we are only two people, and very busy people at that, it’s never been the absolute top priority to get all these things done.

But, when KFM2 is released, you won’t have to wait for us anymore. You will be able to write plugins, or download plugins created by other people, which do all of the above and I’m sure there will be plugins that I simply have not imagined yet, that inventive people will come up with (Conor, I’m sure you’ll be one of those!)

languages

Languages have always been one of the cool things about KFM. Version 1.3 was released in about 13 languages in total. Unfortunately, the code got too complex in 1.4 to easily add more, but KFM2 is starting from scratch with the learning-curve of KFM1 well behind me. I’ve already made most of the big mistakes I’m likely to make, so from now on, things should be much easier.

How languages will work is that a website will be created which allows people to help out open-source projects, not just KFM by providing translations of words and sentences.

The programmer will create a screen-shot showing the text in use, and will provide the untranslated text in a format such as .po. Translators will translate the text on-line, and the translation will be available to download by the programmer as soon as it’s done.

To ensure good translation, each line will be presented to a number of different translators. The translations which agree the most with each other, will be considered the “right” one.

Using this, it will be possible for KFM to organically add languages with no interaction from myself or any other programmers. When first loaded, a plugin would try to translate messages into the user’s language. Finding that it doesn’t have the language in its database, it will try downloading it from the language translation website. If it’s not available, a request will be added that it be created. When translators that speak that language come to the website, they will be automatically given the KFM text to translate to that language. The next time the plugin checks, the translation might be done, and it will be added to the KFM instance’s local language database.

Of course, people don’t just do this stuff for the hell of it. It’s more fun if there’s a competition or reward. It would be nice if I could get a few paying customers to add their own translation requests to the website – that way I could offer a reward to the best translators (those that do the most work, or are the most consistent).

The competition aspect comes in, where you give people points for every translation they do that is matched by someone else’s translation of the same text.

Google does a similar thing, for example, where it shows images to people and asks them to come up with keywords describing the image – the commonest keywords are considered to be correct. Same trick, different target.

Anyway! Without further ado, I have a core to design, and tests to create. I was going to talk about testing with PHPUnit, automatic upgrades, and the like, but to hell with ye all – I’ve work to do 😉