Welcome!

Have a Raspberry Pi or other cool Linux device? Want to turn it into your Personal Cloud or run Indie Web applications on it? The Indie Box Project is creating tools to make this easier. After all, there is more to life than editing configuration files…

Check out our goals, FAQ, code and how to contribute. And do say Hi on the mailing list.

 

Posted in Uncategorized

First Indie Box Meetup

We’re planning the first Indie Box Meetup:

on May 12, 2014
from 6:30-8:00pm
at the Hacker Dojo in Mountain View, CA
cost: none

RSVP here: http://bit.ly/1lJoDyx

If you want to talk Indie Box, come by! We’re probably have a couple demos, and discuss how you run your own box at home. Or write apps for Indie Box that you and others can use.

Bring your own hardware and we’ll work on getting Indie Box software running on it. (x86, Raspberry Pi, and the like)

Posted in Event

Personal Clouds and Indie Box

Personal Clouds are a fascinating idea. Imagine if you could get all the benefits of cloud computing, but with none of the drawbacks?

That’s of course the kind of thing we want to accomplish with Indie Box: running web applications as if they were in the cloud, automatically managed as if they were in the cloud, and without data loss if a hard drive or other device dies. And perhaps, in the longer term, automatically scaling up and down as needed. Except with keeping the control in the hands of the user instead of some third party.

As user @explanoit recently said on Twitter:

I can’t believe you people even used DropBox in the first place. You just share your most important files with random companies?

So the Indie Box Project is happy to support the Personal Cloud Principles:

  1. Users control their data. Users decide whether to store it, transfer it, modify it, or delete it.
  2. Users control sharing. Users decide who is permitted to access their data, and which apps to run on it.
  3. Users set the terms for use of their data and may change them at any time.

which is why we recently have started to display the logo on the right of this page.

Posted in Design, Principles

Indie Box Networking Requirements

Credit: Jono Haysom

The goal is to install Indie Box software on some arbitrary piece of hardware, and be up and running with as little as possible further configuration, certainly without having to edit /etc/I/forgot/what/file.it/was, and restarting daemons. This necessarily includes network configuration.

That is not simple, however. Here are some of the difficulties:

  • Different servers tend to have different network interfaces. Some have a single ethernet interface. Some have two. Some have WiFi, others don’t. Some (particularly lower-end ARM devices) may be using things such as USB-to-Ethernet connectors, sometimes even internally. How can Indie Box code automatically configure hardware that it cannot know upfront?
  • Different networking hardware requires different device drivers. How can we automatically install and maintain the right code, regardless what WiFi dongle a user has bought?
  • Even seemingly identical servers may use different Linux device identifiers. As Linux moves away from /dev/eth0 and so forth to predictable device names, things have become simpler on one hand (no more strangely swapped device names after reboot) and more complicated on another (e.g. plugging the WiFi dongle into a different USB port will change its device name). Depending on the internal USB structure of a server, USB ports may cause rather different device names from one server to another.
  • Different users will use Indie Box in different situations. Some will run Indie Box code on devices that expect to receive a DHCP address via ethernet, and nothing more complicated than that. Some will run Indie Box on devices that expect the same via WiFi (which creates a pesky authentication issue). And others will run Indie Box on full-fledged routers that do things such as Network Address Translation, firewalls, local DNS, run their own local DNS and so forth between several network interfaces.

There’s the additional complication that Linux has a bunch of historically grown, not entirely well-segmented pieces of functionality that is affected by this, from udev to systemctl, netctl and daemons such as dnsmasq.

Somehow Indie Box needs to support all of those needs, without being complex to use.

I think the right approach is this:

  • We define a (hopefully small) set of situations in which an Indie Box-managed server/device can find itself. The most important ones seem to be “I’m just a DHCP client on some network” and “I’m a gateway between a home/office network and the wider world”.
  • We define different hardware profiles. These will map the “situations” onto actual hardware devices and their configurations. For example, a Raspberry Pi configuration will know that it has a single ethernet interface, it knows its device name and device driver. Others will have others.

Then the user can invoke a command like this: “I’d like to use this device as a DHCP client device on some network”. This command will map on a hardware profile-specific set of configurations given the particular hardware on which it is invoked.

The downside is that we’ll have to define different profiles for different hardware. But then, somehow that needs to be done anyway, even if only as documentation, because different users will have different hardware, and unless we help them get it running with Indie Box, they will not be successful. Obviously, the hardware profiles should be an open-ended list, so that it is easy for anybody to support any piece of hardware that they like.

Each hardware profile would presumably consist of a bundle of:

  • udev rules
  • netctl profiles
  • systemd profiles

 

Posted in Design, networking

Indie Box 1 Prototype

Look what arrived today:

IMG_0837

It’s a version of this box sold, appropriately, by mini-box.com, which is just across the bay in Fremont.

It has:

  • A dual-core Atom D2550 processor at 1.86GHz
  • 2 gigabit Ethernet interfaces, so it can act as a home firewall and router and serve websites both inside the home and towards the outside world
  • 4 USB ports on the back
  • HDMI and VGA
  • 2 or 4 GB of DDR3 RAM
  • 2 hard drives, which we’ll run as RAID-1, so valuable personal data (think family pictures) can survive a hard drive crash.

The whole box is just 192 x 210 x 62mm. My son commented “looks just like a Wii”, which I take as “people won’t be embarrassed to be seen with this box”.

Update 2014-02-06: Linux is installed, and the box is working. When idle, Kill-A-Watt tells me it consumes about 19W. Assuming $0.13/kWh, the cost to run it per year is about $22, or less than $2 per month.

Update 2014-02-07: Indie Box software is running just fine. Here’s a screenshot with a number of apps installed. (Yes, the CSS needs work, and we need help with graphics.)

Screen Shot 2014-02-06 at 16.21.55In case you are wondering, it took a single command to install them all, no configuration file editing, and messy stuff like that. In that one command, Indie Box installed:

Posted in Hardware, Progress

Application icons in Indie Box v0.17

Image you install several web applications on the same virtual host. How do you remember how to get to them? Or even which ones you have installed?

For example, on this site (indieboxproject.org) we run WordPress at /blog and MediaWiki at /wiki. On this site, we solved the problem by adding a lot of links from one to the other, as I’m sure you noticed by browsing the site. That takes a lot of content editing time, though, and may not always be appropriate.

Indie Box v0.17 introduces a new feature to solve this issue. A screen shot explains it best:

snapshot1

On this example site, five applications were installed. When you visit the root of the site (here: http://localhost/), it shows you what they are, and gives you links to click on them. For example, if you click on the OwnCloud logo, it sends you to http://localhost/owncloud/, which is the context path at which OwnCloud was installed.

If you want to try this yourself, install Indie Box, then download the corresponding Site JSON file from github:

> curl -O https://raw.github.com/indieboxproject/example-sites/master/a-few-example-apps-at-localhost.json

Edit it, if you like, e.g. to set a different virtual hostname or administrator account credentials, deploy it like this:

indie-box-admin deploy -f a-few-example-apps-at-localhost.json

and then visit http://localhost/.

More details:

This list of applications only shows up if no installed application claims to be at the root of the site. For example, if you installed Mediawiki at a "context" : "" in the Site JSON file, it would be MediaWiki showing up at the root of the site instead. Or, if you declared that GladIWasHere was the default application (with "isdefault" : true in the Site JSON file), then visitors would be automatically redirected from the front page to there. If neither is true, we automatically generate the page above.

This site itself has WordPress set as default, which is why you always get redirected to http://indieboxproject.org/blog/ when you visit http://indieboxproject.org/.

Finally: how come that one of the icons is pretty, and the others aren’t? That’s because those application packages haven’t defined suitable icons. (Volunteers?) It’s simply a PNG file at 72×72, which needs to be put into /srv/http/_appicons/<packagename&gt>/72x72.png, like indie-helloworld does it. (see Github here and here. Pull requests appreciated.)

Tagged with:
Posted in Code, Milestone, Progress

Indie Box Application Testing

Updated 2014-01-02 per recent code changes in v0.2.

One-command application deployment via Indie Box obviously only works with the application is packaged correctly (has all files, specifies correct metadata, files have the right permissions etc. etc.). How do we make sure that it is?

Fear not, we just released indie-box-testing, a package that makes testing application installations real easy. This is how it works:

Let’s say we are interested whether the GladIWasHere example application is packaged correctly. First we update our Indie Box installation and install the indie-box-testing package:

# pacman -Syu
# pacman -S indie-box-testing

We get the application’s test description from Github, and save it as GladIWasHere1AppTest.pm. Then, we run the tests:

# indie-box-testing run GladIWasHere1AppTest.pm

If there’s no error, it is correct. Done. Easy, right?

Let’s see what this actually does. To do so, let’s look at the structure of the test description in GladIWasHere1AppTest.pm:

my $TEST = new IndieBox::Testing::AppTest(
    appToTest   => 'indie-gladiwashere',
    description => 'Tests whether anonymous guests can leave messages.',
    checks      => [
        new IndieBox::Testing::StateCheck(
            name  => 'virgin',
            check => sub { ... }
        ),
        new IndieBox::Testing::StateTransition(
            name  => 'post-comment',
            check => sub { ... }
        ),
        new IndieBox::Testing::StateCheck(
            name  => 'comment-posted',
            check => sub { ... }
        )
    ]
);

You can see that this file defines two “StateChecks” and one “StateTransition“. The first StateCheck is named “virgin”. An application reaches the virgin state as soon as it has been installed for the first time, and before anybody has used it. indie-box-testing will use this StateCheck to check that the application installed correctly, by running the code in the subroutine passed into the constructor. (We’ll look at that code in just a bit.) If the subroutine returns true (well, 1, this is Perl after all), we know that everything went well.

Then, there is a StateTransition. This reflects an action that moves the application into another state. In the subroutine that goes with it, we simulate that a user posts a comment to our guest book. Then, StateCheck “comment-posted” can test whether the comment is actually displayed.

How does a StateCheck perform the test? Let’s look at the code that tests the “virgin” state:

sub {
    my $c = shift;
                
    my $response = $c->httpGetRelativeContext( '/' );
    unless( $response->{content} =~ /Glad-I-Was-Here Guestbook/ ) {
        $c->reportError( 'Wrong front page', $response->{content} );
    }
    if( $response->{content} =~ /This is a great site/ ) {
        $c->reportError( 'Guestbook entry still there', $response->{content} );
    }
    return 1;
}

Method httpGetRelativeContext( '/' ) performs an HTTP GET at the root URL where the application was installed. So, if the application was installed at http://example.com/foobar, it would HTTP GET from http://example.com/foobar/. The rest is self-explanatory.

To go to the next state, let’s look at the code in the StateTransition:

sub {
    my $c = shift;

    my $postData = {
        'name'    => 'Mr. Test User',
        'email'   => 'test.user@example.com',
        'comment' => 'This is a great site!',
        'submit'  => 'submit' };

    my $response = $c->httpPostRelativeContext( '/', $postData );
    unless( $response->{headers} =~ m!HTTP/1.1 200 OK! ) {
        $c->reportError( 'Guestbook entry failed to post', $response->{headers} );
    }
    return 1;
}

As you can see, this script pretends to fill out the guestbook application’s web form. If it executes successfully, the application will go into state “comment-posted”, which is tested by looking for the the posted comment on the page:

sub {
    my $c = shift;

    my $response = $c->httpGetRelativeContext( '/' );
    unless( $response->{content} =~ /This is a great site/ ) {
        $c->reportError( 'Guestbook entry not posted', $response->{content} );
    }
    return 1;
}

Now, you may ask, but who is doing the installing of the site? There’s nothing in the test script about that. And how does it know which URL to access, and do I need to setup DNS first etc. etc.

The short answer is: indie-box-testing does it all for you. It automatically creates a Site JSON that runs the to-be-tested application at a random hostname, deploys the site there, and then accesses the site via curl with the --resolve flag, so no DNS setup is necessary. (There is the case where the application performs HTTP requests on itself, like Owncloud is prone to do. In which case you should perform the test at a known hostname, which can be specified with the hostname => 'myhost' argument to the AppTest constructor. Example is here.)

By default, it sets the site up on the local machine that you are running indie-box-testing on. But, indie-box-testing has a mechanism (so far, still unused, because this is only version 0.1 0.2!) by which the site can be deployed to some other machine instead, such as a virtual server spun up on demand just for the purposes of testing. We do this through the concept of a Scaffold:

# indie-box-test list-scaffolds
here - A trivial scaffold that runs tests on the local machine without any insulation.

As you see, there is only one right now, which is a trivial one. If you know something about Linux containers, we’d love to see a Scaffold that tests on the local machine but inside a Linux container. To add it, you’d just have to implement a different version of the default Scaffold which can be found here. Should not be too hard for somebody who knows how to use containers.

indie-box-testing has one more cool trick up its sleeve: test plans. Currently, there are threefour:

# indie-box-test list-test-plans
default       - Walks through all States and Transitions, and attempts to backup and restore each State.
deploy-only   - Only tests whether the application can be installed.
deploy-update - Tests whether the application can be installed and updated.
simple        - Walks through all States and Transitions in sequence.

To use test plan “simple”, for example, run the test like this:

# indie-box-testing run --testplan simple GladIWasHere1AppTest.pm

In the “simple” test plan, indie-box-testing initially deploys the application, and then marches through the sequence of StateChecks and StateTransitions one by one, and then is done.

This might take a little bit, depending on how complex the test is that’s being performed, and so test plan “deploy-only” only deploys and checks the virgin state but ignores the rest of the tests. This is quite useful when first getting going with Indie Box and a new application. The “deploy-update” test plan focuses on the all-important automatic application upgrades; will the application still run after an unattended upgrade?

Finally, the “default” test plan is the one you really want to run, which is why it is the default. This plan first does the same thing as test plan “simple”, which is to deploy the application and march through the tests in sequence. However, it also creates a backup after each state is reached. Then, once it is done marching forward, it marches backwards from the last to the virgin state. It reaches each state by restoring the backup that was taken during the march forward. Of course, it again runs the StateChecks for those states to make sure restore worked as intended. (You may have noticed above that the code for the “virgin” state checks whether the comment was posted, and wondered how that was possible. Now you know: if the backup/restore were broken, the comment would still be there after marching backwards back to the “virgin” state, and so we better test it.)

indie-box-testing took a little while to write. Well, not so much to write but to rewrite it many times, in the process morphing it from something that was rather imperative (“deploy! curl! update! backup! etc.”) to the current approach that is rather declarative (“when in this state, this needs to be true”). The beauty of this declarative approach is that the test descriptions are rather simple and compact, and that they are useful with a ton of different test plans. We expect to have test plans in the future, for example, that will install multiple applications at multiple sites on the same machine, and test them, all while not requiring additional test descriptions, just new test plans. This is turning out to be a very powerful approach.

Command-line flags --verbose and --interactive spit out more information about what the test does, and give you some basic “single-step” functionality to suspend the currently-running test plan, so you can figure out what just went wrong, for example. These are very useful during development and bug fixing.

Note: both Scaffolds and TestPlans are pluggable. Write your own, they plug right in!

Ok, that’s it! Happy testing!!

Tagged with:
Posted in Milestone, Progress, Testing

Indie Box V0.10 released with better support for cron jobs

Some applications like to run cron jobs. For example, an application might periodically want to pull data from some RSS feeds. This is best and easiest done with a cron job. And Indie Box just made it easier.

Simply define another AppConfigurationItem in the Manifest JSON, e.g. like this:

    {
        "type"         : "file",
        "name"         : "${appconfig.cronjobfile}",
        "template"     : "tmpl/cronjob.tmpl",
        "templatelang" : "varsubst"
    }

where tmpl/cronjob.tmpl is something like this:

SHELL=/bin/bash
PATH=/usr/bin
0-59/5 * * * * ${apache2.uname} ( cd "${appconfig.apache2.dir}" && /usr/bin/php ./cron.php )

Every time the application gets installed on the server, another cron job is launched that, in this case, runs every 5 minutes. They don’t get into each other’s way because they are run in different directories.

And of course they are being removed as soon as the application is removed, or upgraded.

Posted in Code, Milestone, Progress

indie-redirect is working

indie-redirect is a truly simple application: when installed at a particular URL, say http://example.com/foo, it redirects visitors to another URL that can be specified as a customization point.

Trivial, but useful. And it now exists in Github and in the Indie Box software repository. Here is an example Site JSON, which allows us to install the application like this:

# indie-box-admin deploy -f examples/s0008.json

The zoo of supported applications for Indie Box is growing.

P.S. Yes, you could simply add a few lines to your Apache config file, which does the same thing (and in fact that is exactly what indie-redirect is doing), but that’s not the point. The point is that you don’t need to know Apache config file syntax with Indie Box, or even that Apache exists or what it does! You deploy all applications the exact same way, regardless how trivial (like indie-redirect) or complex (like indie-owncloud) they are.

Tagged with:
Posted in Application, Progress

Where Indie Box keeps valuable data

Some Indie Box applications might manage a lot of data. For example, a user might use an Indie Box-managed server to manage their family photo or video collection. In this case, the data might easily swell to 100′s of Gigabytes or even more.

Indie Box stores all of its own management data on the /var partition. We also ask all application developers to store their valuable data on /var. This is consistent with existing Linux conventions, and the default for databases like MySQL (or, for Indie Box, MariaDB).

One of the advantages of this scheme is that it is fairly straightforward to add more data storage to an existing device: all that needs to happen is that /var needs to be expanded. This is easy enough for traditional filesystems like ext4, and even easier for newer filesystems like btrfs.

Another is that all the valuable data can easily be taken from one (say, underpowered) server to a new server. Simply copy /var or, if it is a separate disk, unplug the disk in one place, and mount it in another.

Finally, if you really care about your data, you want to use some kind of RAID with some redundancy. Only /var needs to be RAID, the root partition (where Indie Box also stores all program code) can always be rebuilt.

Tagged with: ,
Posted in Strategies

Waiting until Apache restarts are complete

Turns out the Apache web server doesn’t really have a good way of restarting or reloading from a script that only continues once the reload or restart is complete. This is important because Apache reloads are frequent for Indie Box (e.g. when installing new applications that require additional Apache modules, or when taking down sites temporarily during backup etc.) and race conditions are bad for user experience and for reliability.

We’re not the first ones who needed a solution for this, but Googling didn’t help, and there were no answers on superuser.com either.

So after some trial and error, we implemented our own approach: attempt the restart, and then watch the log files, until a message shows up that seems to say that the operation is complete. It’s been checked into git. So far, it works well.

Posted in Code