Jazkarta Blog

Agile Development with Plone, Revisited

2012-01-20T18:39:54Z

The announcement of the 2012 Plone Symposium East (thank you Weblion! always a great gathering) is reminding me of the talk I gave there last year on Agile Development with Plone. Here at Jazkarta we’re engaged in another large project, similar in many ways to the one that inspired that talk. Because our project management tools and techniques have evolved a bit since last year, I figured it was time for an update.

The Projects

Last year at this time, we were just getting started on a major website redesign for the University of Minnesota Press. There’s lots of information about this project in my Plone Conference presentation How To Get a Fabulous Website on a Modest Budget Using Plone. Briefly, it involved a major data conversion from a FileMaker database into Plone, a small number of complex custom content types, extensive design work, and a featured “explore” section powered by eea.facetednavigation.

Our new project is a major website redesign for Dumbarton Oaks Research Library and Collection, a research institute of Harvard University and the home of a little known but outstanding museum and garden in Washington, D.C. We began with a discovery phase last summer, and have been working on implementation since November.

What’s the Same

Both of these are fixed budget projects with a large number of desired features, and at the outset many of these were inadequately defined for development purposes. In both cases there are multiple stakeholders with differing priorities, but there is an intelligent, proactive project owner who is trusted and empowered to make decisions. The Jazkarta team is the same – Carlos de la Guardia in Mexico, Alec Mitchell in California, and me in Massachusetts – and none of us ever work full time on any of our projects.

This all adds up to the same basic project management challenge: how to adapt agile techniques to manage a fixed price, flexible scope project with a distributed, part time team.

What’s Different

Happily, the Dumbarton Oaks project does not involve a complicated data import from a FileMaker database. For UMP this required a collective.transmogrifier script, many many many trial and error imports, and over 60 hours of Jazkarta development time. Dumbarton Oaks currently has a static HTML website, which is much simpler to import. Funnelweb made this easy – only a few trial imports and some content massaging were required. We’re not a quarter through the project but all the old content is already on staging. Total development effort: 10 hours.

Because the content import was relatively easy and the project mostly requires custom features and content types that are independent of each other, we have been able to divide up the work into several releases. This gives us a series of smaller planning horizons and allows us to be more agile (more on this later). Each release can potentially be deployed to production – and even if we decide not to, new content is developed on staging which gives the new features a thorough vetting. No matter how much you thought you tested, there are always problems that don’t get discovered until the software sees serious use. Having a series of small production deployments avoids piling all the bug fixes and theme tweaks into a single mega deployment phase at the end of the project.

Our Evolving Agile Development Practices

Our basic development process remains the same:

Discovery
Define schedule and resources (how the budget will get spent)
User story development
Story estimation (we use planning poker)
A series of development iterations alternated with evaluation periods
Production deployment

During discovery the project owner and I distributed the feature requirements into a series of releases, based on a combination of priorities and keeping all the stakeholders happy. (Everyone got something in almost every release.) This high level release plan allows us to do our story development and estimation one release at a time, which saves time and money – especially considering that all the developers participate in planning poker sessions. Since we know the customer wants more than their fixed budget will get them, and we also know they will change their mind about what they want as the project progresses, it makes sense to postpone the effort of writing and estimating stories we may never get to.

As in previous projects, I took the first cut at turning the project requirements into stories, then discussed and revised the stories with the project owner. Because I have intimate knowledge of Plone, I can translate requirements into stories that make sense to develop. For example, we don’t need to spend time defining stories for CMS functionality or for things that Plone has great add-ons for (like staff directories and bibliographies). Some things may turn out to need tweaking, but it is better to give the customer stock Plone and add-on features to try out first. They may decide to live with the stock features and put their development dollars into new features instead of making changes.

Our communication tools remain the same: Skype, IRC, chat, Google Calendar, and Google Documents. This combination is the secret sauce that allows a distributed team to work as if they were in the same room. We did switch from ClueMapper (based on svn and trac) to github for source code repository and ticket tracking. We miss a few management features, but the developers are much happier working in a dvcs.

Our development iterations remain the same – one week each, followed by a week of evaluation during which the customer has time to test and reflect on what they want to do next. We kick off each iteration with a planning meeting, which surfaces any questions about the stories we’re tackling and allows us to do task breakdown and assignment while we are all in the virtual room of a Skype conference call. During the iteration we have a short standup meeting every day to keep things on track – everyone says what they did yesterday, what they’ll do today, and raises any blocking issues. My notes from the standups are written in a Google spreadsheet which is shared with everyone. This gives the project owner access to detailed information about how things are going, if they miss a standup meeting.

The biggest change we’ve made for this project is to adopt ScrumDo for story development and iteration planning. ScrumDo is an open source agile project management tool that’s written in Python. We are using the hosted, SaaS version that’s at scrumdo.com – accounts are free for up to 10 users and 10 projects. ScrumDo has a friendly drag and drop UI that makes it easy for for project owners to plan iterations, edit stories, add comments, and see results. Unlike Pivotal Tracker, it allows flexible iteration scheduling and story and task assignments. Google Docs provide better simultaneous editing for multiple users and more flexible formatting, but ScrumDo makes it really easy to add and change tasks, assign them, and mark them as complete. If it managed estimated and actual hours I’d totally love it.

ScrumDo added a Planning Poker tool last March, and this has been a real time saver. It’s a web page where the scrum master selects a story to size and everyone puts down a story point card. Once you play your card you can see the others, and after discussion and reaching consensus, the scrum master assigns the winning value to the story. What set of cards to use is one of the ScrumDo project settings (we use modified Fibonacci).

ScrumDo Planning Poker interface

The Value of Agile Project Management

The planning and standup meetings required by an agile approach do take time. For the UMP project, the meetings and project management activities combined used almost 40% of the total budget. But what is the alternative? Written requirements “thrown over the wall” to developers are a recipe for failure. The customers and the developers have to talk to each other to understand the project and its development trade offs. I’ve never had a customer complain that I’m using too much project management time when I’m running a project in this way. The value of open and frequent communications and iterative development cycles is very clear to them. They get full ownership of the project, choose exactly what it will deliver at every step along the way, and in the end they are delighted to get the system they envisioned.

Plone Quick Start for EC2 now available in all regions

2011-01-20T08:25:18Z

Our Plone Quick Start Amazon EC2 machine images have been in enough demand to warrant availability in all four EC2 regions. We have responded to that demand and now anyone can test drive Plone 4 easily in their preferred region. Just go to our free signup page and fill out the form. You will receive a confirmation email and will be able to create your instance immediately after that.

The Plone 4 Quick Start gives you a fully working Plone 4.0.2 site, running on Ubuntu 10.10. The site has two instances in a high availability configuration using Nginx as an HTTP frontend, Varnish for caching and HAProxy for load balancing. Supervisor is used for starting and stopping the Plone instances and ZEO.

If you’ve been wanting to try out Plone or know someone who is, this is probably as easy as it gets. Give Plone a chance.

Plone on WSGI sprint report

2010-11-22T21:45:11Z

Two laptops, two sprinters and two bottles of whiskey. What could possibly go wrong?

This is my attempt to document what I can only describe as the “Plone on Whisk-gi” sprint… As part of the Plone deployment sprint at the recent Plone Conference in Bristol, Reed O’Brien and I decided to try to get Plone 4 working on WSGI.

Why is this important? Hanno Schlichting (current Zope2 release manager) tells us on his blog post about the recent Zope 2.13 release:

Zope 2.13 comes with native WSGI support. First pioneered in the repoze.zope2 project, this capability finally found its way back into the core and obsoletes the externally managed project. With WSGI, Zope 2 can natively talk to a variety of web servers and isn’t restricted to its own ZServer anymore. It also opens up new possibilities for writing or reusing middleware in Zope 2 or factoring out capabilities into WSGI endware. It’s expected that this new deployment model will over time become the default and the old ZServer implementation will be deprecated. There’s no concrete timeline for this yet.

What is Hanno saying here? Basically, it means that Zope 2.13 (and subsequently Plone) can run on a server without the added complexity of a long-running process (ZServer). Just as PHP-based applications enjoy ease of deployment since Apache already has mod_php enabled, so too could Zope-based applications be easily deployed if mod_wsgi were enabled.

In thinking about how Plone and Zope can be more accessible to systems administrators and hosting companies, moving to a scenario in which Zope does not require a separate long-running process, and can be deployed essentially at the Apache layer is a step in the right direction.

As Hanno alludes to above, with the WSGI deployment model, Zope/Plone are able to tap into a whole ecosystem of 3rd party WSGI middleware tools. Repoze is an initiative to extract many of the core Zope technologies into reusable WSGI middleware components, that can be used in other Python web application frameworks. More WSGI components can be found on PyPi by searching for “wsgi”.

Also, by adopting a more typical deployment method, Zope will not seem so exotic to developers coming to Zope from other Python web frameworks. For example, the official Django documentation recommends deploying Django using mod_wsgi, so Django developers will likely already be familiar with this way of deploying Python web applications. Incidentally, if you’re doing Django development, check out twod.wsgi which makes WSGI a first-class citizen in Django applications.

How does Plone fit in this picture?

Plone, arguably being the most popular Zope2 application on the market, can stand to benefit from the WSGI deployment strategy. There have already been some efforts to do so with repoze.plone and repoze.zope2 (now deprecated with the release of Zope 2.13).

Calvin Hendrix Parker blogged about deploying Plone and Zine together with Deliverance using Repoze, and Wojciech Lichota blogged about his buildout for Plone 3 with Deliverance on WSGI. Nathan Van Gheem has also written a couple blog posts about his experiences running Plone 4 with a Zope2 WSGI and running Plone 4b4 with Zope 2.13.0a1.

Now that Plone 4 is officially out and Zope 2.13 has had a final release, the instructions for getting this working are considerably easier.

Here are the steps required to get Plone 4 running under WSGI using paster.

1. Create a buildout.cfg file with the following:

[buildout]
extends =

http://dist.plone.org/release/4.0.1/versions.cfg

http://download.zope.org/zopetoolkit/index/1.0/zopeapp-versions.cfg

http://download.zope.org/Zope2/index/2.13.0/versions.cfg

parts =
    instance
    paster

versions = versions

eggs +=
    roman
    Products.ZSQLMethods

[versions]
Zope2=2.13.0
Products.ZSQLMethods = 2.13.3

[instance]
recipe = plone.recipe.zope2instance
user = admin:admin
eggs +=
    ${buildout:eggs}
    Plone
    Paste
    PasteScript
    PasteDeploy
    repoze.tm2
    repoze.retry

[paster]
recipe = repoze.recipe.egg
scripts = paster
eggs = ${instance:eggs}

2. Create a zope2.ini file:

[DEFAULT]
debug = True

[app:zope]
use = egg:Zope2#main
zope_conf = %(here)s/parts/instance/etc/zope.conf

[pipeline:main]
pipeline =
    egg:paste#evalerror
    egg:repoze.retry#retry
    egg:repoze.tm2#tm
    zope

[server:main]
use = egg:paste#http
host = localhost
port = 8080

3. Run the buildout and modify the zope.conf file:

Run the buildout which will generate the paster script in the bin dir and the zope.conf file in parts/instance/etc. You then need to comment out the <http-server> part in this file. If you don’t comment out these lines, then paster will complain that there is already a server running on port 8080.

#<http-server>
# valid keys are "address" and "force-connection-close"
#  address 8080#
#</http-server>

4. Start up paster using the zope2.ini file:

$ paster serve zope2.ini
Starting server in PID 4666.
serving on http://127.0.0.1:8080

You can now go to http://127.0.0.1:8080 to see your Plone site served up by Zope on WSGI!

In the next blog post, we will describe how to get this semi-working in a production environment using mod_wsgi. For a sneak preview you can check out this buildout that Reed put together at the sprint. Note that there are a few patches that must be applied to Zope2, and hopefully Tres Seaver is reading this and will respond to them.

What’s the official word on WSGI support for Plone 4?

Plone 4.1 will ship with Zope 2.13, but as Hanno states in PLIP #10776:

While Zope 2.13 provides native WSGI functionality, it is outside of this PLIP to make use of this in any way. There’s a large number of open questions around WSGI and the changes it requires to instance creation and setup, which warrant their own PLIP. A good number of those will also have an impact on the Zope 2 codebase itself, which is going to be already past a beta release by the time this PLIP gets started and thus will have to wait for Zope 2.14. I expect to see more community experimentation being done with WSGI once the capability is there. We might see a PLIP for Plone 4.2 to see those experimentations being solidified into good practice, standards and documentation.

So in other words, it’s still on the bleeding edge to try to use it, and probably not advised for production use just yet. As Hanno mentions, it’s expected that there will be more experimentation around WSGI, and as a community, we can come up with some best practices and documentation for how to do it in a sane way. Please let me know if you have done any experimentation with WSGI already, or if you’re interested in getting involved in these efforts.

Easily launch Plone sites on Amazon EC2 with mr.awsome

2010-11-08T23:02:48Z

While we’ve been promoting our Plone 3 and Plone 4 AMIs which are really great to evaluate Plone on Amazon EC2 quickly, I want to tell you about another way you can get started with Amazon EC2 that gives you more control over where you launch the EC2 instances and what specific commands get executed on the server the first time it boots up. I mentioned this during my Scalable Plone hosting with Amazon EC2 talk at the Plone conference, but in this blog post, I want to go into more details about how to actually get it set up.

Here’s an example of a very simple set up of mr.awsome. You can either add this to an existing buildout (in which case you would name this file aws.cfg instead of buildout.cfg), or you can make a separate buildout as we have done here:

buildout.cfg:

[buildout]
parts = aws

[aws]
recipe = zc.recipe.egg
eggs = mr.awsome
entry-points =
    aws=mr.awsome:aws
    assh=mr.awsome:aws_ssh
arguments = configpath="${buildout:directory}/etc"

Then you put a config file in the /etc directory of your buildout, that might look something like this:

etc/aws.conf:

[securitygroup:demo-server]
description = Demo Server
connections =
    tcp 22 22 0.0.0.0/0
    tcp 80 80 0.0.0.0/0
    tcp 8080 8080 0.0.0.0/0

[instance:demo-server]
keypair = demo-server-keypair
securitygroups = demo-server
region = us-east-1
placement = us-east-1c
instance_type = t1.micro
image = ami-480df921
startup_script = startup-demo-server.sh
fabfile = fabfile.py

There are a few things to note about this configuration:

We open port 8080 in the firewall so that we can access our Plone site directly without making an Apache vhost, but for a production server you would not want to do this. We leave it as an exercise to the reader to extend the startup-demo-server.sh script to create the Apache vhost config file and activate it.
Creating a keypair

You’ll need to have created a keypair in the AWS Management Console or using ElasticFox, that has the name “demo-server-keypair”. You’ll want to save the keypair file someplace safe on your computer, as you will need this in order to SSH into the server.
We specify the “demo-server” security group that we defined at the top of the config file. This is important not only for defining the ports to be open, but also for identifying which EC2 instance is the demo server, since Amazon doesn’t provide any other way to “tag” a server with such an identifier, so we use the security groups for that.
You might have to play around with the settings for the region, placement, instance type and AMI image ID. Not all instance types are available in all regions, and the AMI ID must match the architecture (32-bit or 64-bit) and the region (us-east, us-west, eu-west, ap-southeast). We use the Ubuntu 10.04 LTS AMIs provided by Canonical.
If you choose an AMI that uses EBS as it’s root store (also known as EBS bootable instances), then you will be able to stop the EC2 instance, restart it later and not lose all the data in the meantime. The other advantage is that you don’t pay for the instance while it’s in the stopped state. If you use the “instance store”, then your only option is to terminate the instance at which time you lose all the data, unless you’ve attached an EBS volume and have stored all data you want to persist there.
If you are a new AWS customer, you qualify for free AWS hosting for one year with the smallest t1.micro instance. This is probably enough horsepower to run a very low traffic Plone 4 site, but I wouldn’t recommend trying to run Plone 3, or any site that has a significant amount of traffic. The t1.micro instance size is great for making demo sites or test instances, but should not be considered ideal for a production site.
The startup script is the script that will be run the first time the EC2 instance starts up. This makes it possible to use a very generic AMI, and then use the startup script to bootstrap the server with all the dependencies needed to run Plone. This is also where you can customize the script to take care of things specific to your Plone site, such as importing data.

Here is an example of an atypical startup script, but one that I think we’ll be making more use of in the future. I found out about this at the Plone Conference when Alan Runyan mentioned on a discussion list that his company Enfold Systems had come up with an Ubuntu package for Plone. This means that to install Plone, you literally can type: apt-get install plone-default, which will install Plone 4 and a default Plone site. This startup script makes use of this Ubuntu package to bootstrap the server.

etc/startup-demo-server.sh:

#!/bin/bash
set -e -x
export DEBIAN_FRONTEND=noninteractive
wget http://www.enfoldsystems.com/pubkey.gpg
apt-key add ./pubkey.gpg
echo "deb http://dist.clients.enfoldsystems.com/ubuntu lucid universe" >> /etc/apt/sources.list
apt-get update && apt-get upgrade -y --force-yes
apt-get install plone-default -y --force-yes
/etc/init.d/plone-default

What this script does it essentially add the plone-default and plone4 packages to your apt sources.list file, and then installs Plone 4 on the server and sets up a default Plone site, and starts it up. What’s great about this way of installing Plone is that it’s very fast since all the eggs are already pre-compiled for your platform (no need to install gcc), and you don’t have to run buildout on the server which can often be very error-prone and risky.

The last thing that you need to do is set up your AWS key and secret key. The way I do this is by making a simple bash script with the following:

setkeys.sh

#!/bin/bash
export AWS_ACCESS_KEY_ID="<your_access_key_id>"
export AWS_SECRET_ACCESS_KEY="<your_secret_access_key>"

Obviously, you need to replace the <your_access_key_id> and <your_secret_access_key> with your actual keys which you can find by logging into aws.amazon.com, and clicking on the Security Credentials tab.

Getting your access key and secret key

Then to set these environment variables:

$ source setkeys.sh

Once your keys are set, and you have the aws.conf file and startup-demo-server.sh script in place, then you can run your buildout and try starting up a demo-server on Amazon:

$ ./bin/buildout -v
...
$ ./bin/aws start demo-server

INFO: Instance 'demo-server' unavailable
INFO: Creating instance 'demo-server'
INFO: Instance created, waiting until it's available
....
INFO: Instance 'demo-server' available
INFO: Instance running.
INFO: Instances DNS name ec2-50-16-25-92.compute-1.amazonaws.com
INFO: Instances public DNS name ec2-50-16-25-92.compute-1.amazonaws.com

Your site is now running at http://ec2-50-16-25-92.compute-1.amazonaws.com:8080. You’ll need to substitute the domain name that is returned when you run this command, and remember to put the port :8080 at the end, since we haven’t set up an Apache vhost to proxy requests from port 80 to 8080.

Within a few minutes you should see the Welcome to Plone screen!

Please note that if you are not a new AWS customer, or you already have a t1.micro instance running, you will be charged by Amazon for every hour that this instance is running, so when you’re done evaluating it, you should either stop it (if you used an EBS bootable instance), or terminate it (if you used an instance store).

$ ./bin/aws stop demo-server
$ ./bin/aws terminate demo-server

You can also stop the instance from the AWS console or using ElasticFox. That’s it! Hope you’ve found this little tutorial useful, and let me know if you have any questions about how mr.awsome or Amazon EC2 works for hosting Plone sites.

Tagged: amazon, aws, ec2, Plone

Scalable Plone hosting with Amazon EC2

2010-11-08T01:52:33Z

At the recent Plone Conference in Bristol, UK, I had the privilege of giving a talk about Scalable Plone hosting with Amazon EC2. In particular, the talk centered around a project that we did recently with Rice University. Rice has been at the forefront of online teaching and learning platforms with their Plone-powered Connexions site which serves more than 2 million visitors every month! In fact, I remember first hearing about Connexions at the very first Plone conference in New Orleans, so they’ve been around for a long time, and publish an excellent development blog. The Connexions project is a significant showcase for the power and flexibility of Plone as a development platform.

What’s even more cool is that they’ve open-sourced the software that powers the www.cnx.org site as Rhaptos, so if you want to set up your own open learning platform, you can check out the code and be up and running in minutes.

But it didn’t always used to be this easy though. The suite of software that powers a Rhaptos site has evolved over the years, and has grown into a formidable stack. In order to get this software installed on a server used to require many hours of reading documentation and manual steps.

One reason that Rice hired Enfold Systems and Jazkarta was to get expert assistance in getting the Rhaptos software stack up to modern standards with buildout to make a repeatable deployment and a full test suite to ensure that bugs would be identified more quickly when running tests. Enfold did this beautifully, and then it was Jazkarta’s job to make it easy to deploy Rhaptos up onto an Amazon EC2 instance with just a few commands. For this we used mr.awsome, a buildout recipe written by Florian Schulze from Jarn.

You can see the AWS buildout that we created to deploy not only test/demo instances but full multi-server production deployments of Rhaptos and CNX. There’s also full Sphinx-based documentation that explains how to use the buildout and launch new instances of Rhaptos on Amazon EC2.

Since I completely spaced at the end of my talk, and forgot to invite Mark and Roche to come up and tell people about the conference sprint, I don’t want to forget this time. The next sprint is happening at the Connexions conference on February 10, so if you’re a developer interested in working on Connexions/Rhaptos, this is an excellent opportunity to get involved and meet the core developers.

Here is the video and slides from my talk:

Plone quick-start: easily evaluate Plone using Amazon EC2

2010-09-29T03:44:59Z

Plone 4 is undoubtedly one of the best open source content management systems out there, and it’s also very easy to download and install. However, there are times when you just need a quick evaluation or have to set up a test or demo site that people at different locations can use. In those cases, our Plone quick-start instances may be just what you need.

To quickly and painlessly get your own Plone site, you’ll need an Amazon Web Services account. Running your Plone site there is not free but it’s comparatively cheap and the set up cost is zero. We have prepared a couple of fully featured Plone Amazon Machine Images (AMIs), so all you need to do is fill out this form to add the Plone image that you want to use to your AWS account. You can be up and running in minutes.

What is included with the Plone quick-start AMI:

Latest stable releases of Plone: 4.0 or 3.3.5
Apache is configured to proxy to Varnish for caching
Varnish is configured to proxy to Zope
CacheFu is included (Products.CacheSetup for Plone 3.3.5, plone.app.caching for Plone 4.0)
Supervisor is used to start/stop the Zope instance
Backups of the Data.fs file are run daily (incremental) and weekly (full)
ZODB packing prior to backups
Logfile rotation of Zope logs
Start/stop the instance and only pay Amazon while the instance is running. Stop the instance and not lose your data.
Full SSH access to the server using your keypair for further configuration and customization

Note that at the moment we only have set up the Plone AMIs for the US East region. If you are interested in having them available for other regions, please leave a comment on this post. We’ll consider adding other regions if there’s enough interest.

Handling large files in Plone with ore.bigfile

2010-09-21T20:06:15Z

Plone The Application is great. Out of the box it has a set of sensible defaults and features that are useful for a wide range of use cases. You can install Plone and without having to make any adjustments start using it immediately.

The stack that Plone is built on is very flexible, but if you push the defaults hard enough you will begin to discover the limits. This is the scaling issue: how does the application perform under high load. For Plone, as well as many other web applications, a sure way to increase the load on the system is to have it process large files. What is a large file? The answer is ‘it depends’. For purposes of this post we will assume it is any file large enough to make Plone feel slow for the end user, or worse result in a time out.

To be clear, handling large files is not a problem exclusive to Zope/Plone, it is an issue with many application servers. This description of the issue come straight from the Rails community:

“When a browser uploads a file, it encodes the contents in a format
called ‘multipart mime’ (it’s the same format that gets used when
you send an email attachment). In order for your application to do
something with that file, rails has to undo this encoding. To do
this requires reading the huge request body, and matching each line
against a few regular expressions. This can be incredibly slow and
use a huge amount of CPU and memory.

While this parsing is happening your rails process is busy, and can’t
handle other requests. Pity the poor user stuck behind a request which
contains a 100M upload!”

http://therailsway.com/tags/porter

Plone has the same issue of thread blocking. Let’s take the example of a 15MB file uploaded to Plone, with the file contents ultimately stored as a BLOB in the ZODB.

Browser encodes file data in multipart mime format (payload) and sends this in the request body along with additional parameters such as the name of the file, etc.
Apache receives the file and forwards the request to Zope.
Zope must undo the coding, which is both CPU and memory intensive. During this processing the Zope thread is blocked, thus unable to service additional requests. The risk is that, in a multiple user scenario, you can potentially tie up all available threads.
Once Zope has parsed the request, it can then write the file data – in this case it writes the data to ZODB BLOB storage.

For many sites, having Zope process the file in the above manner is an adequate solution. The size of the uploads is not usually large, and the site traffic is not heavy. Recently however, we had a client who needed a high availability site that could handle large file uploads and downloads. We had to find a way to increase Plone’s stock file handling performance. In order to do this we partnered with Kapil Thangavelu to develop both an implementation strategy and supporting code.

The general strategy can be summed up as:

Offload ﬁle encoding/unencoding and read/write operations from Plone.
Web servers are really good at handling the above tasks.
Unlike application servers, for web servers like Apache ﬁle streaming is fast and threads are cheap.

Learning from Rails

The first step is to move the file processing from Plone to Apache (or your web server of choice). A solution that comes from the Rails community is to use the ModPorter Apache module.

In a nutshell, ModPorter does the following to an incoming request.

Parses the multipart mime data
Writes the ﬁle to disk
Modifies the request to contain a pointer to the temp ﬁle on disk

Clearly ModPorter is powerful magic, but we still have to integrate it with Plone. This is where Kapil’s ore.bigfile package comes into play.

ore.bigfile provides the following:

A modified file upload widget that supports two upload methods
Code to securely move the temp file created by the web server/ModPorter to BLOB Storage and link the data to a Plone File object (Blob.consumeFile).
Code to provide an efficient method of downloading large files.
An alternate (not CMFEditions) way of providing file versioning

With ore.bigfile in place, once ModPorter has finished writing the file data to disk, Blob.consumeFile is called with the necessary parameters to store the data in ZODB BLOB storage. The key is that Plone only has to process a lightweight request body to create the Plone File – the heavy lifting has already been done by Apache.

I mentioned that the file upload widget supports two upload methods. The first method supports uploading via the browser, as described above. The second method supports processing a file that was uploaded using an FTP client. Browsers in general are unreliable for long upload operations. However most FTP clients support continuation in the event that the transfer is interrupted. So for very large uploads (again it depends on the speed and quality of your network), the file is first uploaded via FTP to an incoming directory. When a new File object is created in Plone, the user has a choice to upload a new file using the browser, or select from list of files that have already been uploaded to the incoming directory. Data files are swept from the incoming directory after they are used in the creation of a Plone File.

What goes up, must come down

Solving the upload problem only gets you half way there.

Downloading large files will also tie up a Zope thread until the file is passed entirely to the front-end webserver. The solution is to use the second feature of ore.bigfile, which offloads the serving of the file from Zope to the front-end web server. The initial request is still handled by Plone to ensure the necessary security checks are in place, but instead of having Zope read the data from BLOB storage, ore.bigfile constructs a custom response to send to the webserver. When using Apache, the response consists of an X-Sendfile header and a path to the file data in BLOB storage. This results in Apache reading the file directly from disk, instead of having to read the data from Zope. This ensures that the Zope thread is quickly released so that it can respond to new requests.

Where to go from here

To recap, the default installation of Plone is incredibly useful. It works because it makes as few assumptions as possible about the installation’s environment. However for most production sites you need to begin integrating other tools – such as a caching proxy – to maintain system performance. In the case of caching, Plone uses a combination of internal software (Cache-Fu for example) and supporting software (Varnsih, Squid, etc.) As with caching, efficient large file handling could be implemented in Plone by extending the code and concepts from ore.bigfile to provide utilities that could be configured to work with common front-end webservers.

Would you buy this guy a beer?

2010-05-18T23:03:27Z

Attracting and keeping talented developers is the lifeblood of any open source project. Domen Kožar is one such developer. I met Domen at the 2009 Plone Conference and he showed me a thing or two about Vim that I didn’t know. He has already contributed to a number of Plone related projects. As he is currently a student, he needs a little help from the community so he can attend the Sauna Sprint. I am sure a lot of us can relate to the unique financial circumstances you find yourself in as a student. You can read more about his request here. This community has come through in the past to help a fellow member. I am asking, for a minimum donation of the cost of a beer in your country, help Domen get to the sprint. Let’s make this happen.

Announcing deliveranceproject.org, a new home for Deliverance

2010-05-16T04:26:59Z

At the CMSExpo 2010 conference in Chicago, Chris Johnson from ifPeople asked me what URL to give people who were interested in learning more about Deliverance.

I was embarrassed to say that there wasn’t really a good landing page – the page on coactivate.org was outdated, and the Deliverance documentation was not exactly very welcoming to a newcomers.

I had registered the domain name deliveranceproject.org awhile ago and even had my brother Luke put together this logo you see to the right.

I was intending to make the site a showcase of Deliverance itself, bringing the documentation, mailing list archive, code repository and Trac site under a single umbrella with a unified theme using Deliverance.

But as my colleague Aaron likes to say, “The perfect is the enemy of the good enough,” and I never launched the site because it wasn’t ready. So I pinged Aaron in the morning of the first day of the conference and said, “How quickly can you whip up a static page for deliveranceproject.org”. He said 1 hr, and one hour later, deliveranceproject.org was born! Thanks Aaron for the quick turnaround. Chris Johnson quickly updated his slides to include this URL, as did I.

DeliveranceProject.org

The site is purposely a very simple one-page of information with links and a couple embedded presentations. Eventually we may expand it to have more examples and showcase what cool things you can do with Deliverance, but for now it serves it’s purpose: a place to direct you to other resources.

I also have Aaron to thank for his expertly prepared slides about Deliverance which he put together for World Plone Day Boston, and which I re-purposed for my talk at CMSExpo, with a few extra easter eggs thrown in. Here are the slides:

There’s also a video that I recorded on my Nexus One using Qik. I’m not really sure why the video is rotated, but hopefully your neck won’t get strained too much.

I think we were somewhat successful in spreading the good news about Deliverance to the conference goers, especially the non-Plone people from the Drupal, Joomla! and WordPress communities. A couple guys who were in my session said it was the highlight of the conference for them, and another guy said that learning about Deliverance saved him $9,000 that he otherwise would have had to pay an outside consultant to build a Plone theme add-on.

Deliverance training opportunity

If you’re interested in learning more about using Deliverance to theme your Plone site, or any site for that matter, I’ll be teaching a class right before the Plone Symposium East conference on May 24. There are still spots left, so if you want to take your theming knowledge to the next level, register today for the class.

Take my class on monday and then go to Chrissy’s theming Plone or Steve McMahon’s Javascript, jQuery and jQuery Tools classes on tuesday and wednesday, or Tres and Chris class on WSGI. See all the excellent training classes that are offered at very reasonable prices.

Tagged: cmsexpo, cmsx, deliverance, Plone, pse2010, theming, xdv

Rapid deployment of Plone to Amazon EC2

2010-05-16T02:54:16Z

If you were at my Plone hosting on Amazon EC2 talk at the Plone Conference 2009 in Budapest, you know that I have a keen interest in making it easier to get a Plone site deployed and running on a production server. While there are many hosting providers that offer VPS machines or “slices”, none of them come close to providing the level of scalability and flexibility that Amazon provides. Whether you want a single small server or a cluster of highly available machines that auto-scale as traffic and resources on your site increases, Amazon offers room to grow as your needs change over time.

Here is the video and presentation slides of my talk:

Case study: Deploying Plone-powered Rhaptos software to Amazon EC2

On a recent customer project with Rice University, I had the opportunity to explore using Amazon EC2 to deploy the Rhaptos software. Rhaptos is the open source software built on Plone that powers Connexions, a web-based repository of educational materials. I remember hearing about Connexions at the very first Plone Conference 2003 in New Orleans, and being very impressed with what they had built on top of Plone.

In this 2006 TED talk by Connexions founder Richard Baraniuk, he explains the vision behind Connexions : cutting out the textbook and allowing teachers to share and modify course materials freely, anywhere in the world.

The Connexions team is taking it a step further and making it even easier for prospective universities to evaluate and run Rhaptos by providing a cloud-based hosting solution.

Scalable infrastructure

The Connexions site gets over 1,500,000 visitors per month, so having an infrastructure that is built to handle this amount of traffic is essential. Using Amazon’s Elastic Load Balancing (ELB), auto-scaling and CloudFront as a CDN, plus the ease with which you can launch additional instances to run as Zeo clients, makes Amazon a perfect platform on which to build a highly scalable Plone site.

Fault tolerance and disaster recovery

In addition, the servers that currently host the Connexions site are located at Rice University in Houston, which is in hurricane territory. One reason for exploring a cloud-based hosting strategy is to provide a backup plan in the event that a hurricane were to strike Houston and take out the Rice University data center, the entire operation could be quickly deployed up on Amazon EC2.

Easy testing and demoing

Another advantage of building this deployment strategy is that new EC2 instances can quickly be launched for testing or demo purposes. For example, if another university wants to have their own sandbox to explore the Rhaptos software, they can very easily launch a machine that has all the software pre-installed and ready to use.

Or if you want to run a buildbot farm to run unit tests or functional tests, Amazon EC2 makes that very easy – and you only pay for the time that the machine is active. Once the tests are completed, you can shut down the machine and stop paying for it.

As you will see in this blog post, launching a new server with Plone already installed on it, is a matter of typing in one command, or pointing-n-clicking in a web-based dashboard.

What makes Amazon EC2 different?

Before we get started, there are a few key concepts that I want to explain. Amazon’s Web Services (AWS) differ from traditional hosting providers in several ways:

The virtual machines are intended to be disposable
Any data you want to persist needs to be installed on Elastic Block Storage (EBS) or S3
You only pay for what you use
You can take a snapshot of a running instance and turn it into an Amazon Machine Image (AMI) which can be shared with others

Instead of starting up a machine and putting all of your data on it, you typically start up a machine, attach and mount an EBS volume to it, and store all of your data on that volume. If the EC2 instance shuts down or gets terminated for any reason, your data will be safely persisted on the EBS volume. You can also take snapshots of the EBS volumes and restore from them, and you can create new EBS volumes from a saved snapshot, or share the snapshot with another Amazon AWS customer.

Think of an EBS volume as an external harddrive, that you plug into the machine and access as if it were part of your file system. The performance of the EBS volumes rivals that of the local data store, so it’s advised to use them wherever possible, not only for safety reasons but because you’ll get better performance out of a high data I/O application such as Zope.

Rhaptos Enterprise deployment usage scenarios

For Rhaptos Enterprise, we explored 3 different scenarios for deployment balancing simplicity with flexibility. The following diagram shows these scenarios:

0) Standard AMI bootstrapped with user-data script

After you’ve signed up for an Amazon AWS account, the next thing you would usually do is select a base AMI that you want to use. We chose a standard Debian Lenny AMI put together by the prolific Eric Hammond who also maintains an excellent blog about Amazon EC2. I prefer using the official Ubuntu Karmic AMIs from Canonical but Connexions was already familiar with Debian, so we stuck with that for this project.

We then started up a small instance on EC2 using the Debian Lenny AMI (ami-dcf615b5) and bootstrapped it with a user-data script with all of our Rhaptos/CNX specific dependencies. Then we checked out the buildout on the server which initiated the dataset download and import. Everything was built from scratch which takes 30 min to 1 hr from start to finish to bootstrap the server, run the buildout and import all the data into the PostgreSQL database.

Pros:

the code is always fresh since it’s checked out from trunk
the data is always fresh since it’s downloaded every time from the repository
we can check out a particular branch for testing

Cons:

it takes a long time to run the buildout and import the data every time
it’s risky to run the buildout in place, because it can fail if there any servers that are offline
the data is co-mingled with the application which can make it more tricky to upgrade

Summary: This first step is necessary in order to make snapshots of the AMIs and EBS volumes which are used in the other scenarios we’re about to describe. This approach to deploying the stack takes a long time, so it’s not very suitable for demo instances, but it is useful for testing out a particular branch of the codebase, or a specialized dataset.

1) All-in-one: AMIs containing system dependencies, application and data

In this usage scenario, we create specialized AMIs based on the standard Debian Lenny AMI, and make these AMIs publicly available in the AMI directory. These AMIs contain the entire stack necessary to run an instance of Rhaptos or Connexions software including the data. Everything lives on the EC2 instance, and no EBS volumes are mounted. These instances launch in about 3-7 minutes and Plone is running almost immediately after they’ve finished booting up.

Pros:

instances start up quickly because everything is pre-loaded (no need to copy large data sets and import them into the database)
deployment is simple because all the software and data necessary to run the app is contained on the AMI (no need to create and mount EBS volumes)
management is easy because there is just a single EC2 instance to start/stop in the control panel (no need to track down associated EBS volumes)

Cons:

if instance terminates unexpectedly, all data is lost application and data are co-mingled, so cannot easily upgrade application without moving data out of the way first
system and application/data are co-mingled, so cannot easily upgrade OS dependencies without moving application out of the way

Summary: This approach is good for demo instances and testing instances in which the server is temporal and disposable. No need for backups, redundancy, and failover. Everything is self-contained, so it makes it easy to deploy. Should not be used for production instances, because the risk for data loss is too great, and maintainability is potentially difficult.

2) AMI with system dependencies + EBS volume with application and data

In this scenario, we separate out the system dependencies that don’t change very often (Squid, Postgres, Postfix, etc.) as part of the AMI, and store all of the application code (that changes more often) in an EBS
snapshot. When new EC2 instances are launched, a new EBS volume will be created from the master EBS snapshot, and it will already have all of the application code and data pre-loaded.

Pros:

if the EC2 instance terminates for some reason, all the important data is preserved persistently on the EBS volume.
updating to a new version of the application only requires mounting a new EBS volume from a snapshot of the new application code. (no need to build a whole new AMI)
can update the OS level dependencies by simply launching a newly updated AMI and attaching the EBS volume to it.

Cons:

introduces an EBS volume into the stack, which makes the deployment story more complex, especially for demo sites.
data is co-mingled with the application, so in order to update the application, you either update in-place (risky) or move the data out of the way first.
management is more complicated because you must keep track of which EBS volume is associated with which EC2 instance.

Summary: this solution adds complexity with an EBS volume, but with the benefit of persistent data storage and the added flexibility of being able to upgrade the OS level dependencies or application independently of each other. This solution is suitable for instances that are going to be around for awhile, so persistent data is important, but instances that might not be upgraded so often, so keeping the data co-mingled with the application is acceptable.

3) AMI with system dependencies + EBS volume with application + EBS volume with data

This is similar to #2, except that we introduce a 2nd EBS volume which contains the data (Data.fs and Postgres database). This requires modifying the standard Postgres layout to store the database on a volume other than where the Postgres application is stored, and modifying Zope to store it’s Data.fs file on a separate volume from where the Zope instance is living.

Pros:

all the benefits of #2 separation of concerns: OS, application and data can be managed independently of each other
don’t need to SSH into the machine to update the application or restore data from a backup. This can all be done within the ElasticFox control panel by mounting EBS volumes from snapshots.

Cons:

more EBS volumes to manage means more complexity
more EBS volumes means higher cost potentially
changing the location where Postgres and Zope store their data diverges from convention

Summary: this solution has the most complexity, but also the most flexibility in that each part is separate and can be upgraded independently of the other parts. This way of componentizing the server architecture is advantageous from a redundancy and upgradeability perspective, but comes at the cost of more moving parts, and more places that things can become disconnected due to mismanagement of these resources.

Conclusion

We chose #3 for Enterprise Rhaptos because it’s the most flexible, although it’s likely that an EBS bootable AMI will be created for easy launching of testing and demo instances. Thanks to Florian Schulze’s mr.awsome, with one command, a new site can be created in a matter of minutes:

$ ./bin/aws start rhaptos-partial-32

This command will launch a new EC2 instance on a 32-bit machine, create an EBS volume from a saved EBS snapshot with a partial dataset and start the Plone site proxied behind Squid. If we wanted to launch Rhaptos on a 64-bit machine with a full dataset, we would simply substitute “rhaptos-partial-32″ for “rhaptos-full-64″.

Stay tuned for a future blog post describing how you can get started using mr.awsome to easily deploy your Plone sites to Amazon EC2.

In the meantime, the folks at Connexions have been blogging about this virtualization project here and here. For the adventurous, you can try to follow the instructions to launch your own copy of Enterprise Rhaptos.

If you are coming to the Plone Symposium at Penn State University in State College, PA later this month, I will be giving a talk on Best Practices for Plone Hosting and Deployment where I will discuss these new hosting strategies for Plone and more.

Tagged: amazon, cloud, cms, connexions, courseware, deployment, ec2, education, hosting, Plone, ploneconf2009, rhaptos, zope