Twelve months later we are thirteen more (talented) people, and the quality and quantity of our work has shot up as a result. Being able to brainstorm in planning and swarm around projects during execution, we collectively come up with sharper, cleaner, more beautiful solutions than what any of us could do individually. It's been very enjoyable and empowering for all of us.

Yet this kind of rapid growth has brought with it a huge challenge: How to effectively manage our time and tasks on all the different projects that come in through our doors. Swarming is fun, but it's not the most cost-effective approach. And while we're adding and training new people on a regular basis -- not to mention working with some off-location people -- planning and keeping track of everything has grown way beyond what pencil lead, dry-erase pens and laser printers can provide.

Managing growth is one of those topics that gets lots of surface coverage, but it really is a different thing to face that challenge directly. While I had had hopes, I never would have predicted the kind of growth we've enjoyed so far. As a result, we've found ourselves over the past weeks and months compelled to set aside more company time to focus on our project management processes, our development methodology, and possible tools to enable and empower ourselves moving forward. I'll share some of our thinking here. Thoughts and feedback are welcome.

Our Development Methodology

At the center of everything is the reality that, at any given time, we are working on a diverse range of projects. We do all kinds of websites -- community sites, intranets, B2B sites, publication sites -- and we design and develop them for all kinds of clients.

When it comes to the technical nature of what we do, each client has a different comfort zone. Some want to know everything, while others just wave their hands and say, "Tell me when it's done." What seems to characterize most clients, however -- and I believe this is inevitable given the nature of what we do (i.e., build the kinds of websites they have never had before) -- is that they end up placing change orders, especially when seeing design comps but often very late in the game, such as when they first start working with the test alpha site. Scope creep has become our companion. And as such, "measure twice, cut once" doesn't seem to be as workable as we would like.

Not only that, client change orders are not the only things that might throw a wrench in the development gears. Some other factors we've identified that can disrupt the grand plan include:

• discovery of unspecified requirements or complexity (e.g., form foo should not only send an email, but should also be archived in a downloadable database table) -- this often comes up when a client brings us a specification that seems complete but has details unrevealed or unnnoticed until development has progressed a ways;
• changes in the code base (e.g., a module update that breaks custom module foo) -- something that can happen at any time in the open source world;
• changes in the timeline -- this can affect our workflows on all of our other active projects;
• lack of manpower (e.g., someone gets sick and someone else has to take over and get up to speed on things) -- always expect the unexpected;
• slow client feedback -- if we have to delay work because of an open question, we must change gears and put our staff onto other projects, thus losing the flow;
• mistakes (i.e., we messed up and have to refactor or start over) -- these are painful, and even though the client may never see the mistake and is never billed for it, mistakes happen. In-house, they end up being valuable learning opportunities ... at least that's how we try to treat them.

We've had to be adaptable during the development process. The problem with this runs up against our methodology all too often. We've often found ourselves having to go back and revise our site specs, alter designs, even refactor this or that functionality to accommodate these kinds of disruptions to our planned workflow.

We had to find a better way. Our existing system was simply too static and inflexible for the realities we face.

Waterfall vs. Agile Development

Waterfall. Measure twice, cut once. The Big Design Up Front. This is essentially what we had been doing since I was working alone on websites (which all too often had me burning the midnight oil to cover the extra ground). It wasn't working as a methodology. A Wikipedia entry on the Waterfall model notes this criticism:

For example, clients may not be aware of exactly what requirements they want before they see a working prototype and can comment upon it; they may change their requirements constantly, and program designers and implementers may have little control over this. If clients change their requirements after a design is finished, that design must be modified to accommodate the new requirements, invalidating quite a good deal of effort if overly large amounts of time have been invested into "Big Design Up Front".

This certainly fits a significant part of our experience. The "scope creep" seems to visit many projects.

However, I cannot fault clients for changing their minds, or altering and clarifying their needs once they see something up and running. And given all the other factors noted above, the BDUF approach can give us a solid architecture that ends up being built upon shifting sands.

On the Wikipedia page for Agile development:

Adaptive methods focus on adapting quickly to changing realities. When the needs of a project change, an adaptive team changes as well. An adaptive team will have difficulty describing exactly what will happen in the future. The further away a date is, the more vague an adaptive method will be about what will happen on that date.

And yet there's no replacement for designing a solid architecture up front. It's essential for us to be able to understand and clearly define the client's needs and site goals, and translate that through a user-centered design process, into a planned implementation.

We explored both models, and the many variants out there. To make a long story short, it seemed clear that what we needed is a hybrid approach, starting with the Big Design Up Front of the Waterfall approach, but then working development through an Agile methodology of multiple iterations. Measure twice, cut once -- but for each iteration along the way.

What appealed to me, personally, is that the Agile approach of breaking development down into tasks chunked by "user stories" dovetails quite nicely with the user-centric interactive design we approach in the architecture phase of each project.

The Application Hunt

We needed something that is collaborative and accessible to all of our staff (which immediately rules out desktop apps like Merlin 2 and OmniPlan), but is also richly featured and relevant enough to handle multiple nuanced projects throughout all phases of planning, design, development and review (which rules out commonly referenced but relatively unsophisticated services like BaseCamp). We considered building off of Drupal's case tracker module, or even the Project and Issue modules, but we didn't have time. We needed to get this piece solved so we could get back to the business of developing for clients. (We haven't ruled out such development in the future, though. It's just not in the cards for us in the coming months, given our existing obligations.)

We are now testing out a robust system optimized for Agile development. We'll post more on that in the future. Meanwhile, I look forward to your comments and thoughts on development methodologies. What has worked for you? (More important, perhaps: What hasn't?)