PINGV Creative for Planet Drupal

When will Drupal 8 be released?

When will Drupal 8 be out? It's a question that's asked in many forms.

Sometimes it's phrased as: when will I have to upgrade from Drupal 6? Or, how long will Drupal 7 be supported?

Laura Scott and I were reviewing Dries Buytaert's 2010 State of Drupal and the estimates on when the new Drupal release candidate would be virtually frozen. The graph below is his summary graph of the trends in April 2010.

Fig 1 — Dries Buytaert's April 2010 Prediction of Drupal Release.

It turned out that the trends were not linear, and it was Laura who suggested that bug fixes aren't linear: they tend to be hyperbolic or (for fellow math majors) follow "power functions" that have a logarithmic aspect to them.

This got us thinking. What is the history of time between Drupal releases (TBDR)? In the slide below, Dries provided some this data in his slide from the State of Drupal at the March 2011 Drupalcon Chicago.

Fig 2 — Dries' March 2011 graphic showing Drupal's code size historically.

All that was missing were the dates, which Laura looked up from the Changelogs of each Drupal n.0 release, and from this I created the graph below, which is consistent with a power function. That is, geometric!

Fig 3 — TBDR of core size versus release dates. Graph by Kate Lawrence.

What this means is that the more the code, the longer it takes, and in this case (at least thus far) it is obeying a consistent pattern. Interestingly, in the period from of Drupal 4.0.0 to Drupal 4.6.0, the size of Drupal was pretty constant, especially as compared to the recent growth of Drupal core ... and the TBDR was about 6 months! What do today's longer release cycles mean for Drupal? I say this only because we are hearing that Joomla has a 6 month time horizon between releases.

If Developers were Horses...

What would it look like to have 90 developers contributing, only contributing really, really fast? Is there a marginal utility curve, like we learned in Econ 200, in there someplace? That is, at what point do you have so many people involved, that they are tripping over one another? Throwing more people at the problem does not get quicker results, and may in fact, slow things down. cf. The Mythical Man Month.

The Mythical Man Month premise is: Assigning more programmers to a project running behind schedule will make it even later.

So a question to ponder is to what extent can the growth curve in the above slide be changed?

The final slide from Dries that I think is relevant to this pattern appears below.

Fig 4 — 30 contributors were responsible for 50% of D7 Patches. Graphic by Dries Buytaert.

Most of us have heard of the "80/20 Rule." The rule is attributed to the findings of Italian economist Vilfredo Pareto. Sometimes its the 90/10 Rule or 95/5 Rule, or even 920/30 rule (see Figure 4)!

The challenge may be in making the ski-jump curved line more linear (while pinning the line on the y-axis). The more linear it becomes, the steeper the TBDR curve's slope becomes — meaning that releases can be more rapid.

In short, holding the intersection point on the left axis, as the graph becomes more linear, the faster the TBDR. The challenge we face as individuals within the community is helping in making that happen.

Too busy doing the doing to talk about the doing?

As a strategy and design studio, we're all over communications. It's an uninterrupted conversation with our clients, hashing out ideas, working iterations of goals definition, research, wireframes, designs. But when it comes to development, when we put our heads down and get to work, it's all too easy for us in our focused implementation state to just do it and not communicate enough about what we're doing. Of course, this can be disconcerting to the client who has no idea how things are going. And that's a ripe situation for fear, uncertainty and doubt to creep into the relationship, and suddenly a successful project-in-progress can feel troubled, even if it's not.

The simple solution is to give regular status updates, which is something we've always done. Even just saying, "Things are going well" can help. But on long projects, or even in 3-week iterations, repeating that same message ends up being not that helpful. Better to share what you're thinking with regards to planning and project management, and provide updates on development of the various features/backlog items worked out in the strategy and design phases. This transparency in the project management process is even better when updates are more specific.

We have been using Open Atrium, the powerful intranet CMS built in Drupal, developed initially by our friends at Development Seed, now maintained by Phase2 Technology, for our intranet to handle all client-facing communications, documentation, file sharing and issue tracking. It's been a great tool for this. No missed emails lost in the spam filter. No critical documentation lost on someone's hard drive. No mega-emails containing two dozen specific items. (Well, those still happen on occasion, but we now can push those into the intranet to continue discussion.) We love it.

And now we love it even more.

Rethinking Open Atrium

We had been using almost all of Open Atrium's various features: messages, uploads, case tracker, books for documentation, and of course the groups functionality. Almost all of the features. The part that didn't really fit in with our communications strategy was the OA notion of "Projects". Until now.

You see, in Open Atrium, the basic architecture is thus: You have Clients, which are defined by groups, and you have Projects, which are ways of grouping tickets within the group by project. It can make sense for a large organization using OA for internal communications to manage several projects. But for a professional services firm like a strategy and design company, or a software company, or any other kind of company that defines work by time-boxed iterations or sprints, breaking up stuff by "project" may not be that useful. Projects projects projects blah! At PINGV, we rarely have a series of small projects with a client where it would make sense keeping them all together in one group; we tend to do bigger projects, projects that have phases of strategy, design and planning, and iterations of development. And when we have a new large-ish project with an existing client, we just spin up a new Atrium group. Project was that dunsel feature — the thing that for us had no real use. But—

What if we use "Project" nodes to define phases and iterations?

Wash that nomenclature right out of OA

Ignoring nomenclature, and just looking at what "Project" as a functionality in Open Atrium does for us, it provides a way to group tickets into clumps that you can define by anything. If we define those clumps as iterations, perhaps, or phases, we can organize and parse through backlog items, features, questions and other tickets by iteration, while providing some ready clarity as to how and when these items will be taken on. (NB: It's an approach that works only with Tickets, not other node types, but in our mind that's a minor limitation.)

What's really convenient is that Open Atrium provides via UI the means to rename "Project" to whatever you prefer to call it (e.g., "Iteration"). These string replacements, configured in the group feature settings available to site administrators from within each group's space, work almost everywhere in OA. (The "Add Project" button seems to be unaffected even by string replacements defined in settings.php. Something to investigate later.)

Once we saw this potential for grouping issues into iterations, we ran with it.

Defining iterations in Open Atrium

Using "projects" in Atrium, we can define iterations by close date, and assign tickets to iterations.

As you can see, the Iteration title is compressed down to an abbreviation or acronym in the icon in the tickets display. "Iteration 1" becomes "I1", "General" becomes "GENERA", and so on. After briefly considering using iteration numbers, which we rejected as too arbitrary, we opted to name iterations by the Iteration release date, or end date (e.g., 08/17 or even 11/10/23), which can be abbreviated in a meaningful way in the Tickets views. This gives our clients added insight into our timetable planning at a glance.

Adding a sortable Iteration column to the tickets view involves a simple override to the default ticket View. (If ever we need to revert that Feature during a site update, these changes are easily re-implemented in a few minutes. No actual data is affected.)

You can view all of the iterations planned, and see their descriptions. Each iteration is clickable for detailed viewing.

This approach allows us also to focus just on the tickets that are relevant now, without having to sort through all the other tickets that are further down in the timetable, and without having to mark dozens (hundreds?) of individual tickets as "deferred" and then change those as they come available.

Single iterations of backlog items can be viewed as well, and this is stock functionality for Open Atrium.

The bulk of communications still happens in the individual tickets. Questions, information, updates, concerns, notes can be tied to the ticket itself. New questions and bug reports can also be slotted for attention in a scheduled Iteration. So even though we're grouping our backlog items together, we still avoid the one-ticket-for-8-issues kind of communications that can lead to confusion, frustration and missed messages.

On any comment update, you can change the iteration assignment of a ticket.

All changes are tracked in the comment metadata displayed right there in the thread.

We like this system so far because of this fact that all changes are tracked and visible in each ticket's thread. If we move up a backlog item to a higher priority, we simply do it with a comment and change in iteration designation. This flexibility with traceability provides a nice balance for communicating our planning.

And it's all using stock Open Atrium!

Plan before communicating, and vice versa

I should add that this is not how we do our actual project management. Open Atrium is a great tool for project-related communications. We currently use Brian's recommendation, we tried Liquid Planner (disclosure: affiliate link; go to liquid planner dot com to skip it if you like) for our internal planning. Liquid Planner has an API, and we're considering developing an integration with Open Atrium to allow us to push and pull info between the two. Might be a nice convenience. We do the translation manually now, and while it can take up time on occasion, it also provides an occasion to rethink and re-evaluate, so the manual process does have value.

Feature think, Feature do

Now for the road not taken:

When first considering the challenge of better communicating our iterations plans, I imagined up a very simple Features-based solution using nodes and node reference, which resulted in Open Atrium Iterations. This module works as a drop-in extension to Open Atrium. Not touching any existing Atrium Features, OA Iterations defines a new content type with reference fields to call up tickets (cases), messages (blog posts) and documentation (books) that collectively define and describe an iteration of work. A dedicated view restricts the available referenced nodes to the current group's content.

On Friday afternoon, this seemed like a fine way to communicate our iteration planning. Tickets, as well as blog posts and book nodes, could live on their own, with their own revisions and threads, while the Iteration node would group them into an iteration. Done. Time to relax and prepare for a weekend cleaning the garage.

On Monday morning, however, when I sat down to start defining the iterations for one of our projects, I immediately questioned my own game plan. Open Atrium Iterations was an easy way to define iterations, but making changes would be difficult. What if I wanted to move a ticket from one iteration to another? Yes, this project was never intended for iteration planning, just iteration communication, but even just for communications this approach would mean editing at least two Iteration nodes just to "move" one backlog item. What's more, without implementing Reverse Node Reference and adding that block to one or more Contexts (and thus overriding stock Open Atrium Features), there was no way to look at a backlog item and see what iteration it's slotted for. The simple solution was not turning out to be so simple after all.

And that's when the flicker of the idea described above came to mind. In hindsight, I can smack myself for not thinking of this before. I've been working with Atrium for two years. But I think I got stuck on nomenclature.

Contrib status

Meanwhile, despite our not using it as originally planned, Open Atrium Iterations will remain live on Drupal.org to see if it proves helpful to others. If the Feature can be improved upon to make it more broadly useful to serve more use cases, I look forward to suggestions in the issue queue.

We're frequently setting up servers for development, for staging, for production. I've lately preferred the Debian flavor of Linux, but up until now that had been something of a problem because Debian and Ubuntu did not include the higher-quality php5-gd library, which meant that you either had to compile your own PHP, pull from an alternative source host, or cope with substandard image resizing with limited processing features.

But now we have Ubuntu 10.4 LTS "Lucid" and life is good. Lucid comes with PHP 5.3.x and the proper GD2 library! (Cheers. Applause.)

Still, there's a lot of little steps to set up to get your server (or virtual private server) up and running and ready for Drupal. It's not hard, just detail-oriented work you don't want to do when you're bleary-eyed at the end of the day. In my repetitions of doing this over and over, I've collected some notes, and I thought I'd post them here for my own reference, and perhaps your reference as well. My thanks to our own Brian Vuyk for guidance on the APC installation.

[Edit: Brian also pointed me to Taskel as another way to go. I've not tried it that way yet, so this post doggedly sticks to the step-by-step for now.]

Drupal requires some things, and likes some other things a lot. Here's how to configure the latest Ubuntu Linux to make Drupal happy.

Assumptions

• You already have Ubuntu 10.4 installed on your server.
• You know your server's IP address.
• You already have root or sudo access.
• You want to set up for Drupal. (Of course, many CMSs and blog platforms will have similar requirements, and this post may have some appeal and relevance to people using those tools. But in this post I'm aiming at the particulars for Drupal.)
• You have a computer with shell access to your server, or a properly configured *AMP setup on your computer. (This post does not cover MAMP, WAMP, DAMP or commandline bootcamp.)

Step by Step

1. Initial Preparation

   1. Create a public key.

      If you do this, you can connect to your server without having to log in. It's also necessary if you want to deploy from a provisioning system or repository like GitHub.

      ssh-keygen -t rsa -C "yourname@example.com"
      

      Note: For deployment purposes, you need to generate the key as the user who will be executing the checkouts. For example, if you checkout as root, you will need to be logged in as root when generating the key.

      To print out the key (to copy and paste into GitHub, for example):

      cat ~/.ssh/id_rsa.pub

   2. Install security updates.

      FIrst things first.

      apt-get update
      apt-get upgrade --show-upgraded
      

      Note that if you are using a regular shell user account with sudo, you will want to prepend pretty much all commands in this guide with "sudo". For example: sudo apt-get update would be the first command above.

   3. Set your hostname

      You should give your machine a name. This is a machine name for your server. It does mean anything about your domain or what URL you want to use for your site. For this example, we'll use the name "athena":

      echo "athena" > /etc/hostname
      hostname -F /etc/hostname
      

      Next, edit the file "/etc/hosts". There are a few options for text editor within Linux. I find nano to be easy to use.

      nano /etc/hosts
      

      [NB: In these instructions, your server's IP address is represented by "12.34.56.78". Every time you see that sequence in the examples here, you should replace them with your server's IP address. Also replace "athena" everywhere it appears with your server's name, and "example.com" with your actual domain.]

      127.0.0.1        localhost.localdomain        localhost
12.34.56.78      athena.example.com             athena

      athena.example.com is your "fully qualified domain name" or FQDN. (You will come across this from time to time.)

   4. Set your server's base timezone

      This part is easy:

      dpkg-reconfigure tzdata

      You will get pop-up prompts for country, etc. and that's done!

2. Set up VirtualHost stuff

   1. Install Apache

      First things first.

      apt-get install apache2

   2. Edit /etc/apache2/ports.conf

      nano /etc/apache2/ports.conf
      

      Replace *:80 with your IP address, so it looks like this:

      NameVirtualHost 12.34.56.78:80

   3. Edit /etc/apache2/sites-available/default

      nano /etc/apache2/sites-available/default
      

      Edit as such:

      <VirtualHost 12.34.56.78:80>

      You will also want to update the DocumentRoot value to where you intend to install your Drupal root.

      DocumentRoot /var/www/example.com/html

   4. Configure name-based virtual hosts

      Create a file in

      /etc/apache2/sites-available/

      for each of your sites on the server.

      nano /etc/apache2/sites-available/example1.com
      

      <VirtualHost 12.34.56.78:80>
     ServerAdmin admin@example1.com
     ServerName example1.com
     ServerAlias www.example1.com
     DocumentRoot /var/www/example1.com/html/
     ErrorLog /var/www/example1.com/logs/error.log
     CustomLog /var/www/example1.com/logs/access.log combined
</VirtualHost>

      Repeat this process for each site you are setting up. Note that separate sites will have different DocumentRoot values, while a multisite setup will share the same DocumentRoot value.

      Git note: If you are going to deploy using Git (covered in the next step below), you want to define your DocumentRoot to include your Git repository name and any internal paths to your Drupal document root. For example, if you are installing from a GitHub project titled "foobar" and inside of it you have a folder "html" that contains your Drupal installation, your DocumentRoot value would be /var/www/example.com/foobar/html/.

   5. Create your website folders

      • First, create the folder for your logs.

        mkdir -p /var/www/example.com/logs
        

        This command will create your logs folder, and the domain folder containing it.

      • Now create your website html folder(s).

        How you do this depends upon how you're going to install your site on the server.

        If you are using scp or sftp or otherwise copying your Drupal code files onto the server, you will want to create the folder to hold them. Remember that your DocumentRoot value you entered above must match where your actual document root ends up being on the server.

        If you are going to be deploying via Git, you don't need to do that: Git will create it when you git clone the repository onto the server into /var/www/example.com/.

   6. Enable the virtual domain

      a2ensite example.com
      

      Do this command for each domain or subdomain you're configuring here.

   7. Reload Apache

      /etc/init.d/apache2 reload
      

   Assuming that you have configured the DNS for your domain to point to your server's IP address, virtual hosting for your domain should now work.

   Of course, there's still more server prep to do so you can run Drupal....

3. Build PHP, MySQL and the goodness Drupal loves

   1. MySQL Installation

      First install the thing.

      apt-get install mysql-server
      

      You will be prompted to create the MySQL root password, and re-enter it.

      Now we secure the installation.

      mysql_secure_installation
      

      You will be prompted to set/change the MySQL root password and other things. You won't need to change the MySQL root password, but you probably want to answer "Y" yes to the other questions.

   2. Create your database

      Log into MySQL.

      mysql -u root -p
      

      The -u defines the mysql user, which in this case is mysql root user, 'root'. You will be prompted for the MySQL root password. When you get a prompt like this:

      mysql>
      

      …you're in!

      Now create your database. [In this example, the database name is 'foobar', the database user is 'rumpole', and the user password is 'p4ssw0rd'. Change these to the actual database name, database user and passwords you want for your database. Be sure to note these down, because you'll need this info when you set up your Drupal site.]

      create database foobar CHARACTER SET utf8 COLLATE utf8_general_ci;
      

      [Update: You need to set the CHARACTER SET and COLLATE values, as shown above, or MySQL will default to non-recommended latin1 / latin_swedish_ci.]

      Note the ';' at the end of the line. That is required for MySQL to execute the command.

      Define the user and permissions for the database. I'll keep it easy here:

      grant all on foobar.* to 'rumpole' identified by 'p4ssw0rd';
      

      Now wrap this up and quit MySQL:

      flush privileges;
      
      quit
      

   3. Install PHP

      Use apt-get to pull down the packages.

      apt-get install php5 php5-dev php-pear php5-gd
      

      Note that php5-dev is not necessarily required, except you will need it later for the PECL installation of UploadProgress (which is very nice to have for the Drupal user interface). php5-gd is to enable nice image handling.

   4. Configure the php.ini settings.

      Edit the appropriate php.ini file:

      nano /etc/php5/apache2/php.ini
      

      This is a big file. You will need to search through the file to find these value configurations.

      You will want to boost the default memory limit value.

      memory_limit: 128M

      (128MB is the default for Lucid. If you're running a lot of modules, or some heavy processes, you may need to increase this memory_limit value even higher.)

      Now, while you in php.ini, make sure that the following lines are uncommented and have proper values established.

      max_execution_time = 30
error_reporting = E_COMPILE_ERROR|E_RECOVERABLE_ERROR|E_ERROR|E_CORE_ERROR
display_errors = Off
log_errors = On
error_log = /var/log/php.log
register_globals = Off
safe_mode = Off
session.cache_limiter: nocache

      You will also need to add lines for PDO support. Find the Dynamic Extensions area in the file, and add these lines:

      extension=php_pdo.dll
      extension=php_pdo_mysql.dll

      [Edit: You don't need to add these in Ubuntu 10.04. As it happens, Lucid comes with PDO extensions already enabled. Thanks to Peter Wolanin for pointing out in comments below what should have been obvious about .dll extensions in Linux. #facepalm]

      Refer to http://drupal.org/requirements for details and nuances on php settings.

      To have these take effect, restart apache.

      /etc/init.d/apache2 restart
      

   5. Install PHP MySQL and Security Packages

      1. Let's start with MySQL.

         apt-get install php5-mysql
         

      2. Update your sources.

         Now be sure the following lines in /etc/apt/sources.list are uncommented:

         nano /etc/apt/sources.list
         

         deb http://us.archive.ubuntu.com/ubuntu/ lucid universe
deb-src http://us.archive.ubuntu.com/ubuntu/ lucid universe
deb http://us.archive.ubuntu.com/ubuntu/ lucid-updates universe
deb-src http://us.archive.ubuntu.com/ubuntu/ lucid-updates universe

deb http://security.ubuntu.com/ubuntu lucid-security universe
deb-src http://security.ubuntu.com/ubuntu lucid-security universe

      3. Update what you have so far.

         apt-get update
         

      4. Boost security a bit.

         Now install the php5-suhosin package to provide additional security.

         apt-get install php5-suhosin
         

      5. Restart apache.

         To have these take effect, restart apache.

         /etc/init.d/apache2 restart
         

   6. Enable mod_rewrite

      This is easy peasy in Ubuntu because the PHP module is already installed; you just need to enable it:

      a2enmod rewrite
      

      And restart apache again:

      /etc/init.d/apache2 restart
      

   7. Install PECL UploadProgress.

      This allows you to have that nifty upload progress display when uploading files. It's not required, but is useful user feedback, which I recommend highly.

      1. First, update.

         apt-get update
         

      2. Run the PECL installation of UploadProgress.

         pecl install uploadprogress
         

      3. Now save the setting.

         Note: This is all one line!

         echo "extension = uploadprogress.so" > /etc/php5/apache2/conf.d/uploadprogress.ini 
         

      4. Reload Apache to have it take effect.

         /etc/init.d/apache2 reload
         

      Done!

4. Optional

   1. Install Git

      If you want to use Git to deploy your site code, you will need to install Git itself.

      apt-get install git-core
      

   2. Install Drush and Drush Make

      Drush installation instructions are in the Drush readme.txt file.

   3. Install APC

      apt-get install libpcre3-dev
      pecl install apc
      

      Then edit your php.ini file to add to the extensions area:

      ; APC
extension=apc.so;
apc.shm_size=64M;

      Then you can copy over a php file that offers some nice stats:

      cp /usr/share/php/apc.php /var/www/example.com/html/apc.php
      

      Note that the path of the destination is your webroot.

      Then restart apache.

      /etc/init.d/apache2 restart
      

      And you're done. You will be able to get some nice APC stats at http://example.com/apc.php. (You can always protect that file via .htaccess to put an authentication password on it if you like.)

   4. Install fail2ban

      Fail2ban is a nice little security addition.

      apt-get install fail2ban
      

      That's all on that.

   5. Install bash-completion

      Bash-completion is a new one for me, and is turning out to be very handy. How did I go so long in the dark?

      apt-get install bash-completion
      

And that's it. Now your system is ready for site deployment to the folder you defined. Note that after you've deployed your site you will also want to set up a cron job.

IANASA

I am not a sysadmin by profession. This is just documentation of what I do for my own projects, or when tossing up a staging server for one of our in-house or client projects. Any tips for improvement, corrections, etc. are most welcome.

Update: Clean URLs problem?

Recently I've run into an added complication with setting up Lucid: clean urls were not working. rewrite_module was enabled, .htaccess was there, but no go. As it turns out, .htaccess was not being read by default. If this happens to you, here's a fix:

Edit the default site configuration in sites-available:

nano /etc/apache2/sites-enabled# 000-default 

A few lines down, under the directory pointing to you docroot, you need to change "AllowOverride None" to "AllowOverride all".

Now restart apache.

/etc/init.d/apache2 restart

And you're done. Clean urls should work for you now.

For more information on this, see http://drupal.org/node/945860. To read about other ways to configure Apache and avoid the performance hit of using .htaccess, see http://drupal.org/node/43788.

2010 was an amazing year. But 2011 has come in with such a bang, it's impossible not to look forward.

Drupal 7 is out!

Drupal 7 is a whole new Drupal

It has been a long time coming. Drupal 6 came out some 3 years ago. That in many ways was the time when Drupal emerged from relative obscurity, where it was known only by open source aficionados, PHP geeks and a few untold thousands of independent website owners who knew a good thing when they saw it, to a CMS recognized worldwide as one to be reckoned with. More powerful than Wordpress, more flexible than Joomla!, more affordable (and flexible and, in many ways, powerful) than proprietary systems like Vignette, Drupal 6 came on the scene the quiet hero to many site owners. [Note: I mean no disparagement against Wordpress or Joomla!, which are active open source web software projects, and we like open source software! But each is different; describing how so is something to be left for another blog post another day.]

We here in this shop have had the great fortune of being able to work with Drupal 6 on a number of projects — and each of us individually beforehand — playing a small part in a very large and impressive community.

Drupal 7 promises to step everything up a notch.

Right now, we have a couple of Drupal 7 projects in house that are in the last strategic planning and architecture phases, which means that we — and especially Brian Vuyk, who contributed 33 patches to Drupal 7 and, last year, took over the Primary Term module, while offering up and testing patches for a number of contributed modules — will be bringing a lot of time and attention to some interesting modules needing upgrades, as well as perhaps a new one or two.

Meanwhile, as announced here before, the Square Grid theme and the Floater theme are both available for Drupal 7. (Confession time: They're still not tagged as official stable releases, but Square Grid is actually quite stable currently; Floater has some sprucing up for its 1.0 release. Expect stable releases of each very soon!)

How far we've come in just a few years!

I first discovered Drupal in 2004. I joined the community in November of that year. Back then, Drupal 4.5 was new. Theming was an order of magnitude more difficult than it is today. Modules did not have installation routines, you had to manually execute MySQL commands to add or update tables. PHP 4.whatever was the standard. You had to get into the code quite a bit to set up any kind of site.

Since then, Drupal has grown. Ultimately, the test for Drupal 7 is just beginning. My expectation is that people will respond with enthusiasm and delight, and Drupal will continue its phenomenal growth with every major release.

And yet this is open source, and the drop is always moving. Drupal 8 is coming!