Newsletter Usability: Can a Professional Publisher Do Better?

My recent review of the Bush and Kerry campaigns' email newsletters concluded that both U.S. presidential candidates published newsletters with good content, but both had severe deficiencies in their content's user interface.

The Washington Post has a dedicated newsletter called the Weekly Campaign Report that covers many of the same issues as the candidates' newsletters. So, I set out to answer a question: Does a professional publisher do better with its email newsletter than the campaign sites do?

Result: A Grand-Slam Winner

I evaluated washingtonpost.com's subscribe and unsubscribe interfaces on September 21, 2004, and evaluated the Weekly Campaign Report newsletter itself during a four-week period from September 6 to October 3, 2004. I scored the website and mailings for compliance with the design guidelines for newsletter usability derived from our recent research with users who were reading and subscribing to a large number of email newsletters.

The Washington Post handily outscores both George W. Bush and John Kerry. Not only is its overall rating much better, but the Post also got more points than either candidate in each of the four major newsletter usability areas.

Okay, so 72% is certainly not a perfect score. I prefer to see sites complying with 80–90% of usability guidelines. (A 100% score is not required because any given site often has special circumstances that make it appropriate to deviate from a few guidelines.) Still, 72% is a very respectable level on today's internet, and it's dramatically higher than the 57–58% achieved by Bush and Kerry.

Subscription Interface

Washingtonpost.com scores much better than the presidential campaign sites in setting users' expectations for what they'll get if they subscribe to the newsletter. Basically, Bush and Kerry say "give us your email address, and we'll send you some [unspecified] stuff." Not good in this day and age. The Post follows most of the guidelines for comforting users before they've subscribed, thus ensuring that more users will do so.

The main reason that the Post 's subscription interface scores less than 100% is that it requires user registration, including many nosy questions requesting personal information. This is presumably driven by the myth that user demographics are the way to target ads on the internet, but registration impedes usability and drives away subscribers. The net result is thus lower advertising revenues. (Also, you should target computational advertising based on each user's explicit behavior, which defines a demographic of one; this is much more valuable than stereotyping people as members of broad groups.)

Amazingly, washingtonpost.com partly violates the basic guideline to promote relevant newsletters in context by failing to link to its newsletters from appropriate articles throughout the site. They do link to the Weekly Campaign Report from the "2004 Election" category page, so I gave them half credit for this guideline. Still, there was no mention of the newsletter on Dan Froomkin's September 21 White House briefing column, which was the second-most promoted story on the site's homepage, with the headline, "Let the Debate Spin Begin." The title alone probably attracted many readers who were likely newsletter subscribers.

Washingtonpost.com also scores low on helping users find newsletter information through its user guidance features. I did give it a few points for good implementation of search usability guidelines: when users search "subscribe" or "newsletters," there's a clear best-bets-style link to the newsletters at the top of the results list. But a search for "unsubscribe" provides no results and no advice on what to do — not even a link to Help or a site map. If users do find the site map, it works well and follows many of the guidelines for site map usability, except that it's called "site index" instead of the recommended "site map." Also, the site map fails to offer a link to the site's newsletter area.

Subscription Maintenance and Unsubscribing

User registration comes back to bite the Post in terms of unsubscribe usability. The newsletter doesn't follow the recommended one-click process for unsubscribing. Instead, users must sign in, which requires them to remember their passwords. Nasty.

At least the site offers a decent interface for recovering forgotten passwords, which somewhat alleviates the usability problem of requiring users to sign in before they can get off the mailing list.

Once users sign in, it's easy to unsubscribe from either a single newsletter or from all the newsletters they receive. Good.

In several cases, it's possible to change the frequency of newsletters, going from daily to weekly newsletters for users who feel flooded with email. This is good, though there is no recovery interface that offers a corresponding weekly newsletter as a substitution when users unsubscribe from a daily newsletter. The site relies on having people notice such alternatives in a huge list of possible newsletters — not likely when users are feeling pressed for time and just want to unsubscribe.

Newsletter Content

The Post gets high scores for newsletter content, but then so did the presidential candidates. You would expect a leading newspaper to have people who can write and edit, and they do, collecting a tad more points in this area than the campaign newsletters.

Headlines are short (good), but not always specific enough for the online medium (bad). For example, the September 8 newsletter was entitled "Over the Top," which doesn't quite explain the newsletter's main thrust. A subject line like "Campaign Rhetoric Takes a Nasty Turn" would have generated a higher open rate.

Subheads are often straightforward and to the point (good), such as "Poll Analysis" or "Kerry Blasts Bush on Guns." The Washington Post collects only half credit for the common task of newsletter printing, because the printout's right side is cut off if users simply click the Print button in their email program. The cursed frozen layout strikes again.

Advertisements are handled appropriately and don't damage the user experience, except that, paradoxically, some editorial content is too similar to ads and users might therefore overlook it. Looking like an advertisement has long been one of the top web design mistakes, and should be avoided.

Improving Newsletter Usability

Is it unfair to compare the presidential campaign newsletters with one that is professionally published by a major newspaper? I don't think so. Yes, it would be unfair to impose such a comparison on the internet operations of someone running for mayor of Podunk. But presidential campaigns are another matter: they have budgets of more than $300 million each. That said, the money itself isn't even that important; many of the best designs in our test of 111 email newsletters were published by fairly small companies. Most of the candidates' mistakes could be fixed with minimal resources — often without any programming. One of their biggest problems was not a lack of writing skill, but a lack of editorial judgment.

The Bush and Kerry newsletters have improved their usability scores somewhat since I evaluated them. From a positive perspective, this indicates that the campaign managers are capable of learning from experience. However, there was no reason for the campaigns to make mistakes in August that had been clearly documented in user research published earlier in the year. Whatever subscribers they lost because of these mistakes are probably gone forever. It's very hard to reacquire users after they've decided against an online offering.

The Washington Post clearly shows a good email newsletter's potential. The Post 's web design team and newsletter editors have done a superlative job and beat both George W. Bush and John Kerry by a mile. The Post newsletter is also better than the majority of email newsletters that we've evaluated from corporate websites and internet marketers. Yes, the usability could be even better (I am never satisfied), but my conclusion is: good job, folks.

Full Report

The full report on email newsletter usability with essential design guidelines for newsletters is available for download.