January 5, 2010

We're a wanted man; everyone is searching for us.

http://www.google.com/intl/en/press/zeitgeist2009/cities.html

Look under Minneapolis.

January 26, 2009

Gophermail roadmap

As expected, Gophermail 2.0 went production in summer of 2008. There have been initial rounds of bugfixes and enhancements to the virtual host framework. Some of this work was done at the request of METNET, which may be adopting Gophermail as their webmail platform.

Overall, we have had very good feedback to 2.0, and I'd like to thank everyone who helped with ideas, suggestions, and testing. The mobile version, in particular, has received a lot of comments and we're happy for that!

In the coming months, I'll be finalizing "styles" documentation on the Gophermail wiki. Style handling in 2.0 is very plug-and-play, and I look forward to having that documented for the U Community.

At this time, however, it's looking likely that Gophermail itself (the codebase) will be in bugfix-only mode going forward due to the strategic direction of the University. The lead developer (that's me, Brian Hayden) has moved on to a position with the Minnesota Supercomputing Institute.

In the near future I'll likely be integrating the concepts of Gophermail 2.0 into the latest version of the Prayer webmail package which it was based on. The major advantage to the newest Prayer is HTML templating, so changing the presentation (the 'V' in MVC, for those of you that like that language) will not require code changes.

This is a large project that will not happen overnight, but I consider it worthwhile to have a clean, efficient, modern open-source alternative to the mostly-poor PHP webmail offerings out in the wild. I'm also excited to fully integrate the web-services portions of the Gophermail 2 beta, which did not make it into production as a proper API.

Thanks again to all of you.

May 12, 2008

The usability of a garbage can

It's been a while. The last couple of weeks have been a mix of usability lab, and me being sick. Usability, as always, was exhausting but very productive. Scripted activities in a lab are only part of the interface picture. There are a lot of things that environment isn't, but one thing it is: a great way to shine a light on any basic stumbling blocks that the designers/developers have become immune to during the process.

We had surprisingly few of these; the worst one involved prompting users to try to go into "mobile"/small display mode. It was lumped in with "styles" in the preferences, which nobody expected (and not many found). It's been shifted to more a appropriate heading, and more importantly, a "fast switcher" is now available on the masthead. This has the advantage of being a common element across much of the web. Hopefully we'll catch most cases automatically with a little intelligence on style sheets and a tiny bit of a javascript probe. Google seems to rely exclusively on this--or a user knowing their mobile url--as neither google.com nor gmail seem to have a link or preference anywhere to jump to their mobile versions. I'm not sure if they don't subscribe to the mobile web design community's wariness of detection (as accuracy is always suspect), or they just think they're that much smarter than anyone else... interesting question!

Aside from that issue, the other thorny one was the idea of deleting messages. IMAP has a two-step deletion model; you mark a message deleted, and then you "expunge" to permanently remove all deleted messages from a mailbox. It's a model of both safety and efficiency. You aren't moving a message around to delete it, so you're saving on IO, possibly network bandwidth depending on the environment, and lessening the risk of message corruption.

The downside, as we've heard numerous times in both the usability labs and from users throughout the years, is that most people flat out don't like it. They obviously like not having messages permanently disappear as soon as you hit "Delete". They don't like the message sitting there, with its big red "X" and strikethrough and whatever other "hey, I am deleted! I'm different!" indicator might be used. This isn't surprising. From POP3 clients twenty years ago to the defaults of Yahoo! Mail and Thunderbird today, deleted messages go to a "Trash" can. People are used to it, people want it.

The problem is, that isn't what IMAP is designed for. So, we're left thinking about ways to accomplish this. Thunderbird, Apple Mail, and Outlook all take a common approach. By default, deleted messages are copied to a Trash folder. They are also left in place marked for deletion, in proper IMAP style. The client hides these marked-for-deletion originals from the user's view.

What happens when the Trash is emptied?

All the copied messages in the Trash folder are permanently removed, while the originals are left there for all eternity, or until the user executes some sort of "Compact folder..." command. Which, obviously, nobody ever does unless they specifically know they should, because the client is hiding the messages that the "Compact" will remove. How can you expect someone to be a responsible user of a shared resource when they don't know that their software is leaving a trash heap under their feet?

This approach, however, does have advantages in that it satisfies the user desire for a Trash folder, and it leaves deleted messages alone in the original mailbox until directed otherwise. While the latter is problematic for the reasons noted above, it's a good thing in that it's non-disruptive if a person uses more than just that client to access their mail. Imagine you've been using Gophermail, marking messages for deletion but you aren't ready to get rid of them permanently yet; then you use Thunderbird at your desk, and it unilaterally does an expunge for you after you delete one message there. You'd go back to Gophermail the next day to find all the messages you left there gone, and then you'd end up with an expensive paperweight after giving your laptop a Kruschev-style shoe-banging.

So, there is a method to their madness.

But it has a huge cost in wasted storage.

What's a person to do! We're discussing this in an ongoing fashion. In the end, it seems that this approach may be the best idea if coupled with a clear, well-communicated policy like most e-mail providers (from Google and Yahoo to other universities) of removing any deleted messages after thirty days. This is one that is going to take a lot of talking out, but it seems a reasonable compromise.

April 24, 2008

Accessible, and Mobile

So I've talked a lot here about accessibility, and that work is still ongoing. But, today is a day for talking about something new and different.

Received Mobile Web Design by Cameron Moll in the post yesterday, and have begun working my way through it. As I've mentioned here in the past few weeks, part of the refactoring of Gophermail towards current "semantic" standards is to make it simpler to target towards various media, including mobile. This path is currently being worn by some of the best, from Apple to Google to a hundred smaller outfits. Our goal is to take some inspiration from the things they've done, while incorporating some of our own ideas and targeting the University environment in specific ways that outside folks simply never can.

But! While that thinking is going on, the first step is simply to have something usable and readable, and to that end I'll conclude this little bit of musing (more details later, I promise!) with this screenshot of Gophermail 2.0 running on an iPod Touch. Apologies for the quality, there's no easy way to do a screenshot so this was taken with the built-in camera on my iMac. Click for full-size.

April 18, 2008

Searching for (W3C) validation...

A week from today, we'll be meeting to finalize usability scenarios and go over the prototype. Five days after that evaluation sessions begin.

As we're in the home stretch now, I'm trying to finalize any feature adds that involve changes to interface elements, and beginning to move on to clean-up. Just like a good woodworker, I do my best to keep my area clean as I work. But I'll be the first to admit that during the iterative prototyping process, sometimes the sawdust accumulates around my feet a little. Today began the process of standardizing the CSS class usage across different areas of the app, fleshing out error handling to do more than writing a message to the console, things of that nature. And, of course, running all pages through the W3C validators.

Inherent in this is confirming that accessibility elements such as labels, fieldsets, and tab order have been employed in applicable areas. Heading elements, which can be used for quickly navigating a page with a high density of information (such as our message list screen), are a particularly interesting aspect when it comes to a web application as opposed to a web page. While most textual pages have some sort of outline structure, explicit or implicit, this isn't necessarily true in an application. Apps can lean in the direction of, oh, call it a widget model. There are widgets on the screen, which you try to group logically in relation to each other, but don't necessarily lend themselves to a hierarchical or list structure.

I've talked to a number of people about this issue and it sounds like I'm not the only one who wonders about this. So: for the moment I'm doing the best I can by having a "heading" (not necessarily always visible to non-screen-reader users, when visual clues render it redundant) for each major segment of the page.

For example, in the following screen shot:


  1. H1, not visible, at the top of the page which reads: "Message list for homework".
  2. H2, not visible, at the start of the main toolbar which reads: "Main toolbar".
  3. H2, not visible (likely will become visible), for the search box: "Search messages in homework".
  4. H2 on "My Folders".
  5. H2 on "homework with 8 messages". For an interface that does not use styles, this heading appears at the start of its toolbar, before the Copy/Delete widgets.
  6. H2, not visible, on the bottom toolbar which reads: "Occasional actions". That title will likely change, but it is appropriate!


Click for larger version

April 17, 2008

Learning, and interface

I spent Monday at MinneWebCon, which was a great experience; big thanks to the organizers Kristofer Layon and Peter Fleck, et al. It was a credit to the University to have that quality of a program here.

Moving on to this week's Gophermail progress: work on finalizing some style issues is ongoing. Myself and the alpha test audience have been evaluating some small things that make big differences, such as typeface. Trying to settle on a sans-serif face that is readable without strain in both standard and bold. I'm a bit surprised, honestly, at the challenge this has posed. At least to my increasingly Mr Magoo-like eyes, most of the standard sans-serif fonts seem very thick and compacted in bold. I think we've got it now; if it poses any problems during usability we'll hear about it.

Also continuing work on using bandwidth-freeing javascript insets, rather than full page loads, for a number of input interfaces that don't really need a full page. This would apply to things such as adding a person to your Contacts or changing miscellaneous settings, as opposed to composing a message. For an idea of where we're going with this (click for larger version):

April 13, 2008

Feature add: Web service for searches

Last week saw finalization of a framework for "web services" in Gophermail, and a proof-of-concept that lets you fetch the the newest N messages for a user who is currently logged into CAH.

Today IMAP searches have received a similar treatment. Remember that all details here are preliminary and subject to change, but here's the outline. A url such as:

gophermail.umn.edu/search?status_op=unseen

will return all unread messages. 'date_op' will be available for date-based searches, 'text_op' for text, etc. The current "proof" for this is a toggle on the message list page to show only unread messages. That may or may not be something that would get enough use to justify taking up space on the app's main interface. Have to evaluate that going forward. Details matter, lack of distraction matters, and anything that doesn't really need to be at the fingertips shouldn't be.

April 11, 2008

Web interfaces

AJAX, Web 2.0. We've been through the hype, the outrageously context-inappropriate deployments, more hype, the dismissals, more hype, and the backlash. Now we seem to be getting on to the part where people are using these techniques to add a layer of responsiveness that makes web applications less frustrating than they once were, while keeping in mind fallbacks for accessibility, mobile, and the like.

So, what can these things do for Gophermail?

Two of the major goals for the 2.0 release are addressing accessibility concerns and the mobile environment. The first step in this process took place over the first few months of this year. The late-90s markup style of Prayer was largely gutted. Tables for layout, especially, were mercilessly stripped. What was left was a semantically-meaningful skeleton to be enhanced with a presentation layer (CSS) and an enhanced interaction layer (Javascript).

First, I'll stress this: all functionality is available to a browser that ignores images, styles, and Javascript. In fact, one goal of this process was to make such an environment more useful than it previously was. For example: There are now proper headings to make navigation with a screen reader easier, and fewer redundancies in page content (the 1.x series has no fewer than three identical lists of folders on the message list page for various controls, for example).

That said: the abstraction of the presentation layer has enabled mobile/small-screen and large-font versions, among other things, that hinge only on style sheets. This will enable, going forward, rapid development and modification of environments for a wide array of devices.

On the interaction side, there is a small Javascript library being introduced that attaches asynchronous handlers to a number of actions. For example: deleting a message from the message list screen no longer causes a page reload. Neither does checking mail, or navigating through your folder hierarchy in the new folder view. Not only does this enhance responsiveness and hence usability, but it greatly cuts down on network use.

This is critical going forward as e-mail on mobile devices moves from being a toy--or something people put up with because they had to--to something that is truly usable, and in demand. The more we get devices with readable screens and low-key, usable interfaces, the more true this will become, and with this shift in the "platform" we're on our way to being able to address these needs in an agile, flexible manner.

I'm curious if people have ideas on ways that web-based e-mail interaction, whether in a desktop browser, a mobile device, or anywhere else, could be improved. Most development seems to focus on emulating the classic desktop e-mail clients, and it's fairly obvious that this isn't the most productive approach.

April 10, 2008

Transparent session recovery

Background: Gophermail's underlying engine is the University of Cambridge's Prayer webmail application. Written in C, it is in a sense an IMAP client much like a desktop mail application. It encapsulates its own https daemon which spawns a persistent session on the server on login; for the rest of the user's session, they deal with this persistent child, and the child is a full-fledged IMAP client that "just happens" to use a browser for its output.

This stateful nature (and the C codebase) makes it a very efficient app, light years beyond all the PHP webmails out there. No PHP, no apache, all very self-contained and streamlined.

The downside is that Prayer/Gophermail depends on these sessions for nearly all operations. If the session times out or is otherwise interrupted, the user is faced with a "Sorry, your session went away!" and must start over from the beginning. Whether they were just trying to look at a message, or send a draft they'd spent an hour on. It's as if your postman occasionally decided, after picking up your letters, to stop in at the bar and use them for a beermat.

So: In Gophermail 2.0, the interrupted session is (for almost all cases) a thing of the past. What enabled today's proof-of-concept of a "recent mail" web service is a new regime of transparent session recovery. When a user hits the server with a "session"-type URL and the server finds no session, one is silently established, based on the UMN auth cookie in the browser, and the GET or POST request is then honored as if nothing had happened. GETs are fully functional; POSTs will require a little more work but are mostly functional. Just need a posthole digger and some guywire to finalize them...

Feature add: Web service for listing email

So, as of this afternoon there is a functional proof-of-concept that exposes the IMAP message list as a web service.

Querying a url (which will likely end up being something like gophermail.umn.edu/recent?range=X) will give you an XML doc that contains the most recent X (in range=X) number of messages for the username that is currently logged into UMN Central Auth Hub in that browser. Likely representation will be something like arrival time, sender, and subject, and a url to jump into viewing that message in Gophermail.

This has been something that's been requested for MyU, among other things. Looks like it will be do-able.

BUGFIX: Empty Drafts and Sent mail folders

The bug that was causing Drafts and Sent mail folders to appear empty for standard (mbox format) users in the 2.0 test builds has been fixed. Will be deployed to stable-test this morning.

April 9, 2008

International Character Support functional

International character support (via UTF-8) has been added to today's GopherMail 2.0 test build and has passed early checks. This was a piece of spam I happened to have in my Inbox and I have no idea what it says, if it's vulgar, let me know and I'll switch it out for something else:

EDIT: I've been informed that this was indeed some NC-17 spam :) , so I have two new examples. One is Japanese from the administrative info on the FreeBSD mailing list archives website; the other is Greek Ministry of Tourism transport info. Both were sent and then received in Gophermail 2.0.

More to come....