Dealing with the prospect of a “digital dark age” is one of the reasons why we built Emailchemy, but I’ve found that many have trouble with this new term because they may not be familiar with the definition of the original. The term “Dark Ages” refers to a time period between the fall of Rome and the Age of Enlightenment, and historians often referred to it as “dark” because there is little in the way of contemporary written history or literature.

The concept of the “Digital Dark Age” is similar, in that through the use of proprietary file formats we may be setting the stage for a future Dark Age because we will no longer have the legacy technology required to read those proprietary file formats.

Here are a couple articles that goes into some more detail, but if you search Yahoo! for “Digital Dark Age”, you’ll find plenty of resources:

• ‘Digital Dark Age’ may doom some data

from the Graduate School of Library and Information Science at the University of Illinois at Urbana-Champaign

• Warning of a data ticking time bomb

from the BBC and the National Archives of the UK

In Part 1, I explained how the “Date:” header of an email is actually the “sent date” and introduced a couple of issues with how the sent date is created by different email clients. Now in Part 2, I’ll give a similar rundown of the problems with the received date, starting with a nod to the the title of this series of posts by telling you there is no such thing as a received date.

That’s right – it doesn’t exist. The format of all Internet mail, as it goes over the wire (or over the air, whatever) is specified by RFC-2822 – the standard Internet Message Format. It defines the structure of a message, including the various header fields you may be familiar with, like “From: ” or “To: ” or “Subject: ” and many more that you may never see. The point here is that RFC-2822 does not define a header field for a received date, and this is a problem because then every email client application is free to interpret and create a received date pseudo-header which may not map to any other email client’s interpretation.

For example, Mail for Mac OS X uses the date found in the top (last written) “Received:” header as a received date. It’s a novel approach because it doesn’t require the injection of proprietary non-portable headers like “X-Received-Date:”, but the date it is using is when the last mail server in the delivery chain received the message. Specifically, it is not the date that the email recipient saw or even read the message. The email could have a received date of today, even if you don’t check your email for the next week!

Other email clients will do effectively the same by asking the mail server for the date of receipt, or they will try to derive a received date based on when the message is first “seen” on the server, or the time of actual download from the server (if it is downloaded at all vs. just viewed on the server), or the time that the message was read.

So, when you send a message to multiple recipients, although it is to be expected that they will all receive the message at different times, unless they are all using the same email client configured the same way, chances are they will all have received dates that reflect different events. This all makes answering questions of “Who got it first?” or “When did they read it?” very difficult.

More importantly, from the standpoint of email migration, because there is no standard for a received date, after moving messages to a new email client you may lose all “received date” information. In this case, you can always fall back to the “sent date” for sorting by date.

Why do email clients try to show a received header at all? Well, it’s convenient, and I can only assume that end users asked for it. People like to process messages in the order in which they arrive. Does “sent date” work for this too? Yes, mostly, but because messages may arrive in an order different from how they were sent and email can sometimes take hours to days to be delivered, when processing your inbox you would have to check back in the list every now and then for new arrivals.

I think that last point is crucial — the received date was introduced on the client side as a convenience and not as trace information. Treat it accordingly.

There is no such thing as a “received date”. Not in the standards, anyway. Is this a quibbling technicality? An omission?

Nope. And I’ll tell you why. Why? Because the date of a message matters. It matters more than it probably should, with the advent of email forensics, being that the date is inconsistently defined across email clients, and being that it is easily forged. Still, the date of a message is good enough for most purposes, as long as you understand what it means and where it came from.

First, let me tell you that the “Date:” header field in the RFC-2822 header of every message is the “origination date” field. From section 3.6.1 of the specification:

“The origination date specifies the date and time at which the creator of the message indicated that the message was complete and ready to enter the mail delivery system. For instance, this might be the time that a user pushes the “send” or “submit” button in an application program. In any case, it is specifically not intended to convey the time that the message is actually transported, but rather the time at which the human or other creator of the message has put the message into its final form, ready for transport.”

Note there is significant leeway given here to what event the Date field can represent. In practice, this date is usually when the user hits “Send”, but I’ve seen it also implemented to be the time of message creation — even if the message is sent days later. I argue the latter case is an incorrect usage of the Date field, but that doesn’t stop it from happening.

Now consider if there is an error the first time I hit “Send” — or if it takes a really long time to send. My point is that it’s difficult to build a science around the meaning of this Date. All you can really derive is that the Date header represents when the sender intended to send the message.

True, it often does has some correlation to when the message was actually delivered to a mail server for delivery, but it’s frame of referential consistency is extremely limited. What I mean by that is that if you were to sequence the messages of multiple users — or even the same user who uses different email accounts or email client programs — the sequence could not be considered scientifically sound. At best, you have a sequence of intents, not events.

I’ll talk more about the Myth of the Received Date and how some applications choose to derive a date of message receipt in Part 2. In the meantime, try a few experiments yourself to determine how accurate the Date header reflects what you consider to be the “sent date” of a message. I suggest trying this with accounts on different domains or even different hosting providers and with different email clients.

Mail for Mac OS X format is not the same as mbox files

I think the title says it all, but the problem is bigger than that. The whole idea that the last 3 or 4 letters of a filename are an indication of underlying file format and structure is flawed. More than flawed, it’s wrong, but 3 decades of MS-DOS (yes, it’s still part of Windows) and its usability nightmare known as filename extensions is hard to overcome.

Interestingly, I don’t blame Microsoft for this particular confusion though, since it was Apple that broke the generally accepted, or de facto, standard in this case.

With the release of OS X, Apple introduced a new kind of file — or really a folder that acted and looked like a file to the user — called a package. The idea was that the insides of certain folders were only for system usage and should be hidden from users. For example, applications and all the various libraries and resource files and executables were packaged into a .app folder. To the end user, this .app folder looked and acted like a standard file and it could be double-clicked to launch the application. Early versions of Mac OS X even hid this package extension from the user, but to this day, to see the contents of a package, you have to “right-click” or “control-click” on the package and select “show package contents” to see what’s inside.

I have had 2 big problems with this new package construct:

1. No other OS has this construct and writing cross-platform code to deal with it is more difficult than it should be.
2. The Mac OS X file open/save dialogs do not hide the contents of the packages the same way the Finder does. Being that Mac OS X is otherwise very particular about consistency in its UI, I call this a bug.

Now, back to the topic of this post, Apple’s Mail application for Mac OS X uses “.mbox” as the package suffix for the folders that hold email message files. This is in direct conflict with the de facto standard of using “.mbox” as the file name extension for standard mbox files, and this creates incredible confusion for Mac and PC users alike when they are looking to convert their mail to or from mbox format or Mac OS X Mail format. In particular, Mac users looking to import mail into Mac OS X Mail get very confused with this and Apple should really do something about it. To make matters even more confusing, back with the release of Mac OS X 10.4, Apple removed the “package-ness” of the .mbox folders and now users can browse their contents freely. So, since these folders are no longer packages, why do they still need the .mbox name extension? Again, inconsistency.

So, to help deal with this confusion, I offer up these 4 rules of thumb:

• Mac OS X Mail folders are not Standard mbox files.
• Standard mbox files are not Mac OS X Mail files.
• If you have a folder with a name that ends with “.mbox”, it MAY be a Mac OS X Mail folder, but it is DEFINITELY NOT a standard mbox file.
• If you have a file with a name that ends with “.mbox”, it MAY be a Standard mbox file, but it is DEFINITELY NOT a Mac OS X Mail folder or file.

I hope this helps.