Enough people have trouble with importing email into Outlook or trying to create PST files that we felt it necessary to give an explicit walk-through of detailed instructions. We’ve also been experimenting with answering tech support requests with video “screencasts”, so I’m putting them both in this post for future reference.

Outlook has many advanced features and the user interface presents many different ways to perform any given task, but the problem is that not all results are consistent. After much experimentation, we’ve found a sequence of actions that will work in most circumstances.

The screencast starts with the act of converting Entourage email, but you could substitute the converting of any supported format. For example, you could use these instructions to move a standard mbox file into Outlook, too, as a general way to convert mbox to PST files.

View Screencast here: http://screencast.com/t/ktDmvphFJ6∞

Step 1: Convert your mail to the IMAP ImportServer format

1. do this on the computer you are moving to, or at least the same kind of computer (Windows, Mac, etc.)
2. copy your email from your old computer to the new computer
3. run Emailchemy, use the “Conversion Wizard” tool (you’ll be in it after Emailchemy starts)
4. select the source format, click Next
5. select your old email files, click Convert
6. select the IMAP ImportServer format from the drop-down list
7. enter a name, like “convertedmail”

Step 2: Start the IMAP ImportServer

1. in Emailchemy, click the Toolbox button
2. select the “convertedmail.imapserver” folder — it should already be in the file box, though
3. click the “Start Server” button

Step 3: Connect Outlook to the IMAP server

1. launch Outlook, using the menu, select Tools –> Account Settings…
2. click New to create a new account
3. choose Microsoft Exchange, POP3, IMAP or HTTP, then click Next
4. at the bottom of the next window, check “Manually configure server settings…” and click Next
5. select Internet E-mail and click Next
6. Enter the following:
   • Your Name: import
   • E-mail Address: import@localhost
   • Account Type: IMAP
   • Incoming mail server: localhost
   • Outgoing mail server: localhost
   • User Name: import
   • Password: mymail
7. Click “More Settings…”, click the Advanced tab, and enter:
   • Incoming server (IMAP): 8143
8. Click OK, then Next, then Finish, then Close (to close the Account Settings window)

[Note: at this point you should be able to browse your mail on the server. If you do not see your mail folders inside the mail account folder in Outlooks navigation pane, then you can right-click on the account name (import@localhost) and select "Update Folder List"]

Step 4: Force Outlook to download the mail

1. in Outlook, click on a folder in the new account folder, any folder
2. using the main menu, select Tools –> Send/Receive –> Send/Receive Settings –> Define Send/Receive Groups
3. select All Accounts, then click the Edit button
4. select the option: “Download complete items including attachments…”
   • optional: uncheck “Send mail items” (turn this back on after done importing)
5. click OK, then Close
6. click Send/Receive in the toolbar

[Note: this part can take some time while Outlook is downloading all the email from the server.]

When complete, your mail will be in the PST file found by right-clicking on the account folder, select Properties, then click “Advanced…”. From this point you can copy your mail to an existing PST file or a new one.

Step 5 Option A: Copy the mail to an existing local PST file

1. right-click on the server folder you want to copy — all subfolders will be copied automatically
2. select “Copy (name of folder you selected)…”
3. a window will popup and ask you to select the folder to copy the server folder to, or you can create a new folder
4. click OK

Step 5 Option B: Copy the mail to a new PST file:

1. select the server folder you want to copy
2. using the main menu, select File –> Import and Export…
3. select “Export to a file”, click Next
4. select “Personal Folder File (.pst)”, click Next
5. double check you have the right folder selected, then click “Include subfolders”
6. select where to save the new PST file and what to call it, then click Finish

At this point you can then open the PST file you just created with File –> Open –> Outlook Data File

Some people have written in confused about why Emailchemy converted so many more emails than their Inbox was showing. While perhaps alarming, it’s not cause for concern. To say it simply, Emailchemy extracts deleted messages from many email storage formats, and if you don’t want that to happen you should use the “Compact Mailbox” or “Compact Database” feature of your old email application before doing the conversion.

To be accurate, however, the “deleted” emails were really only marked as deleted and not physically removed from the mailbox or database. Many email applications, including Eudora, Thunderbird, Netscape, and a few versions of Outlook Express, don’t really delete your email when you tell them to. In order to be quick and responsive to the user, these email applications will simply mark the message in a way that tells the user interface to no longer display the message. The email message is still there, and it will be there until the user tells the application to rebuild or compact the database, at which time the application will take the time to physically remove the messages that were marked for deletion.

So, why does Emailchemy assume you want to recover these “deleted” messages? Without getting into the technical details, it was a design decision that helps make the converters more robust, but also there is the idea that it’s better to err on the side of giving the user more rather than less. What we’ve found is that many people appreciate this feature and are even pleasantly surprised when they see email messages that they thought had been lost, and it’s often rather difficult to explain the difference between messages in the Trash folder, messages marked for deletion, and messages that have actually deleted.

Here’s a walkthrough I recently sent to a customer who had questions about how the Google Apps Uploader works. I’m posting it here for future reference.

1) Using either the “Conversion Wizard” or “Advanced Conversion” tool in Emailchemy’s toolbox, convert your mail to Standard mbox Format, giving it a name like “converted”.

2) Switch over to Emailchemy’s Google Apps Uploader tool, select the “converted” folder and enter the target email account’s address (the account that you want to receive the uploaded messages) and leave the default settings for the other options for now.

3) When you click the “upload” button, you’ll have to authenticate to Google Apps. The account you authenticate with must be authorized to upload messages or be the domain administrator. The administrator may also choose to allow specific account holders to
upload their own, too. Also, the “upload email” feature must be enabled from the domain admin console before this works. If you are behind a web proxy server, click the “proxy” button and enter the details.

4) Click “login”

As the upload is going on, you may get a timeout message if you lose connectivity or if a message is taking too long to upload. The upload thread is still going even though it is asking you for what to do next, so you can safely step away from the computer. When the connection is restored and a response comes back from Google, the timeout dialog will go away by itself. In other words, it assumes you want to keep waiting even if you don’t explicity say so.

If an upload error occurs during the upload, Emailchemy will automatically retry before performing the action you specified on the first screen (ask, skip, or stop). If the failure repeats, Emailchemy will perform the specified action. If it asks you, you can choose to skip or stop or always skip.

If you stop an upload for any reason, Emailchemy will ask you if you want to resume the next time you start the Uploader.

When finished, Emailchemy will report the number of messages “processed” — this is the number of messages Emailchemy tried to upload, not adjusted for the ones you told it to skip. This number should be the same as the number of messages converted, reported by the Conversion Wizard tool.

Gmail will take up to an hour to display all the uploaded messages. Then, any messages that was “read” will not be in the Inbox – they will be in the “All Mail” folder. Only “new” or “unread” messages will be in the Inbox.

Emailchemy tags/labels every message with a timestamp of the upload event (something like “upload20090120″). This is useful in that you can click on this label to see the list of all uploaded messages, and even delete all the messages with this tag.

Again, keep in mind that Gmail gives you “conversation” counts, not message counts, so the conversation count will likely not match the number of messages uploaded that Emailchemy reported. Google uses some fuzzy logic to try to group messages into conversations, so things may get grouped together that should not be grouped together, and that is an unavoidable feature of the Gmail interface.

In Part 1, I explained how the “Date:” header of an email is actually the “sent date” and introduced a couple of issues with how the sent date is created by different email clients. Now in Part 2, I’ll give a similar rundown of the problems with the received date, starting with a nod to the the title of this series of posts by telling you there is no such thing as a received date.

That’s right – it doesn’t exist. The format of all Internet mail, as it goes over the wire (or over the air, whatever) is specified by RFC-2822 – the standard Internet Message Format. It defines the structure of a message, including the various header fields you may be familiar with, like “From: ” or “To: ” or “Subject: ” and many more that you may never see. The point here is that RFC-2822 does not define a header field for a received date, and this is a problem because then every email client application is free to interpret and create a received date pseudo-header which may not map to any other email client’s interpretation.

For example, Mail for Mac OS X uses the date found in the top (last written) “Received:” header as a received date. It’s a novel approach because it doesn’t require the injection of proprietary non-portable headers like “X-Received-Date:”, but the date it is using is when the last mail server in the delivery chain received the message. Specifically, it is not the date that the email recipient saw or even read the message. The email could have a received date of today, even if you don’t check your email for the next week!

Other email clients will do effectively the same by asking the mail server for the date of receipt, or they will try to derive a received date based on when the message is first “seen” on the server, or the time of actual download from the server (if it is downloaded at all vs. just viewed on the server), or the time that the message was read.

So, when you send a message to multiple recipients, although it is to be expected that they will all receive the message at different times, unless they are all using the same email client configured the same way, chances are they will all have received dates that reflect different events. This all makes answering questions of “Who got it first?” or “When did they read it?” very difficult.

More importantly, from the standpoint of email migration, because there is no standard for a received date, after moving messages to a new email client you may lose all “received date” information. In this case, you can always fall back to the “sent date” for sorting by date.

Why do email clients try to show a received header at all? Well, it’s convenient, and I can only assume that end users asked for it. People like to process messages in the order in which they arrive. Does “sent date” work for this too? Yes, mostly, but because messages may arrive in an order different from how they were sent and email can sometimes take hours to days to be delivered, when processing your inbox you would have to check back in the list every now and then for new arrivals.

I think that last point is crucial — the received date was introduced on the client side as a convenience and not as trace information. Treat it accordingly.

There is no such thing as a “received date”. Not in the standards, anyway. Is this a quibbling technicality? An omission?

Nope. And I’ll tell you why. Why? Because the date of a message matters. It matters more than it probably should, with the advent of email forensics, being that the date is inconsistently defined across email clients, and being that it is easily forged. Still, the date of a message is good enough for most purposes, as long as you understand what it means and where it came from.

First, let me tell you that the “Date:” header field in the RFC-2822 header of every message is the “origination date” field. From section 3.6.1 of the specification:

“The origination date specifies the date and time at which the creator of the message indicated that the message was complete and ready to enter the mail delivery system. For instance, this might be the time that a user pushes the “send” or “submit” button in an application program. In any case, it is specifically not intended to convey the time that the message is actually transported, but rather the time at which the human or other creator of the message has put the message into its final form, ready for transport.”

Note there is significant leeway given here to what event the Date field can represent. In practice, this date is usually when the user hits “Send”, but I’ve seen it also implemented to be the time of message creation — even if the message is sent days later. I argue the latter case is an incorrect usage of the Date field, but that doesn’t stop it from happening.

Now consider if there is an error the first time I hit “Send” — or if it takes a really long time to send. My point is that it’s difficult to build a science around the meaning of this Date. All you can really derive is that the Date header represents when the sender intended to send the message.

True, it often does has some correlation to when the message was actually delivered to a mail server for delivery, but it’s frame of referential consistency is extremely limited. What I mean by that is that if you were to sequence the messages of multiple users — or even the same user who uses different email accounts or email client programs — the sequence could not be considered scientifically sound. At best, you have a sequence of intents, not events.

I’ll talk more about the Myth of the Received Date and how some applications choose to derive a date of message receipt in Part 2. In the meantime, try a few experiments yourself to determine how accurate the Date header reflects what you consider to be the “sent date” of a message. I suggest trying this with accounts on different domains or even different hosting providers and with different email clients.

Mail for Mac OS X format is not the same as mbox files

I think the title says it all, but the problem is bigger than that. The whole idea that the last 3 or 4 letters of a filename are an indication of underlying file format and structure is flawed. More than flawed, it’s wrong, but 3 decades of MS-DOS (yes, it’s still part of Windows) and its usability nightmare known as filename extensions is hard to overcome.

Interestingly, I don’t blame Microsoft for this particular confusion though, since it was Apple that broke the generally accepted, or de facto, standard in this case.

With the release of OS X, Apple introduced a new kind of file — or really a folder that acted and looked like a file to the user — called a package. The idea was that the insides of certain folders were only for system usage and should be hidden from users. For example, applications and all the various libraries and resource files and executables were packaged into a .app folder. To the end user, this .app folder looked and acted like a standard file and it could be double-clicked to launch the application. Early versions of Mac OS X even hid this package extension from the user, but to this day, to see the contents of a package, you have to “right-click” or “control-click” on the package and select “show package contents” to see what’s inside.

I have had 2 big problems with this new package construct:

1. No other OS has this construct and writing cross-platform code to deal with it is more difficult than it should be.
2. The Mac OS X file open/save dialogs do not hide the contents of the packages the same way the Finder does. Being that Mac OS X is otherwise very particular about consistency in its UI, I call this a bug.

Now, back to the topic of this post, Apple’s Mail application for Mac OS X uses “.mbox” as the package suffix for the folders that hold email message files. This is in direct conflict with the de facto standard of using “.mbox” as the file name extension for standard mbox files, and this creates incredible confusion for Mac and PC users alike when they are looking to convert their mail to or from mbox format or Mac OS X Mail format. In particular, Mac users looking to import mail into Mac OS X Mail get very confused with this and Apple should really do something about it. To make matters even more confusing, back with the release of Mac OS X 10.4, Apple removed the “package-ness” of the .mbox folders and now users can browse their contents freely. So, since these folders are no longer packages, why do they still need the .mbox name extension? Again, inconsistency.

So, to help deal with this confusion, I offer up these 4 rules of thumb:

• Mac OS X Mail folders are not Standard mbox files.
• Standard mbox files are not Mac OS X Mail files.
• If you have a folder with a name that ends with “.mbox”, it MAY be a Mac OS X Mail folder, but it is DEFINITELY NOT a standard mbox file.
• If you have a file with a name that ends with “.mbox”, it MAY be a Standard mbox file, but it is DEFINITELY NOT a Mac OS X Mail folder or file.

I hope this helps.

Inbox Repair Tool on Windows Vista

It can be random, and you may not even notice it right away, but even if a single bit gets flipped from a “0″ to a “1″ in a PST file, you could lose messages or the ability to even open the PST file in Outlook.

How do you know if a PST file is corrupt? There are major tells, like if Outlook tells you, for example, when you try to open it, or, if Outlook crashes when you try to open a particular message or open a particular folder in the PST data. But sometimes it is more subtle, like you may discover messages or attachments have disappeared, search no longer returns any results, or you can’t move messages in or out of the PST file. These are the cases that you may not notice in normal day-to-day use, but when you want to export your email from Outlook PST files with a utility like Emailchemy, you just might.

Emailchemy reads the PST file directly from the disk — not using the Microsoft connectors or API’s, but it reads the data from the PST’s database in much the same way that Outlook would. When Emailchemy finds corrupted entries in the PST data, it will first tell you about it by logging a warning to the console (or to stdout.log or stderr.log file) and then it will try to recover. Most of the time, if an expected data object like an attachment or other message body part is missing, Emailchemy will easily recover and you will get the message in the output with all the data that Emailchemy was able to find. However, because data corruption is random in nature it can sometimes cause Emailchemy to halt or stop the conversion prematurely — but this only happens with files that even Outlook would have trouble opening. With Emailchemy, you would at least get some of the messages out.

However, not getting all your messages out is simply unacceptable. So, what can you do if you have a corrupt PST file? (Remember, if Emailchemy is not completely converting your PST file, then it is likely corrupt.)

Fortunately, Microsoft is aware of how easily the PST file can become corrupted, so they provide the necessary tools for repairing them. The Inbox Repair Tool from Microsoft, aka “scanpst.exe”, can identify PST file corruption and most of the time repair it to the point that Outlook can again open the file — and thus so that Emailchemy can read the PST file too.

Here’s an excerpt from the Microsoft support article on how to use scanpst.exe:

ScanPST mostly validates and corrects errors in the internal data structures of a .pst file. The .pst file is a database file. Therefore, structures, such as BTrees and reference counts, are checked and repaired as necessary. These low-level objects have no knowledge of the upper-level structures, such as messages, calendar items, and so on, that are built upon them. If ScanPST determines a specific block of the structure or table is unreadable or corrupted, ScanPST removes it. If that block was part of a specific item in Outlook, the item will be removed when it is validated. User may not expect this behavior. However, the removal of the item is appropriate given the circumstances. Also, this specific type of situation is probably very rare, and it will always be entered in the ScanPST log file.

Scanpst.exe will also create a backup of the original PST file before attempting repair. Keep this backup in case you experience the rare case of messages or attachments (items) being removed from the database as a result of the repair.

To sum up, in most cases you will not need to run scanpst.exe on your PST file before converting it with Emailchemy, but if you think Emailchemy isn’t converting your PST file correctly, try running scanpst.exe on it first. If you see warnings written to the console logs during the conversion of your PST file, definitely try running scanpst.exe on the PST file and then try the conversion again.

Someone recently wrote in asking how to recover email from a corrupt Entourage Database file and then how to get the email into a new version of Entourage. Specifically, the asker had a file that Entourage 2008 could not import or upgrade from an earlier version, and he had tried all the various methods of rebuilding that Entourage provides. Here is my answer to him, which I thought others would find useful too:

Hi B.,

It sounds like there are a couple questions here, so let me try to address them one at a time.

Regarding corrupt Entourage database files: Emailchemy can still extract messages from generally corrupt databases, but depending on how the file is damaged, it may not get everything out. So, there are a couple of other things you can try.

• Convert the Entourage database file as if it were an Outlook Express 5 for Mac file. You’ll have to do this using Emailchemy’s advanced interface. Doing this will lose all the folders, but it has a good chance of getting more messages out. This works with Database files created by Entourage versions prior to Entourage 2004
• If you have the machine where the old Entourage was running, you can try to grab the cache files that Entourage generates. Emailchemy now has a Entourage Cache conversion wizard that should be able to help.

Regarding how to get the mail into Entourage 2008, there are a few different methods listed in the manual:

• Convert to an Entourage RGE file and then have Entourage import it. This is probably the easiest way.
• Convert to mbox files and then have Entourage import them. It works as well as the first method, but you have to each mbox one at a time (you’ll get an mbox file for each original email folder).
• Convert to IMAP ImportServer format and download the messages into Entourage using an IMAP account that connects to Emailchemy’s IMAP server. This is a fallback that you usually only need for importing into Outlook, but it’s there as an option.

Let me know if this helps, or if you have any more questions.

Thanks,

Matt

Emailchemy's Google Apps Uploader tool

Which to use? I like both. That way, you get to preserve the original hierarchy and still have the flexibility of “tags”.

I’m not sure exactly which update did it, but Entourage has lost the ability to double-click import Entourage archives (.rge files). I for one found this feature very convenient, but that’s probably because I’ll import tens of archives in a single day when I’m testing a new converter. Still, it made it easy to describe the migration steps to customers moving to Entourage: “Convert and Double-click!”

No more.

I tested this with RGE files created by Entourage itself as well as RGE files created by Emailchemy, and every time Entourage 2008 would open when double clicking the RGE file, but no import. I now have to import using Entourage’s Import Wizard, which works, but just isn’t as fast or convenient.