The new Entourage converter has increased throughput (i.e. “it’s faster”) because it now keeps the entire database index in memory. This reduces the amount of jumping around the file, and it doesn’t increase the memory requirements significantly because all Entourage database files have known maximum number of items they can hold.

Also, and perhaps more importantly, the new Entourage converter extracts more data. It preserves all message flags (e.g. read/unread, priority, replied-to, forwarded, flagged) and sent attachments now too. Previous versions only preserved received attachments.

The Outlook PST converter was also modified to be a bit more resilient to corrupted PST files. Still, it’s always a good idea to make sure your PST file still works in Outlook before trying to run it through Emailchemy. See our previous post on working with corrupt PST files for more details.

Inbox Repair Tool on Windows Vista

It can be random, and you may not even notice it right away, but even if a single bit gets flipped from a “0″ to a “1″ in a PST file, you could lose messages or the ability to even open the PST file in Outlook.

How do you know if a PST file is corrupt? There are major tells, like if Outlook tells you, for example, when you try to open it, or, if Outlook crashes when you try to open a particular message or open a particular folder in the PST data. But sometimes it is more subtle, like you may discover messages or attachments have disappeared, search no longer returns any results, or you can’t move messages in or out of the PST file. These are the cases that you may not notice in normal day-to-day use, but when you want to export your email from Outlook PST files with a utility like Emailchemy, you just might.

Emailchemy reads the PST file directly from the disk — not using the Microsoft connectors or API’s, but it reads the data from the PST’s database in much the same way that Outlook would. When Emailchemy finds corrupted entries in the PST data, it will first tell you about it by logging a warning to the console (or to stdout.log or stderr.log file) and then it will try to recover. Most of the time, if an expected data object like an attachment or other message body part is missing, Emailchemy will easily recover and you will get the message in the output with all the data that Emailchemy was able to find. However, because data corruption is random in nature it can sometimes cause Emailchemy to halt or stop the conversion prematurely — but this only happens with files that even Outlook would have trouble opening. With Emailchemy, you would at least get some of the messages out.

However, not getting all your messages out is simply unacceptable. So, what can you do if you have a corrupt PST file? (Remember, if Emailchemy is not completely converting your PST file, then it is likely corrupt.)

Fortunately, Microsoft is aware of how easily the PST file can become corrupted, so they provide the necessary tools for repairing them. The Inbox Repair Tool from Microsoft, aka “scanpst.exe”, can identify PST file corruption and most of the time repair it to the point that Outlook can again open the file — and thus so that Emailchemy can read the PST file too.

Here’s an excerpt from the Microsoft support article on how to use scanpst.exe:

ScanPST mostly validates and corrects errors in the internal data structures of a .pst file. The .pst file is a database file. Therefore, structures, such as BTrees and reference counts, are checked and repaired as necessary. These low-level objects have no knowledge of the upper-level structures, such as messages, calendar items, and so on, that are built upon them. If ScanPST determines a specific block of the structure or table is unreadable or corrupted, ScanPST removes it. If that block was part of a specific item in Outlook, the item will be removed when it is validated. User may not expect this behavior. However, the removal of the item is appropriate given the circumstances. Also, this specific type of situation is probably very rare, and it will always be entered in the ScanPST log file.

Scanpst.exe will also create a backup of the original PST file before attempting repair. Keep this backup in case you experience the rare case of messages or attachments (items) being removed from the database as a result of the repair.

To sum up, in most cases you will not need to run scanpst.exe on your PST file before converting it with Emailchemy, but if you think Emailchemy isn’t converting your PST file correctly, try running scanpst.exe on it first. If you see warnings written to the console logs during the conversion of your PST file, definitely try running scanpst.exe on the PST file and then try the conversion again.

Hi B.,

It sounds like there are a couple questions here, so let me try to address them one at a time.

Regarding corrupt Entourage database files: Emailchemy can still extract messages from generally corrupt databases, but depending on how the file is damaged, it may not get everything out. So, there are a couple of other things you can try.

• Convert the Entourage database file as if it were an Outlook Express 5 for Mac file. You’ll have to do this using Emailchemy’s advanced interface. Doing this will lose all the folders, but it has a good chance of getting more messages out. This works with Database files created by Entourage versions prior to Entourage 2004
• If you have the machine where the old Entourage was running, you can try to grab the cache files that Entourage generates. Emailchemy now has a Entourage Cache conversion wizard that should be able to help.

Regarding how to get the mail into Entourage 2008, there are a few different methods listed in the manual:

• Convert to an Entourage RGE file and then have Entourage import it. This is probably the easiest way.
• Convert to mbox files and then have Entourage import them. It works as well as the first method, but you have to each mbox one at a time (you’ll get an mbox file for each original email folder).
• Convert to IMAP ImportServer format and download the messages into Entourage using an IMAP account that connects to Emailchemy’s IMAP server. This is a fallback that you usually only need for importing into Outlook, but it’s there as an option.

Let me know if this helps, or if you have any more questions.

Thanks,

Matt

Emailchemy's Google Apps Uploader tool

Which to use? I like both. That way, you get to preserve the original hierarchy and still have the flexibility of “tags”.

The new feature is for setting a default sender for Claris Emailer 1.x sent messages that don’t have a sender recorded. Claris Emailer assumed that the email was sent from you, the user of Claris Emailer, so it didn’t (or doesn’t, if there’s anyone still using it) bother to record that bit of information. Having a default sender is a convenience, if anything, so that when you view these messages in another email app, you no longer have to guess who sent them.

Bugs fixed:

• Outlook PST HTML message bodies that define alternate charsets now being transcoded correctly
• Outlook PST attachment file names now being converted to UTF-8 for better handling of international characters
• AOL for Windows wizard no longer case sensitive when searching for PFC files
• Entourage cache files from all versions of Entourage now supported. There were some files from the very earliest Entourage version that were not being picked up.

Entourage Email Archiver: This site talks about the effectiveness of the latest version of Emailchemy!

SoftSea Review: This is a very nice review of Emailchemy 9.8; they gave us a 5 star rating!

Til next time,
Jackie

I originally designed Emailchemy to run in as little memory as possible, simply because at that time most computers had 32MB total of RAM. And, being that I was writing Emailchemy in Java, I had to be extra careful about how much heap space (”working memory”) Emailchemy required. This presented a specific requirement to not load an entire email message into memory for processing, because even back then, an email could be 10MB or larger with big attachments and most computers at the time would have choked under that weight. So, Emailchemy was designed to process the contents of an email in chunks, and it was able to run and convert any email in under 6MB of memory.

Even today when computers routinely ship with 1GB of RAM, I insist on sticking to highly efficient use of memory because it forces developers to write better code and it produces executables that can even run on lower-end, older computers. After all, not everybody buys a new laptop every 2 years.

Now, because Emailchemy is written in Java and therefore runs in a virtual machine, when the application starts it has to tell the virtual machine how much memory to grab from the host computer. The problem is that Java applications can’t see how much real memory the host computer has. So, we’ve intentionally kept the amount of memory that Emailchemy requests as low as possible — just high enough to work for most people’s needs. If we were to tell Emailchemy to grab too much memory, it would probably crash.

Even though Emailchemy processes the message content in chunks, for some formats it has to scan and build an index of all the little parts that make up a message in the original file, and that index has to be kept in memory throughout the conversion of all the messages.

So, we’ve bumped Emailchemy’s base memory footprint to 128MB just to accommodate these file formats. But, if a user has very large files, say bigger than a gigabyte or two, then Emailchemy really needs more memory to convert the email at the fastest rate possible. Then, if a user has extremely large files, say a 17GB PST file, Emailchemy really will need some more memory to even begin extracting and converting messages, because the index for such a file may take up the entirety of Emailchemy’s default memory allocation.

When people were writing in and asking for help with these super huge files, we tried to walk them through the process of editing a config file or starting Emailchemy from the command line to give it more memory. In many cases, the users were able to make the changes, but I can’t imagine trying to walk a marginally computer literate person through that exercise. So, it became very clear very fast that we needed to provide an easy way to change Emailchemy’s memory allocation.

But why a separate application? Why not build this into Emailchemy itself? Well, remember, Emailchemy can’t see how much memory the host computer has, so there was a risk that the user would choose a setting that asked for too much memory and thus prevent Emailchemy from launching again. If Emailchemy can’t start, then the user won’t be able to change the setting back, and Emailchemy wouldn’t be able to change it back itself. Sure, we could have posted instructions on how to completely uninstall and reinstall or manually change the setting back, but just putting this all in a separate application seemed like an easier way to go (for the users) in the long run. This way, if the user selects too much memory (which they might, because do average computer users really knows how much memory their computer has free? Probably not.), then they can always just run the memory setting application again and change Emailchemy’s memory back to the default setting.

Anyway, this is becoming a rather long-winded explanation of why the Emailchemy Memory Boost was needed. Ideally, most people shouldn’t need it, because it’s hard enough to explain why it’s needed at all.

So, in a nutshell, if you have really big email files to convert, and Emailchemy appears to be running slowly (say, only a message or two per second), or if Emailchemy stops completely when opening your giant file, try using the Emailchemy Memory Boost.

When I got out of college, most employers didn’t provide email, but mine did. It was considered somewhat of a perk and using it for personal correspondence was perfectly acceptable. As in college, I was once again saving all my personal email. Integrating it with my college email archives wasn’t a big deal since the company I worked for also used Eudora. But after a while, the big software companies discovered that professional email tools were a big market, and that is when the problems began.

The changes at work were based on corporate policy, but the changes at home were necessary to keep up with my email usage needs. In seemingly rapid succession, I went from using Eudora at work to using Apple’s PowerTalk, and from that to using WordPerfect Office (aka Groupwise), Microsoft Exchange, Lotus Notes, and finally Microsoft Outlook. Then, to further complicate matters, I went from using Eudora at home to using Apple’s PowerTalk, Claris Emailer, and Netscape Mail, back to Eudora again, and then finally Apple’s Mail.app that came with Mac OS X.

All of these new email applications that I had to use seemed to save email to the disk drive in their own special proprietary way. I can only guess that the big software companies created proprietary file formats for some kind of competitive advantage or product lock-in strategy, but regardless, I found myself in a situation where I had years of archived email saved in files created by several different applications that no other application could read.

Sure, some of the new email applications did allow me to import certain types of files, but they often didn’t do a very good job or they required the old email application to be installed. So, the majority of my email was still stranded and unreadable. With most new email applications, even if I could import my email there would be no way for me to export it back out again!

After much frustration, I realized that if I could at least “normalize” my data — that is, convert it all to the same format, it would be a step in the right direction. And if I could get everything into some kind of industry-standard format, any email application that comes along should be able to read it.

So, back in late 1996, I began writing Emailchemy to do exactly that, and ever since I’ve been building more converters and utilities, trying to build out Emailchemy into a powerful yet usable tool for everyone.

No more.

I tested this with RGE files created by Entourage itself as well as RGE files created by Emailchemy, and every time Entourage 2008 would open when double clicking the RGE file, but no import. I now have to import using Entourage’s Import Wizard, which works, but just isn’t as fast or convenient.

Hi,

Your application works and saves me a lot of time with settling the little matter of, I’m a “computer guy” who has to know how to use Entourage, Apple Mail, SeaMonkey mail, Thunderbird mail … etc. Of course, I have to be able to move large amounts of “the stack of stuff” between these e-mail clients. Emailchemy works.

Thank you.

Thank you.

Thank you,

Mike Fish
080425.0955 EDT Friday

(Power Mac G4 1.25GHz, 1 processor - Mac OS 10.4.11 with all of Apple’s software updates)