Today we released the Emailchemy Memory Boost app, and I’m hoping it helps people deal with some of the gigantic email archives they’ve been writing to me about.

I originally designed Emailchemy to run in as little memory as possible, simply because at that time most computers had 32MB total of RAM. And, being that I was writing Emailchemy in Java, I had to be extra careful about how much heap space (“working memory”) Emailchemy required. This presented a specific requirement to not load an entire email message into memory for processing, because even back then, an email could be 10MB or larger with big attachments and most computers at the time would have choked under that weight. So, Emailchemy was designed to process the contents of an email in chunks, and it was able to run and convert any email in under 6MB of memory.

Even today when computers routinely ship with 1GB of RAM, I insist on sticking to highly efficient use of memory because it forces developers to write better code and it produces executables that can even run on lower-end, older computers. After all, not everybody buys a new laptop every 2 years.

Now, because Emailchemy is written in Java and therefore runs in a virtual machine, when the application starts it has to tell the virtual machine how much memory to grab from the host computer. The problem is that Java applications can’t see how much real memory the host computer has. So, we’ve intentionally kept the amount of memory that Emailchemy requests as low as possible — just high enough to work for most people’s needs. If we were to tell Emailchemy to grab too much memory, it would probably crash.

Even though Emailchemy processes the message content in chunks, for some formats it has to scan and build an index of all the little parts that make up a message in the original file, and that index has to be kept in memory throughout the conversion of all the messages.

So, we’ve bumped Emailchemy’s base memory footprint to 128MB just to accommodate these file formats. But, if a user has very large files, say bigger than a gigabyte or two, then Emailchemy really needs more memory to convert the email at the fastest rate possible. Then, if a user has extremely large files, say a 17GB PST file, Emailchemy really will need some more memory to even begin extracting and converting messages, because the index for such a file may take up the entirety of Emailchemy’s default memory allocation.

When people were writing in and asking for help with these super huge files, we tried to walk them through the process of editing a config file or starting Emailchemy from the command line to give it more memory. In many cases, the users were able to make the changes, but I can’t imagine trying to walk a marginally computer literate person through that exercise. So, it became very clear very fast that we needed to provide an easy way to change Emailchemy’s memory allocation.

But why a separate application? Why not build this into Emailchemy itself? Well, remember, Emailchemy can’t see how much memory the host computer has, so there was a risk that the user would choose a setting that asked for too much memory and thus prevent Emailchemy from launching again. If Emailchemy can’t start, then the user won’t be able to change the setting back, and Emailchemy wouldn’t be able to change it back itself. Sure, we could have posted instructions on how to completely uninstall and reinstall or manually change the setting back, but just putting this all in a separate application seemed like an easier way to go (for the users) in the long run. This way, if the user selects too much memory (which they might, because do average computer users really knows how much memory their computer has free? Probably not.), then they can always just run the memory setting application again and change Emailchemy’s memory back to the default setting.

Anyway, this is becoming a rather long-winded explanation of why the Emailchemy Memory Boost was needed. Ideally, most people shouldn’t need it, because it’s hard enough to explain why it’s needed at all.

So, in a nutshell, if you have really big email files to convert, and Emailchemy appears to be running slowly (say, only a message or two per second), or if Emailchemy stops completely when opening your giant file, try using the Emailchemy Memory Boost.

Just got a kind note from Mike, a Mac user, saying that Emailchemy was the only Mac-based method he could find to convert his Windows Outlook PST file to Mac OS X Mail. 1GB in 20 minutes, not bad at all.

Anyway, he wrote to also report another quirk in Mac OS X Mail’s import feature. I say “another” because I’ve already noted (and reported to Apple) a few others. The issue he reported was that when importing mbox folders in Mail 2.0 format (Mac Mail .mbox folders are not the same as standard mbox files), Mail will import all the folders but some folders that are deeper in the hierarchy have all their messages show up blank. Mike reported that importing mbox folders in the Mail 1.0 format worked fine, however.

The problem with the Mail import bugs (I’m not calling them quirks anymore) is that they are often hard to reproduce and there may only be a few other users who ever see this issue.

This is why I recommend using Emailchemy’s embedded IMAP server (the IMAP ImportServer tool) if you can. It bypasses the import wizard by providing you a desktop mail server that you download your converted mail from. It’s as simple as creating a new email account and downloading the mail.

Aside from the bugs in the import wizard, I like using Mac OS X Mail. It’s what I’ve been using for the past several years and I haven’t seen anything else I like on the Mac yet.

Today I noticed we had a few customers, Mac users using Mac OS X 10.5.1, who reported odd behavior until they installed the Mac OS X 10.5.2 update. 

So if you’re using the latest Emailchemy (v 9.6) and Mac OS X 10.5.1, and you are seeing things like missing file dialogs or conversions that seem to never finish, try installing the 10.5.2 update from Apple.One note on installing Mac OS X updates: I am a Mac user too, and I’ve noticed that with my Macs that I have increased system stability if I install the “combo” update instead of the “incremental” update that comes through the Software Update app.

I can’t explain why, but after I’ve re-installed the “combo” update (available from the Mac OS X Support downloads page) over the Software Update update, most “odd” system issues disappear.