Blog

Here’s an updated screenshot with my progress over the past few days:

You’ll notice now that the two emails with subjects in Hebrew and Greek (probably horribly translated, I just used Google Translate) now appear correctly in the main window.

Getting these messages to appear correctly was actually quite tricky to figure out. The key, in the end, was that I had to make a copy of the subject field before the Indy routine ProcessHeaders mangled it into Ansi characters, and then call my DecodeHeader algorithm on the un-mangled header.

The ironic thing is, if I upgraded to the latest version of Delphi, and the latest version of Indy (as in Indy10 rather than as in Indy9.0.53), the whole DecodeHeader and handling international characters would be a non-issue. But then I’d have a whole different can of worms–I’d have to fix all the non-backward compatible changes between Indy 9 and 10, I’d have to figure out how to convert ActionBand Popups into the new Delphi 2010 equivalent. In the end, what it would take to port the app to a newer version of Delphi is probably more difficult than coercing the old version of Delphi and Indy into doing what I want it to do. Sure, in the end it might end up being more robust that way, but I might also break and mangle lots of existing features.

At this point, what you see in the screenshot, I am only processing the Subject field with the new technique that works for any codepage (vs: only the current one or UTF-8), so I need to extrapolate my strategy to the other header fields that might be encoded. I also need to find an equivalent strategy to do the same thing on the Preview window. In the preview window it re-downloads the email through a different Indy code-path, and I haven’t found the right place in that code-path where I can capture the un-mangled header yet. So there’s still work to do but I’m on the right path.

To convert the random code-page to Unicode, I am using the windows library function WideCharToMultiByte, which converts a string to a “wide” (Unicode) string based on a specified code page number. Getting the code page number was also a little bit of a challenge. The library with that function doesn’t have a GetCodePageNumber function to convert the code page *name* to the windows code page *number*. There is a DLL that comes with windows that has (almost) that function, but figuring out how to call it is kind of tricky, and rumors on the internet say it might be buggy in certain cases. So, I’m using the straightforward but ugly strategy: convert the table on MSDN of allowed code-page names/IDs to a data structure and look it up manually. That list isn’t the full IANA supported list of encodings (aliases/alternate names), but the cases it doesn’t handle are likely to be rare, and could be added in the future if future research doesn’t find a better strategy.

I started out storing the list in a record (ie: Delphi equivalent of a struct) with a dead simple sequential search algorithm, per a tip on StackOverflow, but I wasn’t happy with the performance on lookup, because there’s 140 different encodings in my list of encodings so far, and this method is going to be called in a loop for each header of each email unless it’s definitely not encoded. So I did some research and found TStringList which can be used like a sorted map with better string search performance, and THashedStringList, which is basically a hash-map data structure, so even better string search performance–up to the level you need for, get this, INI FILES! Then I had to do more research to figure out what the Delphi equivalent of a static initializer is to use the Hashed String List…But now that I have it working, it does seem noticeably faster even on a small number of emails, but the internet could just be less congested today, it’s hard to say.

I had an email in my inbox today from Shutterfly with a subject in Base-64 encoding. Only, the subject was so long in Base-64 that it didn’t all fit on a single line in the email headers. Which means my base-64 subject decoder needs to be a little more robust.

The raw-header looks like this:

Subject: =?UTF-8?B?WW91ciBUaGFuayBZb3UgZ2lmdCBlbmRzIFdlZC4gfCBFbmpveSA=?=
=?UTF-8?B?JDIwIGp1c3QgZm9yIHlvdcKg?=

Notice anything odd about the format of this header? Each line has it’s own encoding tags, instead of one encoding tag spanning both lines, so it’s a little different than what I’ve seen on long subjects encoded with quoted-printable. Converted from Base 64 to English, the subject should parse into:

Your Thank You gift ends Wed. | Enjoy
$20 just for you

No weird special characters or anything that really necessitates base-64, but none the less to be a robust email parser, this kind of case should probably be handled.

In Delphi, I get a single string with the entire header except the word “Subject: “, but including the line break returned from Indy. This means I can’t just say from after the B? to the very end minus two characters, I need to actually tokenize the ending question-marks and/or whitespace, and encoding begin strings, perhaps in a loop, so I can pull out just the base 64 encoded part:

WW91ciBUaGFuayBZb3UgZ2lmdCBlbmRzIFdlZC4gfCBFbmpveSA=
JDIwIGp1c3QgZm9yIHlvdcKg


And then pass that lovely string into the base64 decoder. And then instead of a subject of

=?UTF-8?B?WW91ciBUaGFuayBZb3UgZ2lmdCBlbmRzIFdlZC4gfCBFbmpveSA=?= =?UTF-8?B?JDIwIGp1c3QgZm9yIHlvdcKg?=

you could have a subject of

Your Thank You gift ends Wed. | Enjoy $20 just for you

Two bugs fixed! The first was trivial, adding one line of code for a case I had overlooked initially. The second was more subtle. I eventually tracked down the cause to a less common variant in the MIME headers.

Here’s a typical MIME header:

Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

And here’s the header I was seeing in the error producing email:

Content-Disposition: inline
Content-Transfer-Encoding: 8bit
Content-Type: text/html

Notice anything majorly different? Take a look at that content-type. In the second-example, it didn’t specify the charset, so there’s no semicolon after the content type string. And that meant that some existing code that parsed out the content-type would fail if the charset was missing. Position of semi-colon returning zero (not found) when you want it to return end of string if not found…slight difference, now it gets the full content-type in the latter case instead of a blank string and displays correctly.

I decided to take a break from fixing up the subject line decoding algorithms (I think it could be more efficient and limitations that could be potentially eliminated) and see how difficult it would be to add HTML message preview to the application.

I started out with some online research about what sort of component could display HTML. The first thing I came across was a third party component that was eventually released into the public domain. It looked promising and easy to use, but I had a couple concerns. First, was overcoming an issue with installing components in the first place, probably a compatibility issue between Delphi 7 and Windows 7. Eventually I realized the solution was to delete the file it was giving an error about and then ignore the error message. Not obvious but it worked. But even at the point where I got to where I could install components, I had concerns about whether the component would meet my needs for rendering HTML emails. It looked in their demo app as if it was rendering all the HTML from scratch, in a very HTML 3.0 kind of way, with lots of emphasis on frames and displaying frame-sets…stuff that’s kind of obsolete these days. But being able to handle frames isn’t the problem. My concern was that, because the component appeared to be dated, it might not handle certain modern widely used HTML features such as stylesheets.

So then I decided to look into an alternative approach, to see whether there was a way to embed the native OS web-browser into the application to render the HTML. It did not take much research to determine that this was quite possible, and with that using only stock components included with Delphi.

Of course, it’s always easier said than done, right? First it turned out when I installed Delphi and skipped a bunch of stuff I didn’t anticipate needing (database and web stuff mostly) to speed up the install, I had not installed a piece I needed to use the TWebBrowser component. And then it turns out that even after you install it, you still have to select “Import ActiveX Control” from the Component menu and select the component to get it to appear on the Components toolbar.

And after I got that fixed, when I added it to the form it added an extra “uses” import to the project for some OleControls thing that I didn’t need, which had an extra feature of causing the entire application to crash immediately on debug. I commented out all the super-questionable new code I’d written, but it still crashed. So I commented out more changes in cycles until I got to the point of diffing the entire file and commenting out anything that wasn’t in the original when I discovered the suspect import (I’m guessing that “import” is probably not the proper Delphi term for it, but coming from a java background that’s how I mentally understand it). At that point it stopped crashing so I started un-commenting the sections of code one at a time to see where it broke down, and I was able to uncomment everything except my suspect import without issue, so I was able to move on.

After that, I had to learn how to add a tab to the tab control, which was pretty straightforward. Though I still have an outstanding issue to look into how translation of the tab names in handled and regression test translation of the tabs to make sure my new tab doesn’t cause the tabs to have the wrong names if the app is not being displayed in English. It would be easy enough to ignore that case for my needs since I speak English, but since I do hope to release my code improvements pubically, I need to think like a developer, and consider other users with use cases other than my own personal one, and not break features that would be important to the users of the application.

Displaying raw HTML in the component was not too difficult, though it relies on a snippet of code from the internet on how to “properly” load raw HTML directly into the browser. It’s a little complicated, but it worked right out of the box. My understanding is that the code chunk I used was based on directions in Microsoft’s knowledge-base about the proper way to display HTML that’s not in a file, starting with showing a blank page and then some other stuff including checking the browser’s ready state, and creating an “IPersistStreamInit” stream to stream the HTML to the page. Having worked on writing Internet Explorer Plugins before, you have to kind of expect there to be some ugly boilerplate code like that to connect to Internet Explorer with long interface names starting with I’s. I found some other similar code that was a bit more object-oriented that would become a wrapper class for the TWebBrowser to give it pretty functions like LoadHtmlFromString(…) instead of having a procedure do it yourself, but for 15 lines of code, using a pre-made wrapper class with a bunch of other pretty functions I don’t need and am unlikely to need in the future might be overkill.

Next roadblock? MIME. Lot of technical information about the Email spec that you have to worry about. You don’t just want to display the entire email in the browser window, because often the message is included in both plaintext and HTML formats, so you only want to display the HTML format part, so you don’t have the whole email twice. Luckily, since the application has options for saving attachments already, there was a place I could hook into to grab the MIME section with the HTML attachment. Unfortunately, it’s also not quite that simple and I still have some corner cases I need to fix.

With the test bank of emails of “whatever is in my inbox today” that I tested with, about 90% of them display correctly with the current code. It’s those corner cases though. At least, email is easy to get more of to test with ;-). There’s this one message that isn’t reading the MIME type correctly, but I’m slightly suspicious that the application isn’t reading the MIME type of that email correctly for other reasons unrelated to my new code. But I’ll still have to look into why it’s picking up the second MIME section as “text” instead of “html” and fix it to make the preview robust and work for as many emails as possible. The other message I’m having trouble with appears to not have a plaintext section, and everything is HTML, and I think it’s an edge case that’s falling through and not hitting the code that copies the HTML chunk into the preview window.

But all in all, I’m very excited that after only a couple hours of mucking around I have a preview window that can display HTML emails for about 90% of what’s in my inbox! That’s leaps and bounds of progress and makes the app even more powerful than it was before and will be a very useful additional feature for the application.