Category: Jessica’s Tech Blog

I had an email in my inbox today from Shutterfly with a subject in Base-64 encoding. Only, the subject was so long in Base-64 that it didn’t all fit on a single line in the email headers. Which means my base-64 subject decoder needs to be a little more robust.

The raw-header looks like this:

Subject: =?UTF-8?B?WW91ciBUaGFuayBZb3UgZ2lmdCBlbmRzIFdlZC4gfCBFbmpveSA=?=
=?UTF-8?B?JDIwIGp1c3QgZm9yIHlvdcKg?=

Notice anything odd about the format of this header? Each line has it’s own encoding tags, instead of one encoding tag spanning both lines, so it’s a little different than what I’ve seen on long subjects encoded with quoted-printable. Converted from Base 64 to English, the subject should parse into:

Your Thank You gift ends Wed. | Enjoy
$20 just for you

No weird special characters or anything that really necessitates base-64, but none the less to be a robust email parser, this kind of case should probably be handled.

In Delphi, I get a single string with the entire header except the word “Subject: “, but including the line break returned from Indy. This means I can’t just say from after the B? to the very end minus two characters, I need to actually tokenize the ending question-marks and/or whitespace, and encoding begin strings, perhaps in a loop, so I can pull out just the base 64 encoded part:

WW91ciBUaGFuayBZb3UgZ2lmdCBlbmRzIFdlZC4gfCBFbmpveSA=
JDIwIGp1c3QgZm9yIHlvdcKg


And then pass that lovely string into the base64 decoder. And then instead of a subject of

=?UTF-8?B?WW91ciBUaGFuayBZb3UgZ2lmdCBlbmRzIFdlZC4gfCBFbmpveSA=?= =?UTF-8?B?JDIwIGp1c3QgZm9yIHlvdcKg?=

you could have a subject of

Your Thank You gift ends Wed. | Enjoy $20 just for you

Two bugs fixed! The first was trivial, adding one line of code for a case I had overlooked initially. The second was more subtle. I eventually tracked down the cause to a less common variant in the MIME headers.

Here’s a typical MIME header:

Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

And here’s the header I was seeing in the error producing email:

Content-Disposition: inline
Content-Transfer-Encoding: 8bit
Content-Type: text/html

Notice anything majorly different? Take a look at that content-type. In the second-example, it didn’t specify the charset, so there’s no semicolon after the content type string. And that meant that some existing code that parsed out the content-type would fail if the charset was missing. Position of semi-colon returning zero (not found) when you want it to return end of string if not found…slight difference, now it gets the full content-type in the latter case instead of a blank string and displays correctly.

I decided to take a break from fixing up the subject line decoding algorithms (I think it could be more efficient and limitations that could be potentially eliminated) and see how difficult it would be to add HTML message preview to the application.

I started out with some online research about what sort of component could display HTML. The first thing I came across was a third party component that was eventually released into the public domain. It looked promising and easy to use, but I had a couple concerns. First, was overcoming an issue with installing components in the first place, probably a compatibility issue between Delphi 7 and Windows 7. Eventually I realized the solution was to delete the file it was giving an error about and then ignore the error message. Not obvious but it worked. But even at the point where I got to where I could install components, I had concerns about whether the component would meet my needs for rendering HTML emails. It looked in their demo app as if it was rendering all the HTML from scratch, in a very HTML 3.0 kind of way, with lots of emphasis on frames and displaying frame-sets…stuff that’s kind of obsolete these days. But being able to handle frames isn’t the problem. My concern was that, because the component appeared to be dated, it might not handle certain modern widely used HTML features such as stylesheets.

So then I decided to look into an alternative approach, to see whether there was a way to embed the native OS web-browser into the application to render the HTML. It did not take much research to determine that this was quite possible, and with that using only stock components included with Delphi.

Of course, it’s always easier said than done, right? First it turned out when I installed Delphi and skipped a bunch of stuff I didn’t anticipate needing (database and web stuff mostly) to speed up the install, I had not installed a piece I needed to use the TWebBrowser component. And then it turns out that even after you install it, you still have to select “Import ActiveX Control” from the Component menu and select the component to get it to appear on the Components toolbar.

And after I got that fixed, when I added it to the form it added an extra “uses” import to the project for some OleControls thing that I didn’t need, which had an extra feature of causing the entire application to crash immediately on debug. I commented out all the super-questionable new code I’d written, but it still crashed. So I commented out more changes in cycles until I got to the point of diffing the entire file and commenting out anything that wasn’t in the original when I discovered the suspect import (I’m guessing that “import” is probably not the proper Delphi term for it, but coming from a java background that’s how I mentally understand it). At that point it stopped crashing so I started un-commenting the sections of code one at a time to see where it broke down, and I was able to uncomment everything except my suspect import without issue, so I was able to move on.

After that, I had to learn how to add a tab to the tab control, which was pretty straightforward. Though I still have an outstanding issue to look into how translation of the tab names in handled and regression test translation of the tabs to make sure my new tab doesn’t cause the tabs to have the wrong names if the app is not being displayed in English. It would be easy enough to ignore that case for my needs since I speak English, but since I do hope to release my code improvements pubically, I need to think like a developer, and consider other users with use cases other than my own personal one, and not break features that would be important to the users of the application.

Displaying raw HTML in the component was not too difficult, though it relies on a snippet of code from the internet on how to “properly” load raw HTML directly into the browser. It’s a little complicated, but it worked right out of the box. My understanding is that the code chunk I used was based on directions in Microsoft’s knowledge-base about the proper way to display HTML that’s not in a file, starting with showing a blank page and then some other stuff including checking the browser’s ready state, and creating an “IPersistStreamInit” stream to stream the HTML to the page. Having worked on writing Internet Explorer Plugins before, you have to kind of expect there to be some ugly boilerplate code like that to connect to Internet Explorer with long interface names starting with I’s. I found some other similar code that was a bit more object-oriented that would become a wrapper class for the TWebBrowser to give it pretty functions like LoadHtmlFromString(…) instead of having a procedure do it yourself, but for 15 lines of code, using a pre-made wrapper class with a bunch of other pretty functions I don’t need and am unlikely to need in the future might be overkill.

Next roadblock? MIME. Lot of technical information about the Email spec that you have to worry about. You don’t just want to display the entire email in the browser window, because often the message is included in both plaintext and HTML formats, so you only want to display the HTML format part, so you don’t have the whole email twice. Luckily, since the application has options for saving attachments already, there was a place I could hook into to grab the MIME section with the HTML attachment. Unfortunately, it’s also not quite that simple and I still have some corner cases I need to fix.

With the test bank of emails of “whatever is in my inbox today” that I tested with, about 90% of them display correctly with the current code. It’s those corner cases though. At least, email is easy to get more of to test with ;-). There’s this one message that isn’t reading the MIME type correctly, but I’m slightly suspicious that the application isn’t reading the MIME type of that email correctly for other reasons unrelated to my new code. But I’ll still have to look into why it’s picking up the second MIME section as “text” instead of “html” and fix it to make the preview robust and work for as many emails as possible. The other message I’m having trouble with appears to not have a plaintext section, and everything is HTML, and I think it’s an edge case that’s falling through and not hitting the code that copies the HTML chunk into the preview window.

But all in all, I’m very excited that after only a couple hours of mucking around I have a preview window that can display HTML emails for about 90% of what’s in my inbox! That’s leaps and bounds of progress and makes the app even more powerful than it was before and will be a very useful additional feature for the application.

Over my Christmas “vacation” I decided I needed a project. Something to do during my son’s naps for my own enjoyment while visiting the in-laws. After a short amount of mulling it over, I decided to attempt to tackle a project I’d been idly thinking about for quite some time.

And when I say quite some time, I mean it. I actually started looking into solving this problem three years ago, and had come up with a proposal back then of how to fix it and posted that to the application’s forum site. But…the original developer of the application stopped updating the app in 2006, rather abruptly.

It was a little bit of a mystery why he stopped developing it, at the time, for a while there were new editions every few months, keeping up with the latest version of the programming environment, adding and improving features with a nice wishlist of changes for future releases, and then all of a sudden, it just stopped in it’s tracks. I never really knew why he stopped–on the forums I saw something about the application “met his needs sufficiently” so he decided to stop updating it. But that was three operating systems ago! And that was before e-mail was frequently sent with subjects encoded in UTF-8, that look almost illegible when displayed as plain-text.

I kind of understand why open-source development can just spontaneously stop, especially when there’s really only one guy doing everything. Maybe it starts out as side-project that just gets out of hand, and starts sapping your free time, until you no longer enjoy working on it. Maybe you get a new job, or your life circumstances change, and you no longer can devote the time. Maybe your computer breaks down or gets stolen, and you no longer have access to the development tools you need. Things happen. Its life.

But that doesn’t mean that one little nagging un-fixed bug in the application doesn’t keep nagging at you. Back in 2008, I took a look at the source-code to see whether I could find the cause of the bug. I had a pretty good hunch where you needed to fix the problem, and what needed to be done. But I didn’t take it a step farther and implement it at that time because I didn’t know the programming language, I didn’t have the programming suite I needed, and there were several steps in my outlined solution that relied on knowledge I didn’t know. There was a little bit of hand-waving as in, I’m not sure how you do this in Delphi, but in programming languages I’m familiar with it’s not too hard, so this should be doable, I think.

So at that time I proposed my solution outline on the forum, hoping someone else who might not want to spend as much time figuring out why it’s a bug, or even the original developer, could take my outline of a solution and implement it.

Three years go by. During that time I’d considered changing mail notifiers a couple times. I even downloaded just about every other free one I could find. But none of them were as good as the one I’d been using for the last ten years. Yes, some of them handled international characters much better, but I was quite content with the user-interface and functionality of the application I was using, and didn’t really want to get used to a different clunkier interface.

If in all this time, nobody else has taken the time to solve the problem and fix it, and nobody else has come out with a better alternative application, maybe it’s just time to bite the bullet, overcome the hurdles, and fix it myself.