Tag: poptray

So to be a little more detailed from my last post about how upgrading Indy affects the protocol plugins for PopTrayU…

PopTray, historically, only included POP3 as a “core” protocol. Anything else (Pop3 with SSL, Pop3 with APOP, IMAP, IMAP with SSL, etc) had to be configured as a seperate plugin. There’s a DLL interface that the plugin has to implement (though many of the methods are optional), which can be implemented in either Delphi or C++.

PopTray had a set of “example” plugins that included the most common additional protocol needs (those mentioned above), implemented in Delphi.

The POP3 SSL plugin for PopTray (a seperate DLL) directly extended pop3.pas (one of the source files for the core executable). That class has a reference to an Indy callback object that updates the a progress meteron the UI. In Indy 10, one of the parameters was changed from Int to Int64 and changed whether it was const. So the older DLL + the newer EXE (or conversely new DLL + older EXE) just don’t quite mix well and initial testing was showing “random” crashes and such undesirable behavior.

Well, if we’re going to have to break the existing plugins for SSL and IMAP, now would be the time to make any other changes/additions to the plugin interface, so we don’t have too many different versions of the plugin interface, wouldn’t it?

One change I’ve been contemplating for some time is including IMAP and SSL as “core” features rather than plugins. It’s becoming increasingly common for mail providers (Gmail, for example) to require SSL to even get POP access to an email account.

Another change I’d been contamplating is offering an “advanced connection options” feature. Almost every email client (even the one on my phone) has a menu or dialog where you can tinker with some various cryptic sounding settings beyond “SSL/TLS enabled”, like STARTTLS, Explicit TLS, SASL, APOP, etc. In PopTray, you had to have a lengthy list of protocols to have a seperate protocol for every permutation of feature combinations you needed, and not only does that lead to having to make the protocol support only the least common denominator of desirable settings, it made configuring options on a per account basis difficult. The plugins were set up so there was a configuration page for the plugin itself (which to be honest I’ve never seen actually used for more than an “about” dialog), but no way to configure settings differently between two different accounts that both require SSL connections, even if I put the plugin settings dialog to better use.

So I spent a bit of time researching what all these crazy-sounding advanced settings acronyms mean that other mail clients use, and then tried to create a list of which ones would be useful to support. One in particular that I was initially undecided about was APOP. Even the iPhone’s default mail app doesn’t have APOP support. In the days of SSL, does anyone really use APOP instead (APOP just uses an MD5 hash to obscure the password on a plain-text connection, which is obviously better than sending any passwords in plaintext over public wifi where there might be hackers looking for your passwords, but far less secure than SSL. But some research seemed to indicate that there is at least one major ISP (Earthlink) that, as best I can tell, flat out does not support SSL even in this day and age, but does support APOP. Well, for the sake of anyone who unfortunate enough to have such an account, I’d rather not deprecate increased security that was already available in previous versions of PopTray and PopTrayU.

And then after deciding which advanced options I’d like to support, the more difficult decision becomes how to group and arrange and label them on the UI. Should I label a checkbox “STARTTLS” or “Explicit TLS” or put slashes between alternate version of the name? They mean the same thing, but different apps label it differently. That particular option opens the connection over the standard insecure port (110 for POP) and then sends the server a command to “explicitly” enter encrypted connection mode, rather than requiring you to use a different port if you want a secure connection. Apparently, there’s some debate about which way is the better way to do things, but if you want to be flexible and accomodate the widest assortment of mail server configurations, ideally you’d support both options. and just to make things confusing, there’s also a mode called Implicit TLS. That means that you still use a TLS secure connection, but the connection doesn’t make an insecure “handshake” first, you connect over the secure port and it’s just assumed you’re going to be in secure mode because you connected on the secure port.

Anyway, I don’t want to bore you with all the details of every single email connection option I ended up learning about and what all those options mean exactly…for most users, they just want to check the box their ISP or geek friend/adviser told them to use, and not know a thing about what it means 😉

But it suffices to say, there’s no one standard UI for setting all those advanced settings. They have different labels and groupings depending on the app and which features it supports. So one of the hardest aspects of addding all these extra advanced connection options is deciding on a usable interface for setting the options, not to mention how the plugins can interact with the options available to enable or disable options that protocol doesn’t support. So I’m taking my time with this and doing a lot of testing, and well, some of this secure stuff is tricky, and I’m still working on some of the kinks that happen if you mix and match combinations of features that just aren’t compatible and shouldn’t be available at the same time.

One of the features in particular that I’m excited to add support for is SASL authentication. SASL wasn’t even invented until 2006, so obviously, it was after PopTray’s time, and still new enough that everyone doesn’t automatically expect SASL support, let alone know what it does. The layman’s version is you don’t need to know a thing about how it works, you just need to know that that SASL + SSL can be more secure than SSL alone, so you should absolutely be using it if it’s an option available. Internally, there’s a whole bunch of different “authentication mechanisms” the app can choose from, and it will send your password using the most secure one of those mechanisms that the server supports. So it’s going to be one of those “enabled by default” kind of features that will make newer version more secure without having to be an IT professional to know what settings to pick.

The one thing I’m probably not going to do is include OpenSSL DLL’s in the base installer for PopTrayU, simply because, it’s a liability to bundle SSL abilities. The US and many other countries have all kinds of weird rules about importing and exporting software that include “strong” encryption. So some people may want the “insecure only” version, to avoid the whole issue of whether it’s legal to travel to xyz country with this software or it’s installer loaded on their PC.

Overall, I am attempting to minimizing changes to the plugin interface, and add the new features as optional methods, in hopes of breaking as few existing plugins as possible. There are poptray plugins for various webmail interfaces that I have no interest in supporting or developing, but if they continue to work as-is, I’d rather leave things that way.

Here’s an updated screenshot with my progress over the past few days:

You’ll notice now that the two emails with subjects in Hebrew and Greek (probably horribly translated, I just used Google Translate) now appear correctly in the main window.

Getting these messages to appear correctly was actually quite tricky to figure out. The key, in the end, was that I had to make a copy of the subject field before the Indy routine ProcessHeaders mangled it into Ansi characters, and then call my DecodeHeader algorithm on the un-mangled header.

The ironic thing is, if I upgraded to the latest version of Delphi, and the latest version of Indy (as in Indy10 rather than as in Indy9.0.53), the whole DecodeHeader and handling international characters would be a non-issue. But then I’d have a whole different can of worms–I’d have to fix all the non-backward compatible changes between Indy 9 and 10, I’d have to figure out how to convert ActionBand Popups into the new Delphi 2010 equivalent. In the end, what it would take to port the app to a newer version of Delphi is probably more difficult than coercing the old version of Delphi and Indy into doing what I want it to do. Sure, in the end it might end up being more robust that way, but I might also break and mangle lots of existing features.

At this point, what you see in the screenshot, I am only processing the Subject field with the new technique that works for any codepage (vs: only the current one or UTF-8), so I need to extrapolate my strategy to the other header fields that might be encoded. I also need to find an equivalent strategy to do the same thing on the Preview window. In the preview window it re-downloads the email through a different Indy code-path, and I haven’t found the right place in that code-path where I can capture the un-mangled header yet. So there’s still work to do but I’m on the right path.

To convert the random code-page to Unicode, I am using the windows library function WideCharToMultiByte, which converts a string to a “wide” (Unicode) string based on a specified code page number. Getting the code page number was also a little bit of a challenge. The library with that function doesn’t have a GetCodePageNumber function to convert the code page *name* to the windows code page *number*. There is a DLL that comes with windows that has (almost) that function, but figuring out how to call it is kind of tricky, and rumors on the internet say it might be buggy in certain cases. So, I’m using the straightforward but ugly strategy: convert the table on MSDN of allowed code-page names/IDs to a data structure and look it up manually. That list isn’t the full IANA supported list of encodings (aliases/alternate names), but the cases it doesn’t handle are likely to be rare, and could be added in the future if future research doesn’t find a better strategy.

I started out storing the list in a record (ie: Delphi equivalent of a struct) with a dead simple sequential search algorithm, per a tip on StackOverflow, but I wasn’t happy with the performance on lookup, because there’s 140 different encodings in my list of encodings so far, and this method is going to be called in a loop for each header of each email unless it’s definitely not encoded. So I did some research and found TStringList which can be used like a sorted map with better string search performance, and THashedStringList, which is basically a hash-map data structure, so even better string search performance–up to the level you need for, get this, INI FILES! Then I had to do more research to figure out what the Delphi equivalent of a static initializer is to use the Hashed String List…But now that I have it working, it does seem noticeably faster even on a small number of emails, but the internet could just be less congested today, it’s hard to say.

I had an email in my inbox today from Shutterfly with a subject in Base-64 encoding. Only, the subject was so long in Base-64 that it didn’t all fit on a single line in the email headers. Which means my base-64 subject decoder needs to be a little more robust.

The raw-header looks like this:

Subject: =?UTF-8?B?WW91ciBUaGFuayBZb3UgZ2lmdCBlbmRzIFdlZC4gfCBFbmpveSA=?=
=?UTF-8?B?JDIwIGp1c3QgZm9yIHlvdcKg?=

Notice anything odd about the format of this header? Each line has it’s own encoding tags, instead of one encoding tag spanning both lines, so it’s a little different than what I’ve seen on long subjects encoded with quoted-printable. Converted from Base 64 to English, the subject should parse into:

Your Thank You gift ends Wed. | Enjoy
$20 just for you

No weird special characters or anything that really necessitates base-64, but none the less to be a robust email parser, this kind of case should probably be handled.

In Delphi, I get a single string with the entire header except the word “Subject: “, but including the line break returned from Indy. This means I can’t just say from after the B? to the very end minus two characters, I need to actually tokenize the ending question-marks and/or whitespace, and encoding begin strings, perhaps in a loop, so I can pull out just the base 64 encoded part:

WW91ciBUaGFuayBZb3UgZ2lmdCBlbmRzIFdlZC4gfCBFbmpveSA=
JDIwIGp1c3QgZm9yIHlvdcKg


And then pass that lovely string into the base64 decoder. And then instead of a subject of

=?UTF-8?B?WW91ciBUaGFuayBZb3UgZ2lmdCBlbmRzIFdlZC4gfCBFbmpveSA=?= =?UTF-8?B?JDIwIGp1c3QgZm9yIHlvdcKg?=

you could have a subject of

Your Thank You gift ends Wed. | Enjoy $20 just for you

Two bugs fixed! The first was trivial, adding one line of code for a case I had overlooked initially. The second was more subtle. I eventually tracked down the cause to a less common variant in the MIME headers.

Here’s a typical MIME header:

Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

And here’s the header I was seeing in the error producing email:

Content-Disposition: inline
Content-Transfer-Encoding: 8bit
Content-Type: text/html

Notice anything majorly different? Take a look at that content-type. In the second-example, it didn’t specify the charset, so there’s no semicolon after the content type string. And that meant that some existing code that parsed out the content-type would fail if the charset was missing. Position of semi-colon returning zero (not found) when you want it to return end of string if not found…slight difference, now it gets the full content-type in the latter case instead of a blank string and displays correctly.

I decided to take a break from fixing up the subject line decoding algorithms (I think it could be more efficient and limitations that could be potentially eliminated) and see how difficult it would be to add HTML message preview to the application.

I started out with some online research about what sort of component could display HTML. The first thing I came across was a third party component that was eventually released into the public domain. It looked promising and easy to use, but I had a couple concerns. First, was overcoming an issue with installing components in the first place, probably a compatibility issue between Delphi 7 and Windows 7. Eventually I realized the solution was to delete the file it was giving an error about and then ignore the error message. Not obvious but it worked. But even at the point where I got to where I could install components, I had concerns about whether the component would meet my needs for rendering HTML emails. It looked in their demo app as if it was rendering all the HTML from scratch, in a very HTML 3.0 kind of way, with lots of emphasis on frames and displaying frame-sets…stuff that’s kind of obsolete these days. But being able to handle frames isn’t the problem. My concern was that, because the component appeared to be dated, it might not handle certain modern widely used HTML features such as stylesheets.

So then I decided to look into an alternative approach, to see whether there was a way to embed the native OS web-browser into the application to render the HTML. It did not take much research to determine that this was quite possible, and with that using only stock components included with Delphi.

Of course, it’s always easier said than done, right? First it turned out when I installed Delphi and skipped a bunch of stuff I didn’t anticipate needing (database and web stuff mostly) to speed up the install, I had not installed a piece I needed to use the TWebBrowser component. And then it turns out that even after you install it, you still have to select “Import ActiveX Control” from the Component menu and select the component to get it to appear on the Components toolbar.

And after I got that fixed, when I added it to the form it added an extra “uses” import to the project for some OleControls thing that I didn’t need, which had an extra feature of causing the entire application to crash immediately on debug. I commented out all the super-questionable new code I’d written, but it still crashed. So I commented out more changes in cycles until I got to the point of diffing the entire file and commenting out anything that wasn’t in the original when I discovered the suspect import (I’m guessing that “import” is probably not the proper Delphi term for it, but coming from a java background that’s how I mentally understand it). At that point it stopped crashing so I started un-commenting the sections of code one at a time to see where it broke down, and I was able to uncomment everything except my suspect import without issue, so I was able to move on.

After that, I had to learn how to add a tab to the tab control, which was pretty straightforward. Though I still have an outstanding issue to look into how translation of the tab names in handled and regression test translation of the tabs to make sure my new tab doesn’t cause the tabs to have the wrong names if the app is not being displayed in English. It would be easy enough to ignore that case for my needs since I speak English, but since I do hope to release my code improvements pubically, I need to think like a developer, and consider other users with use cases other than my own personal one, and not break features that would be important to the users of the application.

Displaying raw HTML in the component was not too difficult, though it relies on a snippet of code from the internet on how to “properly” load raw HTML directly into the browser. It’s a little complicated, but it worked right out of the box. My understanding is that the code chunk I used was based on directions in Microsoft’s knowledge-base about the proper way to display HTML that’s not in a file, starting with showing a blank page and then some other stuff including checking the browser’s ready state, and creating an “IPersistStreamInit” stream to stream the HTML to the page. Having worked on writing Internet Explorer Plugins before, you have to kind of expect there to be some ugly boilerplate code like that to connect to Internet Explorer with long interface names starting with I’s. I found some other similar code that was a bit more object-oriented that would become a wrapper class for the TWebBrowser to give it pretty functions like LoadHtmlFromString(…) instead of having a procedure do it yourself, but for 15 lines of code, using a pre-made wrapper class with a bunch of other pretty functions I don’t need and am unlikely to need in the future might be overkill.

Next roadblock? MIME. Lot of technical information about the Email spec that you have to worry about. You don’t just want to display the entire email in the browser window, because often the message is included in both plaintext and HTML formats, so you only want to display the HTML format part, so you don’t have the whole email twice. Luckily, since the application has options for saving attachments already, there was a place I could hook into to grab the MIME section with the HTML attachment. Unfortunately, it’s also not quite that simple and I still have some corner cases I need to fix.

With the test bank of emails of “whatever is in my inbox today” that I tested with, about 90% of them display correctly with the current code. It’s those corner cases though. At least, email is easy to get more of to test with ;-). There’s this one message that isn’t reading the MIME type correctly, but I’m slightly suspicious that the application isn’t reading the MIME type of that email correctly for other reasons unrelated to my new code. But I’ll still have to look into why it’s picking up the second MIME section as “text” instead of “html” and fix it to make the preview robust and work for as many emails as possible. The other message I’m having trouble with appears to not have a plaintext section, and everything is HTML, and I think it’s an edge case that’s falling through and not hitting the code that copies the HTML chunk into the preview window.

But all in all, I’m very excited that after only a couple hours of mucking around I have a preview window that can display HTML emails for about 90% of what’s in my inbox! That’s leaps and bounds of progress and makes the app even more powerful than it was before and will be a very useful additional feature for the application.

Over my Christmas “vacation” I decided I needed a project. Something to do during my son’s naps for my own enjoyment while visiting the in-laws. After a short amount of mulling it over, I decided to attempt to tackle a project I’d been idly thinking about for quite some time.

And when I say quite some time, I mean it. I actually started looking into solving this problem three years ago, and had come up with a proposal back then of how to fix it and posted that to the application’s forum site. But…the original developer of the application stopped updating the app in 2006, rather abruptly.

It was a little bit of a mystery why he stopped developing it, at the time, for a while there were new editions every few months, keeping up with the latest version of the programming environment, adding and improving features with a nice wishlist of changes for future releases, and then all of a sudden, it just stopped in it’s tracks. I never really knew why he stopped–on the forums I saw something about the application “met his needs sufficiently” so he decided to stop updating it. But that was three operating systems ago! And that was before e-mail was frequently sent with subjects encoded in UTF-8, that look almost illegible when displayed as plain-text.

I kind of understand why open-source development can just spontaneously stop, especially when there’s really only one guy doing everything. Maybe it starts out as side-project that just gets out of hand, and starts sapping your free time, until you no longer enjoy working on it. Maybe you get a new job, or your life circumstances change, and you no longer can devote the time. Maybe your computer breaks down or gets stolen, and you no longer have access to the development tools you need. Things happen. Its life.

But that doesn’t mean that one little nagging un-fixed bug in the application doesn’t keep nagging at you. Back in 2008, I took a look at the source-code to see whether I could find the cause of the bug. I had a pretty good hunch where you needed to fix the problem, and what needed to be done. But I didn’t take it a step farther and implement it at that time because I didn’t know the programming language, I didn’t have the programming suite I needed, and there were several steps in my outlined solution that relied on knowledge I didn’t know. There was a little bit of hand-waving as in, I’m not sure how you do this in Delphi, but in programming languages I’m familiar with it’s not too hard, so this should be doable, I think.

So at that time I proposed my solution outline on the forum, hoping someone else who might not want to spend as much time figuring out why it’s a bug, or even the original developer, could take my outline of a solution and implement it.

Three years go by. During that time I’d considered changing mail notifiers a couple times. I even downloaded just about every other free one I could find. But none of them were as good as the one I’d been using for the last ten years. Yes, some of them handled international characters much better, but I was quite content with the user-interface and functionality of the application I was using, and didn’t really want to get used to a different clunkier interface.

If in all this time, nobody else has taken the time to solve the problem and fix it, and nobody else has come out with a better alternative application, maybe it’s just time to bite the bullet, overcome the hurdles, and fix it myself.