Category: PopTrayU

There was a user requested feature for PopTrayU to add filtering based on read/unread status. This is something that is only technically feasible for IMAP since POP servers don’t store read/unread status. Anyway, I started working on implementing this feature, and decided that if you are only showing unread messages, you might need a mechanism to change the read/unread status–especially if previewing automatically marks the message read on the server.

So whip up some code for this…go to test it…and…nothing. No change to the message status.  Turn on some logging of server communication, and something looks off about the command set to the server…and the server response? Yeah, pretty sure that’s not what’s supposed to happen either.

Sent: C4 UID STORE 1264  (\Seen)<EOL>
Recv: C4 BAD Error in IMAP command UID STORE: Invalid STORE modifiers.<EOL>

This may not be so apparent in a non-monospaced font, but there’s a double space between 1264 and (\Seen) like a token got left out. But not being intimately aquainted with every nuance of the IMAP spec, I had to lookup the section on setting flags in the IMAP RFC for IMAP to be sure I even knew what the command was supposed to look like. It’s supposed to have a +FLAGS.SILENT in that missing token.

So, long story short, I had to update my copy of Indy from SVN to make sure the bug still existed on top of trunk, then I had to debug a variable type that disappeared in the new version of indy (it got replaced with something else), after retesting and trying a couple variants to see if it only exhibited this behavior when certain parameters were used, I filed a bug report to let the Indy team know about this issue, and left my code using a workaround with the better code commented out in the meantime. Remy and the Indy team are pretty responsive to fixing actual bugs, especially ones that are probably pretty trivial to fix like this (now incredibly useful but difficult to implement enhancements on the other hand, like IMAP IDLE I’m not about to hold my breath over…).

Aside from this particular sub-feature, there’s still quite a bit that needs to be done to make “unread only” mode happen. Like, the main window needs a whole new paradigm of basing the read/unread status off of the server for IMAP and off internal tracking for POP. And the message list needs to be using UIDs instead of relative message numbers for a bunch of different commands including but not limited to preview and delete.

And while I’m at changing it to download by UIDs, this would be a good time to evaluate whether it should be downloading only the “message envelope” (a limited set of headers that basically are the headers you see in your email client message list) instead of the “full headers” (including all the headers you usually don’t read like spam checker versions and hosts that redirected the email and so on that could be useful for rules) the way it does now. If you’re not making rules that use the extended headers, this could cut data transfer in about half, which could be a significant time saver on large inboxes, so it’s worth looking into, even though I’d probably still have to also support having the existing modes to support users who use those headers to create complex rules. And while I’m doing that…and so on and so forth it goes 😉

I wanted to give you all a little preview of more of the new features I’ve been working on for the 5.1 version of PopTrayU.

To start here’s a screen-shot of a new panel/category on the Options tab to customize the Preview Window:

Lots of subtle differences. Before we get too far, I’m sure someone is thinking “What’s with the font?” No, 12 point Segoe Print is not a new default (it’s something I picked arbitrarily for testing). But it does illustrate a new feature–large fonts support. 12 point isn’t exactly “that large”, but it’s large enough that things would be overlapping and cut off, and created a horrible user experience for for users who use windows large fonts feature, or just wanted a larger font in this application.

I’ve painstakingly gone through just about every screen in the application and added resizing code to make just about every screen “resize correctly” in both the vertical and horizontal directions if you use larger fonts. As an added bonus, this actually improves the user experience for users of languages other than English as well, because buttons and label areas that used to be fixed size now will resize intelligently, and increase in size, if the translated caption doesn’t fit on default sized button. At first I tried to do this with a grid layout manager, because that would have been the elegant, clean way to do things, but after wrestling with some frustrating bugs in how it auto-sizes, it ended up being more reliable to just position everything manually. So it’s quite possible somewhere I overlooked *something*, but we’ll call that a bug for now, and feedback on resizing issues that need to be addressed will be welcome.

Now, let’s talk about the list of tabs. I gave a minor overhaul to the list of categories, and made some changes to the order of the categories. Defaults has been moved to the first category, because it holds the option to change the language. And as a french-speaking user pointed out, you need to change the language before you change any other setting on a new installation. Seems sensible, so language selection is now at the very top of the first options screen. There are also two entirely new categories of options: Preview Window, and Rules.

The Rules options category gathers together existing options relating to rules that were previously spread across several tabs. Options like whether to log rules, whether messages marked as protected by rules require an extra confirmation to delete, whether to download part or all of the message body to make more sophisticated rules, and what happens when you blacklist something are all gathered in one place under a single logical category.

You may have also noticed that the old “Black List mark as Spam” checkbox has been replaced with the new “Blacklist Action” drop-down. I didn’t feel like it was intuitive what happens if you **don’t** mark blacklisted messages as spam–and the alternative involves things getting deleted. I take emails getting deleted seriously, and don’t want emails deleted because of bugs or because some configuration option was vague or unclear. So the drop down now forces an explicit choice between “Mark as Spam” or “Delete”. And I always do lots of extra testing (including on my own personal email) before releasing any changes that affect message deletion. I don’t want PopTrayU deleting your important mail–unless you really meant to do so.

The other new category in the list is Preview Window. There’s a couple of new features here, but previously, most of these options you had to change from a right-click menu on an actual preview window.

So, going back to that first screen-shot, you’ll see a new Tabs sub-category. This is for a new feature I’ve been wanting for some time. I’ve been torn over the design decision that PopTrayU should preview using the last selected tab or view type. I kept finding myself wanting PopTrayU to behave more like Outlook, where it automatically displays the email in HTML if the email has an HTML section, regardless of what view I was using to preview the previous email. But sine a lot of different people use PopTrayU in completely different ways to accomplish completely different goals, I didn’t want to eliminate the current option to have it display in the last used tab. So now you can choose between previewing with the last selected tab, or to always use a specific tab by default.

And, since Spam is sort of a special case, it gets its own independent selection. The main reason for this is that it’s likely that if you have images enabled, you may not want to display the email with images if it’s spam. So now you can choose whether you’d rather view spam as Plain Text, RAW headers, or any of the options supported for non-spam emails.

Back to the preview tab from the first screen shot. What can be customized for preview but isn’t on this tab? Fonts and Colors. When I first created this tab, initially I had put the options to customize the preview window font and colors (does not apply to HTML messages) on this new panel, however, I decided it would make more sense to have all of the font face, font size, and color customizations on the Visual appearance tab in one place.

And while I was at it, I finally fixed that eyesore that was the fonts selection panel. Originally I’d had the colors for the message list as separate drop-downs, which minimized the number of mouse-clicks needed to customize everything. But I decided it was getting a little too cluttered,  and confusing since only certain UI items can have the colors customized. I changed the previews to have borders–this was actually something I had wanted to do before, but there were some technical hurdles that required research to figure out how to make that happen.

For the upcoming 5.1 release of PopTrayU, I’ve redesigned the accounts screen:

In the old design, which was basically the PopTray 3.2 layout with new features squished in around the edges wherever they’d fit without changing things too much, things didn’t resize well. Look what happens if you increase the font size:

Everything is overlapping, some of the captions don’t fit. It’s a bit…cluttered. And if you use a language other than English, the translations often had to be abbreviated to fit the space available.

The new design manually resizes and positions everything in both horizontal and vertical directions to accommodate the selected font size and language settings. The account settings are broken into logical groups, like splitting apart the server information for your server from the ways PopTrayU notifies you about, to make it the page easier to navigate. The sections are each collapsible as well, so the advanced options or notification options can be hidden if you don’t want to see those sections. Oh, and those buttons at the bottom? They are now intelligently sized…if the caption doesn’t fit, they will be resized larger to fit the caption. Users of languages other than English will also especially appreciate this new feature.

So in a slight twist of irony since my last post about working on PopTrayU, after all the trouble of getting Indy 9 set up on my current machine, I’ve decided to port PopTrayU from Indy 9 to Indy 10.

For those of you unfamiliar with Indy (which is probably pretty much everyone who is not an experienced Delphi developer), Indy is an open source networking library that is available both for .NET and Delphi platforms. It does most of the heavy lifting as far as the technical implementation of connecting to a POP3 server, downloading and parsing the email headers (though I had to override a few little pieces of that to workaround Delphi 7 international character support issues) and that sort of stuff.

Version 9 is, well, a legacy version at this point. You can’t escape being asked “why are you still using Indy 9 anyhow?” if you ask a question pertaining to it on a programming forum. Version 10 has a lot of new features, and performance improvements and so forth. But, they made a lot of changes to the interfaces, so it’s not like you can just download version 10, and your code for version 9 will just work without changes.

I’d already done the vast majority of the porting to version 10 legwork when I was working on my Lazarus port, so there were only a few holes (“todos”) here and there that needed to be fixed. It was probably more painful to actually do enough research/reading to even figure out how to get versions 9 and 10 “somewhat” concurrently installed in delphi (to the degree that’s possible), than it was to makde PopTrayU built on Indy 10 itself work.

But I’ve been holding off on releasing these changes as the next version, because aside from wanting to make sure it’s stable and not buggy, it introduces the first change to the plugin interface since branching from PopTray that will make certain older plugins no longer compatible. I’ll write more about those changes in my next post 😉

I recently finished implementing a new feature for PopTrayU, conditionally disabling automatic checking for an email account based on the time of day. This feature already existed on a global level: you could disable checking for all accounts, so that your computer would not check for email while you’re sleeping or awaken you with a loud “you have new email” notification sound. But some users, particularly users who work from home and use the same computer for both work and pleasure, were making a feature request that they’d like to be able to turn off notifications for their work email outside of normal working hours.

The part of the implementation that was most difficult, of course, was the requirements analysis and the detailed design. On the surface, this feature seemed pretty straight-forward, but as I started to dive into actually implementing it, a lot of nuances come up where it isn’t exactly clear how this feature should work, and the design has to be hashed out in more detail, and the requirements clarified.

Each account has a status icon indicating whether the account is disabled or has new email or no new email. (For the sake of clarify, lets call this new per-account no-auto-check feature “sleeping”.) Should the visual appearance of an account that is “sleeping” appear like a normal account or a disabled account? Something else? What should the status-bar for the account say when an auto-check happens, but a single account is skipped because of the time? If the user presses “Check All Accounts” manually, should that skip or include accounts that are sleeping? What if they press “Check” for a single “sleeping” account?

Once I nailed down the design, implementation was pretty simple. Because the feature existed on a global level, most of the implementation was quite similar to the logic and UI of how it was done for the global level.

As far as how I decided to resolve some of those issues:

An account that is disabled is represented by an envelope with a red minus symbol on it, like this:  Previously, this was the only reason a specific account would be skipped when an auto-check occurred. Now we have two reasons an account might be skipped. It could, for some users, be handy to be able to differentiate the reason the account is not automatically checking, especially should they be troubleshooting settings gone awry. I experimented with making a new symbol that resembled sleeping, like something with zzz’s or a moon, but given the limited amount of space in a 16×16 image, trying to make an image that is clear and recognizable is difficult, so I decided to use the same symbol as a disabled account, but use a contrastingly different color to differentiate the different status.  

As far as when to check/not-check:

• Automatic timer-interval checking, pretty straightforward, sleeping accounts should be skipped.
• Manually pressing the “check all” button: more ambiguous, it’s not really an “auto-check”, but it behaves a lot like one. In the end I decided “check all” should replicate the behavior of auto-check, but on-demand, so accounts that are not in a state allowing automatic checking should be excluded, so it should behave like a disabled account.
• Manually pressing the “Check” (this account) button: this one seemed pretty straightforward, even a disabled account will check when pressing this button, so a sleeping account should definitely check in this case.

Well, I’ve gotten a lot of the “cosmetic” stuff fixed. Here are updated screenshots, that you can compare to the ones my last update about this project:

First, here’s the options screen.

You’ll notice it looks pretty much about right. I had to use a third-party time control (ZVDateTimeControls) to get the time picker working. Lazarus doesn’t have an equivalent of TDateTimePicker. They have a TDateEdit, but that only does calendar and not time only. But the third party one seems to work fine, though I’m not in love with the arrows on it, I’d rather see it using the native windows picker control under the hood, since cross-platform compatibility is not a primary concern of mine at the moment. But at least it’s functional for now.

Here’s the main window at the moment:

You’ll notice the main screen now looks “almost” like the real thing now. The toolbar is there, but the labels are in the wrong spot. I stumbled almost on accident onto the solution for how you do the labels on the right. Apparently you have to use TSpeedButton(s) on a TBevel instead of using a TToolbar to get labels on the right. Sounds pretty doable, but is lower priority than some of the other issues I’m dealing with.

Those tabs also don’t appear totally “right”. You’ll notice there’s some weirdness about the borders for them. The tabs are a TTabControl because they don’t have individual content, it’s really just an array of accounts it’s representing. But there’s a key difference in Lazarus from Delphi that the TTabControl is not designed to have child components, so it’s resizing it wrong to make room for the components that should be nested on the tab. This might actually be kind of involved to fix. One guy supposedly submitted a patch at one time that made it work more like Delphi, but it got rejected because it didn’t work “right” for all the different platforms Lazarus supports. So looks like I could try to track down that patch and make it into a third party control, or I could change the TTabControl to a TTabSheet, and replicate the buttons onto every tab, but that would lead to a lot more UI overhead. But either way, it’s not worth fixing unless I find a way to fix the blocking issue that could be a showstopper.

Right now the blocker issue is Indy 9. In 2005, Marco van de Voort ported Indy9 to Free Pascal, at least for 32 bit architecture, not 64 bit. But then he decided to upgrade to Indy10 so it wasn’t maintained. So, showstopper issue, Indy9 isn’t working right. I’m focusing on the “test account” button right now since that’s about the simplest network-operation the app does. Connect to the Pop3 server, login, and disconnect. Login is actually working. The problem is disconnecting. It’s reading garbage instead of “+OK” for the disconnect message, so it fails to disconnect and throws an exception that causes other problems, like skipping actually closing the connection. The problem is, using Wireshark to sniff what’s actually going on in the network, the server is actually sending the correct response back. But what PopTrayU is seeing when I step through the debugger is half of the account’s domain name, “bear.net” Stepping through the routines where it moves pointers around to find the response, it looks like it’s reading the wrong place in memory, it’s skipping right over where it should be reading, and reading from some random spot a little bit farther out that presumably contains memory that was previously used for something else. Tricky right? The code doing this is totally obscure, and has very few comments, so it might take a while to understand it well enough to figure out *why* it’s reading the wrong address when it goes to disconnect. The warnings in Lazarus that it’s doing Pointer to Integer conversions could be a hint though. Theoretically that shouldn’t mess it up on 32 bit, but you never know, seems to be doing some bad pointer math somewhere.

Interestingly, poking through the Indy10 codebase, they appear to have reworked the whole “managed buffer” that’s giving me issues and eliminated this weird pointer math. And Indy10 is officially supported in Free Pascal, so I may have to bite the bullet and upgrade. Apparently there were some non-trivial issues when Renier tried to port to Indy10 once long ago, but perhaps they would be less nasty to resolve than buffer overrun issues.

Oh, and unrelated to all of that, I also have the tray icon partly working now. The right click menu is working, but changing the icon images is not. The rules and about screens are also on the todo list if I can resolve blocker issues. The about screen lost all the strings in the tables, because they’re encoded in binary in Delphi, so I have to retype them. And the rules stuff was using some components that have to be manually ported/fixed.

Yesterday I started working on trying to port PopTrayU to Free Pascal/Lazarus. The built-in converters seemed to do a good job for the most part, though they did leave hundreds of errors for me to clean up before I could even say it compiled, and its pretty far from, well, working.

But here’s where we’re at right now:

This is the main screen, and it does look fairly close to what it’s supposed to look like, but as you’ll notice, a few key UI items (the toolbars and list of messages) are missing.

On the options screen, things look even worse:

Though I’d be willing to bet that in the end, the problems with the options screen turn out to mostly be cosmetic (a list that needs to be manually repopulated), and that the main screen will turn out to be way more work to get working.

So what issues have I been running into as I port? Thanks for asking.

The CoolTrayIcon component that PopTrayU relies on for the tray icon in the windows tray is super-broken to the point where it was easier to comment out everything relating to the tray icon rather than cobble the Tray Icon component into running skeleton code. The Tray icon relies heavily on Application.Handle and Application.Hook methods to get references to the application to do snazzy things like minimizing to the tray. Those methods are conveniently not available in Lazarus. So I’m going to have to do more research about workarounds for that, because there doesn’t appear to be a straightforward “just do this instead” option.

When the Lazarus porting guide said most of the things that aren’t implemented are “things that are very windows specific” it turns out to be quite the list, and some of those components that are “wontfix” missing components are ones that the app uses.

For example, THotKey has no equivalent. THotKey is a class that hooks into the system settings to set system wide hotkeys to control PopTrayU when you’re using other programs. That’s not to say it can’t be done, but if it can be done the code is probably going to look a lot more ugly. But that’s a low-priority feature compared to say, checking your email and the tray icon being also broken.

I actually came across this handly list that shows which Delphi components do and do not have equivalents in Lazarus. Unfortunately for me, the “not included” list includes TActionManager, TActionToolBar, TPopUpActionBar, and TCustomizeDlg. I may be forgetting others. So basically, in Delphi, all the menus and toolbars used to use Actions to link the menu items to the code it runs when you click on them, and then you could drag and drop menu items into toolbars and vice versa and fully customize the toolbars, which is a really nice to have but not essential feature. Having menus and toolbars is, however, an essential feature, so I’m going to have to check out TToolbar, TActionList (that is available) and TPopupMenu which are candidate replacement classes.

Some of the more straightforward replacements that the auto-code-fixer couldn’t figure out for me were things like:

• TTabSet -> replace with TTabControl
• TColorBox -> add import ColorBox
• GetTime() in SysUtils -> replace w/ Time()

There were also a couple type (struct) definitions that I had to manually add (related to hotkeys), which wasn’t a big deal other than figuring out what was missing.

One of the tricks I found handy for fixing things was keeping Delphi open at the same time and using the “Browse Symbol at Cursor” feature to plug in the name of whatever it was complaining was missing to find out where that was defined, so I could figure out whether it’s an import name that’s missing or something from one of the dependencies or delphi or wherever it’s from.

But overall, most of the issues have been with UI stuff and NOT indy stuff, so it’s looking optimistic that this is a feasible port worth continuing to pursue. I was having a couple minor issues with indy, but I think they’re minor, figuring out where to store the data.

On with the porting!