Diving into the Bohemouth

One of the things I’ve been finding most challenging as I develop PopTrayU is how to test and debug issues with emails. There’s no easy way to select an arbitrary email and paste in it’s raw headers to test the email.

Or just in general, situations like this: a user reports some bizarre error message that doesn’t make sense and is poorly worded about how dollar sign underscore isn’t an integer…they got this email from ebay.de that caused the error message. A little searching the code and google reveals the error message is probably caused by trying to convert a string to a number. Well, ok, so have a good suspicion of a couple places that might do this, such as where it’s converting dates around…but I need an email with the bad headers to cause this behavior… no easy way in. This bug has been around for some time (since before I started messing with the code), and it seems, in my opinion, that if it’s going to give an error message it should at least make sense, eg: “this email message is corrupt” and it should try to display the bad data as best it can excepting the corrupted header if it can’t be salvaged.

On the one hand, I could modify that code to try to fix the bug, but without creating a repeatable test-case, I have no way of verifying the fix actually fixes anything and isn’t just an incorrect stab in the dark.

As I think about how could I make the code more testable, I keep coming back to the need to break the code up into modular pieces, rather than a 7000+ line uMain.pas file. If I could break apart the pieces where it’s doing the message processing from the UI and input data source, I could make a second unit test project that fakes the input data and does something different with the output rather than display it on the UI.

So, now that I fixed the internationalization code so all my new labels are translatable (unrelated, but needed to be done), I think some refactoring will be my next sub-project…see if I can break down the mammoth main file into more bite-size chewable testable sections.

Tables

I was looking online to refresh my memory how to make a table with only horizontal lines and not vertical lines between columns. I was kind of surprised the top results made it sound really complex and like it required all kinds of hacks to make it work in a cross-browser way. Really? What about just setting the table’s border property to 0 and then adding this css?

table { border: 1px solid #AAAAAA; border-collapse:collapse; }
td, tr { border-bottom: 1px solid #AAAAAA; }

Keep it simple.

Where to Store PopTrayU ini/user files?

One area where PopTray/PopTrayU is currently less than perfect when it comes to Vista/Win7 functionality is that it stores the poptray.ini file (and other files such as rules and whitelist) in the program files folder with the application. In Vista and above, writing to program files when you’re running normal user applications is a big no no. Those kinds of files are supposed to go in a per-user folder under the users/appdata folder to support multi-user environments, and protect users from malicious applications overwriting things they shouldn’t. It doesn’t make my job as a developer easier, but from a security perspective, it definitely helps cut down on certain opportunities for viruses and trojans to be destructive.

But that leaves me with a dilemma. The old design where everything was in the same folder, and the app doesn’t use the registry makes copying the app to a memory stick to use on other machines (say, internet cafe) with locked down machines pretty easy and doesn’t require an installer. Also makes side-by-side use with PopTray easy, since you can just plop poptrayu.exe in the PopTray folder and go.

But, with Vista/Win7, that model where everything goes in the app folder makes a lot of headaches for users who accidentally ended up with a virtual program files folder and can’t figure out why their changes aren’t “sticking” to poptray.ini or the plugins they are trying to install are suddenly “missing” and they have to approve all these UAC prompts to even install a plugin. Not the best of user experiences.

In order to support Vista/Win7 properly, the default storage location for those files needs to be appdata (aka a user-specific folder). I’ve been debating whether it would be best to keep things simple–now poptray.ini HAS to go in user/appdata/roaming/poptrayu/ unless they specify otherwise by using the command line options to specify the “ini folder”, or whether to come up with some more elaborate scheme such as falling back to the app directory if the user/appdata/roaming/poptrayu/ folder doesn’t exist.

Then, if I change the default storage location, there’s another ball of worms, as far as updating the installer to move/copy the ini file to the new storage location, whether it should import the ini file from poptray and/or poptrayu’s app folder if it’s there, etc.

So, please discuss in the comments, what you think the “right” behavior of the app should be. Do you actually use the use-case where having fall-back folders would be handy, or do you just run the installer and expect it’s going to magically put everything in the right place and you don’t care where it is so long as it works?

PopTrayU Beta version 4.0.1

About a week ago, I uploaded the first completed beta version of PopTrayU up onto Sourceforge. I sent a link to Renier to test/comment, and he sent me back some comments about some bugs–mostly minor stuff to fix, like an incorrect link to the help file. But a couple are bigger/trickier issues.

In particular, one is the issues with storing user-editable files in Program Files under Vista/Win7… well, you can, but it can cause quirky issues, like “virtual”/duplicate files to be created depending on the security settings, and that can cause it to look like there’s bugs in the app when there isn’t. On the one hand, having everything in the app’s directory is nice because you can run it off a memory stick with no installing required, on the other…well, trying to cooperate better with the OS can be a boon and not confuse users… 😉

I’m still contemplating the best way to fix that. How should the app decide where the default ini file storage goes? Should it always be in the appdata/user folder? Should it look in the user folder and if that folder doesn’t exist, fall back to the app folder? Use a registry key to store where the config files are? Make the ini file only have one line that points to the location of the “real” config file.

This problem doesn’t stop at just an ini file though. (and as an aside I’m not going to get into the holy war about “outdated” ini files vs “non-portable” registry entries at the moment), For PopTrayU, files the user might be more interested in editing than an INI file (no need to edit that one manually) would be to install plugins, sounds, or the skin files. So ideally, that should be in a user writable folder without needing to use UAC dialogs or administrator permissions, so not in program files. Maybe I need to explore how other open source software that have similar support files (Miranda IM comes to mind) have dealt with changes to OS restrictions on newer versions of Windows.

The second bigger one is a bug more than a design issue. Use the SSL plugin to connect to Gmail, and bam, get an access violation memory error. I was actually able to reproduce this the first time I set up my gmail account and accessed it. But then I turned break on exceptions back on and restarted the app, and it wouldn’t reproduce anymore, and additional testing of using my gmail account since has not reproduced the issue. So this one is going to be tricky to fix because I’m not sure what causes it. I guess I’ll have to keep my eyes out for additional hints on what factors are necessary to reproduce the bug.

Today I got an email from a sourceforge user thanking me for continuing the development of PopTray and letting me know about an issue the user was having with custom sound notifications, an area I haven’t thoroughly tested (nor made changes to). You mean people are actually using and finding my app even though I haven’t done a lick of promotion (aside from creating it’s sourceforge page) yet? Awesome. I kind of purposely haven’t been promoting the site yet, as I’d rather have bugs trickle in than flood in, so I have time to look at them and fix them rather than just queuing piles of duplicate bugs. That’s why you do beta versions, right? You want enough exposure to find the bugs, but not so much that you’re overwhelmed with duplicated feedback.

Sourceforge Project Set Up

Yesterday I logged onto Sourceforge account and created a project for PopTrayU, so I’d have somewhere to release my modified PopTray. Yes, I could probably do it all from scratch on my own site, but why bother when they have all the tools right there, it’s a respected site for looking for free software, and it’s also where PopTray’s verison history is hosted as well.

For some reason I had presumed that Sourceforge only provided hosting for files and I’d have to zip up complete versions of source code to save on Sourceforge (probably because that’s what PopTray did), but lo and behold, it has options to set up SVN or Mercurial. I wish I’d known about this sooner!

I’d previously looked into setting up SVN on my website for this purpose, but although creating the repository worked flawlessly without installing any additional tools, figuring out the username/password issues and proper URL repository path on the shared hosting environment was not successful. So I had been doing daily zip file backups every time I work on the project up until now with the hope that I would eventually get real version control set up where you can browse all the incremental changes to each file to track where a change was introduced, etc.

Now I am chugging away at setting up the repository structure, and going back to the original source code, checking that in, and checking in incremental bits of the changes in managable chunks. It may or may not compile on any of the intermediate versions because I’m mass importing changes via diff’s, but it’s a small price to pay, I’ll probably add a tag when I get to the point where I have something checked in that builds and compiles. Mostly I’m checking in incrementally so I can better document what changes I’ve made and isolate related changes in case any of them later need reverting or debugging…slightly more useful than here’s every change that’s in version 4.0

It just feels really good to get everything checked in, so I have a good, easy, up to date, not on my computer backup of everything, so there’s less risk of having to start over if my hard drive explodes.

Creating an Installer

Another thing I’ve been working on lately is a couple of finishing touches to my PopTray fork, so I can distribute it (icons, installer, graphics for the installer, etc).

In particular, my top priority was creating an installer. I thought I might have to start from scratch there, but it turns out Renier was using NSIS for the installer, and the configuration file used to create the installer is included, so while I tend to prefer applications that are a bit more visual than NSIS and have menu options and check-boxes to configure how your installer will work and function, this one was almost all the way there. It was easier to just fix the problems with the existing installer script (like creating images and folders needed by the installer) than to start over from scratch with a different installer.

I see the advantages in certain environments that the install script system used by NSIS is very customizable for specific install procedure needs and would be easy to track changes to the install script in version control systems since everything is stored in one text file until you compile it into a setup.exe, however, it is also a bit fragile by the same nature.

It would be quite easy to overlook or make a typo in the name of a file that needs to be deleted when uninstalling and have it not uninstall completely because of that. That, and I’d kind of rather not have to pull out a manual every time I want to know what a particular line of the script does, or to figure out whether the line is actually necessary. And then if you comment out an image here from the script, then the macro it creates needs to be commented out there as well or it just breaks and won’t compile. It makes a nice small installer without a lot of overhead, but less flexibility would also mean less opportunity to introduce a mistake into the installer script. I don’t want to worry about registry keys needed to uninstall the app being in my script, I want the compiler to take care of that part, I want to be spending my time specifying that the following files go in program files and the following files go in application data 😉

Moms Club Website

One of the many projects I’ve been up to lately is updating the website for the moms club I belong to. It probably doesn’t surprise you that that moms clubs are not typically havens full of geeks, so it probably also doesn’t surprise you that I got recruited into updating the website.

It’s something I don’t mind doing though, I think it will help other moms looking for their niche where they really find community. There’s a lot of stuff the club does that you wouldn’t realize from the website, and who’s going to pore over years worth of back-issues of newsletters (unless, say, one was updating a website and looking for content…) to learn about them?

But before tackling content, the UI was in sore need of a makeover. Here’s a before/after thumbnail:

Certainly, there are still things I think could be better UI-wise, but its a huge improvement, and I’m going for incrementally better changes. By the time I got done with the new homepage, it had been so long since I’d gone back and looked at what I started with, that I’d almost forgotten some of the things I had fixed early on, like the random checkerboard pattern, and half of the headings being images without alt text, in a different color and font-style than the other headings on the page.

I’m going to have a featured article in next month’s member newsletter letting them know about some of the changes. Here’s an excerpt from my article:

Website Update

If you’ve been to our public website recently, you’ve probably noticed some changes. […]

Behind the scenes I’ve been working on sprucing up the website, doing lots of things that you probably haven’t noticed like search engine optimizations and new “404 error” pages, as well as some much more apparent changes like the new look for the home page and additional content throughout the site.

We now have a new contact form where you can easily contact a variety of club officers without having to keep track of who is fulfilling that role at the moment.

There is also a new playgrounds and outings guide that I’m particularly excited about. I’m always looking for ideas on fun things to get out and do, and being new to the area, I don’t automatically “just know” everything there is to do around here, so it was a great discovery to find a list of local attractions collecting dust in the files area. The list has been updated and now is on the website where you can easily share it with your friends, including ones who aren’t members—yet!

Another new addition to the website is a section with visual tutorials on how to use BigTent. To be honest, even I, a total geek who “should” know how to do that kind of stuff, had to search the help area to figure out how to leave a subgroup, so if there’s something that seems like it ought to be easy to do, but isn’t obvious, it’s probably not just you! So I’ve taken some of Ivette’s wonderful BigTent tutorials and added screen-shots and put them on the public website where they should be easy to find, and added direct links to some of the other existing tutorials. Same content, now easier to find.

There are still many more changes in the pipeline, so check back from time to time to see what’s new! And feel free to use the new contact form to send in your comments and suggestions on how we could make the website even more awesome!

Aside from the stuff mentioned in that article, other changes included things like a password protected file upload form with PHP script to let the member discount coordinator update the member discount PDF without me having to upload it and re-link it for her every time (FTP + non-techy = just not going to happen).

I also made “user friendly” 404 pages (and 401, 500, etc), that clearly let you know the page wasn’t found, but still has the navigation available to get back into the rest of the site even if you hit an unexpectedly bad link.

I also updated the entire site’s URL structure to make the URLs more logical. Having a folder called /pages/ in the url is kind of useless information to the person viewing the site. So the new urls follow a hierarchical structure and use folder names instead of filenames (so they don’t have to worry about whether it’s .php or .html), and of course I set up all the redirects in apache so incoming links from search engines don’t break and we don’t lose the page-ranks. And added the right meta-tags to make the snippets shown on google be the important part of the page, so the stuff shown is a bit more meaningful to people searching for the page, etc.

Embedded Iframes

Working with 3rd party embeddable widgets can often enhance the user experience by allowing you to show information you wouldn’t otherwise have access to, or save a click or two to get to a third-party login page, etc. However, not all 3rd party embedding APIs are created equal.

One in particular that I’ve been working with has absolutely no customization options, and the only option for how to embed it is with an iframe. This severely limits the options I have as far as integrating the widget into the rest of the page. The rest of the page has headings in blue, in Arial, and the widget? No way to customize that font.

I’ve looked into it multiple times, but without control over the domain hosting the widget, there’s no way to add CSS to within the iframe, because even if you add the CSS with javascript, you still have to have the CSS on the same domain as the iframe because of cross-site scripting safety.

I did at least find some answers about the limited customizations I do have. You can change the margins, padding, and border, basically, the frame itself. The couple interesting things you can do is use CSS3 features to add shadows to the border, or even rotate the frame (though in my testing the rotation looks terrible and isn’t up to the quality needed to look decent) to at least make the sticking out like a sore thumb widget look intentionally stand-out from the rest of the page…sort of…I’d rather have it at least have matching fonts…but until I figure out some way to do that using an API with no “attach stylesheet” option, my hands are tied.

Delphi 7 for Java Programmers

Comments

Comment Style Java Delphi
// comment yes yes
/* comment */ yes no
(* comment *) no yes
{ comment } no yes

Delphi convention: (* code *) vs {Textual Comment}

Procedure vs Function
Method in Java = either “function” (if it has a return value) or “procedure” (if it has no return value) in Delphi

Overloading a procedure:
In the interface section of your unit, add “overload;” at the end of both overloaded versions of the procedure.

    procedure TranslateForm(form : TTntForm); overload;
    procedure TranslateForm(form : TForm); overload;

Exceptions

Java Delphi
Try Try
Catch Except
Finally Finally
Throw Raise

In order to have both an Except and Finally block for the same line of code, you have to nest Try blocks, like this:

Try
  Try
    causeSomeRandomException();
  except
    on E:Exception do ErrorMsg(num,'Error:',E.Message,True);
  end;
finally
  doSomeCleanup();
end;

Other suggested reading:
http://techinorg.blogspot.com/2008/09/delphi-for-java-developers.html
Comments on some of the differences the author found less enjoyable

http://www.webbasedprogramming.com/Java-Developers-Reference/ch5.htm
It’s going the other way Delphi to Java, so it’ll point out areas of difference, but not the Delphi syntax.

http://www.marcocantu.com/papers/ooplang.htm
Compares C++, Java, and Object Pascal (Delphi)’s OOP Features.

PoptrayU: Updated Screenshot

Here’s an updated screenshot with my progress over the past few days:

You’ll notice now that the two emails with subjects in Hebrew and Greek (probably horribly translated, I just used Google Translate) now appear correctly in the main window.

Getting these messages to appear correctly was actually quite tricky to figure out. The key, in the end, was that I had to make a copy of the subject field before the Indy routine ProcessHeaders mangled it into Ansi characters, and then call my DecodeHeader algorithm on the un-mangled header.

The ironic thing is, if I upgraded to the latest version of Delphi, and the latest version of Indy (as in Indy10 rather than as in Indy9.0.53), the whole DecodeHeader and handling international characters would be a non-issue. But then I’d have a whole different can of worms–I’d have to fix all the non-backward compatible changes between Indy 9 and 10, I’d have to figure out how to convert ActionBand Popups into the new Delphi 2010 equivalent. In the end, what it would take to port the app to a newer version of Delphi is probably more difficult than coercing the old version of Delphi and Indy into doing what I want it to do. Sure, in the end it might end up being more robust that way, but I might also break and mangle lots of existing features.

At this point, what you see in the screenshot, I am only processing the Subject field with the new technique that works for any codepage (vs: only the current one or UTF-8), so I need to extrapolate my strategy to the other header fields that might be encoded. I also need to find an equivalent strategy to do the same thing on the Preview window. In the preview window it re-downloads the email through a different Indy code-path, and I haven’t found the right place in that code-path where I can capture the un-mangled header yet. So there’s still work to do but I’m on the right path.

To convert the random code-page to Unicode, I am using the windows library function WideCharToMultiByte, which converts a string to a “wide” (Unicode) string based on a specified code page number. Getting the code page number was also a little bit of a challenge. The library with that function doesn’t have a GetCodePageNumber function to convert the code page *name* to the windows code page *number*. There is a DLL that comes with windows that has (almost) that function, but figuring out how to call it is kind of tricky, and rumors on the internet say it might be buggy in certain cases. So, I’m using the straightforward but ugly strategy: convert the table on MSDN of allowed code-page names/IDs to a data structure and look it up manually. That list isn’t the full IANA supported list of encodings (aliases/alternate names), but the cases it doesn’t handle are likely to be rare, and could be added in the future if future research doesn’t find a better strategy.

I started out storing the list in a record (ie: Delphi equivalent of a struct) with a dead simple sequential search algorithm, per a tip on StackOverflow, but I wasn’t happy with the performance on lookup, because there’s 140 different encodings in my list of encodings so far, and this method is going to be called in a loop for each header of each email unless it’s definitely not encoded. So I did some research and found TStringList which can be used like a sorted map with better string search performance, and THashedStringList, which is basically a hash-map data structure, so even better string search performance–up to the level you need for, get this, INI FILES! Then I had to do more research to figure out what the Delphi equivalent of a static initializer is to use the Hashed String List…But now that I have it working, it does seem noticeably faster even on a small number of emails, but the internet could just be less congested today, it’s hard to say.