Add/Remove Gmail Labels using Indy 10 in Delphi

According to support threads, Indy’s TIdImap4 supports fetching Gmail labels but doesn’t have support built in for adding and removing labels. I’m still working on figuring out how to fetch and read the fetched labels, but by using the SendCmd function to manually send the IMAP commands I got adding and removing labels working. So I thought I’d share that code:

function AddGmailLabelToMsgs(const uidList: TStrings; labelname : string) : boolean;
begin
  try
    if HasCapa('X-GM-EXT-1') and (uidList.Count > 0) and (labelname <> '') then begin
      IMAP.SendCmd(ImapCmdNum(),'UID STORE '+uidList.CommaText+' +X-GM-LABELS ("'+ labelname + '")',['OK','BAD','NO'], true);
      Result := IMAP.LastCmdResult.Code = 'OK';
    end else
      Result := false;
  except
    on E : Exception do
     begin
       //Dialogs.ShowMessage('Exception class name = '+E.ClassName);
       //Dialogs.ShowMessage('Exception message = '+E.Message);
       Result := false;
     end;
  end;
end;

At the heart of it, I am using Indy’s SendCmd function to manually send the command. It was a little tricky figuring out how to use this function since documentation and examples are a little scarce. The first parameter is a sequence number so that the response can be matched up with the command. Second is the remainder of the actual command. Third parameter is what responses to look for to know this command is “complete”, and the fourth is whether to look for a single line response or multi-line response.

I also have some commented out exception handling that was helpful to debugging when my SendCmd wasn’t formatted correctly, but is neither needed nor useful for production code.

Remove label is basically the same thing but with a minus sign:

function RemoveGmailLabelFromMsgs(const uidList: TStrings; labelname : string): boolean;
begin
  try
    if HasCapa('X-GM-EXT-1') and (uidList.Count >0) and (labelname <> '')  then begin
      IMAP.SendCmd(ImapCmdNum(),'UID STORE '+uidList.CommaText+' -X-GM-LABELS ("'+ labelname + '")',['OK','BAD','NO'], true);
      Result := IMAP.LastCmdResult.Code = 'OK';
    end else
      Result := false;
  except
    Result := false;
  end;
end;

And here’s the helper functions, minus the global/class/whatever variables:

ImapCmdNum() generates a sequence number for the IMAP transaction, essentially a replacement for NewCmdCounter in IdImap4, since it’s a protected property and client code can’t use it. At first I tried using “A1”, “A2”, etc. but it turned out that using a C prefix, “C1″, C2”, etc. like Indy was important for using Indy to process the response when I got into debugging why things weren’t working right at first…but now I’ve completely forgot exactly where it goes wrong if you don’t use a C.

function ImapCmdNum(): string;
begin
  Result := 'C'+IntToStr(cmdNum);
  inc(cmdNum)
end;

And HasCapa() is a function I added to make sure that the Gmail capabilities are actually available before trying to set/unset labels…Might not be totally necessary, but it seemed like a good idea. Under the hood, I just used Indy’s built in Capability function that calls CAPABILITY on the server, with a little caching so I wouldn’t need to do this more than once per session.

function TProtocolIMAP4.HasCapa(capability: string) : boolean;
begin
  if (capabilities.count = 0) then begin
    IMAP.Capability(capabilities);
  end;
  Result := (capabilities.IndexOf(capability)<>-1);
end;

How to Use Indy 10 Imap Intercept function for Logging

It took me a while to track down enough information to figure out how to use Imap.Intercept to log raw IMAP client/server communication to troubleshoot an SSL account issue, so thought I’d share a basic code snippet on how to have IMAP log to disk the communications.

var
idLogFile1 : TidLogFile;
begin
IMAP := TIdIMAP4.Create(nil);

idLogFile1 := TidLogFile.Create(nil);
idLogFile1.Filename := 'C:\temp\imaplog.txt';
idLogFile1.active := True;

IMAP.Intercept := idLogFile1;

//...
end;

Log4D Basic FileLogger Example

Since there is lacking on “getting started” documentation for Log4D, I thought a basic example might be helpful to others who might be intimidated about getting started with logging to a file in Delphi. Log4D is very simple, and similar to other Log4J related logging packages but lacks basic “getting started” documentation.

I used XE4 Starter for this. At the time of this writing, I and others have submitted some patches to fix the compile errors in Log4D affecting XE4, but they have not been merged in yet. For the quick fix to make it compile in whatever recent version of Delphi you have, see my previous post, my previous post.

First, create a project. I created a VCL application project for this example, because I’m not writing console applications so it’d be easier to copy and paste from 😉

Open the .dproj file (right click on the .exe name in the Project Manager and select “view source”) and add the highlighted lines:

<pre>
program Log4DbasicExample;</code>

uses
Vcl.Forms,
Log4D,
Unit1 in 'Unit1.pas' {Form1};

{$R *.res}

var
Logger : TLogLogger;
begin

TLogBasicConfigurator.Configure;
TLogLogger.GetRootLogger.Level := All;
Logger := TLogLogger.GetLogger('myLogger');
Logger.addAppender(TLogFileAppender.Create('filelogger','log4d.log'));

Logger.Debug('initializing logging example program');

Application.Initialize;
Application.MainFormOnTaskbar := True;
Application.CreateForm(TForm1, Form1);
Application.Run;
end.
</pre>

When you run this application, it will create a log file called log4d.log in the directory the .exe file is in. Good enough for quick logging and debugging. Not adequate for production code.

Anywhere else in the project you need to log something, now it’s easy. I added one button to my VCL form, and double clicked the button to add an onClick handler, and then updated my VCL form unit as follows:

unit Unit1;

interface

uses
  Log4D, 
  Winapi.Windows, Winapi.Messages, System.SysUtils, System.Variants, System.Classes, Vcl.Graphics,
  Vcl.Controls, Vcl.Forms, Vcl.Dialogs, Vcl.StdCtrls;

type
  TForm1 = class(TForm)
    Button1: TButton;
    procedure Button1Click(Sender: TObject);
  private
    { Private declarations }
  public
    { Public declarations }
  end;

var
  Form1: TForm1;

implementation

{$R *.dfm}

procedure TForm1.Button1Click(Sender: TObject);
begin
  TLogLogger.GetLogger('myLogger').Debug('Button 1 clicked!');
end;

end.

If you’re doing a bunch of logging in one part of your code, you probably don’t want to use the chaining syntax because calling GetLogger repeatedly is unnecessary overhead. Instead store the Logger in a variable:

procedure TForm1.Button2Click(Sender: TObject);
var
  Logger : TLogLogger;
begin
  Logger := TLogLogger.GetLogger('myLogger');
  Logger.Debug('Button 2 clicked!');
end;

Well, there you have it. Basic, just enough to get started example. Because next time I want to add logging to a different project, I’m sure I will have completely forgotten how to do it and what and where I added my code.

File-Logging in Delphi

I wanted to do some quick logging to help me understand some code flow in Delphi. But having the Starter Edition of Delphi was crimping my style! Starter Edition doesn’t include the Event Log Viewer to see the output from OutputDebugString() while running in the debugger.

So I kept coming back to “I really just need to get over the learning curve and set up a ‘proper’ logging tool like Log4J”, a tool I’d used for Java projects in times long past. StackOverflow suggested Log4D and Log4Delphi as good possible candidates.

After reviewing both packages, I decided to go with Log4D.

Log4D seemed a bit simpler in that everything you need for basic logging is in one .pas file and one .inc file. I can see advantages both ways as a developer, but for just get me some logging quick, only having to add two files rather than a handful, and only needing one new unit in the uses statement simplifies some things.

Neither one had been updated in the last few years, so when you’re talking about “make this compile in a current version of Delphi” it looked like similar level of work.

Log4Delphi claims to be “loosely based on” Log4J. Log4D claims to be a direct port of Log4J.

Log4Delphi appears to be fairly well documented, whereas Log4D has almost no documentation. And while usually I prefer to go with the well-documented version, I felt like the simplicity of one file would make it worth seeing if I could muddle through the lacking documentation to make Log4D work. A few Delphi experts on StackOverflow swear by Log4D.

There were two compile errors I had to fix to make Log4D compile on XE4:

The IFDEF around Contnrs is missed by newer versions of Delphi, so you need to add {$DEFINE DELPHI5_UP} to the end of the .inc file (or in the .pas file works too). And ShortDateFormat needed to change to FormatSettings.ShortDateFormat

That’s the bare minimum fix, a better fix that maintains backwards compatibility isn’t a whole lot harder See my StackOverflow Discussion here about fixing the IFDEF so “and above”. And the ShortDateFormat issue, just needs another IFDEF around the FormatSettings for XE+.

The only basic “how to get this up and running” example of Log4D I could find online was this one blog entry. It needed a couple changes to compile (“Trace” wasn’t one of the log levels in the downloaded Log4D package). And then a little more puzzling together from the code and documentation for Log4Delphi and Log4J to make a file logger. I’ll write more about those changes in my next post ;-).

Installing Indy9 from Source Code

I’m finally working on getting Delphi 7 set up on my Win7 x64 machine. I think it’s been over a year since I last did this on my Vista machine or my Win7 x32 netbook, so my mind totally went blank trying to remember how to install the lastest version of Indy 9.

I already had pulled the code from Indy’s SVN server and been able to transfer that from the old machine, so I was one step ahead there.

I pulled up the “ManualInstall.txt” document in the Indy source directory and tried to follow the directions, only to find it didn’t work. At first it looked like it was a permissions issue, I needed to be running a batch file called fulld7.bat from a command line running on elevated privileges to get the script to run without errors, however, even after sorting all that out, when you get to the step of Installing the *.bpl, even after you’ve correctly remembered which one is the right menu option install such, it just doesn’t work. Complains about a missing file (that just doesn’t seem to exist in the provided repository or compiled files directory).

Well, after much fiddling and trying to figure it out on my own, since the internet had no solutions, it turns out, those directions probably are obsolete, and doing it the way that actually *worked* was much easier…just not obvious if you haven’t been working with Delphi recently.

In the Indy 9 directory from the SVN, there’s a .DPK package file for each delphi version. So, since I’m using Delphi 7, double click Indy70.DPK, it opens in Delphi with a package window visible displaying a list of all the files. At this point you’re two clicks away from a working Indy install. Click “Compile” and then “Install” on the package window, and dismiss the dialogs that confirm those steps work, and suddenly Indy is on the component toolbar without even restarting Delphi.

So there you have it. I’m thinking documenting the steps on installing this that work will be helpful, because by the next time I get around to trying to install Indy on a new machine, I almost certainly will have forgotten how you install it, since you do it once and never touch it again, and odds are good that it will need re-installing after a hiatus where I haven’t touched Delphi in a while period and can’t remember off the top of my head the difference  between installing a component and installing a package, since that’s very delphi-specific terminology.

Upgrading from Indy9 to Indy10

Translating the source code to Indy10 only took about an hour. It would be nice if there were a more comprehensive upgrade guide, rather than having to google one piece after another to figure things out, and pulling up grep to search the source code to figure out what include is needed now.

Here’s a summary of a few of the changes I encountered needing (hope this helps someone save some time!)


  • POP3.MaxLineLength becomes Pop.IOHandler.MaxLineLength (watch out for IOHandler being null though)
  • wsOK moved, add uses IdIMap4
  • Pop3.Connect(Timeout) breaks into two commands, Pop3.ConnectTimeout := TimeOut; Pop3.Connect()
  • StoredPathName not found. Change TIdAttachment to TIdAttachmentFile
  • POP.Capture(Dest); moved to POP.IOHandler.Capture(Dest);
  • Signatures of OnWork and OnProcessWork changed. Change const Integer param to Int64 (not const)
  • EIDSocketError not found. Import/uses IdStack
  • CommaSeperatedToStringList not found. Add import/uses IdGlobalProtocols
  • TIdText not found. Add import IdText

A few articles about porting to Indy10:
http://conferences.embarcadero.com/article/32160
http://www.indyproject.org/docsite/html/MigratingIndy10.html
http://www.indyproject.org/docsite/html/ChangesObjectHierarchy.html

But when all was said and done, that didn’t fix my disconnect problem. Looking a little more into why it was crashing disconnecting, I changed closing the connection from being conditional on the correct response to unconditional, and instantly, things start working and I have email showing up in my client! Yay.

Debugging Forms Ported from Delphi to Lazarus

One problem I was encountering in converting a few of the Delphi forms into Lazarus forms is that after I got to the point where everything was compiling, and even the point where I could run the application and bring up the main window, I still couldn’t successfully edit the forms properly.

I’d open the converted LFM (Lazarus form) and it would only display in code-view and not the “Unit View” where you can visually edit the UI. Go to View->Toggle Form/Unit View, which should cause it to switch to Unit view, and nothing would happen. Not so helpful. Or one time the option was greyed out entirely. With no real clear hints about why.

Clearly, something was wrong, and not a lot of feedback about what. I was unsuccessful in finding any guides online about how to troubleshoot such, so it took a bit of tinkering and guessing. So here’s what steps were actually useful in solving the problem.

1. Make sure all expected dependencies are listed in the Project Options (under Compiler Options->Paths)

First, since the form was fully working in Delphi before converting, there’s a very good chance the problem is an unknown property name or an object of an unknown type. This is probably not a totally unfamiliar to a Delphi developer who’s ever had a missing dependency on a form before. But since you’ve probably already done that or you wouldn’t be still trying to fix it…

2. Tools -> Check LFM in Editor

This brings up a window that checks for various errors in the LFM. If anything is found here, you should definitely fix them. In this example, I’d already replaced TDateTimePicker with TDateEdit in the corresponding .pas unit file, since Lazarus doesn’t have quite the exact same component available. But the form needed to be updated too.

Mind you, that wasn’t enough to make the form actually load. But now at least the form check has fixed all the problems it can. The wording on the informational dialog that tells you this could probably stand to be more clear, but  if you thought to read the title, that would be a big hint that this isn’t an error message.

In some cases I found it better to edit the fixes I wanted into the file in notepad rather than letting it automatically delete the objects/properties.

3. Make sure the LFM is open and saved, then select File -> Reload

Hmm, that’s interesting, all of a sudden I’m seeing a new error it didn’t tell me about when I first loaded the form.

It’s not the most easy to decipher error message, but essentially it’s the same sort of message you see in Delphi when you try to load a form that uses a component that’s not in the project path correctly. Stream position (presumably either in bytes or characters) is not nearly as helpful as a line number would be. But it did give me enough information to know what I’m looking for. Unfortunately, TabOrder is all over the file, because almost everything has a tab order. So I decided to take the easy way out and just wipe out all the TabOrder properties (Later I’ll recreate the necessary tab order properties using the GUI builder) and be done with it rather than trying to decipher the Stream position.

4. Turn on Lazarus logging

For one stubborn form, the one that was conveniently almost 3,000 lines long (not easy to find the problem by eye!), I found all these other steps were giving me no further errors or hints as to what was wrong. I researched whether there was any way to turn on a debug log or something for Lazarus. Maybe there’s an error happening or bug it’s not telling me about.

So (after some googling) I copied the shortcut for Lazarus from my start menu onto the desktop, added “ --debug-log=c:\temp\lazarusdebuglog.txt” to the run target, and restarted Lazarus from my new shortcut. I tried the menu option to switch the view that failed, quit lazarus to end caching on the log, and looked at the log. The log actually gave me a very good hint what was wrong.

TMainIDE.DoLoadLFM loading nested class TIdAntiFreeze needed by C:\projectpath\uMain.lfm
TMainIDE.DoLoadLFM DoLoadComponentDependencyHidden NestedClassName=TIdAntiFreeze failed for C:\projectpath\uMain.lfm

There were a couple other lines too, but clearly this was a problem. Dependency problem. I searched the lfm in text view for TIdAntiFreeze and quickly located the offending object on the form, which was easy to delete (to be fixed later). After editing that object out and saving, it was able to load in the GUI builder, and fix some of the other more important UI issues.

So hopefully these troubleshooting steps will be helpful to me or someone else down the line having to resolve similar types of problems.

New Project: Porting PopTrayU to FreePascal?

Poking around on the internet for one reason or another (maybe I was browsing posts about Delphi on the Programmers Stack Overflow/Exchange site?), I came across something interesting. Lazarus, the open-source IDE for Free Pascal. 

Lazarus is an open source equivalent to Delphi. Well, not exactly equivalent, but pretty close.. It comes with (not terribly well documented at the moment) tools to port Delphi code to Free Pascal, and comes with the Lazarus Component Library (LCL) which is very similar to Delphi’s Visual Component Library (VCL).

The idea of porting PopTrayU to an open source development platform instead of an expensive closed source one (namely: Delphi XE) is very much in the spirit of open-source development. It was always one of the dreams of Renier (the original PopTray developer) to port PopTray to a free platform, and he indicated he might be interested in continuing to develop the app if it could be ported. And it would save me the expense of upgrading to Delphi XE to fix some various bugs.

I’d previously looked into porting PopTray to the free version of Turbo Delphi 2006, but my feasibility analysis results were that it was not feasible. That version of Delphi, aside from being discontinued/no longer supported or “officially” available, was that it left out a key feature (namely, 3rd party component support).

Free Pascal and Lazarus, from what I’ve said so far looks more promising as far as being feasible. Indy components has been ported to Free Pascal, which is a key stepping stone, though investing whether I’d have to port to Indy 10 first or not remains to be determined. So far everything else I’ve been reading looks like it would “probably” be possible without significant loss of critical features, but it wouldn’t be a plug and play transition, you have to rename a bunch of things and change the imports, and some of the UI might get mangled porting it so you might have to redraw parts of the UI. But so far haven’t come across any definite showstoppers.

At the moment, the biggest risk for showstopping problems would be if there is a missing library eg “things that are very windows specific” may be missing. But nothing specific yet. There would definitely be some hurdles to fix the unicode issues, but that’s to be expected and certainly fixable. Even though Free Pascal is multi-platform, the probram would still only run on Windows, because of Indy and it’s limited compatibility with other platforms.

So that’s on my todo list. And in between downloading Lazarus and now, they’ve already gone from version 0.8 to 1.0 of the IDE. Actively developed and being improved regularly is a positive sign to look for.

HTML Email Preview

One of the complaints I’ve heard about the feature of having html preview of emails is that spam-bots often track whether emails are read by embedding a unique image, and tracking if the image gets downloaded. It’s a legitimate concern, one I was well aware of when I added the HTML preview option. But it came down to being a value vs. risk proposition.

The HTML preview window in PopTrayU
The HTML preview window in PopTrayU

For version 1.0 of the feature, that seemed like a perfectly reasonable risk to accept. When I check my email from my phone, it displays in HTML and doesn’t have an option to disable loading images either. And, if like me, you run your email through a really good spam filter (like Gmail), you aren’t getting a lot of spam in your inbox in the first place. But that’s not necessarily the case for people using PopTrayU to check their ISP email account, which is a very common use case.

But that doesn’t mean it shouldn’t be somehow addressed in subsequent releases.

Initially, when I’d looked into HTML components, my first choice was a native Delphi HTML rendering component. But upon investigation, the component was too old and did not support CSS at all, which was a deal-breaker since font tags are completely deprecated these days. I mean, I could have added CSS support myself, but that is an extremely complex feature to implement correctly, with many corner cases. You’re pretty much writing a web-browser minus the network connections part. Much bigger scope than I was willing to take on.

So my second strategy was to use the TWebBroswer control to render the HTML, which is essentially a wrapper for an embedded internet explorer window. You get the HTML rendering almost for free with that technique. But…Internet explorer has this known bug that the “offline mode” flag in the component is completely ignored, and it decides whether you are online by a global registry setting or whether an internet connection is present. Less than ideal for this purpose. Any external content is really undesirable for spam email previewing.

So then I looked into maybe I could just pre-process the HTML and strip out the images. Removing <img> tags wasn’t too difficult. You can either do that with a regular expression or a for loop iterating through the characters in the message. Well, it turns out to be a little more complex than that. Images don’t just come from <img> tags anymore. Images are embedded in CSS that’s slapped in the body section of the message using url(…) syntax. Images are embedded through inline CSS. Almost any CSS component could have an image as part of a background or bullet image or border or any number of other places. And then you need to strip out any javascript, because that might be loading images or external content, so there go onclick, onload, and other properties that could be on any element in the document. And you need to get rid of external css files, you need to get rid of object tags, maybe even image maps. Point is, it gets really complex really fast.

After realizing how many loopholes there could be in pre-processing, I came back to the conclusion that pre-processing isn’t the “right” strategy. It’s a hack, that might work most of the time, but it’s never going to be as sure-fire as doing it right. The right way would be to block the requests at the network level or the browser level. Oh, you want to request a file off the internet? No, you haven’t clicked the magic button that says this is trusted content, forbidden.

Which brings me back to offline mode and that annoying bug that IE ignores the offline mode parameter. Why did they even bother putting that in their spec for the interface to their WebBrowser control in the first place? I’m sure there’s some technical reason they decided to not handle the parameter correctly. They even have a knowledge base article for a very old version of IE (IE5?), explaining the issue, and that there are no known workarounds. Very helpful.

Then I started to look into other replacements for the TWebBrowser control. There was one that initially looked promising, a firefox component that attempted to be a direct replacement for the TWebBrowser. So close, in fact, that you can patch a compiled EXE to put it in without touching the code. But…there were some big buts. But the newest version it comes for is Firefox 1.5, a seven year old version. This code is not maintained, it does not have bug fixes, it’s poorly documented, nevermind the possible security issues. Oh, plus it doesn’t implement any of the “more complex” aspects of the TWebBrowser interface, for example, it can’t create the page from a memory stream rather than from a file on the hard drive, so then I’d have to start littering the hard drive with unnecessary temp files. And even if I changed the code to do that, there’s no guarantees I’d be able to get it to work at all.

Then there’s a Google Chrome Frame. It’s not a direct replacement, and is not an activex control like the TWebBrowser or the Firefox browser component. It only supports HTTP requests (not file) unless you muck around with some settings, probably have to start writing temp files, etc. I haven’t played with it enough to assess whether it would be a feasible option or not.

One thing I did implement was adding some logic that if a message is marked as spam when you hit preview, it switches to plain-text view instead of HTML automatically. At least that solves the problem of how do you preview a message that might be spam, without having to open another non-spam email and switch tabs and then close the window and preview the spam message. That was a step in the right direction.

And then I also figured out that if you set “offline mode” in any internet explorer window, they ALL go offline, including the embedded browser window in PopTrayU. So, if you really don’t want external content to load, all you have to do is set IE offline. Of course, then you cant surf the internet in IE at the same time. But if you’re pretty serious about security, there’s a good chance you don’t use IE as your primary browser anyhow. So that’s one option. A workaround. Less than ideal for some users. But may be totally sufficient for others. Potentially I could pragmatically enter offline mode before rendering the page and then return to online mode after, affecting other IE windows as well, though that would affect other IE windows doing stuff in the background (hope you don’t have a big file download going on in the background!).

So then I get back to debating whether I should just nix the whole experimental “hide images” feature entirely, because it’s just not worth the trouble of making it work, or keep pressing in to try to find a way to make it work that’s going to be a lot of work for a modest return. There are plenty of other features and bugs and refactoring that could easily occupy my time instead. Decisions…

Writing a string to a stream in Delphi

There are two tricks to this. Most of this comes from what Embarcadero suggests on how to write a string to a stream. Another website suggested multiplying the length times the number of bytes to make sure the code works on both unicode and non-unicode builds, something one would be wise to consider when developing legacy code on Delphi 7, that may someday get ported to a newer version. One less thing to worry about later.

procedure WriteStringToStream(stream: TStream; const appendText: string);
begin
  stream.WriteBuffer(Pointer(appendText)^, Length(appendText)*SizeOf(Char));
end;