How to Configure SpamAssassin Bayesian Filter to Work with Exchange

Some organisations may have a Unix email gateway server and choose SpamAssassin to filter out the spam emails before the Internet email get delivered to the Exchange system. To achieve better spam filter result, you need to use the Bayesian filter and feed it with spam and ham. However, manually pulling spam and ham out of the Exchange mailboxes and import them to train the Bayesian filter can be a fairly time consuming process.

You can create two folders (one for spam, one for legitimate email) in the Public Folder and ask people to put spam and ham to the folders. Then, use a Perl script to pull all the spam and ham out to train the Bayesian filter each time you run this script. This way, everybody in your organisation can put the spam he/she received to the spam folder in the Public Folder.


Figure 1

Then you can put the Perl script (learn-spam.pl) to the SpamAssassin server and modify it a bit to work for you. (I found this script in a forum on the Internet.)


Please change the following accordingly.

$imapserver = "YOUR IMAP SERVER";

-uid="USERNAME"
-pwd="PASSWORD"

learn_mail ($HOME."/spam/", ".spam", "This is SPAM/", 1, "--spam --showdots")

learn_mail ($HOME."/ham/", ".ham", "Legitimate Email/", 1, "--ham --showdots");

----------------------------------------------------------
#!/usr/bin/perl -w
use strict;
use Mail::IMAPClient;
use Shell;
use Env qw(HOME);
use Getopt::Long;

use File::Temp qw/ tempfile tempdir /;

my $imapserver = "exchangeserver.allaboutexchange.net";

# set to 1 to enable imapclient debugging
my $debug = 0;

# set to 1 if running under cron (disables output)
my $cron = 0;

my $filename;
my $fh;

my %options =
(
uid => "username",
pwd => "password"
);

my $cmdsts = GetOptions ("uid=s" => \$options{uid}, "pwd=s" =>
\$options{pwd});

if (!$options {uid}) { die "[SPAMASSASSIN] uid not set
(-uid=username)\n"; }
if (!$options {pwd}) { die "[SPAMASSASSIN] pwd not set
(-pwd=password)\n"; }

my $uid = $options{uid};
my $pwd = $options{pwd};

# login to imap server
my $imap = Mail::IMAPClient->new (Server=>$imapserver, User=>$uid,
Password=>$pwd, Debug=>$debug)
or die "Can't connect to $uid\@$imapserver: $@ $\n";

if ($imap)
{
my $count;

# Deal with spam first
learn_mail ($HOME."/spam/", ".spam", "This is SPAM/", 1, "--spam --showdots");

# Now deal with ham
learn_mail ($HOME."/ham/", ".ham", "Legitimate Email/", 1, "--ham --showdots");

}
else
{
die "[SPAMASSASSIN] Unable to logon to IMAP mail account!
$options{uid}\n";
}

exit;

#
# read and learn mail from imap server
#
# arguments
# $dir directory to place retrieved messages in
# $ext file extension to use on retrieved messages
# $folder imap folder name on server
# $shared 0 if imap folder is in users mailbox
# 1 if imap folder is in shared name space or
# $sa_args additional arguments to specify to sa-learn
# (e.g. --spam or --ham)
#
sub learn_mail {
my $dir = shift (@_);
my $ext = shift (@_);
my $folder = shift (@_);
my $shared = shift (@_);
my $sa_args = shift (@_);

my $count = 0;

# tidy up directory before run
clear_directory ($dir, $ext);

# read mail from server
$count = read_mail ($dir, $ext, $folder, $shared);
if ($count > 0)
{
# learn about mail
sa_learn ($dir, $ext, $sa_args);

# tidy up files after sa-learn is called
clear_directory ($dir, $ext);
}
}


#
# reads mail from an imap folder and saves in a local directory
#
# arguments
# $dir directory to place retrieved messages in
# $ext file extension to use on retrieved messages
# $folder imap folder name on server
# $shared 0 if imap folder is in users mailbox
# 1 if imap folder is in shared name space or
sub read_mail {
my $dir = shift (@_);
my $ext = shift (@_);
my $folder = shift (@_);
my $shared = shift (@_);
my $count = 0;
my $target = "";

if ($shared)
{
# use a shared public folder instead
my ($prefix, $sep) = @{$imap->namespace->[2][0]}
or die "Can't get shared folder namespace or seperator: $@\n";

$target = $prefix.
($prefix =~ /\Q$sep\E$/ || $folder =~ /^\Q$sep/ ? "" : $sep).
$folder;
}
else { $target = $folder; }

$imap->select ($target) or die "Cannot select $target: $@\n";

# If a shared public folder is required uncomment the following
# lines and comment out the previous $imap->select line

# read through all messages
my @msgs = $imap->search("ALL");
foreach my $msg (@msgs)
{
($fh, $filename) = tempfile (SUFFIX => $ext, DIR => $dir);
$imap->message_to_file ($fh, $msg);
close $fh;
$count++;
}

if ($cron == 0) { print "Retrieved $count messages from $target\n"; }

return $count;
}

#
# Removes files in directory $dir with extension $ext
#
sub clear_directory{
my $dir = shift (@_);
my $ext = shift (@_);

opendir (DIR, $dir) or die "Couldn't open dir: $dir\n";
my @files = readdir (DIR);
close (DIR);

for (my $i = 0; $i <= $#files; $i++ ) { if ($files[$i] =~ /.*?$ext$/) { unlink ($dir.$files[$i]); } } } # # execute sa-learn command # sub sa_learn { my $dir = shift (@_); my $ext = shift (@_); my $type = shift (@_); my $learncmd = "/usr/bin/sa-learn ".$type." --dir ".$dir; if ($cron == 0) { $learncmd .= " --showdots"; } else { $learncmd .= " > /dev/null 2>&1"; }

#
# Run sa-learn script on spam directory
#
my $sh = Shell->new;
my @args = ($learncmd);

system (@args) == 0 or die "system @args failed: $?";
}

----------------------------------------------------------
I am very happy with the spam filtering results after implementing this.

Configure Mailbox Recipient Policy to Work with GFI MailArchiver

Once the GFI MailArchiver is up and running, all the new incoming and outgoing emails will be archived (if you have configured it to do so). What about the old emails prior the installation of GFI MailArchiver? The answer is to import all the old emails to a database or databases. The GFI MailArchiver for Exchange Manual details all the procedure from configuring the GFI MailArchiver Import Service to how to use the GFI PST-Exchange Email Export wizard etc.

However, the importation of old emails does not purge the emails in the current Exchange databases. It simply makes a copy of the email and put it into the GFI databases.

Once you have imported the old emails to the GFI MailArchiver databases, you need to create Mailbox manager Recipient polities to delete the old emails.

• In System manager, expand the Recipients node, and the select Recipient Policies.
• Right click Recipient Policies, point to New, and then click Recipient Policy.
• In the new Policy dialog box, select the Mailbox Manager Settings check box and then click on OK.
• In the Name field, type a name for the recipient policy.



• Click on Modify button, and you can select the recipient types that you want the new policy to apply to.
• Click on mailbox manager Settings (Policy) tag. You can specify the Age Limit to serve the purging old email purpose.
• Click OK to finish.

Remember, before you apply this mailbox Recipient Policy, make sure you get the permission from your boss and let all the users know what is going to happen, as they will need to log to the web page to access their old email, but not from their email client software.

Archive Exchange Email by Using GFI MailArchiver

Archiving email can be a challenge many companies and organisations face as email volumes increase everyday. Treating the email system as a storage repository (sending each other large attachments and keep them in the email system) by many end users certainly does not help the situation. You keep educating your staff members to delete the old old old emails they never bother to read again or simply empty the Deleted Items on some occasions, and everyone is still requesting to increase his/her email quota. Instructions have been given out to users, and the Desktop Support team have been trained to help users to archive old emails to PST files, but the store databases size still keeps growing. Also, there could be a policy in place to keep everybody’s email for five years. It is time for you to find a solution to archive the email from your end. One option is to archive the email to PST files, but managing the PST files can be another challenging task to do.

GFI MailArchiver can be a corporate email archiving solution enabling the email to be archived to a MS SQL database or an NTFS drive (with GFI MailArchiver version 4). It also provides users a web interface to check the archived email.

End users can log on to the web page by their AD accounts. They can browse and search the archived email (Now sitting in a different database).

Figure 1


Figure2

GFI MailArchiver also comes handy when you need to recover a single email. No matter it is out of the deletion retention period or deleted while holding the shift key. As long as the email is archived, you can recover it to the current email box or any email box like a charm.


Figure3

You also manage and configure GFI MailArchiver through the web interface, but it provides you more options than a normal user after you log in.

Figure 4

You can decide what to archive, set up archive store databases and archived email retention period, assign different access level to different groups and etc.

I will talk about how to set up the Mailbox Manager Recipient Policy to work with the GFI MailArchiver in my next post.

Distribution List Migration Tool – A Simple VB Program

In article Export Members of a Distribution List (10 March, 2007), I talked about two ways to export a Distribution List members to a file. However, both ways do not export people’s email addresses. I have written a VB program in MS Access which is used in conjunction with the tools such as DSGET, LDIFDE to actually migrate the members’ email addresses to a Distribution List to a TXT or a CSV file.

First, I use DSGET or LDIFDE export the members of the Distribution List to a TXT file and then import it to a Table in Access. It should look like this. (Figure 1)


Figure 1

Then I have included this piece of program in a module.(Sorry about the indenting. It does not work very well here.)


Sub dlmigration()
Dim fs
Dim fsOut
Dim i
Dim strFile
Dim oAccount
Dim arrFields
Dim strFull
Dim strLocation
Dim strUserName
Dim intLocationLength
Dim dbs As Database
Dim strLDAP
Dim intNameLength
Dim strOutputName
Dim Output

Dim rsQuerySQL As Recordset
Dim adQuerySQL
Set dbs = CurrentDb

adQuerySQL = "Select * From WildlifeDL"
Set rsQuerySQL = CurrentDb.OpenRecordset(adQuerySQL, dbOpenDynaset)

strFile = "c:\temp\outdata.txt"
Set fs = CreateObject("Scripting.FileSystemObject")
Set fsOut = fs.OpenTextFile(strFile, ForWriting, True)

Do Until rsQuerySQL.EOF Or rsQuerySQL.BOF

strFull = rsQuerySQL("Name")
arrFields = Split(strFull, ",")
strUserName = arrFields(0)

intNameLength = Len(strUserName)
intNameLength = intNameLength - 4
strOutputName = Right(strUserName, intNameLength)

For i = 1 To UBound(arrFields)
strLocation = strLocation & arrFields(i) & ","
Next

intLocationLength = Len(strLocation)
intLocationLength = intLocationLength - 1
strLocation = Left(strLocation, intLocationLength)

strLDAP = strUserName & "," & strLocation

Set oAccount = GetObject("LDAP://" & strLDAP & "")


Output = strOutputName & ": " & oAccount.EmailAddress
fsOut.WriteLine (Output)
strUserName = ""
strLocation = ""

rsQuerySQL.MoveNext
Loop
On Error Resume Next

fsOut.Close
Err.Clear

rsQuerySQL.Close
Set rsQuerySQL = Nothing
End Sub

Then I get this output TXT file. (Figure 2)


Figure 2

It has saved me a lot of hair tearing time.

Create Contact Objects in AD in Batch with a Simple VB Script

I have come across a few occasions that I need to create a list of Contact objects for external email addresses in our Active Directory. Most of the time, I ask them to give me the name list in the electronic form. I would then export it to a table to my Access database. I have written a few lines of VB code in this Access database. What it does is that it reads the information from the table and creates Contact objects in the AD.

Here is the code. (Sorry about the indenting. It does not work very well here.)

Private Sub cmdAddContact_Click()

Dim objNewContact ' New Contact to create.
Dim objADAMPath ' Active Directory Application Mode - needed for binding to AD
Dim sPath

sPath =”LDAP:// ou=AddressBook,dc=allaboutexchange, dc=net" ‘where I put all the new Contacts

Dim fName
Dim lName
Dim email
Dim displayname

Dim rsAdContact As Recordset
Dim adContactSQL
Set dbs = CurrentDb

adContactSQL = "Select * From ContactList" ‘ContactList is the table name
Set rsAdContact = CurrentDb.OpenRecordset(adContactSQL, dbOpenDynaset)

Do Until rsAdContact.EOF Or rsAdContact.BOF

fName = rsAdContact("FirstName")
lName = rsAdContact("LastName")
displyname = rsAdContact("DisplayName")
email = rsAdContact("Email1Address")

Set objADAMPath = GetObject(sPath)
sCN = fName & " " & lName & "."
Set objNewContact = objADAMPath.CREATE("Contact", "CN=" & sCN)
If lName <> "" And lName <> vbNullString Then
objNewContact.Put "sn", lName
End If

If fName <> "" And fName <> vbNullString Then
objNewContact.Put "givenName", fName
End If

If email <> "" And email <> vbNullString Then
objNewContact.Put "mail", email
End If

If displyname <> "" And displyname <> vbNullString Then
objNewContact.Put "mailNickname", displyname
End If

If displyname <> "" And displyname <> vbNullString Then
objNewContact.Put "displayName", displyname 'This appears in the GAL
End If

If email <> "" And email <> vbNullString Then
objNewContact.Put "targetAddress", "SMTP:" & email
End If

objNewContact.SetInfo
objNewContact.SetInfo
rsAdContact.MoveNext
Loop

Set objNewContact = Nothing
Set objADAMPath = Nothing

End Sub

Just one more thing, when you run this Access application, you need to have the domain administrator permission to run since you are creating new AD objects.

Fight Spam with GFI MailEssentials

GFI MailEssentials is an effective anti-spam software package to fight against spam emails. It is fairly simple to set up and after tweaking and training it, it pretty much runs by itself. What I am going to talk about in this article is the feature which allows users to tune the Bayesian engine and blacklist spam emails via Outlook Public Folders.

Open GFI MailEssentials and you will see the filtering functions it provides listed on the left pane. (Figure 1)


Figure 1

From the list you can see GFI MailEssentials uses several filtering technologies to determine what is considered as spam and what is not, including anti-phishing, whitelist, DNS blacklist, custom blacklist, keyword checking, Bayesian analysis etc. Click on one of them and you can see detailed configuration options. (Figure 2)


Figure 2

You can choose to delete the spam or simply move it to the Junk Email folder.

GFI MailEssentials uses Bayesian technology to analyze the content of each message based on certain mathematical rules to decide if the mail is spam. It learns from the outbound email and adapts itself over time by learning about new spam and new valid email. For this feature to work efficiently, you need to train the Bayesian engine. After training it for a while, it results in a 98% spam detection rate.

GFI MailEssentials adds GFI AntiSpam Folders to client’s Outlook Public Folder. When a client receives spam emails in the inbox, he/she can simply drag the spam to This is spam email or Add to blacklist sub-folders. (Figure 3)


Figure 3

After GFI MailEssentials has processed the spam emails, it will put them to the Processed sub-folder.

You can check the detailed spam email detection rate from GFI MailEssentials reporters. (Figure 4, Figure 5, Figure 6)


Figure 4



Figure 5


Figure 6

It is as simple as that.

Export Members of a Distribution List

There are different ways to export members of a Distribution List. You can buy commercial tools, use CSVDE, LDIFDE, DSGET tools or even write you own scripts.

I have found using Outlook itself to retrieve DL members is simple and fast. If you want to directly export the members of a DL to a csv file, LDIFDE is also a good tool to use.
Open Outlook 2003 and create a new email with the address of the DL. Right click on the DL appearing in the address field, and choose Expand DL. Or click on the + sign in front of the DL address. (Figure 1)


Figure 1
If the DL does not expand to show members, you need to unhide the membership in Active Directory Users and Computers snap-in. Right-click the DL, and click Exchange Tasks. Click Next and Unhide Membership,
Then you can copy all the members to a csv or txt file.

LDIFDE is also simple to use. The syntax required looks like this:

ldifde -f export.ldf -d "cn=\distribution group DN" -l member -s server

-f specifies the file name to save the data to.
-d specifies the RootDN (basically where the search starts).
-l specifies what parameter you are interested in (in this case we are looking at the 'member' attribute.)
-s specifies which Active Directory Domain Controller to use.

On a Domain Controller server or your Exchange server, open Command prompt and type in

ldifde -f c:\temp\wildlifeDL.csv –d “cn=\wildlife, ou=Distribution Lists, dc= allabboutexchange, dc=net” –l member

Now go to c:\temp and get your file.

Exchange 2003 Disaster Recovery by Using Symantec Backup Exec

Before Christmas, an electrician was trying to fix the UPS in our server room. Accidentally, he took the power out and brought down all the servers. We certainly did not appreciate what he did. After the power resumed, I found three stores in a storage group would not mount. Then our service desk was flooded with phone calls. This time, it was a bit hard to send out a notice via email. Here is how you should recover your exchange server if you experience a scenario like this.

1. Confirm all you previous online backups are successful, including last Full backup and then Differential or Incremental backup(s).

2. Run a offline backup of the corrupt database and all transaction logs. This may take a while depending on how big the store is.

3. Ran eseutil and check the State of the database. In this scenario, it should show as Dirty Shutdown.

4. Do not try to run a repair of the database if you have successful backups. Run a restore instead.

This is how you restore the store database from the tape by using Symantec Backup Exec 10.1.

a. On the Exchange server
--- Open Exchange System Manager
--- Right click on the corrupt store and go to Properties
--- Click on Database tag and tick This database can be overwritten by a restore (Figure 1)


Figure 1

b. Open Symantec Backup Exec and choose Restore
--- In Source -- > Selection, choose the store you need to restore
--- In Settings -- > Microsoft Exchange, tick No loss restore (do not delete existing transaction logs) under Exchange 2000 and Exchange 2003
--- Choose a temporary location for log files
--- If it is the last restore set, tick Commit after restore completes. This will allow the transaction logs to be replayed after restore. (In our work scenario, we run a full backup once on the weekend and differential backup every week day. Since the transaction logs will be purged after each full backup, the differential backup will only backup the transaction logs. Therefore, the full backup is the last restore set in this case.) (Figure 2)


Figure 2

c. On the Exchange server
--- Open Exchange System Manager
--- Right click on the corrupt store and go to Properties
--- Click on Database tag and un-tick This database can be overwritten by a restore

d. Re-mount the store.

Now the store should mount. This is not finished yet. You should now run a new online full backup.

Be noted, backing up the corrupt database, restoring and replaying transactions all take time. You should have a detailed disaster recovery plan and always run a fire drill on you test environment once a while.

This outage affected three out of five stores. Some staff members asked why some people could still send and receive email but they could not. Some people even thought it happened because their work was less important than others. Why they think like that? I don't know. Even you explained to them what had happened, they still gave you a suspicious look. My suggestion to you is, always think of something to say before this kind of thing happens.

Good Luck in recovering.

My Very First Message

This is my very first blog on this website. I have been doing Exchange design and administration for quite a while. I would like to share my Exchange knowledge and exprience with others. Also, I hope this blog will be a bridge connecting me and other Exchange users.