SAGATOR

(c) 2003-2008 Jan ONDREJ (SAL) <ondrejj(at)salstar.sk>

 This program is free software; you can redistribute it and/or modify
 it under the terms of the GNU General Public License as published by
 the Free Software Foundation; either version 2 of the License, or
 (at your option) any later version.

Contents
  1. Introduction
    1.1. Features
    1.2. Requirements
    1.3. Benchmark
  2. Installation
    2.1. Getting started
    2.2. Creating chrooted environment
    2.3. Upgrading from previous versions
  3. Sagator configuration
    3.1. Service configuration
    3.2. Scanner configuration
  4. Detailed scanner configuration
  5. SMTPd configuration
    5.1. Postfix configuration
    5.2. Sendmail configuration
    5.3. Another smtpd configuration
  6. Antivir/Antispam installation
    6.1. ClamAV installation
  7. Staring sagator
  8. Testing
  9. Scanning mailboxes
  10. Retunrning badly quarantined mail to mailq/user mailbox
  11. Removing old files from quarantine
  12. Reporting summaries to users
  13. Graphs
    13.1. RRDtool
    13.2. MRTG
  14. Web quarantine
  15. Done

1. Introduction

This program is an email/web antivirus/antispam gateway. It is an interface
to the postfix or sendmail (or any other smtpd), which runs antivirus
and/or spamchecker. It also can be used as http proxy with
filtering abilities. It's modular architecture can use any
combination of antivirus/spamchecker according to configuration.

It currently supports clamav, nod32d, AVGlinux, sophie, trophie,
symantec antivirus scan engine (via ICAP), Bitdefender,
spamassassin, bogofilter and QuickSpamFilter.
It has some internal checkers (string_scanner and regexp_scanner).
Sagator can parse MIME mails and decompress archives, if it is configured so.

---------------------------------------------------------------------

1.1. Features

  - simple chroot support
  - modular antivirus/spamchecker support
     - currently supports clamav, nod, sophie, trophie, AVGlinux,
         ICAP protocol (Symantec Antivirus), ...
     - any other antivirus/antispam can be implemented
     - spamaassassin support
  - database support
     - SQL logging
     - dynamic scanner (antivirus/antispam) configuration
  - nice RRDtool or MRTG statistics
  - daily reports for users
  - web quarantine accessible for all users
  - smtp policy service (greylist)
  - mailbox/maildir scanning and cleaning
  - easy installation and configuration
  - SMTP and LMTP protocol support
     - LMTP can be used to use different scanners for each recipient
  - you can use simple scanners to determine files which you don't need:
     - attach_name - define a virus by mime extension
     - file_type - define a virus by file type (content is checked)
     - max_file_size - define max. attachment or max. file size
     - string_scan - define a virus by string
     - regexp_scan - define a virus by regular expression
  - you don't need any perl modules or any other modules, only python2
  - you can return any quarantined mail to mailq/user mailbox

---------------------------------------------------------------------

1.2. Requirements

  - python2 (python 2.3 or higher is recomended)
  - an supported antivir/antispam (clamav, spamassassin, ...)
  - SMTP daemon (one from following list)
    - postfix 1.1.11 (or may be later or newer)
    - sendmail with milter and python milters
    - any other SMTPd with special configuration.

---------------------------------------------------------------------

1.3. Benchmark
  Hardware & Software:
    Athlon 2000+, 
    Fedora Core 2
    clamav 0.80rc4 (used as libclam)
  Pattern:
    90000 emails processed in 312 minutes
    speed ~= 4.8 emails/second

---------------------------------------------------------------------

2. Installation

  You can skip this part if you are using rpm or deb packages.
They can be downloaded from SAGATOR's homepage or via a package
installer (yum, apt-get or similiar).

2.1. Getting started

  Unpack the package. Go into main directory and run:
    ./configure
    make
    make install

2.2. Creating chrooted environment.

  Create user vscan:
    useradd vscan -M -s /sbin/nologin

  Use script:
    ./scripts/mkchroot.sh /some/where/new/root/
The directory can be for example: "/var/spool/vscan".
This script creates a small copy of your files, which are required
for antivirus, unpacker, ...

2.3. Upgrading from previous versions

  Upgrade information is available only for sagator 0.5.0 and higher.
After upgrade run "updatecfg.py" script from your installation directory.
This script updates your configuration files automatically. Not all changes
can be fixed automatically, but many of them can.

- upgrading to SAGATOR 1.0.0
  Please check, if you are using an obsolete scanner or service. All
obsolete scanners/services have been removed. If your configuration has
been not fixed automatically by updatecfg script, do it manually.

- upgrading to SAGATOR 0.8.0
  There are many of changes in scanner names. For detailed description see
ChangeLog file, or run updatecfg.py script.
  You need to move '/tmp/status' to '/var/lib/sagator/status' in your CHROOT
directory, or define '/tmp/status' manually in collector() service.
  Also RRD template has been changed. There is no way to update your
sagator.rrd file now. You must remove or rename this file and a new rrd
file will be created automatically.

- upgrading to SAGATOR 0.5.0
  In this version you need to add a SRV array in sagator's configuration.
If you are using COLLECTOR_SERVER and BIND_ADDR variables, you need to add:
  SRV=[
    collector(COLLECTOR_SERVER[0],COLLECTOR_SERVER[1]),
    smtpd(SCANNERS,BIND_ADDR[0],BIND_ADDR[1])
  ]

3. Sagator configuration

3.1. Service configuration

  Services are subprograms to communicate with an SMTP daemon or with other
client. For more help see Services.txt. Use SRV variable in config.py.

  SRV varialble is a python array, defined as:

    SRV=[ service1, service2, ... ]

  Services are defined in separate files, you must include it into
config.py. You can do it with this command:
                                                                                
  from srv import *

Examples:

  a) smtpd service (good for postfix and useable for any smtpd) with
     combination for collector (for statistics):
       SRV=[
         collector('0.0.0.0',28),
         smtpd(SCANNERS,'127.0.0.1',27)
       ]

  b) lmtpd service (if you want different scanners for each recipient)
       SRV=[
         collector('0.0.0.0', 28),
         lmtpd(LMTP_SCANNERS, '0.0.0.0', 27)
       ]

  c) milter service (for sendmail) with combination with statistics
     collector:
       SRV=[
         collector('0.0.0.0',28),
         milter(SCANNERS,"sagator","inet:3333@127.0.0.1")
       ]

  d) http sanning service:
       SRV=[
         collector('0.0.0.0',28),
         http_proxy(SCANNERS,'0.0.0.0',3128)
       ]

3.2. Scanner configuration

  Edit config.py. There are some parameters, described in config.py file.
Scanner configuration:
  Use SCANNERS variable in config.py. You can use more scanners.
If one of them found a virus, scanning is stopped. If one of them
fails, an error is raised.

  SCANNERS variable is a python array, defined as:

    SCANNERS=[ scanner1, scanner2, ... ]

  Scanners are defined in separate files, you must include it into
config.py. You can do it with:

  from scanners import *

Some examples:
  a) Use only clamscan binary to scan for viruses:
       SCANNERS=[
         clamav(['/usr/bin/clamscan','--stdout','--infected',
                 '--disable-summary','-r','--mbox'])
       ]

  b) Adding spamassassin tester:
       SCANNERS=SCANNERS+[ spamassassind(['localhost',783]) ]

  c) If you want to add some special stuff, like scan for
     names of attachments, and if you don't want any attachments,
     use mime_ext scanner (YOU NEED PYTHON 2.2 FOR THIS@):
       EXECUTABLES='(\.exe|\.COM|\.com|\.COM|\.pif|\.PIF|\.lnk|\.LNK|\.scr|\.SCR)'
       SCANNERS=[ mime_ext(EXECUTABLES) ]

There are also subscanners ... they are scanners, which parses emails
and runs another scanner. For example there are parsemail and
decompress scanners.

Example for parsemail:
  I wan't to test parsed files by clamd. Clamd can parse mail,
but I don't want to use its scanner now.

  parsemail(clamd(['localhost',3310]))

4. Detailed scanner configuration

There are three types of scanners:
  - bufferscanner
  - filescanner
  - combination (filescanner+bufferscanner)

Scanners are also divided into:
  - realscanners
     These scanners are real scanners, which scans the buffer/file
     and then returns its status.
  - interscanners
     These scanners are an interface between main scanning engine
     and realscanners. You also can attach interscanner to another
     interscanner.

By default bufferscanner is used. If you wan't to use filescanner,
there is a interscanner named buffer2file().

If you need to know, which scanner is which, look at this table:

  ------------------------------------------------------------------
  | name                  | module       | inter | buffer |  file  |
  ------------------------------------------------------------------
  | report()              | report       |   *   |   *    |   -    |
  | report_recipients()   | report       |   *   |   *    |   -    |
  | quarantine()          | actions      |   *   |   *    |   -    |
  | drop()                | actions      |   *   |   *    |   -    |
  | deliver()             | actions      |   *   |   *    |   -    |
  | deliver_to()          | actions      |   *   |   *    |   -    |
  | rename()              | actions      |   *   |   *    |   -    |
  | time_limit()          | actions      |   *   |   *    |   -    |
  | modify_subject()      | header       |   *   |   *    |   -    |
  | modify_header()       | header       |   *   |   *    |   -    |
  | add_header()          | header       |   *   |   *    |   -    |
  | remove_headers()      | header       |   -   |   *    |   -    |
  | file2buffer()         | file         |   *   |  -/out |   in/- |
  | buffer2file()         | file         |   *   | in(m)  |  -/out |
  | buffer2mbox()         | file         |   *   | in/-   |  -/out |
  | match_all()           | match        |   *   | in/out |   -    |
  | alternatives()        | match        |   *   | in/out |   -    |
  | match_any()           | match        |   *   | in/out |   -    |
  | nothing()             | match        |   *   | in/out |   -    |
  | recover()             | match        |   *   | in/out |   -    |
  | parsemail()           | parsemail    |   *   | in/out |   -    |
  | attach_name()         | parsemail    |   -   |   *    |   -    |
  | mimeparse()           | mimeparse    |   *   | in/out |   -    |
  | mime_ext()            | mimeparse    |   -   |   *    |   -    |
  | decompress()          | decompress   |   *   |   -    | in/out |
  | cache()               | cache        |   *   | in/out |   -    |
  | log()                 | logger       |   *   | in/out |   -    |
  | log_syslog()          | logger       |   *   | in/out |   -    |
  | log_sql()             | logger       |   *   | in/out |   -    |
  | sql_find()            | condition    |   *   | in/out |   -    |
  | regexp_find()         | condition    |   *   | in/out |   -    |
  | check_level()         | condition    |   *   | in/out |   -    |
  | stat()                | stats        |   *   | in/out |   -    |
  ------------------------------------------------------------------
  | clamscan()            | clamav       |   -   |   -    |   *    |
  | clamd()               | clamav       |   -   |   *    |   -    |
  | libclam()             | clamav       |   -   |   -    |   *    |
  | avgd()                | avg          |   -   |   -    |   *    |
  | bdc()                 | bdc          |   -   |   -    |   *    |
  | kav()                 | kav          |   -   |   -    |   *    |
  | kavclient()           | kav          |   -   |   -    |   *    |
  | nod2()                | nod          |   -   |   -    |   *    |
  | nod2dazuko()          | nod          |   -   |   -    |   *    |
  | nod2pac()             | nod          |   -   |   -    |   *    |
  | esets()               | nod          |   -   |   -    |   *    |
  | dazuko()              | dazuko       |   -   |   -    |   *    |
  | savse()               | icap         |   -   |   -    |   *    |
  | sophie()              | sophie       |   -   |   -    |   *    |
  | trophie()             | sophie       |   -   |   -    |   *    |
  | const()               | basic        |   -   |   *    |   *    |
  | smtp_comm()           | basic        |   -   |   *    |   *    |
  | regexp_scan()         | basic        |   -   |   *    |   -    |
  | string_scan()         | basic        |   -   |   *    |   -    |
  | sender_regexp()       | basic        |   -   |   *    |   -    |
  | max_file_size()       | basic        |   -   |   *    |   *    |
  | file_type()           | basic        |   -   |   *    |   *    |
  | file_magic()          | basic        |   -   |   *    |   *    |
  | sanitize()            | sanitize     |   -   |   *    |   -    |
  | cmd()                 | cmd          |   -   |   -    |   *    |
  | cmd_bdc()             | cmd          |   -   |   -    |   *    |
  | cmd_clamav()          | cmd          |   -   |   -    |   *    |
  | cmd_drweb()           | cmd          |   -   |   -    |   *    |
  | cmd_fprot()           | cmd          |   -   |   -    |   *    |
  | cmd_kavscanner()      | cmd          |   -   |   -    |   *    |
  | cmd_trendmicro()      | cmd          |   -   |   -    |   *    |
  | cmd_uvscan()          | cmd          |   -   |   -    |   *    |
  | cmd_vbuster()         | cmd          |   -   |   -    |   *    |
  | filter()              | filter       |   -   |   *    |   -    |
  ------------------------------------------------------------------
  | spamassassin()        | spamassassin |   -   |   *    |   -    |
  | spamassassind()       | spamassassin |   -   |   *    |   -    |
  | bogofilter()          | bogofilter   |   -   |   *    |   -    |
  | qsf()                 | qsf          |   -   |   *    |   -    |
  | dspam()               | dspam        |   -   |   *    |   -    |
  ------------------------------------------------------------------

  For detailed scanner description see Scanners.txt.

There are some opearators, which you can use over scanners:

  ------------------------------------------------------------
  | operator |  "+"  |  "-"  |  "*"  |  "/"  |  "|"  |  "&"  |
  | S1 | S2  |       |       |       |       |       |       |
  ------------------------------------------------------------
  | OK   OK  | L1+L2   L1-L2   L1*L2   L1/L2    L1      (1)  |
  | F    OK  | Fail    Fail    Fail    Fail     L2     Fail  |
  | OK   F   | Fail    Fail    Fail    Fail     L1     Fail  |
  | F    F   | Fail    Fail    Fail    Fail    Fail    Fail  |
  ------------------------------------------------------------

  ------------------------------------------------------------
  | operator |  "<"  |  ">"  |  "="  |  "<=" |  ">=" |  "!=" |
  | S1 | S2  |       |       |       |       |       |       |
  ------------------------------------------------------------
  | L1 < L2  | 1.0   | 0.0 * | 0.0 * | 1.0   | 0.0 * | 1.0   |
  | L1 = L2  | 0.0 * | 0.0 * | 1.0   | 1.0   | 1.0   | 0.0 * |
  | L1 > L2  | 0.0 * | 1.0   | 0.0 * | 0.0 * | 1.0   | 1.0   |
  ------------------------------------------------------------

  S1 - status of first scanner
  S2 - status of secondary scanner
  L1 - returned level of first scanner (on left side of operator)
  L2 - returned level of second scanner (on right side of operator)
  NA - not available, can't occur
  (1) - if L1>=1.0: L1*L2; else 0.0
  *  - virus name is cleaned (set to empty string "")

When you are using interscanners, you can't attach scanners of
wrong type. This means, that you can't attach a scanner with
input of buffer to scanner, which has only file output.

For example you can use:
  parsemail(string_scan(...))
but can't:
  parsemail(decompress(...))

5. SMTPd configuration

5.1. Postfix configuration

# /etc/postfix/main.cf
mynetworks = 127.0.0.0/8, ...
content_filter = smtp:[127.0.0.1]:27
# for lmtpd() service use: content_filter = lmtp:[127.0.0.1]:27

# /etc/postfix/master.cf
# Local smtpd without scanning
127.0.0.1:26    inet n - n - 30 smtpd
  -o content_filter=
  -o myhostname=sagator.mydomain.sk
  -o local_recipient_maps=  -o relay_recipient_maps=
  -o mynetworks=127.0.0.0/8  -o mynetworks_style=host
  -o smtpd_restriction_classes=  -o smtpd_client_restrictions=
  -o smtpd_helo_restrictions=  -o smtpd_sender_restrictions=
  -o smtpd_recipient_restrictions=permit_mynetworks,reject
  -o receive_override_options=no_unknown_recipient_checks,no_header_body_checks

Myhostname can't be equal with your hostname, otherwise postfix
detects a loop.

  Add sagator to startup scripts. In this package there is
a sagator.init scipt, which can be used in /etc/init.d/ .
You can start it also manually.

5.2. Sendmail configuration

  As first step, please install required packages:
    sendmail with milter support
    python milters (http://www.bmsi.com/python/milter.html)

Configure your sendmail to use sagator's milter. Add these line into
  /etc/mail/sendmail.cf:
    O InputMailFilters=sagator
    Xsagator, S=inet:3333@127.0.0.1, F=T
For more information see milter documentation.

  Configure sagator to run as milter filter. Add this service to SRV in
sagator.conf:
  milter("sagator","inet:3333@127.0.0.1")

Limitations:
  Milter can't modify Subject and other headers, only can add some new.

5.3 Another smtpd configuration.

  For other smtpd, you can use sagator as main smtpd on port 25.
Configure your smtpd on another port as 25. Then configure
sagator to run on port 25 on your IP.

For example:
  Configure smtpd on port 26 on localhost and on your internal ip on port 25.
Allow relaying from your site on internal IP, but don't forget to deny
relaying on localhost:26 or from 127.0.0.0/8.
Use following configuration for sagator:

  SMTP_SERVER=('127.0.0.1',26)
  SRV = [ smtpd('YourExternalIP',25) ]

6. Antivir/Antispam installation

6.1. ClamAV installation

  Download and install clamav - clam antivirus.
URL: http://www.clamav.net/. You will also need some
decompressors, like unrar, unace, unarj, zoo, lha, tar, ...

  If you are using clamd, do not forget to start it in chroot,
and set these parameters in chrooted clamav.conf:
  DatabaseDirectory /var/lib/clamav
  LocalSocket /var/run/clamav/clamd.sock
  StreamSaveToDisk # recomended
  ScanMail # required
  ScanArchive # recomended
If you are using chroot, set in /etc/freshclam.conf:
  DatabaseDirectory /var/spool/vscan/var/lib/clamav
where /var/spool/vscan is your chroot. If you do this, freshclam
will update clamav database in your chroot.

7. Staring sagator

  There are init scripts in scripts/ directory named sagator.init.rhfc or
sagator.init.suse. Copy one of them into your init.d directory and set it.
For RedHat based system do:
  cp scripts/sagator.init.rhfc /etc/init.d/sagator
Then edit file /etc/sysconfig/sagator:
  CHROOT="/var/spool/vscan"

Warning: You must have your antivirus installed in chroot.

8. Testing

  There some test scripts. Go into test directory and try:

    ./smtptest localhost 25 Eicar
    ./smtptest localhost 25 spam
    ...

  With these scripts you can also test your sagator daemon,
and you don't need to configure content_filter in postfix.
You can test it without real traffic. I recomend for you to
start sagator as command line program, and see its reports first.
Only if it works really, then use it as daemon.

9. Scanning mailboxes

  In some case an virus can pass over sagator (for example, if your
antivirus is not actual). If you detect this, you need to find viruses
in user mailboxes. There is an program named sgscan. If scans over
mailboxes and return finded viruses/spams. You can use:

  sgscan /var/spool/mail/*

  If you can scan only for viruses, you can use --av-only command
line parameter. Another parameter is --clean, if you want to create
clean mailboxes as /var/spool/mail/*.clean.
For renaming these mailboxes to real names use a shell script:
  for i in *.clean; do mv $i `echo $i | sed 's/\.clean$//'`; done
Warning: Don't use this script, if an user of an SMTPd has access
         to mailbox!!!

10. Retunrning badly quarantined mail to mailq/user mailbox

  If by chance is any mail quarantined, which should not, you can return
to queue or deliver to user mailbox by running command:

  sqback quarantined_file1 [quarantined_file2 [...]]

This can be done also via netcat program:

  nc localhost SMTP_PORT_WITHOUT_FILTER < .../quarantine/filename

11. Removing old files from quarantine

  Use tmpwatch. For example, if you want to delete older files,
than aprox. one month, use crontab command in /etc/crontab:

  0 0 * * * vscan /usr/sbin/tmpwatch 768 /var/spool/vscan/tmp/quarantine

or same line for vscan user's crontab:

  0 0 * * * /usr/sbin/tmpwatch 768 /var/spool/vscan/tmp/quarantine

Replace "/var/spool/vscan/tmp/quarantine" with path to quarantine directory.
For more information read "man tmpwatch" manual page.

12. Reporting summaries to users

  Many of emails are simply dropped or rejected and users are not
informed about these actions. If you want send summaries periodically,
add a special log() scanner into your configuration and run
reporter script from cron. Here is an example of SCANNERS configuration:

SCANNERS=[
  log(1,log.SUMMARY_REPORT,
    ... your scanner configuraion follows here ...
  )
]

And also add something similiar into /etc/crontab:

  0 5 * * * vscan /usr/share/sagator/reporter.py

You can define your own custom report. Do this in reporter() service.
For more information see reporter() service documentation.

Example (all parameters are optional, something similiar is default):

reporter(
begin='''\
Subject: Report mail

This is a report of your undelivered emails.
''',
body='''
$DATETIME  Sender: $SENDER [$SENTBY_IP]
                 Status: $VIRNAME $STATUS [$SCANNER_NAME=$LEVEL]
''',
end='''
End of report
''',
include='@my-domain.com$',
exclude='^root@'
)

13. Graphs

  There are two ways, how to use Sagator's statistic graphs.
You can use MRTG or it's newer version RRDtool. It's suggested to use
RRDtool, because it's graphs are very nice, easy to install and can display
more information.

13.1. RRDtool

  RRDtool usage is simplest than MRTG. Make /var/www/mrtg/sagator/rrd
directory into your web server configuration and add following lines
into your /etc/crontab:

# SAGATOR's RRD
*/5 * * * * root /usr/share/sagator/stats.py rrdtool /var/www/mrtg/sagator \
                 > /var/log/sagator/rrd.log 2>&1

  You can see your RRDtool graphs at http://localhost/mrtg/sagator/ .
Rrdtool 1.2 is used by default in sagator 0.8.0 and higher, if you are using
rrdtool 1.0, please change "...stats.py rrdtool" to "...stats.py rrdtool10".

13.2. MRTG

  Make /var/www/mrtg/sagator directory into your web server configuration.
Copy a file named mrtg.cfg to /etc/mrtg/sagator.cfg and add this line
into your /etc/crontab:

# SAGATOR's MRTG
0-59/5 * * * * root /usr/bin/mrtg /etc/mrtg/sagator.cfg

  You can see your MRTG graphs at http://localhost/mrtg/sagator/ .

14. Web quarantine

  This service can be used to access quarantine files. Each user can see,
if some emails have been quarantined, can see it's content, check it again
with a virus/spam scanner and get it or resend as email.

  Requirements: python-kid-0.9.5 or higher is required for this service.

  webq() service requires to log information about quarantined emails
into a SQL database supported by sagator. You can see supported databases in
Databases.txt documentation file.

  If you are using apache on port 80, you can use mod_proxy module to
redirect webq requests to port defined in sagator, for example:
  ProxyPass /webq http://localhost:8008
  ProxyPassReverse /webq http://localhost:8008

  Each user must authenticate to sagator's web quarantine. It is done in
webaccess table. Users must have CRYPT_MD5 password in this table. You
can create admin user by inserting this row:
  INSERT INTO webaccess (email, pass, perms, datetime, lang, showrows)
  VALUES ('admin', '$1$WA6srD5g$h4XRTVZiH/goIEuNzdegC1',
          'A', CURRENT_TIMESTAMP, 'en_US', 50);
This is an admin account with full access to all users! It's default
password is "sagator". If you want regular users, replace 'A' flag by
empty string ''.

15. Done

  Congratulations. You have sucessfully installed sagator. :-)
