Plugins¶

Overview¶

The fuglu core does nothing except receiving mails from postfix and sending them back. All functionality is written in plugins which can be enabled or disabled at will. Fuglu provides plugins for the most common mail filtering requirements, but if some functionality is missing, it is easy to add new plugins without knowing all of fuglu’s internals.

Types of plugins¶

Scanner Plugins¶

Scanner plugins are the most important type of plugins. They do the actual mail content filtering. Plugins are run in the order specified in the plugins= configuration option. Each plugin returns an action for the message:

DUNNO : This plugin decides not to take any final action, continue with the next plugin (this is the most common case)

ACCEPT : Whitelist this message, don’t run any remaining plugins

DELETE : Silently delete this message (The sender will think it has been delivered)

DEFER : Temporary Reject (4xx error), used for error conditions in after-queue mode or things like greylisting in before-queue mode

REJECT : Reject this message, should only be used in before-queue mode (in after-queue mode this would produce a bounce / backscatter)

If one of the plugins returns something other than DUNNO, all remaining scanner plugins are skipped. If all plugins return DUNNO, the message is accepted and re-injected to postfix.

Prepender Plugins¶

Prepender plugins run before the scanner plugins and have the ability to alter the list of scanner plugins to be run. This can be used for example to have different plugins run for incoming or outgoing mails or to skip whitelisted messages.

Appender Plugins¶

Appender plugins are run after the scanner plugins when the message has already been re-injected into postfix (or rejected or deleted or...) so they don’t increase your scanning time . They are mostly used for statistical tasks, updating counters in a database etc.

Plugin configuration¶

Each plugin has its own configuration section in /etc/fuglu/fuglu.conf or any *.conf in /etc/fuglu/conf.d. The section is usually named like the plugin itself.

For example, the Spamassassin Plugin’s Name is SAplugin, so it would search for a [SAPlugin] config section.

Suspect Filters¶

SuspectFilter are special rule files used by many fuglu plugins. Often they define actions a plugin should take based on a message header or suspect tag.

The format is : <property> <regex> <optional argument>

<property> can be any of...

a email header name or header:headername , e.g. Received, To, From, Subject ... also supports ‘*’ as wildcard character
mime:headername to get mime Headers in the message payload eg: mime:Content-Disposition
one of fuglus builtin names to get envelope data:

envelope_from or from_address (the envelope from address)

from_domain (domain part of envelope_from)

envelope_to or to_address (envelope to address)

to_domain (domain part of envelope_to)

a message Tag prepended by the @ symbol, eg. @incomingport
body:raw to match the the decoded message body (only applies to text/* parts)
body:stripped or just body to match the the message body (only applies to text/* parts), with stripped tags and newlines replaced with space (similar to SpamAssassin body rules)
body:full to match the full body

<regex> is a standard python regular expression applied to <property>. if the regex is not enclosed by forward slashes, be sure not to include any whitespace in the regex, it must be one continous string. To match whitespace, use \s. all regexes that are not enclosed in forward slashes are automatically case insensitive and support multiple lines (re.DOTALL is enabled, so the newline character is matched by a dot). with slashes, the regex flags can be defined manually, eg.

Subject /hello world/i match hello world, case insensitive

<optional argument> depends on the plugin that reads this header filter file, some don’t need arguments at all. Please refer to the plugin documentation.

Filter files are automatically reloaded if you change anything, you don’t need to restart fuglu.

Example:

#normal header test
Subject hello   Hello in the subject!
MIME-Version ^1\.0$ Mime Version is 1.0

#builtin special fields
to_domain       (\.)?fuglu\.org$        Sent to fuglu.org or any subdomain
envelope_from ^sender@example\.com$

#match a tag from a previous plugin
@SAPlugin.report MISSING_HEADER

#wildcard
X-Spam-* .*     a X-Spam-<something> header exists

#decoded body text parts
body    Viagra

#full body
body:full ^--SPAMMY-MIME-BOUNDARY

#mime-headers
mime:Content-Type ^application\/x-msdos-program$

see Debug a suspect filter for info on how to debug suspect filter files on specific messages.

Template Variables¶

Some plugins allow you to create templates (errormessages in bounces, reject reasons, ...). Here is a list of commonly used template variables that should be available in those templates. Plugins may not support all those variables and can add more variables not listed here. If unsure, check the plugin documentation.

${from_address} : Envelope Sender Address

${to_address} : Envelope Recipient Address

${from_domain} : Envelope Sender Domain

${to_domain} : Envelope Recipient Domain

${timestamp} : Unix Timestamp when the message was received

${subject} : Message subject

${date} : Current date

${time} : Current time

${blockinfo} : Reason why this message was blocked (available in the attachment plugin)

As of fuglu 0.8.1, properties as documented in Suspect Filters can be used as well

Plugins included in Fuglu¶

SpamAssassin¶

Plugin: fuglu.plugins.sa.SAPlugin

This plugin passes suspects to a spamassassin daemon.

Prerequisites: SPAMD must be installed and running (not necessarily on the same box as fuglu)

Notes for developers:

if forwardoriginal=False, the message source will be completely replaced with the answer from spamd.

Tags:

reads SAPlugin.skip, (boolean) skips scanning if this is True

reads SAPlugin.tempheader, (text) prepends this text to the scanned message (use this to pass temporary headers to spamassassin which should not be visible in the final message)

sets spam['spamassassin'] (boolean)

sets SAPlugin.spamscore (float) if possible

sets SAPlugin.skipreason (string) if the message was not scanned (fuglu >0.5.0)

sets SAPlugin.report, (string) report from spamd or spamheader (where score was found) depending on forwardoriginal setting

Configuration¶

[SAPlugin]
#how often should fuglu retry the connection before giving up
retries=3

#action if there is a problem (DUNNO, DEFER)
problemaction=DEFER

#reject message template if running in pre-queue mode
rejectmessage=message identified as spam

#enable scanning of messages larger than maxsize. all attachments will be stripped and only headers, plaintext and html part will be scanned. If message is still oversize it will be truncated. Also enable forwardoriginal or truncated version of large messages will be forwarded
strip_oversize=0

#forward the original message or replace the content as returned by spamassassin
#if this is set to True/1/Yes , no spamassassin headers will be visible in the final message.
#"original" in this case means "as passed to spamassassin", eg. if 'scanoriginal' is set to 0 above this will forward the
#message as retreived from previous plugins
forwardoriginal=0

#consult spamassassins(or any other) sql blacklist for messages that are too big for spam checks
#requires the sql extension to be enabled
check_sql_blacklist=0

#sqlalchemy db connect string
sql_blacklist_dbconnectstring=mysql:///localhost/spamassassin

#SQL query to get the blacklist entries for a suspect
#you may use template variables: ${from_address} ${from_domain} ${to_address} ${to_domain}
sql_blacklist_sql=SELECT value FROM userpref WHERE prefid='blacklist_from' AND username in ('$GLOBAL',concat('%',${to_domain}),${to_address})

#maximum size in bytes. larger messages will be skipped
maxsize=256000

#what header does SA set to indicate the spam status
#Note that fuglu requires a standard header template configuration for spamstatus and score extraction
#if 'forwardoriginal' is set to 0
#eg. start with _YESNO_ or _YESNOCAPS_ and contain score=_SCORE_
spamheader=X-Spam-Status

#hostname where spamd runs
host=localhost

#what should we do with high spam (spam score above highspamlevel)
highspamaction=DEFAULTHIGHSPAMACTION

#how long should we wait for an answer from sa
timeout=30

#spamscore threshold to mark a message as high spam
highspamlevel=15

#enable user_prefs in SA. This hands the recipient address over the spamd connection which allows SA to search for configuration overrides
peruserconfig=1

#tcp port number or path to spamd unix socket
port=783

#should we scan the original message as retreived from postfix or scan the current state
#in fuglu (which might have been altered by previous plugins)
#only set this to disabled if you have a custom plugin that adds special headers to the message that will be
#used in spamassassin rules
scanoriginal=1

#what should we do with low spam (eg. detected as spam, but score not over highspamlevel)
lowspamaction=DEFAULTLOWSPAMACTION

Clam AV¶

Plugin: fuglu.plugins.clamav.ClamavPlugin

This plugin passes suspects to a clam daemon.

Actions: This plugin will delete infected messages. If clamd is not reachable or times out, messages can be DEFERRED.

Prerequisites: You must have clamd installed (for performance reasons I recommend it to be on the same box, but this is not absoluely necessary)

Notes for developers:

Tags:

sets virus['ClamAV'] (boolean)

sets ClamavPlugin.virus (list of strings) - virus names found in message

Configuration¶

[ClamavPlugin]
#reject message template if running in pre-queue mode and virusaction=REJECT
rejectmessage=threat detected: ${virusname}

#process timeout
clamscantimeout=30

#hostname where clamd runs
host=localhost

#tcp port number or path to clamd.sock for unix domain sockets
#example /var/lib/clamav/clamd.sock or on ubuntu: /var/run/clamav/clamd.ctl
port=3310

#how often should fuglu retry the connection before giving up
retries=3

#action if there is a problem (DUNNO, DEFER)
problemaction=DEFER

#*EXPERIMENTAL*: fallback to clamscan if clamd is unavailable. YMMV, each scan can take 5-20 seconds and massively increase load on a busy system.
clamscanfallback=0

#the path to clamscan executable
clamscan=/usr/bin/clamscan

#maximum message size, larger messages will not be scanned.
#should match the 'StreamMaxLength' config option in clamd.conf
maxsize=22000000

#socket timeout
timeout=30

#*EXPERIMENTAL*: Perform multiple scans over the same connection. May improve performance on busy systems.
pipelining=0

#action if infection is detected (DUNNO, REJECT, DELETE)
virusaction=DEFAULTVIRUSACTION

Attachment Blocker¶

Plugin: fuglu.plugins.attachment.FiletypePlugin

This plugin checks message attachments. You can configure what filetypes or filenames are allowed to pass through fuglu. If a attachment is not allowed, the message is deleted and the sender receives a bounce error message. The plugin uses the ‘’‘file’‘’ library to identify attachments, so even if a smart sender renames his executable to .txt, fuglu will detect it.

Attachment rules can be defined globally, per domain or per user.

Actions: This plugin will delete messages if they contain blocked attachments.

Prerequisites: You must have the python file or magic module installed. Additionaly, for scanning filenames within rar archives, fuglu needs the python rarfile module.

The attachment configuration files are in /etc/fuglu/rules. You should have two default files there: default-filenames.conf which defines what filenames are allowed and default-filetypes.conf which defines what content types a attachment may have.

For domain rules, create a new file <domainname>-filenames.conf / <domainname>-filetypes.conf , eg. fuglu.org-filenames.conf / fuglu.org-filetypes.conf

For individual user rules, create a new file <useremail>-filenames.conf / <useremail>-filetypes.conf, eg. oli@fuglu.org-filenames.conf / oli@fuglu.org-filetypes.conf

To scan filenames or even file contents within archives (zip, rar), use <...>-archivefilenames.conf and <...>-archivefiletypes.conf.

The format of those files is as follows: Each line should have three parts, seperated by tabs (or any whitespace): <action> <regular expression> <description or error message>

<action> can be one of:

allow : this file is ok, don’t do further checks (you might use it for safe content types like text). Do not blindly create ‘allow’ rules. It’s safer to make no rule at all, if no other rules hit, the file will be accepted
deny : delete this message and send the error message/description back to the sender
delete : silently delete the message, no error is sent back, and ‘blockaction’ is ignored

<regular expression> is a standard python regex. in x-filenames.conf this will be applied to the attachment name . in x-filetypes.conf this will be applied to the mime type of the file as well as the file type returned by the file command.

Example of default-filetypes.conf :

allow    text        -
allow    script    -
allow    archive        -
allow    postscript    -
deny    self-extract    No self-extracting archives
deny    executable    No programs allowed
deny    ELF        No programs allowed
deny    Registry    No Windows Registry files allowed

A small extract from default-filenames.conf:

deny    \.ico$            Windows icon file security vulnerability
deny    \.ani$            Windows animated cursor file security vulnerability
deny    \.cur$            Windows cursor file security vulnerability
deny    \.hlp$            Windows help file security vulnerability

allow    \.jpg$            -
allow    \.gif$            -

Note: The files will be reloaded automatically after a few seconds (you do not need to kill -HUP / restart fuglu)

Per domain/user overrides can also be fetched from a database instead of files (see dbconnectstring / query options). The query must return the same rule format as a file would. Multiple columns in the resultset will be concatenated.

The default query assumes the following schema:

CREATE TABLE `attachmentrules` (
  `rule_id` int(11) NOT NULL AUTO_INCREMENT,
  `action` varchar(10) NOT NULL,
  `regex` varchar(255) NOT NULL,
  `description` varchar(255) DEFAULT NULL,
  `scope` varchar(255) DEFAULT NULL,
  `checktype` varchar(20) NOT NULL,
  `prio` int(11) NOT NULL,
  PRIMARY KEY (`rule_id`)
)

action: allow, deny, or delete

regex: a regular expression

description: description/explanation of this rule which is optionally sent back to the sender if bounces are enabled

scope: a domain name or a recipient’s email address

checktype: one of filename,``contenttype``,``archive-filename``,``archive-contenttype``

prio: order in which the rules are run

The bounce template (eg /etc/fuglu/templates/blockedfile.tmpl ) should start by defining the headers, followed by a blank line, then the message body for your bounce message. Something like this:

To: ${from_address}
Subject: Blocked attachment

Your message to ${to_address} contains a blocked attachment and has not been delivered.

${blockinfo}

${blockinfo} will be replaced with the text you specified in the third column of the rule that blocked this message.

The other common template variables are available as well.

Configuration¶

[FiletypePlugin]
#inform the sender about blocked attachments.
#If a previous plugin tagged the message as spam or infected, no bounce will be sent to prevent backscatter
sendbounce=1

#comma separated list of archive extensions. do only process archives of given types.
enabledarchivetypes=

#sqlalchemy connectstring to load rules from a database and use files only as fallback. requires SQL extension to be enabled
dbconnectstring=

#sql query to load rules from a db. #:scope will be replaced by the recipient address first, then by the recipient domain
#:check will be replaced 'filename','contenttype','archive-filename' or 'archive-contenttype'
query=SELECT action,regex,description FROM attachmentrules WHERE scope=:scope AND checktype=:checktype ORDER BY prio

#Mail template for the bounce to inform sender about blocked attachment
template_blockedfile=/etc/fuglu/templates/blockedfile.tmpl

#directory that contains attachment rules
rulesdir=/etc/fuglu/rules

#enable scanning of filenames within archives (zip,rar). This does not actually extract the files, it just looks at the filenames found in the archive.
checkarchivenames=0

#what should the plugin do when a blocked attachment is detected
#REJECT : reject the message (recommended in pre-queue mode)
#DELETE : discard messages
#DUNNO  : mark as blocked but continue anyway (eg. if you have a later quarantine plugin)
blockaction=DELETE

#extract compressed archives(zip,rar) and check file content type with libmagics
#note that the files will be extracted into memory - tune archivecontentmaxsize  accordingly.
#fuglu does not extract archives within the archive(recursion)
checkarchivecontent=0

#only extract and examine files up to this amount of (uncompressed) bytes
archivecontentmaxsize=5000000

Archive¶

Plugin: fuglu.plugins.archive.ArchivePlugin

This plugins stores a copy of the message if it matches certain criteria (Suspect Filter). You can use this if you want message archives for your domains or to debug problems occuring only for certain recipients.

Examples for the archive.regex filter file:

Archive messages to domain ‘’test.com’‘:

to_domain test\.com

Archive messages from oli@fuglu.org:

envelope_from oli@fuglu\.org

you can also append “yes” and “no” to the rules to create a more advanced configuration. Lets say we want to archive all messages to sales@fuglu.org and all regular messages support@fuglu.org except the ones created by automated scripts like logwatch or daily backup messages etc.

envelope_from logwatch@.*fuglu.org no

envelope_to sales@fuglu.org yes

from backups@fuglu.org no

envelope_to support@fuglu.org yes

Note: The first rule to match in a message is the only rule that will be applied. Exclusion rules should therefore be put above generic/catch-all rules.

Configuration¶

[ArchivePlugin]
#change owner of saved messages (username or numeric id) - this only works if fuglu is running as root (which is NOT recommended)
chown=

#Archiving SuspectFilter File
archiverules=/etc/fuglu/archive.regex

#filename template for the archived messages
filenametemplate=${id}.eml

#subdirectory within archivedir
subdirtemplate=${to_domain}

#if true/1/yes: store original message
#if false/0/no: store message probably altered by previous plugins, eg with spamassassin headers
storeoriginal=1

#storage for archived messages
archivedir=/tmp

#set file permissions of saved messages
chmod=

#change group of saved messages (groupname or numeric id) - the user running fuglu must be a member of the target group for this to work
chgrp=

Vacation¶

Plugin: fuglu.plugins.vacation.VacationPlugin

Sends out-of-office reply messages. Configuration is trough a sql database. Replies are only sent once per day per sender. The plugin will not reply to any ‘automated’ messages (Mailingslists, Spams, Bounces etc)

Requires: SQLAlechemy Extension

Required DB Tables:

vacation (fuglu reads this table only, must be filled from elsewhere)
- id int : id of this vacation
- created timestamp : creation timestamp
- enabled boolean (eg. tinyint) : if disabled, no vacation reply will be sent
- start timestamp: replies will only be sent after this point in time
- end timestamp: replies will only be sent before this point in time
- awayuser varchar: the email address of the user that is on vacation
- subject: subject of the vacation message
- body : body of the vacation message
- ignoresender: whitespace delimited list of domains or email addresses that should not receive vacation replies
vacationreply (this table is filled by fuglu)
- id int: id of the reply
- vacation_id : id of the vacation
- sent timestamp: timestamp when the reply was sent
- recipient: recipient to whom the reply was sent

SQL Example for mysql:

CREATE TABLE `vacation` (
  `id` int(11) NOT NULL auto_increment,
  `created` timestamp NOT NULL default now(),
  `start` timestamp NOT NULL,
  `end` timestamp NOT NULL ,
  `enabled` tinyint(1) NOT NULL default 1,
  `awayuser` varchar(255) NOT NULL,
  `subject` varchar(255) NOT NULL,
  `body` text NOT NULL,
  `ignoresender` text NOT NULL,
  PRIMARY KEY  (`id`),
  UNIQUE(`awayuser`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 ;


CREATE  TABLE `vacationreply` (
  `id` int(11) NOT NULL auto_increment,
  `recipient` varchar(255) NOT NULL,
  `vacation_id` int(11) NOT NULL,
     `sent` timestamp not null default now(),
  PRIMARY KEY  (`id`),
  KEY `vacation_id` (`vacation_id`),
  CONSTRAINT `vacation_ibfk_1` FOREIGN KEY (`vacation_id`) REFERENCES `vacation` (`id`) ON DELETE CASCADE ON UPDATE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

Configuration¶

[VacationPlugin]
#sqlalchemy connectstring to load vacations
dbconnectstring=

Sophos¶

Plugin: fuglu.plugins.sssp.SSSPPlugin

This plugin scans the suspect using the sophos SSSP protocol.

Prerequisites: Requires a running sophos daemon with dynamic interface (SAVDI)

Configuration¶

[SSSPPlugin]
#how often should fuglu retry the connection before giving up
retries=3

#hostname where the SSSP server runs
host=localhost

#reject message template if running in pre-queue mode and virusaction=REJECT
rejectmessage=threat detected: ${virusname}

#socket timeout
timeout=30

#action if there is a problem (DUNNO, DEFER)
problemaction=DEFER

#maximum message size, larger messages will not be scanned.
maxsize=22000000

#tcp port or path to unix socket
port=4010

#action if infection is detected (DUNNO, REJECT, DELETE)
virusaction=DEFAULTVIRUSACTION

F-Prot AV¶

Plugin: fuglu.plugins.fprot.FprotPlugin

This plugin passes suspects to a f-prot scan daemon

Prerequisites: f-protd must be installed and running, not necessarily on the same box as fuglu though.

Notes for developers:

Tags:

sets virus['F-Prot'] (boolean)

sets FprotPlugin.virus (list of strings) - virus names found in message

Configuration¶

[FprotPlugin]
#additional scan options  (see `man fpscand` -> SCANNING OPTIONS for possible values)
scanoptions=

#maximum retries on failed connections
retries=3

#hostname where fpscand runs
host=localhost

#reject message template if running in pre-queue mode and virusaction=REJECT
rejectmessage=threat detected: ${virusname}

#if fpscand runs on a different host than fuglu, set this to 1 to send the message over the network instead of just the filename
networkmode=0

#network timeout
timeout=30

#plugin action if scan fails
problemaction=DEFER

#maximum message size to scan
maxsize=10485000

#fpscand port
port=10200

#plugin action if threat is detected
virusaction=DEFAULTVIRUSACTION

ICAP AV¶

Plugin: fuglu.plugins.icap.ICAPPlugin

ICAP Antivirus Plugin This plugin allows Antivirus Scanning over the ICAP Protocol (http://tools.ietf.org/html/rfc3507 ) supported by some AV Scanners like Symantec and Sophos. For sophos, however, it is recommended to use the native SSSP Protocol.

Prerequisites: requires an ICAP capable antivirus engine somewhere in your network

Configuration¶

[ICAPPlugin]
#how often should fuglu retry the connection before giving up
retries=3

#hostname where the ICAP server runs
host=localhost

#reject message template if running in pre-queue mode and virusaction=REJECT
rejectmessage=threat detected: ${virusname}

#ICAP Av scan service, usually AVSCAN (sophos, symantec)
service=AVSCAN

#socket timeout
timeout=10

#action if there is a problem (DUNNO, DEFER)
problemaction=DEFER

#name of the virus engine behind the icap service. used to inform other plugins. can be anything like 'sophos', 'symantec', ...
enginename=icap-generic

#maximum message size, larger messages will not be scanned.
maxsize=22000000

#tcp port or path to unix socket
port=1344

#action if infection is detected (DUNNO, REJECT, DELETE)
virusaction=DEFAULTVIRUSACTION

DrWeb AV¶

Plugin: fuglu.plugins.drweb.DrWebPlugin

This plugin passes suspects to a DrWeb scan daemon

EXPERIMENTAL Plugin: has not been tested in production.

Prerequisites: Dr.Web unix version must be installed and running, not necessarily on the same box as fuglu though.

Notes for developers:

Tags:

sets virus['drweb'] (boolean)

sets DrWebPlugin.virus (list of strings) - virus names found in message

Configuration¶

[DrWebPlugin]
#maximum retries on failed connections
retries=3

#hostname where fpscand runs
host=localhost

#reject message template if running in pre-queue mode and virusaction=REJECT
rejectmessage=threat detected: ${virusname}

#network timeout
timeout=30

#plugin action if scan fails
problemaction=DEFER

#maximum message size to scan
maxsize=22000000

#DrWeb daemon port
port=3000

#plugin action if threat is detected
virusaction=DEFAULTVIRUSACTION

Action Override¶

Plugin: fuglu.plugins.actionoverride.ActionOverridePlugin

Override actions based on a Suspect Filter file. For example, delete all messages from a specific sender domain.

Configuration¶

[ActionOverridePlugin]
#Rules file
actionrules=/etc/fuglu/actionrules.regex

ScriptFilter¶

Plugin: fuglu.plugins.script.ScriptFilter

This plugin executes scripts found in a specified directory. This can be used to quickly add a custom filter script without changing the fuglu configuration.

Filterscripts must be written in standard python but with the file ending .fgf (“fuglu filter”)

Scripts are reloaded for every message and executed in alphabetic order. You do not need to restart fuglu to load any changes made to these files.

The API is basically the same as for normal plugins within the examine() method, with a few special cases:

there is no ‘self’ which means:

access the configuration by using config directly (instead of self.config)

use debug('hello world') instead of self._logger().debug('hello world'), this will also automatically write to the message debug channel

the script should not return anything, but change the available variables action and message instead (DUNNO, REJECT, DEFER, ACCEPT, DELETE are already imported)

use stop(action=DUNNO, message='') to exit the script early

Example script: (put this in /etc/fuglu/scriptfilter/99_demo.fgf for example)

#block all messages from evilsender.example.com
if not suspect.from_domain=='evilsender.example.com':
    suspect.add_header("x-fuglu-SenderDomain",suspect.from_domain,immediate=True)
    stop()
debug("hello world")
action=REJECT
message="you shall not pass"

Configuration¶

[ScriptFilter]
#Dir that contains the scripts (*.fgf files)
scriptdir=/etc/fuglu/scriptfilter

DKIM Sign¶

Plugin: fuglu.plugins.domainauth.DKIMSignPlugin

EXPERIMENTAL Add DKIM Signature to outgoing mails

Setting up your keys:

mkdir -p /etc/fuglu/dkim
domain=example.com
openssl genrsa -out /etc/fuglu/dkim/${domain}.key 1024
openssl rsa -in /etc/fuglu/dkim/${domain}.key -out /etc/fuglu/dkim/${domain}.pub -pubout -outform PEM
# print out the DNS record:
echo -n "default._domainkey TXT  \"v=DKIM1; k=rsa; p=" ; cat /etc/fuglu/dkim/${domain}.pub | grep -v 'PUBLIC KEY' | tr -d '\n' ; echo ";\""

If fuglu handles both incoming and outgoing mails you should make sure that this plugin is skipped for incoming mails

known issues:

setting canonicalizeheaders = simple will cause invalid signature.

signbodylength causes a crash in dkimlib “TypeError: sequence item 1: expected string, int found”

Configuration¶

[DKIMSignPlugin]
#comma separated list of headers to sign. empty string=sign all headers
signheaders=From,Reply-To,Subject,Date,To,CC,Resent-Date,Resent-From,Resent-To,Resent-CC,In-Reply-To,References,List-Id,List-Help,List-Unsubscribe,List-Subscribe,List-Post,List-Owner,List-Archive

#include l= tag in dkim header
signbodylength=False

#selector to use when signing, supports templates
selector=default

#Location of the private key file. supports standard template variables plus additional ${header_from_domain} which extracts the domain name from the From: -Header
privatekeyfile=/etc/fuglu/dkim/${header_from_domain}.key

#Type of body canonicalization (simple or relaxed)
canonicalizebody=relaxed

#Type of header canonicalization (simple or relaxed)
canonicalizeheaders=relaxed

DKIM Verify¶

Plugin: fuglu.plugins.domainauth.DKIMVerifyPlugin

EXPERIMENTAL This plugin checks the DKIM signature of the message and sets tags... DKIMVerify.sigvalid : True if there was a valid DKIM signature, False if there was an invalid DKIM signature the tag is not set if there was no dkim header at all

DKIMVerify.skipreason: set if the verification has been skipped

The plugin does not take any action based on the DKIM test result since a failed DKIM validation by itself should not cause a message to be treated any differently. Other plugins might use the DKIM result in combination with other factors to take action (for example a “DMARC” plugin could use this information)

It is currently recommended to leave both header and body canonicalization as ‘relaxed’. Using ‘simple’ can cause the signature to fail.

Configuration¶

[DKIMVerifyPlugin]

SPF Check¶

Plugin: fuglu.plugins.domainauth.SPFPlugin

EXPERIMENTAL This plugin checks the SPF status and sets tag ‘SPF.status’ to one of the official states ‘pass’, ‘fail’, ‘neutral’, ‘softfail, ‘permerror’, ‘temperror’ or ‘skipped’ if the SPF check could not be peformed. Tag ‘SPF.explanation’ contains a human readable explanation of the result

The plugin does not take any action based on the SPF test result since. Other plugins might use the SPF result in combination with other factors to take action (for example a “DMARC” plugin could use this information)

Configuration¶

[SPFPlugin]

DomainAuth¶

Plugin: fuglu.plugins.domainauth.DomainAuthPlugin

EXPERIMENTAL This plugin checks the header from domain against a list of domains which must be authenticated by DKIM and/or SPF. This is somewhat similar to DMARC but instead of asking the sender domain for a DMARC policy record this plugin allows you to force authentication on the recipient side.

This plugin depends on tags written by SPFPlugin and DKIMVerifyPlugin, so they must run beforehand.

Configuration¶

[DomainAuthPlugin]
#action if the message doesn't pass authentication (DUNNO, REJECT)
failaction=DUNNO

#File containing a list of domains (one per line) which must be DKIM and/or SPF authenticated
domainsfile=/etc/fuglu/auth_required_domains.txt

#reject message template if running in pre-queue mode
rejectmessage=sender domain ${header_from_domain} must pass DKIM and/or SPF authentication

Plugin Skipper¶

Plugin: fuglu.plugins.p_skipper.PluginSkipper

Skips plugins based on standard filter file This can be used for example to skip spam filters on outgoing messages. e.g. put this in /etc/fuglu/skipplugins.regex:

@incomingport 1099 SAPlugin

Configuration¶

[PluginSkipper]
filterfile=/etc/fuglu/skipplugins.regex

Plugin Fraction¶

Plugin: fuglu.plugins.p_fraction.PluginFraction

Runs only a fraction of loaded scanner plugins based on standard filter file Use this if you only want to run a fraction of the standard plugins on a specific port for example eg. put this in /etc/fuglu/pluginfraction.regex:

@incomingport 1100 SAPlugin,AttachmentPlugin

Configuration¶

[PluginFraction]
filterfile=/etc/fuglu/pluginfraction.regex

Debugger¶

Plugin: fuglu.plugins.p_debug.MessageDebugger

Message Debugger Plugin (Prepender).

This plugin enables the fuglu_debug functionality. Make sure fuglu listens on the debug port configured here.

Configuration¶

[MessageDebugger]
#messages incoming on this port will be debugged to a logfile
#Make sure the debugport is also set in the incomingport configuration option in the main section
debugport=10888

#debug log output
debugfile=/tmp/fuglu_debug.log

#don't re-inject debugged messages back to postfix
noreinject=1

#debugged message can not be bounced
nobounce=1

#don't run appender plugins for debugged messages
noappender=1

Statsd Sender: Plugin Time¶

Plugin: fuglu.plugins.a_statsd.PluginTime

EXPERIMENTAL: Send Plugin execution time to a statsd server

Configuration¶

[PluginTime]
#statsd host
host=127.0.0.1

#statsd port
port=8125

Statsd Sender: Global Message Status¶

Plugin: fuglu.plugins.a_statsd.MessageStatus

EXPERIMENTAL: Send message status to a statsd server

Configuration¶

[MessageStatus]
#statsd host
host=127.0.0.1

#statsd port
port=8125

Statsd Sender: Per Recipient Message Status¶

Plugin: fuglu.plugins.a_statsd.MessageStatusPerRecipient

EXPERIMENTAL: Send per recipient stats to a statsd server

Configuration¶

[MessageStatusPerRecipient]
#statsd host
host=127.0.0.1

#statsd port
port=8125

#domain: send stats per recipient domain. email: send stats per recipient email address
level=domain

Extra plugins¶

Note, in addition to the plugins included in the fuglu distribution there are additional contributed plugins available in the fuglu-extra-plugins repository: https://github.com/fumail/fuglu-extra-plugins/

Writing your own plugins¶

Assuming you know python basics, writing plugins for fuglu is very easy. All you have to do is create a new class which extends from ScannerPlugin, override __str__ to provide a nice human readable name and override examine to do the actual work of your plugins. examine should return one of the action codes above (DUNNO, DEFER, DELETE, ....) and optionally a reason for the action. (e.g. return DEFER, 'please try again later' In plugin you usually only have to import things from fuglu.shared , so it’s probably a good idea to get familiar with that module.

This is a quick example of how your plugin code skeleton would look like:

from fuglu.shared import ScannerPlugin,DUNNO

class DemoPlugin(ScannerPlugin):
    """Copy this to make a new plugin"""
    def __init__(self,config,section=None):
        ScannerPlugin.__init__(self,config,section)
        #config example
        #self.requiredvars={
        #    'maxsize':{
        #        'default':'1337',
        #        'description':'Maximum message size',
        #    }
        #}

    def examine(self,suspect):
        #config Example
        #maxsize=self.config.getint(self.section, 'maxsize')

        #debug example
        #self._logger().debug('hello world from DemoPlugin')

        #PUT PLUGIN CODE HERE

        return DUNNO

First of all, you need a few imports. ScannerPlugin (so you can extend from it), and possible return values for your Plugin, DUNNO might be enough depending on what your plugin does.

from fuglu.shared import ScannerPlugin,DUNNO

in __init__ you only call BasicPlugin’s __init__ for now. This sets self.config and self.section on the object where you later can read config options (eg. self.config.get(self.section,'somevalue'). Do NOT load the plugin configuration here. __init__ is only called once when fuglu starts up. Always load plugin config in examine.

def __init__(self,config,section=None):
    ScannerPlugin.__init__(self,config,section)

then code your examine method. You will have to work with The Suspect class, which is a representation of the message being analyzed. The suspect has tags that are read and written by plugins. You can tag a message as virus, as spam, define your own tags, read tags from previous plugins... it’s probably a good idea to look at the Suspect class in fuglu.shared to get a list of things you can do with the suspect.

Common Tasks (“API” FAQ)¶

Define configuration options for your plugin¶

In order to make ‘lint’ and ‘fuglu_conf’ work with your plugin it should tell the core what config options it expects. this is done by creating a dictionary named ‘requiredvars’ in the plugins init:

Example:

def __init__(self,config,section=None):
        ScannerPlugin.__init__(self,config,section)
        self.requiredvars={
            'host':{
                'default':'localhost',
                'description':'hostname',
            },

            'username':{
                'default':'root',
            },

            'password':{
                'default':``,
                'confidential':True,
            },

       }

This would tell fuglu that your plugin has three config options: host, username and password.

The ‘dict of dicts’ uses your config option name as key for the outer dict. The inner dict supports the following keys:

default - default value, used if the option is not specified in the config file

section - config section to check. by default fuglu assumes that the plugin reads its own config section. override this if your plugin requires a config option from a different plugin or from the main config

confidential - set this to True if fuglu_conf should treat this option confidential and redact it in ‘fuglu_conf -n’ (passwords etc)

validator - function that should be called to validate if the configured value is valid. the function will receive the value as argument and must return True or False

deprecated - mark a config option as deprecated

Read the config¶

Configs in fuglu are stored in /etc/fuglu/fuglu.conf (or any .conf file in /etc/fuglu/conf.d ) in ini-style format. Your plugin gets its own section named like the plugin class.

Example:

[DemoPlugin]
maxsize=10382

You can then read the config in your plugin with the usual methods from a python ConfigParser object ( http://docs.python.org/library/configparser.html )

maxsize=self.config.getint(self.section, 'maxsize')

Important: always load the configs in examine (and not in init !). Reading the config in init breaks loading default values and prevents on-the-fly config reload

Get the message source¶

Use suspect.get_source() to get the message source. The maxbytes option allows you to get only a part of the source. Reading the whole source can slow down scanning of big messages.

Add headers¶

suspect.add_header(headername,headervalue)

By default, headers are added to the message shortly before it is re-injected into postfix. add immediate=True to add the header immediately, so other plugins can see it.

Replace headers¶

Here’s an example on how to change the subject of a message. Note: It’s generally not recommended to change existing headers, this breaks DKIM.

msgrep = suspect.get_message_rep()
msgrep.replace_header('Subject', 'Scanned: '+msgrep['Subject'])
suspect.set_message_rep(msgrep)

Write to the log¶

Your plugin has a _logger method that returns a standard python logger object, so you can use the info / debug / error / fatal methods.

Example:

self._logger().info('hello world from DemoPlugin')

Write debug info¶

To make the plugin write special debug info when fuglu_debug is used, use:

suspect.debug('debug info from DemoPlugin!')

Make plugin ‘–lint’-able¶

By default, lint() only validates the plugin’s configuration settings from self.requiredvars. You can override lint() to do more stuff.

use simple print in this method, no logging stuff.

if you override lint() you should run self.checkConfig() to test the configuration

lint() must return True or False

Example of a plugin that would check if an imap account is available:

def lint(self):
    allok=(self.checkConfig() and self.lint_imap())
    return allok

def lint_imap(self):
    try:
        M=imaplib.IMAP4(self.config.get(self.section, 'host'))
        (type,data)=M.login(self.config.get(self.section, 'user'),self.config.get(self.section, 'password'))
        if type!='OK':
            print('Could not login to imap review account: %s',data)
            return False
        return True
    except Exception, e:
        print "Could not login to imap host:%s - Error %s"%(self.config.get(self.section, 'host'),e)
    return False

Use the ‘SuspectFilter’¶

SuspectFilters are a common way for all plugins to perform an action based on conditions defined in a filterfile . These files are automatically re-loaded if changed.

import SuspectFilter from fuglu.shared

define a config variable for your plugin which holds the name of the filter file (not strictly required, you could hardcode the path)

create a plugin property for the filter

from fuglu.shared import ScannerPlugin,SuspectFilter
[...]

class MyPlugin(ScannerPlugin):
    def __init__(self,config,section=None):
        ScannerPlugin.__init__(self,config,section)

        self.requiredvars={
            'myrulesfile':{
                'default':'/etc/fuglu/mypluginrules.regex',
                'description':'Filter file for my plugin',
            },

           [...]
        }
        self.filter=None

In examine create the filter if necessary

if self.filter==None:
    self.filter=SuspectFilter(self.config.get(self.section, 'myrulesfile'))

Run the filter in examine: (match,arg)=self.filter.matches(suspect) match is a boolean, telling you if one of the rules matched arg` is an additional argument which have been appended to the filter rule in the config. lets say, the filter rule reads to_address example@fuglu.org hello world!, you would get match=True and arg=’Hello world!’ if the message is sent to example@fuglu.org

(match,arg)=self.filter.matches(suspect)
if match:
    if arg!=None:
        self.logger.debug("""we got a match with arg %s"""%arg)
    else:
        self.logger.debug("""We got a match without an arg""")
else:
    suspect.debug("no rule matches")

Use the sql extension¶

TODO (DBFiles, sqlalchemy connections)

Debugging¶

Get a stacktrace¶

if something went wrong you should see a stacktrace in the fuglu log (/var/log/fuglu/fuglu.log) with fuglu >=0.6.0 you can also get the most recent exceptions with the following command:

fuglu_control exceptionlist

Debug the plugin while fuglu is runnig¶

run fuglu --console to enter an interactive python console after fuglu startup. Your plugin is then available via the list mc.plugins

Debug a plugin without running fuglu¶

plugdummy.py is a tool that makes plugin development and testing much easier by creating a minimal fuglu environment for the plugin to run. it doesn’t require a running fuglu or postfix. it will create a dummy suspect, call the plugin’s examine method and print the result (and debug output).

the generated input messag is stored as: /tmp/fuglu_dummy_message_in.eml

if your plugin modified the message source, the resulting message can be found at /tmp/fuglu_dummy_message_out.eml

plugdummy.py is located in the develop/scripts directory.

simple usage:

assuming your plugin file (‘myplugin.py’) is in /usr/local/fuglu/plugins you can run plugdummy.py <pluginname>

#./plugdummy.py myplugin.ExamplePlugin

INFO:root:Input file created as /tmp/fuglu_dummy_message_in.eml
INFO:root:Running plugin: ExamplePlugin
INFO:fuglu.plugin.ExamplePlugin:sender@fuglu.local greets recipient@fuglu.local: hello world!
INFO:root:Result: DUNNO
INFO:root:Suspect a7babc1e4cfe49c36710065966e6ed0a: from=sender@fuglu.local to=recipient@fuglu.local size=231 spam=no virus=no modified=no tags={'virus': {}, 'spam': {}, 'highspam': {}}

Advanced usage:

run plugdummy.py --help to get a list of all options

Examples:

Running plugins from a different directory

./plugdummy.py -p /tmp/ myplugin.ExamplePlugin

Change sender / recipient the ‘-s’ and ‘-f’ options change the envelope sender/recipient. -r can be specified multiple times to simulate a multi-recipient message

./plugdummy.py -s me@example.org -r you@example.net  myplugin.ExamplePlugin
INFO:root:Input file created as /tmp/fuglu_dummy_message_in.eml
INFO:root:Running plugin: ExamplePlugin
INFO:fuglu.plugin.ExamplePlugin:me@example.org greets you@example.net: hello world!
INFO:root:Result: DUNNO
INFO:root:Suspect 423e02e1461cd1c314ac9a409176c4f4: from=me@example.org to=you@example.net size=221 spam=no virus=no modified=no tags={'virus': {}, 'spam': {}, 'highspam': {}}

Adding headers to the input message

./plugdummy.py -h 'subject:yo! whassup' myplugin.ExamplePlugin
[...]
cat /tmp/fuglu_dummy_message_in.eml
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
From: sender@fuglu.local
To: recipient@fuglu.local
Subject: yo! whassup
Date: Fri, 01 Jun 2012 12:58:34 -0000

hello, world!

Adding tags:

./plugdummy.py -t 'mytag:myvalue' myplugin.ExamplePlugin
INFO:root:Input file created as /tmp/fuglu_dummy_message_in.eml
INFO:root:Running plugin: ExamplePlugin
INFO:fuglu.plugin.ExamplePlugin:sender@fuglu.local greets recipient@fuglu.local: hello world!
INFO:root:Result: DUNNO
INFO:root:Suspect 168268d6ff2c2748454183efcb554242: from=sender@fuglu.local to=recipient@fuglu.local size=231 spam=no virus=no modified=no tags={'spam': {}, 'virus': {}, 'mytag': 'myvalue', 'highspam': {}}

Setting a config option:

./plugdummy.py -o 'greeting:go away!'  myplugin.ExamplePlugin
INFO:root:Input file created as /tmp/fuglu_dummy_message_in.eml
INFO:root:Running plugin: ExamplePlugin
INFO:fuglu.plugin.ExamplePlugin:sender@fuglu.local greets recipient@fuglu.local: go away!
[...]

Setting the message body:

#set body
./plugdummy.py -b 'hi there, whassup!' myplugin.ExamplePlugin

#read body from file
./plugdummy.py -b bla.txt myplugin.ExamplePlugin

#read headers & body from eml file
./plugdummy.py -e /tmp/bla.eml myplugin.ExamplePlugin

Running a interactive console in the dummy enrivonment:

./plugdummy.py -c  myplugin.ExamplePlugin
INFO:root:Input file created as /tmp/fuglu_dummy_message_in.eml
INFO:root:Running plugin: ExamplePlugin
INFO:fuglu.plugin.ExamplePlugin:sender@fuglu.local greets recipient@fuglu.local: hello world!
INFO:root:Result: DUNNO
INFO:root:Suspect 3cf496cbe2a1097abc37ebda5a645cd2: from=sender@fuglu.local to=recipient@fuglu.local size=231 spam=no virus=no modified=no tags={'virus': {}, 'spam': {}, 'highspam': {}}
Fuglu Interactive Console started

pre-defined locals:
{'config': <ConfigParser.ConfigParser instance at 0x1ac9e60>, 'suspect': <fuglu.shared.Suspect object at 0x1ac8750>, 'result': 0, 'plugin': <myplugin.ExamplePlugin object at 0x1ac8290>}

>>> plugin.requiredvars
{'greeting': {'default': 'hello world!', 'description': 'greeting the plugin should log to the console'}}
>>> plugin.examine(suspect)
INFO:fuglu.plugin.ExamplePlugin:sender@fuglu.local greets recipient@fuglu.local: hello world!
0
>>> config.set('ExamplePlugin','greeting','Greetings, Earthling')
>>> suspect.from_address='me@example.org'
>>> plugin.examine(suspect)
INFO:fuglu.plugin.ExamplePlugin:me@example.org greets recipient@fuglu.local: Greetings, Earthling
0
>>>

Deploying Plugins¶

If there is no plugindir set in fuglu.conf yet, define a new directory for custom plugins. eg /usr/local/fuglu/plugins.

Copy your plugin file to this directory

Depending on the type of your plugin, add it to the plugin/prependers/appenders config option. Eg. if your scanner plugin class is MyHeloPlugin in the file myplugin.py you would add myplugin.MyHeloPlugin to the plugins config

If your plugin reads configuration entries, make sure those are present in fuglu.conf or in a custom conf-file in /etc/fuglu/conf.d

Run fuglu --lint to check if fuglu is happy with your new plugin

(Re)start fuglu