Full-Text Search

Overview

Full-text search allows all messages to be indexed, so that mail clients can quickly and efficiently search messages by their full text content. Dovecot supports a variety of community supported FTS indexing backends.

DMS comes pre-installed with two plugins that can be enabled with a dovecot config file.

Please be aware that indexing consumes memory and takes up additional disk space.

Xapian

The dovecot-fts-xapian plugin makes use of Xapian. Xapian enables embedding an FTS engine without the need for additional backends.

The indexes will be stored as a subfolder named xapian-indexes inside your local mail-data folder (/var/mail internally). With the default settings, 10GB of email data may generate around 4GB of indexed data.

While indexing is memory intensive, you can configure the plugin to limit the amount of memory consumed by the index workers. With Xapian being small and fast, this plugin is a good choice for low memory environments (2GB).

Setup

To configure fts-xapian as a dovecot plugin, create a file at docker-data/dms/config/dovecot/fts-xapian-plugin.conf and place the following in it:

mail_plugins = $mail_plugins fts fts_xapian

plugin {
    fts = xapian
    fts_xapian = partial=3 full=20 verbose=0

    fts_autoindex = yes
    fts_enforced = yes

    # disable indexing of folders
    # fts_autoindex_exclude = \Trash

    # Index attachements
    # fts_decoder = decode2text
}

service indexer-worker {
    # limit size of indexer-worker RAM usage, ex: 512MB, 1GB, 2GB
    vsz_limit = 1GB
}

# service decode2text {
#     executable = script /usr/libexec/dovecot/decode2text.sh
#     user = dovecot
#     unix_listener decode2text {
#         mode = 0666
#     }
# }

adjust the settings to tune for your desired memory limits, exclude folders and enable searching text inside of attachments

Update compose.yaml to load the previously created dovecot plugin config file:

  services:
    mailserver:
      image: ghcr.io/docker-mailserver/docker-mailserver:latest
      container_name: mailserver
      hostname: mail.example.com
      env_file: mailserver.env
      ports:
        - "25:25"    # SMTP  (explicit TLS => STARTTLS)
        - "143:143"  # IMAP4 (explicit TLS => STARTTLS)
        - "465:465"  # ESMTP (implicit TLS)
        - "587:587"  # ESMTP (explicit TLS => STARTTLS)
        - "993:993"  # IMAP4 (implicit TLS)
      volumes:
        - ./docker-data/dms/mail-data/:/var/mail/
        - ./docker-data/dms/mail-state/:/var/mail-state/
        - ./docker-data/dms/mail-logs/:/var/log/mail/
        - ./docker-data/dms/config/:/tmp/docker-mailserver/
        - ./docker-data/dms/config/dovecot/fts-xapian-plugin.conf:/etc/dovecot/conf.d/10-plugin.conf:ro
        - /etc/localtime:/etc/localtime:ro
      restart: always
      stop_grace_period: 1m
      cap_add:
        - NET_ADMIN

Recreate containers:

docker compose down
docker compose up -d

Initialize indexing on all users for all mail:

docker compose exec mailserver doveadm index -A -q \*

Run the following command in a daily cron job:
```
docker compose exec mailserver doveadm fts optimize -A
```
Or like the Spamassassin example shows, you can instead use cron from within DMS to avoid potential errors if the mail server is not running:

Example

Create a system cron file:

# in the compose.yaml root directory
mkdir -p ./docker-data/dms/cron # if you didn't have this folder before
touch ./docker-data/dms/cron/fts_xapian
chown root:root ./docker-data/dms/cron/fts_xapian
chmod 0644 ./docker-data/dms/cron/fts_xapian

Edit the system cron file nano ./docker-data/dms/cron/fts_xapian, and set an appropriate configuration:

# Adding `MAILTO=""` prevents cron emailing notifications of the task outcome each run
MAILTO=""
#
# m h dom mon dow user command
#
# Everyday 4:00AM, optimize index files
0  4 * * * root  doveadm fts optimize -A

Then with compose.yaml:

services:
  mailserver:
    image: ghcr.io/docker-mailserver/docker-mailserver:latest
    volumes:
      - ./docker-data/dms/cron/fts_xapian:/etc/cron.d/fts_xapian

Further Discussion

See #905