Full Text Search (FTS)

What is FTS?

FTS allows all emails to be indexed so that mail clients can quickly and efficiently search messages by their full text content directly from the IMAP server. As the size of your mail storage grows, the benefits of FTS are especially notable.

Without FTS, Dovecot would perform a search query by checking each individual email stored for a match, and then repeat this process again from scratch for the exact same query in the future.
Some mail clients (like Thunderbird) may provide their own indexing and search features when all mail to search is stored locally; otherwise, Dovecot needs to handle the search query (for example webmail and mobile clients, like Gmail).
FTS indexes each mail into a database for querying, where it can skip the cost of inspecting irrelevant emails for a query.

Please be aware that indexing consumes memory and takes up additional disk space.

Dovecot supports a variety of community-supported FTS indexing backends. DMS provides different levels of support for them.

Do not enable two FTS indexers simultaneously

Dovecot supports only one FTS backend at a time. If you migrate, remove the old configuration first, (optionally) prune the old index data to free up space, and only then configure the new indexer.

Indexing will take a while depending on how large your mail folders are.

FlatcurveSolrXapian

About

fts-flatcurve is an FTS backend that became part of Dovecot core with the 2.4 release.

Comparison to Xapian

Like Xapian, Flatcurve uses the Xapian library to store indexes locally, so no additional service is needed (unlike solr). But Flatcurve

is maintained by the Dovecot developers as part of Dovecot itself, and it is the FTS backend recommended by upstream for local indexing.
uses the Dovecot 2.4 language settings for tokenizing, stemming and stop words, so search behaviour is consistent with the other Dovecot FTS backends.
stores its index in an fts-flatcurve directory alongside each mailbox's Dovecot index files (so it is retained in your mail-data volume).
does not require a recurring job to run fts optimize on a schedule.

Support Status

Flatcurve was officially introduced with DMS 16.0.0. It will be the preferred FTS indexer going forward because of its straightforward integration into and its first-party support by Dovecot.

Setup

Configure Dovecot to use Flatcurve

Create a fts-flatcurve-plugin.conf file in your ./docker-data/dms/config/dovecot/ folder with the following content:

# Enable FTS Flatcurve
mail_plugins {
  fts = yes
  fts_flatcurve = yes
}

# Index new mail as it is delivered
fts_autoindex = yes

# If FTS lookup or indexing fails, Dovecot
# falls back to a slow non-indexed search.
# After the initial indexing (see below)
# has completed you may prefer to fail such
# searches instead.
fts_search_read_fallback = yes

# Skip autoindexing of folders that grow
# quickly and are rarely searched:
mailbox Trash {
  fts_autoindex = no
}
mailbox Junk {
  fts_autoindex = no
}

# Tokenizing and stemming. Add a `language xx { }`
# block for each language you expect in your mail.
# Refer to the Dovecot docs for the languages
# supported by the `snowball` stemmer.
#
# DO NOT enable stopwords together with multiple
# languages: searches can then miss matches.
language en {
  default = yes
  language_filters = lowercase snowball english-possessive stopwords
}

language_filter_stopwords_dir = /usr/share/dovecot/stopwords
language_filters = normalizer-icu snowball stopwords
language_tokenizers = generic email-address
language_tokenizer_generic_algorithm = simple

fts flatcurve {
  # Match any part of a word (e.g. `mail` matches
  # `mailserver`) at the cost of a much larger index.
  # The default `no` only matches from the start of a word.
  substring_search = yes

  # Further optional tuning (commit_limit, min_term_size,
  # optimize_limit, rotate_count, rotate_time) is documented
  # upstream. The defaults are sensible for most users.
}

service indexer-worker {
  # Limit the size of an indexer-worker's RAM usage
  vsz_limit = 1G
}

Add a volume mount for that config to your DMS service in compose.yaml:

services:
  mailserver:
    volumes:
      - ./docker-data/dms/config/dovecot/fts-flatcurve-plugin.conf:/etc/dovecot/conf.d/90-fts-flatcurve.conf:ro

Alternatively, put the same snippet in dovecot.cf (DMS copies it to /etc/dovecot/local.conf). That uses the existing config volume and does not need an extra bind-mount.

Trigger Dovecot FTS indexing

After following the previous steps, restart DMS and run this command to have Dovecot index all existing mail for every account:
```
docker compose exec mailserver doveadm index -A -q '*'
```
The -q flag queues the work through the indexer service instead of running it in the foreground. You can watch progress in the Dovecot logs (docker compose logs -f mailserver). Once complete, you should be able to search your mail using the Dovecot FTS feature!

Maintenance

Flatcurve automatically rotates and optimizes its Xapian databases as mail is indexed (controlled by the rotate_* and optimize_limit settings), so unlike fts-xapian, a scheduled doveadm fts optimize job is not required.

Some doveadm commands specific to Flatcurve that may be useful:

# Show index statistics (size, number of mails indexed)
# per mailbox for a user:
docker compose exec mailserver doveadm fts flatcurve stats -u user@example.com '*'
# Verify the index databases of a user are not corrupt:
docker compose exec mailserver doveadm fts flatcurve check -u user@example.com '*'
# Drop the index of a user (rebuild it afterwards with `doveadm index`):
docker compose exec mailserver doveadm fts flatcurve remove -u user@example.com '*'
# Rebuild the index for everyone from scratch (for example after
# changing `language` or `substring_search` settings)
docker compose exec mailserver doveadm fts rescan -A
docker compose exec mailserver doveadm index -A -q '*'

About

Apache Solr is a fast and efficient multi-purpose search indexer.

Support Status

Support for Solr is entirely community-driven. The build scripts for DMS currently install the dovecot-solr package to help users of Solr; the package may be removed if DMS encounters problems with it (especially when building for arm64). Bug reports for Solr are not accepted unless they concern the documentation and are accompanied by a pull request to fix the issue.

Setup

Firstly you need a working Solr container

The official docker image will do:

services:
  solr:
    image: solr:10
    container_name: dms-solr
    environment:
      # As Solr can be quite resource hungry, raise the memory limit to 2GB.
      # The default is 512MB, which may be exhausted quickly.
      SOLR_JAVA_MEM: "-Xms2g -Xmx2g"
      # Current dovecot solr config needs the analysis-extras solr module,
      # so add it with this env var.
      SOLR_MODULES: "analysis-extras"
    volumes:
      - ./docker-data/solr:/var/solr
    restart: always

DMS will connect internally to the solr service above. Either have both services in the same compose.yaml file, or ensure that the containers are connected to the same docker network.

Configure Solr for Dovecot
1. Once the Solr container is started, you need to configure a "Solr core" for Dovecot:
```
docker exec -it dms-solr /bin/sh
solr create -c dovecot
```
  Stop the dms-solr container and you should now have a ./docker-data/solr/data/dovecot folder in the local bind mount volume.
2. Solr needs a schema that is specifically tailored for Dovecot FTS.
  
  As of writing of this guide, Solr 10 is the current release. Dovecot provides the required schema configs for Solr. Copy the following two v9 config files, which also work with Solr 10, to ./docker-data/solr/data/dovecot/conf/ and rename them accordingly:
  - solr-config-9.xml (rename to solrconfig.xml)
  - solr-schema-9.xml (rename to schema.xml)
  Additionally, remove any generated managed-schema or managed-schema.xml file from ./docker-data/solr/data/dovecot/conf/ and ensure the two files you copied have a UID and GID of 8983 assigned.
  
  Start the Solr container once again, you should now have a working Solr core specifically for Dovecot FTS.
3. Configure Dovecot in DMS to connect to this Solr core:
  
  Create a 90-fts-solr.conf file in your ./docker-data/dms/config/dovecot/ folder with this content:
```
language en {
  default = yes
}

mail_plugins {
  fts = yes
  fts_solr = yes
}

fts solr {
}

fts_solr_url = http://solr:8983/solr/dovecot/

fts_autoindex = yes
fts_search_add_missing = yes
fts_search_read_fallback = no

mailbox Trash {
  fts_autoindex = no
}

#fts_decoder_driver = script
#fts_decoder_script_socket_path = decode2text

#service decode2text {
#  executable = script /usr/libexec/dovecot/decode2text.sh
#  user = dovecot
#
#  unix_listener decode2text {
#    mode = 0666
#  }
#}
```
  Excluding Trash from indexing is optional and so is including attachment text. The decode2text script may or may not work, upstream prefers Tika which SOLR should be able to do but is outside of scope for this tutorial.
  
  Starting with dovecot 2.4 dovecot fts-solr needs a default language to initialize solr searching. In this example langcode en was set as default, but any langcode will do. If you want to enable additional languages add them like this:
```
language de {
}
```
  Add a volume mount for that config to your DMS service in compose.yaml:
```
services:
  mailserver:
    volumes:
      - ./docker-data/dms/config/dovecot/90-fts-solr.conf:/etc/dovecot/conf.d/90-fts-solr.conf:ro
```
  Alternatively, put the same snippet in dovecot.cf to use the existing config volume instead of an additional bind mount.
Trigger Dovecot FTS indexing

After following the previous steps, restart DMS and run these commands to reconcile the Solr index and index all existing mail:
```
docker compose exec mailserver doveadm fts rescan -A
docker compose exec mailserver doveadm index -A -q '*'
```

About

fts-xapian is a community-maintained plugin that makes use of Xapian. Xapian enables embedding an FTS engine without the need for additional backends.

The indexes are stored in a subfolder named xapian-indexes inside each user's mailbox directory in your local mail-data folder (/var/mail internally). With the default settings, 10GB of email data may generate around 4GB of indexed data.

While indexing is memory intensive, you can configure the plugin to limit the amount of memory consumed by the index workers. With Xapian being small and fast, this plugin is a good choice for low memory environments (2GB).

Support Status

Xapian was the officially supported FTS indexing option until DMS 16.0.0. With 16.0.0, Flatcurve is preferred and Xapian is not officially supported anymore. We urge you to migrate to Flatcurve. New issues concerning Xapian cannot be worked on.

Setup

To configure fts-xapian as a dovecot plugin, create a file at docker-data/dms/config/dovecot/fts-xapian-plugin.conf and place the following in it:

mail_plugins {
  fts = yes
  fts_xapian = yes
}

fts_autoindex = yes

language en {
  default = yes
}

fts xapian {
  verbose = 0
  partial = 3
}

service indexer-worker {
  # Limit the indexer-worker's virtual memory size.
  vsz_limit = 1G
  # Xapian requires an unlimited number of indexer-worker processes.
  process_limit = 0
}

Adjust the settings to tune for your desired memory limits. To index attachments, configure an attachment decoder as described in the Dovecot FTS documentation.

Update compose.yaml to load the previously created Dovecot plugin config file:
```
services:
  mailserver:
    volumes:
      - ./docker-data/dms/config/dovecot/fts-xapian-plugin.conf:/etc/dovecot/conf.d/90-fts-xapian.conf:ro
```
Alternatively, put the same snippet in dovecot.cf to use the existing config volume instead of an additional bind mount.

Recreate containers:

docker compose down
docker compose up -d

Initialize indexing on all users for all mail:

docker compose exec mailserver doveadm index -A -q '*'

Run the following command in a daily cron job:

docker compose exec mailserver doveadm fts optimize -A

Or like the Spamassassin example shows, you can instead use cron from within DMS to avoid potential errors if the mail server is not running:

Example

Create a system cron file:

# in the compose.yaml root directory
mkdir -p ./docker-data/dms/cron # if you didn't have this folder before
touch ./docker-data/dms/cron/fts_xapian
chown root:root ./docker-data/dms/cron/fts_xapian
chmod 0644 ./docker-data/dms/cron/fts_xapian

Edit the system cron file nano ./docker-data/dms/cron/fts_xapian, and set an appropriate configuration:

# Adding `MAILTO=""` prevents cron emailing notifications of the task outcome each run
MAILTO=""
#
# m h dom mon dow user command
#
# Everyday 4:00AM, optimize index files
0  4 * * * root  doveadm fts optimize -A

Then with compose.yaml:

services:
  mailserver:
    volumes:
      - ./docker-data/dms/cron/fts_xapian:/etc/cron.d/fts_xapian