Setting up a Synapse Server for Matrix Chat

Setting up a Matrix Homeserver with Synapse

Goal:

Setting up my own Matrix Homeserver using Docker and Docker Compose - to get more practice with dockerized setups and to try out Matrix Federation.

Intro:

For more information on Matrix and its benefits see here: https://matrix.org/
It is not necessary to run your own server to join Matrix chat rooms. You can also sign up on an existing public server.
However, one of the interesting Matrix features is its federation capability. It follows a decentralized approach and you can join a conversation from your own server that is under your control.

While Matrix is an open communication standard, the “Synapse” server currently is the one major Matrix server implementation that supports all features:
https://matrix.org/docs/projects/server/synapse
-> I am also using Synapse in this setup.

Prerequisites:

To run the setup below I needed to have:

• My own domain and an A record pointing to a publicly reachable server
• About 1 GB of RAM available for the Synapse server, in case I want to join some of the larger chat rooms - my server is a Digital Ocean Droplet with 2 GB of RAM in total and it is working fine with this spec.
  For reference: I have joined a few larger rooms but very few people are using my server.
• Docker and Docker Compose installed
• A firewall rule to allow communications on port 8448. This is the port used for the federation protocol.
  As the communication will be secured via a TLS channel, port 80 should also be open at least temporarily so that I can get a certificate from Let’s Encrypt.

High-level Steps:

1. Create the initial “homeserver.yaml” configuration for the Synapse server
2. Use Certbot to get a Let’s Encrypt certificate
3. Create the docker compose file for the Synapse container and a Postgres container.
   Synapse by default runs with an SQLite database, which is probably fine for small instances. I wanted to use Postgres directly.
4. Try out Synapse and Matrix
5. Automate certificate renewal

1. Creating the “homeserver.yaml” configuration

I followed the instructions here in the “Generating a configuration file” section for creating a config file: https://hub.docker.com/r/matrixdotorg/synapse/
This will create a volume that contains a default homeserver.yaml configuration. Then I will later mount that volume into the Synapse container.
As the matrix domain name is a parameter in the generate command, the configuration file will already contain my correct domain. To change other parts of the file, exec into the container with sudo docker exec -it <container_name> /bin/bash. Then install an editor and change the file /data/homeserver.yaml.

Settings I configured in the file:

• Using Postgres instead of SQLite:
  I used the instructions here https://github.com/matrix-org/synapse/blob/master/docs/postgres.md.
  I commented out the SQLite part, and added the Postgres part instead:

database: 
  name: psycopg2
  args:
    user: synapse
    password: <yourSecureDatabasePassword>
    database: synapse
    host: db
    cp_min: 5
    cp_max: 10

At this point I generated a secure password for the database. Later on, I will add the database user with that password.
I will call my database host “db”. That hostname is later configured in the docker-compose file when setting up the database container.

• Using TLS connections only:
  By default, Synapse listens on port 8008 for HTTP connections. But to have a secure setup, I only allow HTTPS connections on port 8448. Alternatively, you could place synapse behind a reverse proxy terminating the TLS there.
  There already is a default HTTPS configuration (TLS is set to true), that I commented in instead of the HTTP part:

- port: 8448
    type: http
    tls: true
    resources:
      - names: [client, federation]

For that to work I also need to specify paths to my certificate chain and private key. That follows later, after receiving my Let’s Encrypt certificate.

There are a bunch of other settings that you can tweak via the homeserver.yaml file. I left the rest on default for now.

Debugging: If the homeserver.yaml contains a mistake or some of the configuration is not valid yet, the synapse container might crash.
To still be able to exec into the container and change values in the homeserver.yaml, I use the “entrypoint” parameter in docker or docker-compose. Setting the entrypoint to a command that does not terminate, such as tail -F <some file in the container>is useful for debugging.

2. Getting the certificate with certbot

I needed a certificate for my matrix domain, so I decided to get a free Let’s Encrypt certificate using the certbot client. Synapse had a built-in feature to answer Let’s Encrypt certificate challenges, but unfortunately the matrix team does not recommend this right now. Instead you have to set it up yourself: https://github.com/matrix-org/synapse/blob/master/INSTALL.md#tls-certificates

Luckily this is not a big deal with certbot. I use another container to run certbot in standalone mode. This way, I don’t need another web server, but certbot can answer the Let’s Encrypt challenge itself. Then I will create a cron job to regularly check for expiring certificates.

This is the docker-compose file I use for certbot. As it is just one container, I could also use just a docker command, but I found this easier to read.

version: "3.7"
services:
  certbot:
    image: certbot/certbot:v1.8.0
    volumes:
      - certs:/etc/letsencrypt
      - lets-encrypt:/var/lib/letsencrypt
    ports:
      - "80:80"
    command: certonly --standalone --agree-tos --email <youraddress@mail.com> --no-eff-email -d your.matrix.domain --post-hook 'chown -R 991:991 /etc/letsencrypt/live /etc/letsencrypt/archive' --force-renewal
    restart: "no"

volumes:
  lets-encrypt:
    name: lets-encrypt
  certs:
    name: certs

Details:

The image I use is here: https://hub.docker.com/r/certbot/certbot/.

• According to the documentation https://certbot.eff.org/docs/install.html#running-with-docker it needs two volumes. The volume I named “lets-encrypt” will contain logs and files that certbot needs. The volume “certs” will contain the certificate, private key and certificate chain.
• Port 80 needs to be open. The Let’s Encrypt server will send its HTTP challenge on port 80.
• The entrypoint of this container is “certbot”, so I can directly specify the parameters for certbot as command. I am using “standalone” as I am not running another webserver.
• I also had to get around a file permission issue for using the certificates later in the synapse container, which is why I added the post-hook. Details are described below. After running certbot, the certificate is in the “certs” volume.

I found this post about running Certbot in a container, which describes a slightly more complex setup including nginx very nicely:
https://www.humankode.com/ssl/how-to-set-up-free-ssl-certificates-from-lets-encrypt-using-docker-and-nginx
A good tip in there is to run the certbot command against the Let’s Encrypt staging server first. The Let’s Encrypt API is rate limited, so I used the staging server to test my command before “using up” my attempts on the real endpoint. A certbot command for the staging server could look like this:
certonly --standalone --register-unsafely-without-email --agree-tos --staging -d <your.matrix.domain>
I register without e-mail here, because that does not matter for the staging test certificates.

Root or screwed
One difficulty I ran into is that certbot needs to run as root in its container. As a consequence, files it creates in the “certs” volume will also belong to root. Synapse however does not run as root, but uses UID 991. But after some searching I found that certbot offers a handy post-hook feature that can run after new certificates have successfully been issued. I use the post-hook to change the owner of the relevant folders (live and archive) to user 991. To make further automation easier, I also use the –fore-renewal option in certbot. With this flag, certbot will not ask for confirmation when you are renewing the certificate early. I find this useful for testing, and I want to be able to automate the certificate renewal, as well as renew whenever I want to.

Completing the Synapse TLS config
I already enabled TLS in the homeserver.yaml, but I didn’t specify yet where the certificate and private key is. Now that I have all required files I can complete the config. I decided that the “certs” volume will be mounted under /data/letsencrypt. That means the correct paths in the homeserver.yaml will be:

tls_certificate_path: "/data/letsencrypt/live/<path/to/your/cert>/fullchain.pem"

and

tls_private_key_path: "/data/letsencrypt/live/<path/to/your/key>/privkey.pem"

Note: Make sure to use the fullchain file, as you also need to send the intermediate certificate during the TLS handshake.

3. Creating the docker-compose file

I am configuring two containers in that file:

• one for Synapse itself
• and one for the Postgres database

Let’s start with Synapse:
The documentation here specifies what the Synapse container needs to work: https://hub.docker.com/r/matrixdotorg/synapse/

docker run -d --name synapse \
    --mount type=volume,src=synapse-data,dst=/data \
    -p 8008:8008 \
    matrixdotorg/synapse:latest

For me, this command translates into the following docker-compose config:

synapse:
    image: matrixdotorg/synapse:v1.20.1
    volumes:
      - synapse-data:/data
      - certs:/data/letsencrypt
    ports:
      - "8448:8448"
    restart: unless-stopped
    depends_on:
      - db

Details:

• image: matrixdotorg/synapse:v1.20.1:
  I decided to specify an exact version number instead of using the “latest” tag to stay in control of updates myself
• The container will have its synapse-data volume containing also the homeserver.yaml file. Additionally, it will share the “certs” volume with my certbot container, so that it can access my TLS certificate and private key. See the complete docker-compose file further below for the volume definition
• For my simple setup I decided that I don’t need a reverse proxy, so I directly expose port 8448
• The synapse container will depend on the db container for the Postgres database. Its configuration is described below.

Container Health:
While the above section of the docker-compose file would work for the synapse server, I noticed that my container always reported a status “unhealthy” when checking it with docker ps. This was bothering me, and I found out why this happens.
The synapse image is build using a Dockerfile published here:
https://github.com/matrix-org/synapse/blob/master/docker/Dockerfile
This file specifies a healthcheck command to test, if the synapse container has started correctly and is running fine. But it tests the http endpoint and I am using https, which is why the healthcheck always failed for me.
To fix this, I added another healthcheck in the docker-compose file, which will be used instead of the default one:

healthcheck:
  test: ["CMD", "curl", "-fSsk", https://localhost:8448/health]
  interval: 1m
  timeout: 5s
  start_period: 40s

This check is basically the same as in the synapse Dockerfile, but it uses the https endpoint (ignoring the certificate check with the -k parameter). I also added a start period, as my synapse container has a rather long startup time.

Postgres:
Basically all information is documented here: https://github.com/matrix-org/synapse/blob/master/docs/postgres.md but that guide does not assume a docker container. One small challenge in a docker setup is how to run initialization scripts for the database. The Synapse documentation describes that a specific encoding is required for the database. Also I need to setup the user and password.
Additionally, I wanted to change the password hashing mechanism from MD5 to scram-sha-256. That does improve the hash algorithm for storing the password inside Postgres, however hopefully no one should have access to that anyway.
To really improve security in a distributed setup, I would have to configure TLS between Synapse and the database, so that no credentials are sent over the network in clear text. I skip that step here as both containers run on the same host and no data is sent over an external network connection.
I ended up with a config like this:

 db:
    image: postgres:13.0
    environment:
      - POSTGRES_INITDB_ARGS="--auth=scram-sha-256"
      - POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
      - POSTGRES_USER=synapse
      - POSTGRES_DB=default
      - POSTGRES_HOST_AUTH_METHOD=scram-sha-256
    volumes:
      - db-synapse-data:/var/lib/postgresql/data
      - ./init.sql:/docker-entrypoint-initdb.d/init.sql
    ports:
      - 5432:5432
    restart: unless-stopped

Details:
The docker hub documentation for Postgres is pretty helpful: https://hub.docker.com/_/postgres?tab=description

• The container hostname will be “db” as specified in the homeserver.yaml. Docker-compose will automatically put the Synapse and Postgres container on the same network, as they are in the same docker-compose file.
• Environment variables:
  • POSTGRES_INITDB_ARGS: Send arguments to the initdb call when setting up the database. I specify here that I want to use scram-sha-256 for my database users instead of MD5
  • POSTGRES_PASSWORD: The same password that I configured earlier in the homeserver.yaml for the Postgres connection. I put it in a .env file which is automatically read by docker-compose. This way I can check in my docker-compose file into version control without exposing the password
  • POSTGRES_USER: The name of the database user.
  • POSTGRES_DB: One challenge I found is that postgres automatically tries to create a database with the same name as the user. While that might be handy in normal cases, I had to prevent that. I want to set up the database with the correct encodings as mentioned in the Synapse documentation. My workaround is to use the POSTGRES_DB environment variable to just create another database by default. I am not sure if there is a better way to do this.
  • POSTGRES_HOST_AUTH_METHOD: That value will be written to the “pg_hba.conf” file for user authentication as mentioned in the Synapse documentation.
• Volumes: Besides the volume for storing the actual data, I will put an init.sql file into /docker-entrypoint-initdb.d. The Postgres documentation describes, that this init.sql file is executed when the database is created. My init.sql file contains the statement to create the synapse database with correct encoding:

CREATE DATABASE synapse
 ENCODING 'UTF8'
 LC_COLLATE='C'
 LC_CTYPE='C'
 template=template0
 OWNER synapse;

The initdb script will only run once when there are no database files yet on the specified volume.

The final docker-compose.yaml looks like this:

version: "3.7"
services:
  synapse:
    image: matrixdotorg/synapse:v1.20.1
    volumes:
      - synapse-data:/data
      - certs:/data/letsencrypt
    ports:
      - "8448:8448"
    restart: unless-stopped
    depends_on: 
      - db
    healthcheck:
      test: ["CMD", "curl", "-fSsk", https://localhost:8448/health]
      interval: 1m
      timeout: 5s
      start_period: 40s

  db:
    image: postgres:13.0
    restart: unless-stopped
    environment:
      - POSTGRES_INITDB_ARGS="--auth=scram-sha-256"
      - POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
      - POSTGRES_USER=synapse
      - POSTGRES_DB=default
      - POSTGRES_HOST_AUTH_METHOD=scram-sha-256
    volumes:
      - db-synapse-data:/var/lib/postgresql/data
      - ./init.sql:/docker-entrypoint-initdb.d/init.sql
    ports:
      - 5432:5432

volumes:
  db-synapse-data:
    name: db-synapse-data
  synapse-data:
    name: synapse-data
  certs:
    name: certs
    external: true

4. Try it

With a docker-compose up this config should start up fine now - using a Postgres database and Let’s Encrypt certificate.
There should be a welcome site under https://<your.matrix.domain>:8448
Additionally, there is this website to check, if the server is configured correctly and ready for federation:
https://federationtester.matrix.org/

I use the “Element” client to log into my server:
https://matrix.org/docs/projects/client/element

To actually create an account onto your own server, one last thing is needed:
Edit the homeserver.yaml again. Set enable_registration: true in your configuration. That will allow you to create yourself an account via the Element client. If you are the only person using your server, you can set it to false again after you registered.

5. Bonus: Automate the certificate renewal

Fully automated certificate renewal is always good but it is especially nice to have with Let’s Encrypt certificates, as they expire every 90 days. Let’s Encrypt sends the first reminder 20 days before expiry. That means, if I renew my certificate every 60 days, I should never get a reminder if everything works as planned. I am already using a certbot command that runs without any prompts and it is setting the correct permissions for synapse. Only a few more steps are needed for a full automation:

• run the certbot container every 60 days
• restart the synapse container afterwards, so that it will pick up its new certificates

I will do that with a little script, which then gets executed as a cron job. As my own user on my server is not in the docker group, I will need to use the root user’s crontab. My restart script should therefore also only be writable for root. I don’t have any sophisticated monitoring yet for the synapse server or the certificate renewal. I should however still get the expiry notifications if my automated renewal fails.

The script looks like this:

cd /path/to/your/compose/files
docker-compose -f certbot-docker-compose.yaml up --no-recreate
docker-compose stop synapse
docker-compose start synapse

And my cronjob runs on the first day of every second month at 0:30 am. It logs the outputs into a file.
The notation looks like this:

30 0 1 */2 * /path/to/script.sh >> /path/to/script.log