Skip to main content

Backend

Security

Securing and hardening of your server and infrastructure is up to you and will not be covered in this documentation.

You can install the backend on a server you like. Its your choice. You can use kubernetes, .net or docker compose to deploy the backend services. In this documentation we will focus only on the setup of the backend with docker compose on a ubuntu 22.05 LTS. Here we show you a step by setp guide you can use to setup the backend manually. Feel free to automate the process. However we do not provide a setup script.

The backend consists of multple services which need to be setup seperate. We provide one mono docker-compose file and a default .env-configuration file. In the .env file defaults are defined. Feel free to change the configuration on your neeeds.

info

We assume that you know how to use linux and docker compose. If not, first look at official docu of docker compose.

Copy monodeploy files​

In this documentation all files and folders will be placed in ~/medialib. Create this folder first.

mkdir ~/medialib
cd ~/medialib

Copy docker-compose.yml and the .env from the repository to ~/medialib on server.

Instance name​

Set a name for your instance in the .env file. The name identifies the owner of records and your instance for partner instances. It may contain letters, digits, ., _ and - with up to 64 characters and must not be changed later. The media service does not start with an invalid name.

The Media service stores the name at the first start and refuses to start, if INSTANCE differs from it later, since the owner of records, the registrations of partners and the access tokens depend on the name. The names are compared case sensitive. Identity and Media must use the same name.

Rename an instance on purpose

Only rename an instance if you accept to register all partners again. Stop the services and replace old-name and new-name in the Media database:

UPDATE settings SET value = 'new-name' WHERE key = 'InstanceName';
UPDATE records SET owner_instance = 'new-name' WHERE owner_instance = 'old-name';

Then set INSTANCE=new-name in the .env file and start the services. Admins and clients have to log in again, since the access tokens contain the name. Partners still know the instance by its old name: remove the registration of the instance at every partner and register it again with a new invitation. Entries of the audit log keep the old name.

## The system tenant name to be used e.g in federation. The INSTANCE value should not contain spaces.
INSTANCE=my-media-lib

Create mml user​

All docker contaienrs will run for user id 1001 and group id 1001. So cretae group and user first.

sudo groupadd -r -g 1001 mml && sudo useradd -r -d /nonexistent -s /bin/false -u 1001 -g 1001 mml

Install docker​

To install docker and create the docker network wekode.mml run following commands.

sudo apt-get remove docker docker-engine docker.io containerd runc
sudo apt-get install \
    ca-certificates \
    curl \
    gnupg \
    lsb-release
sudo mkdir -p /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-compose-plugin
sudo usermod -a -G mml media
sudo docker network create wekode.mml

Configure firewall (optional)​

Default the ssl port 443 will be used for extern communication. Create firewall rules based on you infrastructure and your needs. We do not provide any suggestions on this. You are responsible for the security of your infrastructure.

Domain and certificates​

SSL/TLS​

All services are running using ssl/tls encryption. We need a fullchain cert with it's private key and a pfx cert file to run the service.

Generate a ssl/tls certificate based on your needs for your custom domain and copy it to the certs folder.

touch ~/medialib/certs
*.crt *.key *.pfx

Update the required values in the ~/medialib/.env file.

cat .env

## ssl certification configuration
## path, where to find the ssl certificate on host
VOL_CERTS=/home/mml/medialib/certs
## the ssl certificate name on host for .crt and .key without extension.
CERT_NAME=domain.name

Encryption and Signing​

The identity service signs and encrypts its tokens with two certificates. They are created automatically on the first start in the folder IDENTITY_SECRETS_PATH as wekode.mml.signing.pfx and wekode.mml.encryption.pfx. No certificate has to be generated manually.

Keep this folder when updating the service. If the certificates are lost, new ones are created and all admins have to log in again.

The certificates can be protected with a password by setting SIGN_SECRET and ENCRYPT_SECRET in the .env file. You can generate a random secret using openssl rand -base64 32. Certificates created without password are protected automatically when a secret is set later. Do not change a secret afterwards, otherwise the certificate cannot be read anymore.

Configure database service​

Create folders​

As database server postresql is used. Each service has it's own database. Create first some fodler to store configuration and databases persistence on host.

mkdir db db/secrets db/config db/data

Database inital scripts​

Copy the database initial scripts to the server into ~/medialib/db/initdb.d. This scripts will create the databases for the services. Make the scripts executable.

chmod u+x ~/medialib/db/initdb.d/identity-db.sh ~/medialib/db/initdb.d/media-db.sh

Postgres configuration file​

Copy the postres configuration file to ~/medialib/db/mml-postgres.conf. Remove the .default part of the filename. Here you can configure your database server on your needs or just use the default configuration.

Root database user​

Create an user and password file and enter your root db user and root db password

touch db/secrets/user
touch db/secrets/password

Identity database configuration​

Create file with filename identity.conf and configure your user, password and database name for the identity service.

# create identity.conf with content
touch db/config/identity.conf

IDENTITY_DB=identity
IDENTITY_USR=identity
IDENTITY_PW=identity

Media database configuration​

Create file with filename media.conf and configure your user, password and database name for the media service.

# create media.conf with content
touch db/config/media.conf

MEDIA_DB=media
MEDIA_USR=media
MEDIA_PW=media

Update .env file and start service​

Update the required values in the ~/medialib/.env file.

cat .env

## db configuration section
## The docker tag to be used.
# POSTRES_VERSION=latest
## Path to the file on host where the root user credentials are stored.
SECRETS_VOL=/home/mml/medialib/db/secrets
## Path to the file on host where the root user password is stored.
POSTGRES_PASSWORD_FILE=password
## Path to the file on host where the root user name is stored.
POSTGRES_USER_FILE=user
## Path to the file of tzhe postres configuration file on host based on the mml-postgres.default.conf
PG_CONFIG_VOL=/home/mml/medialib/db/mml-postgres.conf
## Path to the database persistent data on host.
PGDATA_VOL=/home/mml/medialib/db/data
## Path to the folder on host where to persist db configs for dbs and users (configured files in ./db/default-db-configs).
DB_CONFIG_VOL=/home/mml/medialib/db/config

Set user rights of the folder to the docker user and start the database service.

sudo chown -R 1001:1001 db/
sudo docker compose up wekode.mml.db -d

Upgrading postgres version​

It is recommended to use a fixed version of the postgres docker image and not the latest tag. When you want to upgrade to newer version of postgres follow the following steps:

Stop all running services​

docker compose down

Create dumps of all databases you want to backup​

docker compose up wekode.mml.db -d
docker exec -it wekode.mml.db pg_dump <database> -U <superuser> > <database>.sql

Stop Database service​

docker compose down wekode.mml.db

Backup the db data directory​

mv <db-folder-on-host>/ db-old/
mkdir <db-folder-on-host>

Prepare the new version​

Increment the version of postgres in the env file and start the container.

docker compose up -d wekode.mml.db

Import the db dumps​

docker exec -i wekode.mml.db psql -U <superuser|dbuser> -d <database> -f <database>.sql

Clean up​

rm -rf db-old
rm <database>.sql
docker compose up -d

Export records to csv​

Run \copy (Select record_id, title, date, track_number, duration, a.album_name as album, ar.name as artist, g.name as genre FROM records r JOIN albums a ON a.album_id = r.album_id JOIN artists ar ON ar.artist_id = r.artist_id JOIN genres g ON g.genre_id = r.genre_id) TO '/data/records.csv' WITH DELIMITER ',' CSV HEADER; as media database user inside the db container to export the list of records and the metadata to csv to use it for reoporting or other tools.

Configure cache service​

Redis is used as cache. Setup redis with following steps.

Create folder​

Create the folder where the cache configuration will be.

mkdir cache

Redis configuration​

Copy the default.redis.conf file to ~/medialib/cache/redis.conf. Remove the default. part of the filename. Replace <password> with your own password.

Update .env file and start service​

Update the required values in the ~/medialib/.env file.

cat .env

## redis cache configuration section
## The docker tag to be used.
#REDIS_VERSION=latest
## Path to the folder on host which stores the default.redis.conf file
CONFIG_PATH=/home/mml/medialib/cache

The docker of the redis image runs for user with id 999. Update the user rights of the folder to the redis user and group 999 and start the cache service.

sudo chown -R 999:999 cache/
sudo docker compose up wekode.mml.cache -d

Configure message bus​

RabbitMQ is used as message bus between backend services.

Create folder​

Create the folder where the configuration will be.

mkdir mbus

Configuration​

Copy enabled_plugins to ~/medialib/mbus/enabled_plugins on the server. Copy default.rebbitmq.conf to ~/medialib/mbus/rabbitmq.conf on the server. Remove the default. part of the filename. Change the default user guest and password guest in rabbitmq.conf.

Update .env file and start service​

Update the required values in the ~/medialib/.env file.

cat .env

## mbus configuration section
## The docker tag to be used for the rabbitmq image.
# MBUS_VERSION=latest
## Path to the enabled_plugins file on host.
ENABLED_PLUGINS_FILE=/home/mml/medialib/mbus/enabled_plugins
## Path to the default.rabbitmq.conf file on host.
CONFIG_FILE=/home/mml/medialib/mbus/rabbitmq.conf

Set user rights of the folder to the docker user and start the service.

sudo chown -R 1001:1001 mbus/
sudo docker compose up wekode.mml.mbus -d

Configure reverse proxy​

Nginx is used as reverse proxy and api gateway. This is the entrypoint to the backend from extern requests of the admin and mobile app.

Create folder​

Create the folder where the configuration will be.

mkdir proxy

Configuration​

Copy default.conf.template to ~/medialib/proxy\default.conf.template on the server.

Update .env file and start service​

Update the required values in the ~/medialib/.env file.

cat .env

## proxy configuration section
## The docker tag to be used.
# NGINX_VERSION=latest
## The ssl port on the host machine for requests to the mml-app from extern. Recomended to be changed to another port.
SSL_PORT=5555
## The port the reverse proxy is listen on in the docker file.
#INTERN_SSL_PORT=5050
## Path to the folder where the default.conf.template file is stored on host.
CONF_TEMPLATE=/home/media/medialib/proxy
## Upload file limit to prevent huge files to be uplaoded to server.
## Must be larger than Federation:ChunkSizeMb of every partner instance (default 32 MB), which sends records to this instance.
#FILE_UPLOAD_LIMIT=100m
...

Set user rights of the folder to the docker user and start the service.

sudo chown -R 1001:1001 proxy/
sudo docker compose up wekode.mml.reverseproxy -d

Generate App Keys​

App keys will be used to verify requests to the backend. Only requests containing the app-key in the header will be accepted. Two app keys are required. One for the admin app and one for the mobile app. The app keys are Guids. Generate two random Guids as app keys.

Configure identity service​

The identity service is used for managing the users, clients and logins of My Media Lib.

Create folder​

Create the folder where the configuration will be.

mkdir identity

Configuration​

Copy the default.appsettings.json to ~/medialib/identity/appsettings.json on server. Remove the default. part of the filename.

Replace the app keys.​

cat ~/medialib/identity/appsettings.json

...
  "ADMIN_APP_KEY": "<admin_app_key>", 
  "APP_KEY": "<app_key>",
...

Replace the name of the pfx certificate and set your pfx password.​

"TLS": {
    "Cert": "/certs/<name of cert>.pfx",
    "Password": "<password>"
  },

Set issuer url to your domain and port you are using as ssl port.​

"OpenId": {
    "AccessTokenLifetimeMinutes": "60",
    "RefreshTokenLifetimeMinutes": "43200",
    "RefreshTokenReuseLeewaySeconds": "10",
    "TokenLifespanMinutes": "15",
    "CleanOrphanTokenDays": "1",
    "Issuer":  "https://<url>:<port>/"
  },

Replace the connection string with values you defined when creating the database for the service. And set the redis password you defined when setting up the cache service.​

"ConnectionStrings": {
    "IdentityConnection": "Server=wekode.mml.db;Port=5432;Database=identity;User Id=identity;Password=identity;",
    "DistributedCache": "wekode.mml.cache:7379,password=<password>"
  },

Replace the rabbitmq user and password with the values you defined when setting up the message bus service.​

"MessageBus": {
    "Host": "wekode.mml.mbus",
    "User": "guest",
    "Password": "guest",
    "VirtualHost": "/",
    "WaitUntilStarted": true,
    "StartTimeoutSeconds": 10,
    "StopTimeoutSeconds": 30
  },

Create api client for media service​

To allow the media service to validate access tokens one oauth client need to be generated, so the media service can send requests to the identity service. The values are just random Guids. The clients defined here will be created automatically on startup.

"ApiClients": [
    {
      "ClientId": "<ClientId>",
      "ClientSecret": "<ClientSecret>"
    }
  ]

Update .env file and start service​

Update the required values in the ~/medialib/.env file.

Actual Version

Actual docker image can be found on docker-hub.

cat .env

## identity configuration section
## The docker tag to be used.
IDENTITY_VERSION=1.0.2
## Path and name on host machine where the appsettings is stored.
## for example /app/config/appsettings.json
IDENTITY_APPSETTINGS=/home/mml/medialib/identity/appsettings.json
## Optional. Secrets to protect the encryption and signing certificates.
#ENCRYPT_SECRET=
#SIGN_SECRET=
## The path where the certificates of the identity are stored.
IDENTITY_SECRETS_PATH=/home/mml/medialib/identity/secrets

The encryption and signing certificates are created in IDENTITY_SECRETS_PATH on the first start. Set user rights of the folders to the docker user and start the service.

mkdir -p identity/secrets
sudo chown -R 1001:1001 identity/
sudo docker compose up wekode.mml.identity -d

Configure media service​

The media service is used for managing the uploaded records of My Media Lib.

Create folder​

Create the folder where the configuration and uploaded records will be.

mkdir media media/records

Configuration​

Copy the default.appsettings.json to ~/medialib/media/appsettings.json on server. Remove the default. part of the filename.

Replace the app keys.​

cat ~/medialib/identity/appsettings.json

...
  "ADMIN_APP_KEY": "<admin_app_key>", 
  "APP_KEY": "<app_key>",
...

Replace the name of the pfx certificate and set your pfx password.​

"TLS": {
    "Cert": "/certs/<name of cert>.pfx",
    "Password": "<password>"
  },

Replace the connection string with values you defined when creating the database for the service.​

"ConnectionStrings": {
    "MediaConnection": "Server=wekode.mml.db;Port=5432;Database=media;User Id=media;Password=media;"
  },

Set issuer url to your domain and port you are using as ssl port.​

"OpenId": {
    "Issuer": "https://<url>:<port>/"
  },

Set the api client id and secret you defined when setting up the identity service.​

 "ApiClient": {
    "ClientId": "<ClientId>",
    "ClientSecret": "<ClientSecret>"
  },

Replace the rabbitmq user and password with the values you defined when setting up the message bus service.​

"MessageBus": {
    "Host": "wekode.mml.mbus",
    "User": "guest",
    "Password": "guest",
    "VirtualHost": "/",
    "WaitUntilStarted": true,
    "StartTimeoutSeconds": 10,
    "StopTimeoutSeconds": 30,
    "ConcurrentMessageLimit":  10
  }

Federation​

Federation synchronizes records with other My Media Lib instances. It is always enabled. Partners reach this instance at the issuer address configured in OpenId:Issuer. Set PublicUrl only if partners use another public HTTPS address, with the ssl port if it is not 443.

"Federation": {
    "PublicUrl": "",
    "AllowPrivateNetworks": false,
    "KeysPath": "/secrets/federation",
    "KeyRotationDays": 90,
    "KeyPublishLeadDays": 7,
    "KeyRetireDays": 14,
    "ChunkSizeMb": 32,
    "MaxRecordSizeMb": 2048,
    "UploadPath": "/tmp/records/federation",
    "UploadRetentionDays": 7,
    "MaxOpenTransfers": 10
  },
SettingDescription
PublicUrlPublic HTTPS address partners reach this instance at. Empty uses OpenId:Issuer. When the address changes, linked partners are informed after the next start, see address changes.
AllowPrivateNetworksAllows partners with addresses in private networks, e.g. in a local network. Disabled by default.
KeysPathFolder of the signing keys. Must be inside the secrets volume MEDIA_SECRETS_PATH configured below.
KeyRotationDays, KeyPublishLeadDays, KeyRetireDaysRotation of the signing keys: a new key every 90 days, published 7 days before use, the old key is removed 14 days later.
ChunkSizeMbSize of the parts records are sent in. Must be smaller than the FILE_UPLOAD_LIMIT of every partner.
MaxRecordSizeMbLargest record accepted from partners.
UploadPath, UploadRetentionDaysFolder of unfinished transfers from partners and the days they are kept without progress.
MaxOpenTransfersUnfinished transfers one partner may have at the same time. Further transfers are rejected until one is finished or removed.
Instance name

Partners are identified by the INSTANCE name in the .env file. Every instance needs a unique name consisting of letters, digits, ., _ or - with up to 64 characters. Do not change the name after the first start, the Media service refuses to start with another name, see Instance name.

Update .env file and start service​

Update the required values in the ~/medialib/.env file.

Actual Version

Actual docker image can be found on docker-hub.

cat .env

## media configuration section
## The docker tag to be used.
MEDIA_VERSION=1.2.1
## Path and name on host machine where the appsettings is stored.
## for example /app/config/appsettings.json
MEDIA_APPSETTINGS=/home/mml/medialib/media/appsettings.json
## Path to the folder on host where to backup the uploaded and indexed records.
VOL_RECORDS=/home/mml/medialib/media/records
## The path where the federation keys of the media service are stored.
MEDIA_SECRETS_PATH=/home/mml/medialib/media/secrets
## Optional. Secret to encrypt the federation keys stored in MEDIA_SECRETS_PATH.
#FEDERATION_KEY_SECRET=

The signing keys of the federation are created automatically in MEDIA_SECRETS_PATH. Keep this folder when updating the service, otherwise partners have to load new keys. The folder must be writable by the docker user; the service cannot create keys otherwise and federation requests fail.

FEDERATION_KEY_SECRET encrypts the stored keys. You can generate a random secret using openssl rand -base64 32. Keys stored without encryption are encrypted automatically when the secret is set later. If the secret is changed or removed, existing encrypted keys cannot be read anymore and the service reports an error. In this case delete the files in MEDIA_SECRETS_PATH: new keys are created and partners load them automatically.

Set user rights of the folders to the docker user and start the service.

mkdir -p media/secrets
sudo chown -R 1001:1001 media/
sudo chown -R 1001:1001 media/records
sudo chown -R 1001:1001 media/secrets
sudo docker compose up wekode.mml.media -d

Start backend​

You can now start all services.

docker compose up -d

Create first admin client and user​

No default user exists at the beginning. First you have to create one. It can be done with the command line interface inside the mml.identity project. When the service is running inside docker the call will be:

sudo docker exec -it wekode.mml.identity /bin/bash
root@6712536aabd:/app# create

You will be ask for a username and a password. If no admin app client id exists already, a new one will be created and the client id will be printed on the console.

Password length

The password of one admin user must be at least 12 characters long. Shorter passwords are not accepted.

Manage admin clients​

You can install the admin app on several computers for example if you have multiple admins. The admin users can be managed in your admin app. If you want to give them a different client id, you can generate a new one by using the command line interface inside the mml.identity project.

Admin clients can be created, listed and removed by the command line interface. To create one client hwne the service is running in docker call

docker exec -it wekode.mml.identity /bin/bash
root@6712536aabd:/app# admin-create

To list all admin client ids call:

docker exec -it wekode.mml.identity /bin/bash
root@6712536aabd:/app# admin-list

And to remove one client call and replace <client id> with your client id you want to remove:

docker exec -it wekode.mml.identity /bin/bash
root@6712536aabd:/app# admin-remove <client id>