skald/skald.md

# Skald manual

Please see the file [skald-install.md][1] for installation instructions.
Please see the section *Server administration* below for detailed
administration instructions.

## Usage

Skald includes a web server that, by default, listens on port 8443.
Skald is therefore accessed on URLs such as:

    https://skald.example.org:8443/

### The landing page

There is a landing page at the root of the server.  The landing page
contains a list of all halls marked "public" in their configuration, as
well as a form that allows joining an arbitrary hall.

Going through the landing page is not required: you are welcome to point
your browser directly at the desired hall URL.

### The hall pages

A hall named e.g. *city-watch* has an associated page
`/hall/city-watch/`.  After login, it presents an interface consisting
of a user list, chat, and connection controls:

  - the left side pane contains the list of users who have joined the
    hall; every username doubles as a menu of hall and user actions;
  - the chat pane contains the messages published to the hall;
  - the controls area contains login, connection, microphone, and settings
    controls.

On mobile, the interface is collapsed to fit the smaller screen.  The top
controls expose the user list, chat, and settings.

### Buttons

There are up to two buttons at the top.  *Enable* starts the microphone
when no local microphone stream is active.

The *Mute* button mutes or unmutes the microphone; the microphone can be
muted remotely by the hall moderator, but it cannot be unmuted remotely.
To stop and release the local microphone stream, use *Turn microphone off*
from the menu opened from your own name.

### Side menu

There is a menu on the right of the user interface.  It allows choosing
the microphone, selecting whether to receive audio, and adjusting audio
processing options.

### User list

There is a user list on the left, starting with the current user and
followed by all the users that have joined the current hall.

The user list doubles as a set of menus.  Clicking on the current user (the
first entry in the user list) opens the *hall menu*, a menu with actions
that apply to the hall as a whole.  Clicking on a different user opens
a *user menu*, a menu that applies to that specific user.

For operators who are allowed to record, the hall menu includes the current
recording control and an *Open recordings (opens in new tab)* action.  *Open
recordings (opens in new tab)* opens the hall's recording archive in a new tab,
which lists completed recordings when there are multiple files.

### Chat pane

The centre pane is a traditional chat interface, with an input form at the
bottom and the chat history above it.  Chat history is never saved to
disk, and is erased after four hours (or whatever is specified in the
`"max-history-age"` field of the hall definition).

Double-clicking on a message opens a contextual menu.

The chat form doubles as a command-line interface, which is especially
important for visually-impaired users, and more generally is often faster
than navigating the user interface.  Commands start with a slash character
"`/`".  The most important command is `/msg`, which sends a private
message to a given user.  Type `/help` to display the list of available
commands.

Below the chat input is a session-only virtual chalkboard.  It is a
multiline text area suitable for commands, code, and notes that should be
visible to everyone in the hall.  Operators and administrators may edit
the chalkboard and may grant or revoke chalkboard editing for connected
users with `/chalkboard user` and `/unchalkboard user`.  Users without
edit permission can read and focus the chalkboard, but it is marked
read-only and does not trap the Tab key.

### Inviting users

In order to generate an invitation link, choose the entry *Invite user* in
the hall menu.  This generates a link of the form

    https://skald.example.org:8443/hall/city-watch/?token=XXX

where the *XXX* part, known as the *token*, is a shared secret. Such
a link allows password-less login to the hall, and may therefore be
shared e.g. over e-mail or instant messaging.

The invitation functionality is usually restricted to the moderator;
however, halls may be configured with the `"unrestricted-tokens"` option,
which allows all users to generate tokens.

Tokens can be created, modified, and expired using the `/invite`,
`/reinvite`, and `/revoke` commands.

### File transfer

Skald includes a peer-to-peer, end-to-end encrypted file transfer protocol.
In order to transfer a file, click on the receiver's entry in the user
list and choose *Send file*.

### Hall moderation

If a user has the *op* permission (short for *operator*), then they have
access to a number of moderation tools.  The contextual menu that opens
when clicking on an entry in the user list is expanded with commands for
muting a user, sending them a warning, retrieving their IP address, or
kicking them out from a hall.

The hall menu (opened by clicking on one's own entry in the user's list)
is extended with options to lock or to unlock a hall (a locked hall is
one that non-operator users cannot join).  Operators with recording
permission can also start or stop recording and open the hall's recording
archive from the hall menu.

All of the moderation commands are also available as command-line commands
(see above), which is helpful when moderating large halls.

# Server administration

## The global configuration file

The server may be configured in the JSON file `data/config.json`.  This
file may look as follows:

```json
{
    "users":{"vetinari": {"password":"lagniappe", "permissions": "admin"}},
    "canonicalHost": "skald.example.org",
    "writableHalls": true,
    "recordingRetention": "168h"
}
```

or, better, with a hashed password:

```json
{
    "users": {
        "vetinari": {
            "password":{"type":"bcrypt","key":"$2a$10$bTWW..."},
            "permissions": "admin"
        }
    },
    "canonicalHost": "skald.example.org",
    "writableHalls": true,
    "recordingRetention": "168h"
}
```

The file is initially created using `skaldctl initial-setup`, but may be
manually edited at any time (there is no need to restart the server).  The
fields are as follows:

 - `users` defines the users allowed to administer the server, and has the
   same syntax as user definitions in halls (see below), except that the
   only meaningful permission is `"admin"`;

 - `writableHalls`: if true, then the HTTP administrative API can be used
   to modify hall definitions; if unset or false, then the HTTP API is
   read-only.  This does not affect local `skaldctl` commands;

 - `recordingRetention`: if set, completed recording files older than this
   duration are automatically deleted from the recordings directory.  The
   value uses Go duration syntax, for example `"24h"` for one day or
   `"168h"` for seven days.  If unset, recordings are kept until deleted
   manually;

 - `allowOrigin` is an array containing the list of HTTP origins that
   are allowed to access the server;

 - `allowAdminOrigin` is like `allowOrigin`, but applies to the HTTP
   administrative API;

 - `proxyURL`: if running behind a reverse proxy, this specifies the root
   URL that will be visible outside the proxy;

 - `canonicalHost`: the canonical name of the host running the server;
   clients that attempt to access the server using a different host name
   will be redirected to the canonical one.


## Hall definitions

Halls are described by JSON files in the `halls/` directory.  These
files are normally administered using the `skaldctl` utility, but may
also be edited manually (there is no need to restart the server).
`skaldctl` edits these files directly and must be run on the server as a
user that may write the hall directory.  By default it uses `./data` and
`./halls`; use the global `-data` and `-halls` options for other paths.

### Managing halls using `skaldctl`

#### Creating, modifying, and deleting halls

A hall is created using `skaldctl create-hall`:

```sh
skaldctl create-hall -hall city-watch
```

For a guided setup that asks for the hall name, common settings, operator
accounts, and an optional shared participant password, use:

```sh
skaldctl create-hall -interactive
```

There are a number of options to customise the behaviour of the hall, see
`skaldctl create-hall -help` for a full list.  For example, in order to
create a hall that allows unrestricted creation of tokens, say:

```sh
skaldctl create-hall -hall city-watch -unrestricted-tokens
```

For more advanced configuration, `skaldctl create-hall` can be invoked
with the `-json` flag, in which case it takes a JSON template from standard
 input.  The syntax of a JSON template is just like that of a hall
definition file (see below), except that it must not contain the fields
`users` and `wildcard-user`.  For example, in order to create a redirect
(see the section *Hall description reference* below):

```sh
echo '{"redirect": "https://skald.example.org:8443/hall/city-watch/"}' | skaldctl create-hall -hall amcw -json
```

Halls are modified using `skaldctl update-hall`:

```sh
skaldctl update-hall -hall city-watch -unrestricted-tokens=false
```

If a JSON template is provided to `skaldctl update-hall`, then it is
merged with the existing hall configuration.  Entries may be deleted
by setting them to `null` in the template:

```sh
echo '{"redirect": null}' | skaldctl update-hall -hall amcw
```

A hall is deleted using `skaldctl delete-hall`:

```sh
skaldctl delete-hall -hall amcw
```

#### Creating, modifying, and deleting users

A user entry is created with the `skaldctl create-user` command:

```sh
skaldctl create-user -hall city-watch -user Username -permissions op
```

If the `-permissions` flag is not specified, it defaults to `present`,
meaning that the user can participate in the chat and present audio to
the hall.  Other useful values are `message`, which allows a user
to participate in the chat only, and `observe`, which doesn't allow any
active participation.

A user is modified using `skaldctl update-user`, and deleted using
`skaldctl delete-user`.

In order to be useful, a user entry needs to be assigned a password.  This
is done with the `skaldctl set-password` command:

```sh
skaldctl set-password -hall city-watch -user Username
```

#### The fallback user

It is sometimes useful to allow multiple users to log in using the same
password.  This can be achieved by defining the *wildcard* user:

```sh
skaldctl create-user -hall city-watch -wildcard
skaldctl set-password -hall city-watch -wildcard
```

For open halls, where any user can login with any password, the wildcard
user's password is set to the password of type `wildcard`:

```sh
skaldctl set-password -hall city-watch -wildcard -type wildcard
```

See the section *Client authorisation* below for more information about
password types.

#### Automatic subhalls

It is sometimes necessary to create a large number of identical halls.
For example, the author has been using Skald to supervise computer
science practicals, where up to 40 students are working in pairs.

While it is possible to automate the creation of halls by scripting calls to
`skaldctl`, by accessing Skald's HTTP administrative API, or by generating files
directly under `halls/`, Skald provides a facility called *automatic
subhalls* that can be used to generate halls on demand.

Automatic subhalls are enabled by setting the `"auto-subhalls"`
field in the hall description:

```sh
skaldctl create-hall -hall unseen-university -auto-subhalls
```

Whenever a user attempts to access a subhall of `unseen-university`, for
example `unseen-university/hex`, the hall is created in memory and
persists until it is empty and its chat history has expired.  The main
hall's operator can view the list of populated subhalls with the command
`/subhalls`.

#### Managing tokens

Tokens are normally managed using the `/invite`, `/reinvite`, and `/expire`
commands in Skald's user interface, but they may also be managed using
the `skaldctl` utility's `create-token`, `revoke-token`, `delete-token`
and `list-tokens` commands:

```sh
skaldctl create-token -hall city-watch
skaldctl list-tokens -l -hall city-watch
```

A token that is generated with the `-include-subhalls` flag applies to
the whole hierarchy rooted at the given hall, including both ordinary
halls and automatically generated subhalls.

```sh
skaldctl create-token -hall city-watch -include-subhalls
```

Such a token can be attached to the root of the hall hierarchy, and
therefore be valid for any hall on the server:

```sh
skaldctl create-token -hall '' -include-subhalls
```

### Hall description reference

The definition for the hall called *hallname* is in the file
`halls/hallname.json`; it does not contain the hall name, which makes
it easy to copy or link hall definitions.  You may use subdirectories:
a file `halls/teaching/networking.json` defines a hall called
*teaching/networking*.

Every hall definition file contains a single JSON dictionary.  All fields
are optional.  The following fields are allowed:

 - `users`: a dictionary that maps user names to user descriptions (see
   below);

 - `wildcard-user` a user description that will be used for usernames
   with no matching entry in the `users` dictionary;

 - `authKeys`, `authServer` and `authPortal`: see *Authorisation* below;

 - `public`: if true, then the hall is listed on the landing page;

 - `displayName`: a human-friendly version of the hall name; this is
   displayed at the top of the hall page;

 - `description`: a human-readable description of the hall; this is
   displayed on the landing page for public halls;

 - `contact`: a human-readable contact for this hall, such as an e-mail
   address, ignored by the server;

 - `comment`: a human-readable string, ignored by the server;

 - `max-clients`: the maximum number of clients that may join the hall at
   one time;

 - `max-history-age`: the time, in seconds, during which chat history is
   kept (default 14400, i.e. 4 hours);

 - `not-before` and `expires`: the times (in ISO 8601 or RFC 3339 format)
   between which joining the hall is allowed;

 - `allow-recording`: if true, then recording is allowed in this hall;
   recordings are single mixed Ogg Opus audio files and require `ffmpeg`
   on the server;

 - `unrestricted-tokens`: if true, then ordinary users (without the "op"
   privilege) are allowed to create tokens;

 - `allow-anonymous`: obsolete and ignored.  Empty usernames are rejected;

 - `auto-subhalls`: if true, then subhalls of the form `hall/subhall`
   are automatically created when first accessed;

 - `autolock`: if true, the hall will start locked and become locked
   whenever there are no clients with operator privileges;

 - `autokick`: if true, all clients will be kicked out whenever there are
   no clients with operator privileges; this is not recommended, prefer
   the `autolock` option instead;

 - `redirect`: if set, then attempts to join the hall will be redirected
   to the given URL; most other fields are ignored in this case;

 - `codecs`: a list of codecs allowed in this hall, see below for
   possible values.  The default is `["opus"]`.

A user definition is a dictionary with entries `password` and
`permission`.  The value of the `password` field is either a plaintext
password, or a hashed password generated for example by the `skaldctl
hash-password` command.  The value of the `permissions` field can either
be an array of individual permissions (not recommended), or one of the
following strings:

 - `op`: a hall operator, with all rights except administering the hall;
 - `present`, an ordinary user with the right to publish audio streams
   and send chat messages;
 - `message`: a user with the right to send chat messages;
 - `observe`: a user that receives media streams and chat messages, but
   is not allowed to send them;
 - `admin`: a user with the right to administer the hall (only).

Usernames must be non-empty and unique among connected non-system clients in a
hall.  If a `wildcard-user` is configured, users may choose arbitrary names
that are not listed in `users`, but two connected users may not use the same
name.

The value of the `codecs` field is an array of codecs allowed in the
hall.  Supported audio codecs include `"opus"`, `"g722"`, `"pcmu"` and
`"pcma"`.  Opus is the default and is the codec used by Skald's
single-file recording path.

## Recording

When `allow-recording` is enabled and an operator starts recording, Skald
creates one recording file for the hall session under the configured
recordings directory.  The file is named
`skald-recording-YYYYMMDD-HHMMSS.ogg` and contains a single mixed Ogg Opus
audio stream.

Skald uses one `ffmpeg` decoder process for each active incoming Opus RTP
audio source, mixes decoded 48 kHz stereo PCM every 20 ms, and sends the
mixed stream to one `ffmpeg` encoder process that writes the Ogg Opus file.
The mixer treats missing source frames as silence, applies deterministic
headroom and limiting before encoding, and logs recording metrics when the
file closes: active sources, decoder starts and stops, dropped source frames,
stale source ticks, dropped mixed frames, and encoder write errors.

The encoder is fed through a bounded queue so a stuck encoder cannot block
WebRTC packet handling indefinitely.  If the queue fills, Skald drops mixed
frames and logs the condition.  If `ffmpeg` is not installed or cannot be
started, recording fails with an operator-visible error instead of silently
creating a broken file.
The recorder closes when recording is stopped explicitly or when the hall no
longer has any non-system clients.

Maintainers can exercise the recording mixer and encoder without browser
clients by running:

```
go run ./cmd/skald-recording-smoke -out /tmp -sources 5 -duration 10s
```

This creates a real mixed Ogg Opus file from synthetic PCM sources and prints
the same core counters that are useful when checking larger halls.

If `recordingRetention` is set in `data/config.json`, Skald periodically
deletes completed recording files older than the configured duration.

## Client Authorisation

Skald implements three authorisation methods: a username/password
authorisation scheme, a scheme using stateful tokens, and a mechanism
based on cryptographic tokens.  The former two mechanisms are intended to
be used in standalone installations, while the cryptographic mechanism is
designed to allow easy integration with an existing authorisation
infrastructure (such as LDAP, OAuth2, or even Unix passwords).

### Password authorisation

When password authorisation is used, authorised usernames and passwords are
defined directly in the hall configuration file, in the `users` and
`wildcard-user` entries.  The `users` entry is a dictionary that maps user
names to user descriptions; the `wildcard-user` is a user description
that is used with usernames that don't appear in `users`.  These two
entries are usually managed by the `skaldctl` utility.

Every user description is a dictionary with fields `password` and
`permissions`.  The `password` field may be a literal password string, or
a dictionary describing a hashed password or a wildcard.  The
`permissions` field should be one of `op`, `present`, `message` or
`observe`.  (An array of Skald's internal permissions is also allowed,
but this is not recommended, since internal permissions may vary from
version to version.)

For example, the entry

```json
{
    "users": {"Username": {"password": "Password", "permissions": "op"}}
}
```

specifies that user "Username" may log in as operator with password
"Password", while

```json
{
    "wildcard-user": {"password": "1234", "permissions": "present"}
}
```

allows any username with the password *1234*.  Finally,

```json
{
    "wildcard-user":
        {"password": {"type": "wildcard"}, "permissions": "present"}
}
```

allows any username with any password.

### Hashed passwords

For security reasons, passwords are usually hashed before being stored in
hall descriptions (in fact, the `skaldctl` utility does not even support
storing plaintext passwords).  A hashed password is represented as a JSON
dictionary with a field `type` and a number of type-specific fields.

A user entry with a hashed password looks like this:

```json
"users": {
    "Username": {
        "password": {
            "type": "pbkdf2",
            "hash": "sha-256",
            "key": "f591c35604e6aef572851d9c3543c812566b032b6dc083c81edd15cc24449913",
            "salt": "92bff2ace56fe38f",
            "iterations": 4096
        },
        "permissions": "op"
    }
}
```

Hashed passwords are normally generated transparently to the user by the
`skaldctl set-password` command.  When editing hall description files
manually, hashed passwords can be generated with the `skaldctl hash-password`
utility.

### Stateful tokens

Stateful tokens are created by the `/invite` command in the Skald user
interface or by the `skaldctl create-token` command; see the section
*Managing tokens* above.  They are stored in the file
`data/var/tokens.jsonl`, which, on most filesystems, can be safely backed
up without stopping the server.

### Cryptographic tokens

In many cases, it is useful to delegate authorisation decisions to a third
party, such as an LDAP or OAuth2 client.  Skald implements delegation of
authorisation decisions using cryptographic tokens generated by a third
party known as an *authorisation server*.  Two authorisation servers are
available: an [LDAP client][2], and a [sample server written in Python][3].

When an authorisation server is used, the `"authKeys"` entry of the hall
configuration file specifies one or more public keys in JWK format (with
the restriction that the "alg" key must be specified explicitly):

```json
{
    "authKeys": [{
        "kty": "oct",
        "alg": "HS256",
        "k": "MYz3IfCq4Yq-UmPdNqWEOdPl4C_m9imHHs9uveDUJGQ"
    }, {
        "kty": "EC",
        "alg": "ES256",
        "crv": "P-256",
        "x": "dElK9qBNyCpRXdvJsn4GdjrFzScSzpkz_I0JhKbYC88",
        "y": "pBhVb37haKvwEoleoW3qxnT4y5bK35_RTP7_RmFKR6Q"
    }]
}
```

If multiple keys are provided, then they will all be tried in turn, unless
the token includes the "kid" header field, in which case only the
specified key will be used.

The hall file should also specify either an authorisation server or an
authorisation portal.  An authorisation server is specified using the
`"authServer"` key:

```json
{
    "authServer": "https://auth.example.org"
}
```

If an authorisation server is specified, then the client, after it prompts
for a password, will request a token from the authorisation server and
join the hall using token authentication.  The password is never
communicated to the server.

Alternatively, the hall file may specify an authorisation portal using
the `"authPortal"` key.  If an authorisation portal is specified, then the
default client will redirect initial client connections to the
authorisation portal.  The authorisation portal is expected to authorise
the client and then redirect it to Skald with the `username` and `token`
query parameters set.

[1]: <skald-install.md>
[2]: <https://git.stormux.org/storm/skald-imap/>
[3]: <https://git.stormux.org/storm/skald-sample-auth-server/>