The Updater¶

The process of updating for administrators is described in Upgrading. Here, we will describe the developer side of the update process.

Motivation¶

Some updates do not only change the code of Fietsboek, but adapt the data that is stored. New features for example might require new columns in the database, or other adaptions might move data around (commit 7a60619d for example moves the GPX data out of the database and into the data directory).

This process is commonly called data migration. In Fietsboek, we use Alembic to do the heavy lifting of SQL migrations. Alembic can automatically create migration scripts and provides an abstraction over the different database backends.

With the image uploads, and more concretely with commit 5a205756, Fietsboek has been extended to store data on disk. This has resulted in issue 20, which raises the question about how to deal with update scripts now: Alembic handles the SQL updates well, but now we have a second place to run migrations for.

This has resulted in the fietsupdate tool, which implements a similar logic as Alembic, but for general purpose Python code instead of SQL. It also acts as the user interface for administrators, providing some abstractions over the Alembic CLI.

Update scripts¶

An update script contains the logic that fietsupdate applies when going from one version to the next. Such a script looks like the this:

"""Revision upgrade script v0.8.0

Date created: 2023-06-05 21:00:37.464583
"""
from fietsboek.updater.script import UpdateScript

update_id = 'v0.8.0'
previous = [
    'v0.7.0',
]
alembic_revision = '3149aa2d0114'


class Up(UpdateScript):
    def pre_alembic(self, config):
        pass

    def post_alembic(self, config):
        pass


class Down(UpdateScript):
    def pre_alembic(self, config):
        pass

    def post_alembic(self, config):
        pass

Each script imports the UpdateScript class as a base.
Each update has a unique ID, which is used on the command line to identify the correct script. For most scripts, this is a randomly generated string, but some specific revisions have a more readable ID.
Each update (except for the initial one) has references to its previous versions. As such, the commits form a DAG. This allows fietsupdate to check which updates have been applied, and which still need to be applied.
Each update has a reference to the underlying alembic version. When fietsupdate applies the update, it will make sure that the database will have the given alembic version after.
Finally, each update has the actual update logic, in four sections: Migrations that are run before the alembic update and after the alembic update, each for the upgrade and the downgrade direction.

Creating Alembic versions¶

There is nothing special about how we use Alembic to generate SQL revisions. As such, we refer to the Alembic documentation.

In general, a command like this should work:

alembic -c development.ini revision --autogenerate

Creating Fietsupdate versions¶

The fietsupdate has a hidden command to generate a revision from the template, and it will populate it with the current alembic revision:

fietsupdate revision -c development.ini

When to create revisions¶

If you change Fietsboek code or documentation that does not change the way that data is stored, you do not need to bother with Alembic or Fietsupdate.

If you change the database schema, or the way that the data is stored in the existing schema, you need to create an Alembic revision.

If you change the data that is stored in the data directory, you need to create a Fietsupdate script.

If a new Fietsboek version is released, a new Fietsupdate migration is created with the version as revision ID, to ensure that fietsupdate update v0.1.2 will work. This script is usually empty, and only contains a reference to the relevant previous migration ID and the correct alembic revision.

Further notes¶

When running Fietsboek from master, it might happen that the repository is in a state in which Alembic revisions exist, but no Fietsupdate revision that refers to them. In this case, you must run

alembic -c development.ini upgrade head

Before creating revisions, make sure that your repository and database are in the correct state. Otherwise your revision might refer to an outdated revision. In particular, you should ensure that you have applied all previous updates.

While both Alembic and Fietsupdate have support for “branches” and “merges”, it is still preferable to use this feature as little as possible. If possible, rebase the changes to give them a linear order.

We cannot guarantee that migration scripts will always continue to work (e.g. if they depend on functionality which is later removed from Fietsboek, dependencies that are removed, …). For this reason, big update jumps might be hard to support. However, all scripts should at least stay loadable (that means their metadata is readable, there are no syntax errors), and for empty databases they should work (to allow bootstrapping of fresh instances).