fietsboek.fstrans module

Filesystem transactions.

Motivation

Fietsboek does a lot of filesystem stuff, such as saving images and track backups. Like database actions, we want to ensure that these actions happen “atomically” – if an error occurs during one action, we want to undo the previous ones. Similarly, if an error occurs after things have been sent to the database/filesystem, we want to ensure that we “clean up” (see issue 98).

By having “transactionized” file system actions, we can ensure that we do not not have such issues:

• Actions are reversible in case changes need to be rolled back.

• Actions are done all-or-nothing.

• Transactions are combined with other transactions (such as SQLAlchemy); if one fails, the others will be rolled back.

Implementation

The main mechanism is Transaction. It provides the commit/abort interface as required by the transaction module, and user-facing methods to enqueue filesystem modifications.

A transaction is started by begin(), which also joins the transaction to the other transactions managed by transaction.

A transaction records Action, which represent modifications to be done to the filesystem. Each action knows how it should be executed. Additionally, each action records how to undo it – the WriteBytes action, for example, records the previous file state so it can be restored.

This implementation has some drawbacks:

• Modifications are kept in-memory, which might be an issue for larger files.

• Extra time is needed to record previous states (read a file before it is overwritten).

• Changes are not visible to other programs until they are committed.

But the advantage is the ease of implementation: Cancelling a transaction just involves clearing the in-memory buffer, and there are no additional temporary files needed.

To further ensure that the filesystem is in a consistent state, the transactions use a lock file to get exclusive access. Currently, this lock spans the complete data directory. In the future, more fine-grained locking can be implemented.

Usage

The transaction is automatically used by the fietsboek.data.DataManager.

Interface

class fietsboek.fstrans.Action

Bases: object

Base class for any actions that can be applied to the filesystem.

commit_1()

Commit this action (phase 1).

This corresponds to tpc_vote, and may raise an exception if committing should be cancelled.

commit_2()

Commit this action (phase 2).

This corresponds to tpc_finish, and should not raise an exception.

undo()

Undo this action.

This is called if commiting fails, to undo any changes that were already committed.

class fietsboek.fstrans.MakeDir(path, exist_ok=False)

Bases: Action

Create the given directory.

commit_1()

Commit this action (phase 1).

This corresponds to tpc_vote, and may raise an exception if committing should be cancelled.

undo()

Undo this action.

This is called if commiting fails, to undo any changes that were already committed.

class fietsboek.fstrans.Purge(path)

Bases: Action

Purge (recursively remove) the given directory.

commit_2()

Commit this action (phase 2).

This corresponds to tpc_finish, and should not raise an exception.

class fietsboek.fstrans.RemoveDir(path)

Bases: Action

Remove the given (empty) directory.

commit_1()

Commit this action (phase 1).

This corresponds to tpc_vote, and may raise an exception if committing should be cancelled.

undo()

Undo this action.

This is called if commiting fails, to undo any changes that were already committed.

class fietsboek.fstrans.State(*values)

Bases: Enum

State of the transaction.

COMMITTED = 3

Transaction has been committed.

COMMITTING = 2

Transaction is in the process of being committed.

OPEN = 1

Transaction is open for further actions.

TAINTED = 4

Transaction is tainted.

class fietsboek.fstrans.Transaction(lock_path)

Bases: object

A transaction, recording pending filesystem changes.

abort(_trans=None)

Abort this transaction.

commit(_trans)

Commit this transaction.

This is required by the interface of transaction.

commit_1()

Start the first phase of committing.

This method is called automatically by the transaction manager.

commit_2()

Start the second phase of committing.

This method is called automatically by the transaction manager.

make_dir(path, *, exist_ok=False)

Creates the directory.

This is a transactioned version of Path.mkdir().

Parameters:

• path (Path) – The directory to create.

• exist_ok (bool) – If True, no error will be raised if the directory already exists.

purge(path)

Completely remove (recursively) the given path.

This uses shutil.rmtree() to delete the path.

Unlike other actions, this cannot be undone!

Parameters:

path (Path) – The directory to remove.

remove_dir(path)

Removes the (empty) directory.

This is a transactioned version of Path.rmdir().

Parameters:

path (Path) – The directory to remove.

sortKey()

Returns the sort key to sort this transaction in relation to others.

tpc_abort(_trans)

Abort the transaction, undoing all previously done changes.

This is required by the two-phase commit protocol of transaction.

tpc_begin(_trans)

Begin the transaction.

This is required by the two-phase commit protocol of transaction.

tpc_finish(_trans)

Commit (phase 2) the pending transaction.

This is required by the two-phase commit protocol of transaction.

This method should not raise an exception.

tpc_vote(_trans)

Commit (phase 1) the pending transaction.

This is required by the two-phase commit protocol of transaction.

This method may raise exceptions to signal that the transaction (and all linked transactions) should be aborted.

undo()

Undo all actions that have already been applied.

unlink(path)

Unlinks (removes) the given file.

This is a transactioned version of \(Path.unlink\).

Parameters:

path (Path) – The path to the file to unlink.

write_bytes(path, data)

Write the given bytes to the given path.

This is a transactioned version of Path.write_bytes().

Parameters:

• path (Path) – Path where to write the bytes to.

• data (bytes) – The data to write.

exception fietsboek.fstrans.TransactionalError

Bases: Exception

An exception that occurs when committing filesystem transactions.

class fietsboek.fstrans.Unlink(path)

Bases: Action

Remove the given file.

commit_1()

Commit this action (phase 1).

This corresponds to tpc_vote, and may raise an exception if committing should be cancelled.

undo()

Undo this action.

This is called if commiting fails, to undo any changes that were already committed.

class fietsboek.fstrans.WriteBytes(path, data)

Bases: Action

Write bytes to the given file.

commit_1()

Commit this action (phase 1).

This corresponds to tpc_vote, and may raise an exception if committing should be cancelled.

undo()

Undo this action.

This is called if commiting fails, to undo any changes that were already committed.

fietsboek.fstrans.begin(lock_path, tm=None)

Begin a new transaction and register it.

Parameters:

• lock_path (Path) – The path to the lock file to use in order to synchronize the transaction.

• tm – The transaction manager from transaction in which to join this transaction.

Return type:

Transaction

Returns:

The Transaction.