Workflow Management System

This portion of the project followed the development of a workflow manaagement system. The purpose of this system was to effectively support and manage the tasks present within the Zamani Project. The supported tasks within the system are user tasks and automated server tasks. Tasks are topologically organised and the system is able to dynamically execute the workflow based on task completion.

User Tasks

Dataflow for User Tasks
Data Transfer in User Tasks.

This refers to tasks executed by the user, on a remote workstation. User tasks require files to be copied to the user's workstation. Once the files are copied the user would then be able to finish the task. Once the user has completed a task, the transfer process is initiated in reverse.

In order to improve task accuracy, all user tasks need to be validated by senior members of the team. This prevents incomplete or invalid tasks from contaminating the rest of the process.

Tasks are assigned to users during the site set up. Users are then able to interact with the system via the interface provided. This allows users to:

  • Update Task Status
  • Obtain information about the task
  • Initiate file downloads and uploads


Server Tasks

These tasks are executed on the server itself. This is done in order to prevent unnecessary file transfer. Tasks are split into two main categories: One-to-One task and Many-to-One tasks.

One-to-One

Dataflow for User Tasks
Types of server tasks

For this task type, the execution script is run once for every input file in the list. This allows for effective mapping of input files.

Many-to-One

Tasks for this type send all the input files to the execution script at once. The process of merging these input files to a set of output files are left to the execution script allowing for maximum flexibility.

Workflow Setup

In order for the system to execute the tasks. The workflow needs to be setup first. The system provides an interface in which task can be set up. These are visualised as nodes in a graph. Dependencies can be added on the visualisation, by dragging the appropriate arrow.

Dataflow for User Tasks
Sample workflow

The status of the task is represented by the colour of the nodes. Tasks the that have not been started are grey, in progress tasks are blue, completed tasks are green and tasks with errors are represented with red.

Every task has a type. This is known as a Job. Jobs define the script that gets called when the task is executed. It also contains the description of the task that would be presented to users.

Task Failure

All activity that the task performs is logged. This includes the transfer logs for user tasks and all activity on stderr and stdout for server task. In the event of a task failing users can use these logs to determine the cause of the fault. Once this has been remidied the task can be restarted.

Implementation

The system was written in Python using the Django Framework. Various frontend tools are also used:

  • JQuery
  • jsPlumb
  • jqxWidgets

Evaluation

A User Experience evaluation, performed on 24 users, showed that 18 users found the system easy to use, 19 found it easy to learn. Overall users were satisfied using the system. Observations made during the evaluation showed that the system is effective and useful in completing the task at hand. It was shown that the system was able to successfully execute a portion of the tasks present within the Zamani Project. This set included a mix of user tasks and automated server tasks.

Source:
workflow_source.zip