PipelineJobs Manager

This Reactor manages updates to PipelineJobs once they are created by other processes using the ManagedPipelineJob and ReactorManagedPipelineJob classes.

Update Job State

PipelineJobs Manager can update a job’s state via three mechanisms:

  1. Receipt of a JSON-formatted pipelinejob_manager_event
  2. Receipt of URL parameters sufficient to form a pipelinejob_manager_event
  3. Receipt of a agave_job_callback POST combined with uuid, status and token URL parameters

These named document formats are documented below in JSONSchemas.

JSON Event Messages

The default method for updating a PipelineJob’s state is to send a JSON message to the Manager actor. Here is an example of a finish event for job 1073f4ff-c2b9-5190-bd9a-e6a406d9796a, which will indicate the job has completed primary computation and archiving steps.

{
  "uuid": "1073f4ff-c2b9-5190-bd9a-e6a406d9796a",
  "name": "finish",
  "token": "0dc73dc3ff39b49a"
}

This can be sent directly over HTTP like so:

curl -XPOST -H "Authorization: Bearer 969d11396c43b0b810387e4da840cb37" \
    --data '{"uuid": "1073f4ff-c2b9-5190-bd9a-e6a406d9796a", \
    "token": "0dc73dc3ff39b49a",\
    "name": "finish"}' \
    https://api.tacc.cloud/actors/v2/<actorId>/messages

It can also be sent from within a Recator like so:

rx = Reactor()
manager_id = '<actorId>'
finish_mes = { 'uuid': '1073f4ff-c2b9-5190-bd9a-e6a406d9796a',
               'name': 'finish',
               'token': '0dc73dc3ff39b49a'}
rx.send_message(manager_id, finish_mes)

URL Parameters Event

The uuid, and event, token fields can be sent as URL parameters in an HTTP POST. The contents of the POST body will be attached to the event if one is present. Our finish event expressed as URL parameters looks like:

curl -XPOST --data '{"arbitrary": "key value data"}' \
    https://api.tacc.cloud/actors/v2/<actorId>/messages?uuid=1073f4ff-c2b9-5190-bd9a-e6a406d9796a&\
    event=finish&token=0dc73dc3ff39b49a

Agave Jobs Notification

HTTP POST body and URL parameters are combined to link the Agave Jobs system with PipelineJobs, which is quite handy as Agave jobs often are enlisted to do the computational heavy lifting in analysis workflows. This approach is demonstrated in the demo-jobs-reactor-app repository.

PipelineJobs Events

The state of every PipelineJob proceeds through a defined lifecycle, where transitions occur in response to receipt of named events. This is illustrated in the following image:

PipelineJob States

PipelineJobs Manager accepts any of the events (lower case words attached to the graph edges) save for create, which is reserved for other agents.

Authentication

POSTs to a PipelineJobs Manager must be authenticated by one of two means:

  1. Send a valid TACC.cloud Oauth2 Bearer token with the request
  2. Include a special URL parameter called a nonce with the HTTP request

JSON Schemas

agave_job_callback
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
{
    "$schema": "http://json-schema.org/draft-07/schema#",
    "$id": "https://schema.catalog.sd2e.org/schemas/agave_job_callback.json",
    "title": "AgaveJobsPost",
	"description": "A job status POST from the Agave API jobs service",
    "type": "object",
    "properties": {
        "owner": {
            "type": "string"
        },
        "appId": {
            "type": "string"
        },
        "executionSystem": {
            "type": "string"
        },
        "archivePath": {
            "type": "string"
        },
        "archiveSystem": {
            "type": "string"
        },
        "status": {
            "type": "string"
        },
        "inputs": {},
        "parameters": {},
        "_links": {}
    },
    "required": [
        "appId",
        "executionSystem",
        "owner",
        "archivePath",
        "archiveSystem",
        "status",
        "inputs",
        "parameters",
        "_links"
    ]
}
pipelinejob_manager_event
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
{
	"$schema": "http://json-schema.org/draft-07/schema#",
	"$id": "https://schema.catalog.sd2e.org/schemas/pipelinejob_manager_event.json",
	"title": "PipelineJobManagerEvent",
	"description": "A state-change event for a PipelineJob",
	"type": "object",
	"properties": {
		"uuid": {
			"$ref": "pipelinejob_uuid.json"
		},
		"name": {
			"$ref": "pipelinejob_eventname.json"
		},
		"data": {
			"description": "Additional information to attach to the event (optional)",
			"type": "object"
		},
		"token": {
			"$ref": "update_token.json"
		}
	},
	"required": ["uuid", "name"],
	"additionalProperties": false
}