User:Roshi/QA/Taskotron Overview

Taskotron Overview

Taskotron is the CI system Fedora currently uses for running automated tests. Currently it runs tests for all Fedora packages, docker base images, as well as Atomic and Cloud images. Taskotron is able to support a wide variety of tasks and reporting mechanisms. While it's fairly straight forward to write tasks for local execution, it's not entirely clear how your task fits into Taskotron in Production. This document aims to be an overview of how data gets from one component to another inside a production instance of Taskotron. Here's a high level flow chart:

The following sections define what happens at each numbered stage in the flow chart.

1) FedMsg Input

Pretty much every aspect of Fedora infrastructure emits status messages to the FedMsg bus. We use this bus to coordinate and kick off a variety of tasks, from desktop notifications to kicking off automated testing. It's a constant stream of event data. Taskotron listens to the FedMsg bus with the "taskotron-trigger" package.

2) Taskotron-trigger

Trigger sets up a local fedmsg-hub with several custom "consumers." These comsumers look for specific FedMsg topics. For instance, the cloud_compose_complete_msg.py jobtrigger listens for 'org.fedoraproject.prod.pungi.compose.status.change' messages, and then kicks off the CloudComposeJobTrigger.

3) Massage the Data

After an fedmsg is consumed, the "JobTrigger" parses the data from the FedMsg to prepare it for use by the execution engine. In our CloudComposeJobTrigger example, it first checks to see if the message is saying that the Pungi compose is "FINISHED." If it's "FINISHED," it'll continue to process the data, otherwise it discards the message. That particular fedmsg only contains the following fields: 'status,' 'location,' and 'compose_id.'

When the JobTrigger is done, it returns a python dictionary with the following fields (this is not an exhaustive list - these can be anything):

• _msg: The raw fedmsg we parsed
• message_type: the type of message, which is CloudCompose or AtomicCompose for our example
• item: This is the data we want to hand to our task for execution. In our case, it's a URL that points to the qcow2 image the Pungi compose generated.
• item_type: What kind of item the task will run on, in our case a 'compose'
• name: The human readable name of what's being passed in the item. In our example it'd be Fedora-Cloud or Fedora-Atomic.
• version: What version of the thing are we using? In our example it'll be 24 (or whatever release of Fedora this fedmsg was for). Other JobTriggers operate on packages, and the version and release fields would reflect which package is being referenced in the fedmsg.
• release: This is very similar to version, and for our example it would be the date the compose was executed on. For example, 20170310.0.

Once this data is ready, it's paired with the task defined in the 'trigger_rules.yml' config file. For our example, it looks like this:

- when:
    message_type: AtomicCompose
  do:
    - {tasks: [upstream-atomic, fedora-cloud-tests]}

The 'message_type' inside the dict the JobTrigger generated maps directly to the trigger rule. The next question is where does the task itself live and how does trigger know what it is? Each task lives in it's own repository, which taskotron mirrors locally to speed up execution times. For our example, upstream-atomic and fedora-cloud-tests live on pagure.

Let's take a look at what exactly a "task" is and what they're made of. There are already docs for this, but here's a brief overview. Each task contains at least 2 things: a task.yml file and some bit of code that is the task. The task.yml contains all the information required by the runner to execute the task. Our fedora-cloud-tests example looks like this:

---
name: fedora-cloud-tests
desc: Run the Fedora Cloud or Atomic tests against a qcow2 image
maintainer: roshi

environment:
  rpm:
    - ansible
    - testcloud
    - git

actions:
  - shell:
    # Clone the upstream test repo
    - git clone https://pagure.io/fedora-qa/cloud-atomic-ansible-tests.git

  - python:
      file: run_cloud_tests.py
      callable: run
      test_image: ${compose}
      artifactsdir: ${artifactsdir}
    export: output

  - shell:
      - ignorereturn:
          - rm inventory

  - shell:
      - rm -rf cloud-atomic-ansible-tests

  - name: report results to resultsdb
    resultsdb:
      results: ${output}

The 'environment' section defines what the runner needs to have installed in order to run. Namely it needs to have the ansible, testcloud and git packages installed on the buildslave to run the task. The 'actions' section defines what steps the task needs to take in order to run. You see a couple different items in there, namely the 'shell' and 'python' directives. The 'shell' calls act exactly like you'd expect, they run raw shell commands. The 'python' directive takes a couple options to run.

First is 'file' which tells it which python file to execute (this is located in the same directory as task.yml). 'callable' is which method inside the 'file' to call. Those are the only two options required when it comes to running python files inside a task. The last two options (test_image and artifactsdir) are keyword arguments that get passed into the method 'callable' points to (you can set as many of these as you like - they just have to match the keyword arguments required as input to the method referenced in 'callable').

'${compose}' is a passed in argument from the runner , and '${artifactsdir}' is an absolute path to where the runner stores task artifacts (like log files), which is configured in the libtaskotron installation. The 'item' in the data dictionary created by the JobTrigger ends up in the '${compose}' variable for our task when it's run by the execution engine.

The last section of the task.yml file is how we get the restults of the task into resultsDB. The output of the python call gets save to 'output' (in the 'export: output' section of the python directive), which is then referenced for submission to resultsDB. Now that we've had an overview of what a task is, let's get back to the rest of the overview.

4-5) Queue the Task

These two things, the task to be run, and the data dict created by the JobTrigger, are bundled together and sent to the execution engine to be queued for execution. The execution engine gets these two bits of information, and then finds an available job runner to execute the task on.

6) Run the Task

The job runner (buildslave) receives the data from the buildmaster and then executes the task. If we were to run this locally, it would look like this:

 runtask -t data['message_type'] -i data['item'] runtask.yml

The '-t' flag is the type of item we want to run on (in this case, a 'compose') and the '-i' flag is for the item we're going to run against. The value passed in via the '-i' flag gets referenced by the task in task.yml as '${compose}' which corresponds to the 'message_type' passed in via the '-t' flag.

7) Report to resultsDB

The actual reporting of the results to resultsDB is handled in the task itself, but the job runner (buildslave) will report that it's completed running the task to the buildmaster so it knows that the buildslave is available for another task.

Conclusion

That's the basic overview for how data flows through Taskotron. As one final note, I've also added a graphic showing actual data, and what it looks like at each stage. Drop by the #fedora-qa channel on Freenode if you have any questions! Happy hacking!