Zuul-based-ci

Zuul Based CI

News

August 28, 2019 - Created Taiga EPIC

• Created the Taiga EPIC with the stories to achieve in the goal to provide Pagure PR Zuul jobs for Fedora distgit: https://teams.fedoraproject.org/project/ci/epic/14

Flock 2019

• Talk: Pagure CI based on Zuul:
  • slides: File:Slides-flock2019.pdf
  • video: not published yet

What is Zuul/Nodepool

Zuul [1] is the CI and gating system from the Open Infrastructure Project [2]. It is able to scale fine and handles by default features such as artifacts sharing between jobs and cross Git repositories testing. You can see Zuul in action here [3].

Below is a list of features proposed by Zuul and its companion Nodepool:

• Event-driven pipelines based on Code-Review or Pull-Request workflow: jobs can be triggered automatically when a PR is submitted, changed, approved, merged, or when the repository is tagged.
• CI-as-code: jobs are defined as YAML + Ansible playbooks, pipeline definitions as YAML files. Zuul reads and loads those definitions directly from Git repositories.
• Support for jobs inheritance, jobs dependencies, jobs chaining (with artifacts sharing).
• Speculative testing of new jobs before merging: jobs will be run as they are submitted to make sure they behave as expected.
• Cross repositories dependencies: a jobs' workspace can include unmerged patches from other projects if specified
• Parallel job run, only capped by resources available or predefined quotas
• Automated jobs resources lifecycle management: resources like VMs or containers needed by a given job can be defined in-repository, spawned on demand at a job's start, and destroyed when the job is finished, or held for debugging
• Job resources support of OpenStack, OpenShift, K8S, Static nodes, AWS.
• Well-defined, reproducible job environments to eliminate flakiness
• Speculative testing before merging (gating): if several patches are about to land at the same time, they are tested on the repository's future state.

Until now, Zuul was only able to listen to Gerrit or Github events. Recently a new driver [4] allow Zuul to interface with Pagure as well. Pagure, Zuul and Nodepool could therefore combine into a very efficient CI/CD stack.

How to Zuul attach a Pagure repository on Zuul

Configure the repository for Zuul

In the project settings:

• Add "zuul" as admin in settings/Users & Groups
• In Project Options
  • Check "Notify on pull-request flag"
  • Web-hooks:
    • For a repository on pagure.io https://softwarefactory-project.io/zuul/api/connection/pagure.io/payload
    • For a repository on src.fedoraproject.org https://softwarefactory-project.io/zuul/api/connection/src.fedoraproject.org/payload
  • (For gating, optional): check "Always merge"
  • Tags:
    • Add the "gateit" tag

Add the repository into the Zuul configuration

Open a Pull Request on https://pagure.io/fedora-project-config

Edit resources/fedora.yaml and add the repository such as!

• For a repository hosted on pagure.io:

resources:
  projects:
    Fedora-Zuul-CI:
      ...
      source-repositories:
        ...
        - repository-name

• For a repository hosted on src.fedoraproject.org:

resources:
  projects:
    Fedora-Zuul-CI:
      ...
      source-repositories:
        ...
        - repository-name:
            connection: src.fedoraproject.org

Once the Pull Request is accepted and merged, the repository should be available in the Zuul project list: https://fedora.softwarefactory-project.io/zuul/projects

How to attach packaging jobs for a distgit repository on src.fedoraproject.org

We have managed to provide some standard Pull Request's jobs and workflow. This idea is to reduce the amount of manual steps to propose packaging changes to Fedora by taking advantage of CI.

We propose a set of templates (template can be seen as workfow of jobs reacting to Pull Request events). Templates are available here https://pagure.io/fedora-zuul-jobs/blob/master/f/zuul.d/templates.yaml.

The templates attach jobs through three type of pipeline (not the same as Jenkins/Gitlab pipeline concept):

• check: Jobs within that pipeline are triggered when a Pull Request is Opened or Updated
• gate: Jobs within that pipeline are triggered when a Pull Request is approved (prior to be merged and closed)
• promote: Jobs with that pipeline are triggered when a Pull Request is closed and merged

The available jobs are:

• rpm-scratch-build: Run a scratch build on Koji (target based on the PR's branch) and retrieve artifacts (rpms).
• rpm-build: Run a regular build on Koji (target based on the PR's branch)
• rpm-lint: Run a rpmlint on artifacts (rpms) passed by the parent job
• rpm-rpminspect: Run rpminspect on artifacts (rpms) passed by the parent job. The job also find the latest build on the related koji tag and pass it the rpminspect so you get the rpminspect diff.
• rpm-test: Execute distgit embedded test tests/tests.yml on the related Fedora node (rawhide VM for master branch, Fedora 30 VM for f30 branch, ...). It is compatible with the Fedora standard-test-interface as dependencies are available on the node.

Available templates:

• build: run the rpm-scratch-build (scratch koji) job in the check pipeline.
• build-lint: run the rpm-scratch-build job then run the rpm-linter + rpm-rpminspect jobs against the rpms built by the first job.
• build-lint-test: run the rpm-scratch-build job then run the rpm-linter + rpm-rpminspect + rpm-test jobs against the rpms built by the first job.
• build-lint-test-gate: same as build-lint-test + the CI will merge the Pull Request if the Pull Request receive the metadata tag "gateit".
• build-lint-test-gate-promote: same as build-lint-test-gate + rpm-build (regular koji build) when the Pull Request is merged and closed.

There are some other templates available but now you should have a clear understanding of them.

To attach a template to a repository you need to open a Pull Request on https://pagure.io/fedora-zuul-jobs-config.

Edit zuul.d/projects.yaml to add:

- project:
    name: repository
    templates:
      - build-lint-gate-promote

repository should be like "rpms/python-gear".

Once the Pull Request is merged should should the jobs of the chosen template displayed by clicking on the related project on the page https://fedora.softwarefactory-project.io/zuul/projects.

Currently f29, f30, f31 and master branches are supported. Only the x86_64 arch is supported.

Nodepool nodes

Jobs are configured to execute on a Fedora VM. The default jobs we provide are configured to use the right node. For instance the rpm-test job

• Fedora rawhide cloud image (VM) https://softwarefactory-project.io/cgit/config/tree/nodepool/elements/virt-customize/fedora-rawhide-cloud.yaml
• Fedora 31 cloud image (VM) https://softwarefactory-project.io/cgit/config/tree/nodepool/elements/virt-customize/fedora-31-cloud.yaml
• Fedora 30 cloud image (VM) https://softwarefactory-project.io/cgit/config/tree/nodepool/elements/virt-customize/fedora-30-cloud.yaml
• Fedora 29 cloud image (VM) https://softwarefactory-project.io/cgit/config/tree/nodepool/elements/virt-customize/fedora-29-cloud.yaml