OpenStack Infra: Jenkins Jobs

A few days ago, while adding a new job to OpenStack Infra, I realized how difficult it must be for newcomers ( to OpenStack) to understand how OpenStack CI works and make new changes. The OpenStack Infra documentation coverage of each project is great and very detailed , but connecting the dots, which  assembles the complete work-flow can be a complex task for anyone.

Hopefully this post can help for those who unfamiliar with OpenStack Infra. This is written in a form of ‘Q & A’. If you read this and find yourself still wondering about additional subjects, please let me know and I’ll make sure to add it here.

What is a gate job?

Before I answer this, let’s describe shortly how patches submitted and merged in OpenStack.

A developer clones the relevant OpenStack project (e.g neutron, nova, etc), making some changes to the code and submits the changes with git-review ( which uploads the patch to Gerrit).

At this point the developer work is done (unless fixes  are required) and the patch is automatically tested by Jenkins. Jenkins might run one or several jobs that will execute different type of tests to verify it’s safe to merge the patch. These jobs are known as ‘gate jobs’.

Jenkins can be described here as the “Gatekeeper”. The change that submitted by the developer will not be merged unless jenkins, the gatekeepr, approved so. This approval is based on the gate job’s tests result. Jenkins approval is simply a vote of -/+1 that delivered to Gerrit. It’s enough for only one  job to fail for jenkins to vote -1, unless job is marked as non-voting. In such case, jenkins would still vote +1.

To learn more on the process, please take a look here.

What type of tests there are?

There is quite a lot, but often you will want to know what jobs exactly run when you submit patch to specific project.

But if you insist on the original question, you’ll find the answer in the project: ‘project-config’ . This project (which is also to Infra as you guessed) is where all the configuration files can be found. Those configuration files are consumed by other project to deploy and configure different parts in OpenStack Infra.

The Jenkins jobs (which include the gate jobs) configuration can be found here. This page can give you pretty good overview of what type of jobs there are.

The configuration files are yaml files which used by anther project, Jenkins Job builder, to build the actual Jenkins jobs. The yaml files describe how exactly the job should be configured.

Which jobs will execute when I submit patch to a specific project?

Let’s say you are interested in neutron project and you are curious what jobs exactly will run if decide to submit a patch  to the project. Sure, one way to know is simply submit a dummy patch and see what runs, but that’s a boring way to check and you’ll also have to wait.

The better way to find out, is by looking in project-config/zuul/layout.yaml. There you will find list of templates for the project ‘openstack/neutron’:

- name: openstack/neutron
    - name: python-jobs
    - name: python-constraints-jobs
    - gate-neutron-dsvm-api
    - gate-neutron-dsvm-api

‘gate’ and ‘check’ are pipelines and used to specify individual job.

‘template’ is used for multiple pipelines and jobs that share similar the properties.

so a followup for ‘name: python-constraints-jobs’ is:

- name: python-constraints-jobs
    - 'gate-{name}-pep8-constraints'
    - 'gate-{name}-docs-constraints'
    - 'gate-{name}-python27-constraints'
    - 'gate-{name}-docs-constraints'
    - 'gate-{name}-pep8-constraints'
    - 'gate-{name}-python27-constraints'

What will the job execute?

Let’s say you want to know what the job ‘gate-{name}-docs-constraints’ (from the previous section) executes.

One way would be to look in the logs, but that’s a specific because the logs are of one spefici job. You can’t really tell what is the original builder for all jobs by looking on the logs.

To see the exact commands, you’ll want to take a look in project-config/jenkins/jobs/python-jobs.yaml

There, you’ll see the following lines:

- job-template:
    name: 'gate-{name}-docs-constraints'

      - print-template-name:
          template-name: "{template-name}"
      - revoke-sudo
      - zuul-git-prep-upper-constraints
      - docs-env:
          env: venv-constraints

Builders are the actual block of code that the job will execute.

You can view the builders in project-config/jenkins/jobs/macros.yaml

Let’s for example take a look on docs-env builder:

- builder:
    name: docs-env
      - shell: "/usr/local/jenkins/slave_scripts/ {env}"

You can see the only thing the ‘docs-env’ builder  will executes is a script called ‘’

How do I add a new job?

First,  choose which project will the job belong to. You’ll find all the projects in project-config/jenkins/jobs/projects.yaml

Let’s say we want to add new job under the project ‘devstack-jobs’, which is simply a logical definition to group all devstack related jobs under one project.

Now let’s add our new job:

  - gate-tempest-dsvm-{name}{job-suffix}:
      pipeline: gate
      name: abregman-new-job
      node: devstack-centos7
      job-suffix: ''

Our job will be called ‘gate-tempest-dsvm-abregman-new-job’. You can see the suffix is an empty string. Usually when you add new job you add it to ‘experimental’ pipeline and you giving it the suffix: ‘-nv’ which stands for ‘non-voting’. So other will know this job is only experimental and has no influence on Jenkins vote.

node: devstack-centos7 is a way to tell Jenkins on which nodes the job will run. devstack-centos7 is simply a label that part of nodes have. So when the job will start it will look for nodes with the label ‘devstack-centos7’.

But that’s not all. In order for the job to actually run when a patch submitted we need to specify it in project-config/zuul/layout.yaml

Let’s say that we want the job to run each time a patch submitted to ‘openstack-dev/devstack’ project.

- name: openstack-dev/devstack
    - gate-tempest-dsvm-abregman-new-job

We added our job (gate-tempest-dsvm-abregman-new-job) under ‘openstack-dev/devstack’ which was already in the file and we added the job under the gate pipeline.

That’s it. After your changes are merged, you can go and submit a patch to openstack-dev/devstack to see how your job is running.

Where can I follow the job execution?

You submitted a patch and your job is running but you don’t have a link from gerrit to follow the job execution and you don’t even if it started to run.

You can follow the jobs your patch triggered, in zuul status page. Simply enter your chnage number and you’ll see the jobs your patch triggered, each in the pipeline it belongs to.

Click on one of jobs in zuul status page and you will get to Jenkins job page. In jenkins job page, click on console output to follow the job execution.  console_output

You can of course always look on the logs after the job finished. You’ll have the links on Gerrit, in the change page. But for that you might need to wait a while for Jenkins to vote after all the jobs finished.

How do I run experimental jobs?

You added a new job to ‘experimental’ pipeline and submitted a new patch, but you don’t see your new job running.

Experimental jobs will execute only when you comment with “check experminetal” on your change in Gerrit. Once you submit such comment, the job will be placed in experminetal pipeline and will be executed.

Usually new, not stable jobs will be placed in experimental pipeline. Once verified to be stable, after a period of time, the job can be placed in other pipelines (e.g gate) to vote and run upon each change submitted to a specific project.

7 thoughts on “OpenStack Infra: Jenkins Jobs

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s