CI Best Practices

The CI (Continuous Integration) is a feature provided by gitlab.
By including the .gitlab-ci.yml file in the root of the project we can define the functionality.
It runs the specified tasks or jobs depending on certain conditions.
We currently use it to run every time code is pushed to gitlab.

This defines the docker image in which jobs run by default. Jobs can override the image that is to be used by defining the image within the job.

In the .gitlab-ci.yml file of the controller_software, the image is set to: registry.gitlab.com/ebee_smart/ebee-controller-meta/linux-4.14:latest

This image is built by the ebee-controller-meta CI and includes the built buildroot submodule as well as the linux kernel submodule compiled and installed. From there it should have the necessary tools to build the controller_software.

A template can define rules and conditions for which jobs will be run and more. Maintenance branches use a different template than the master branch.

Stages run chronologically and consist of a list The definition looks like this:

stages: 
  - build
  - test
  - deploy

Jobs can declare themselves as part of a stage (see jobs) and those jobs will be run when it is time to run that stage.
The CI will run all jobs in the build stage, then the test stage and then the deploy stage.

Jobs define tasks or scripts that should be accomplished. In its most basic definition it is the definition of a bash script.

jobs can declare themselves to run at a certain stage

my-job: 
  - stage: test
  - script: 
    - echo "this is a test"

When it is time to run the test stage, the script will be executed to echo 'this is a test'.

It is possible to use the results of one job in another job by declaring a dependency on another job.

some-job: 
  dependencies:
    - check_hashes

This should allow you to use the build artifacts of the check_hashes job in the some-job job.
If you want to store dynamic variables in one job and use them in another job then they can be exported in the script stage of a job

my-job: 
  script: 
    - echo "MYVARIABLE=1" >> build.env

and must be defined in the artifacts of that same job:

my-job: 
  script: 
    - echo "MYVARIABLE=1" >> build.env
  artifacts:
    reports:
      dotenv: build.env
    

Now any job that depends on 'my-job' can use the 'MYVARIABLE'

another-job: 
  dependencies:
    - my-job
  script: 
    - 'echo $MYVARIABLE'

Note: declaring a job as a dependency for another job does not mean the job that is dependent upon is actually run. You must ensure that the dependent job is run by setting the rules accordingly.

Allows to set conditions under which a job is run or ignored.

For this it's best to refer to the documentation here as it can get quite tricky:
https://docs.gitlab.com/ee/ci/jobs/job_control.html

We can also set up rules by evaluating Gitlab predefined Variables

to see all Environment variables run '- env' in the script section:
https://www.shellhacks.com/gitlab-ci-cd-print-all-environment-variables/

gitlab lets us configure which runners we want to use for the CI. We have installed the gitlab-runner on build.elinc.de (tag: buildbot) and the local server at ebee berlin (tag: bob)
They are private runners and are configured in the project settings. You can specify to use either of them in the job definition by setting the 'tags: bob'

Running the gitlab CI usually takes a while because the gitlab runner creates a docker container and downloads/git-clones the repository for the task. This can take a while and if debugging a task at the later stages, it may take a while to even get to that stage, figure out what went wrong, repair the issue and try again.

Running the gitlab-runner locally is possible as outlined here, in order to speed up development:

cd to the project (~/Ebee/ebee-controller-meta/controller_software)

docker run --privileged --rm --name gitlab-runner
  -v ~/.gitlab-runner/:/etc/gitlab-runner -v $PWD:$PWD
  -v /dev/shm:/dev/shm -v ~/runner-caches:/cache
  -v /var/run/docker.sock:/var/run/docker.sock
  -v ~/.docker:/root/.docker
  gitlab/gitlab-runner:latest

in a different shell run:

docker exec -it -w $PWD gitlab-runner bash

gitlab-runner exec docker --docker-privileged job_by_job_name

you may have to run 'gitlab-runner register' first. Please see the help on this topic at stackoverflow: https://stackoverflow.com/questions/32933174/use-gitlab-ci-to-run-tests-locally