Introduction To Jenkins
Jenkins is a Continuous integration tool and is used to integrate your code repositories such as Git to build servers such as maven, it usually resides between these two to ensure that as soon the code is pushed, the maven is triggered by Jenkins to built it and perform test at the same time, hence small small pieces of codes are built, checked and deployed, hence called CI tool.
** This tutorial was made after creating a dockerfile using which jenkins/jenkins image was installed in a docker container and then all below things were done. Even though jenkins was inside the docker container, still jenkins GUI was accessible on localhost:8080 on the Mac’s Chrome browser just like normal installation since jenkins that we see inside the docker is just a process that is running on your mac/windows in daemon mode, you can check this using the Task Manager on Mac. Now since containers are just processes running on your OS, they can easily access all the resources of OS such as browsers/app just like normal installed software. This is what separates VMs vs containers. If same was done on a VM, then jenkins from inside could have not accessed the Chrome browser from inside in normal VM machines.
System Configuration Settings
Home directory
This is where the jenkins is present
System Message
Enter here a message to be displayed on Jenkins dashboard
# of executors
Executors are like single process that jenkins starts to run the build. More number of executors means more processes can be started for multiple builds.
# of executors basically defined the max number of threads/jobs that can be run on that particular instance. For single core, 2 is recommended. If number of jobs that are run is more than number of executors, then no need to worry, eventually jenkins will run those jobs. keep an key out of disk/memory/CPU utilization though
________________________________________________________________________________
Variable and Functions In bash
To use variable in bash, just write like below
$ variable_name=”variable value”
Please remember there should not be any space in between
ex:
$name=”robert downey jr”
echo $name
Output:
robert downey jr
To mention functions , use $(function_name)
ex: echo $(date)
output: Sun 21 Mar 2021 07:02:31 AM UTC
Please check the difference when using echo with variable and when using echo with function. When we use echo with function, we use braces, but with variables we dont use braced, so:
echo $variable_name
vs
echo $(date)
To give value inside a script while executing the script
- First create a ‘.sh’ file for the script and enter your script
- $ vi script.sh
- Now in your script, remember to use #!/bin/bash in the start to let the cmd know which bash/shell we will be using. Let’s suppose we want to give 2 variable names while executing the script. So we need to mention those variables in the script also, so that they can take the value from outside and past it inside the script when needed. Ex: First press”i” to enter insert mode after $vi script.sh
————————–
#!/bin/bash
fname=$1
lname=$2
echo “Hi Mr. $fname $lname, today’s date is $(date)”
Now press ESC to exit the insert mode and the write :wq and press enter to save and exit
- Now we want to execute the script, but we dont have permissions yet, so just enter
- chmod +x ./script.sh
This gives execution permissions to us on the script
- Now we supply the variables and execute the script
./script.sh jai mishra
and press enter. This is supply “jai” to $1 and “mishra” to $2. The output would be:
hello, My name is jai mishra and today is Sun Mar 21 14:00:31 IST 2021
- Now if we want to send this file inside of docker container ie inside container where jenkins is currently installed, we will need to use docker cp command to send a file from outside the container ie present on the OS where the docker is installed, into the jenkins container, for that use the below command:
docker cp path_of_script name_of_container:full_path_inside_container_to_copy_script
ex: docker cp script.sh jenkins:/tmp/script.sh
- please note to include the file name in the full path
Now although we have the script in the /tmp directory. However, if we mention to execute this command using Build->execute shell-> command in jenkins job configure, we will need to make sure to enter the complete path of this script since the pwd of the command shell shows some other directory.
So just past below command on the jenkins configure -> execute shell-> command:
/tmp/./script.sh jai mishra
output on jenkins console:
Started by user jai mishra
Running as SYSTEM
Building in workspace /var/jenkins_home/workspace/run_script_inside_jenkins
[run_script_inside_jenkins] $ /bin/sh -xe /tmp/jenkins1227381274226922704.sh
+ /tmp/./script.sh jai mishra
hello, My name is jai mishra and today is Sun 21 Mar 2021 09:04:03 AM UTC
Finished: SUCCESS
So we got the output from jenkins by executing this script inside jenkins after creating it outside the jenkins and pasting it inside using docker cp command.
We can also just mention the variable names and pass like this:
way 1:
/tmp/./script.sh jai mishra
way 2:
f_name=fomu
l_name=baby
/tmp/./script.sh $f_name $l_name
output of way 2:
Started by user jai mishra
Running as SYSTEM
Building in workspace /var/jenkins_home/workspace/run_script_inside_jenkins
[run_script_inside_jenkins] $ /bin/sh -xe /tmp/jenkins7907701380278172534.sh
+ /tmp/./script.sh jai mishra
hello, My name is jai mishra and today is Sun 21 Mar 2021 09:11:01 AM UTC
+ f_name=fomu
+ l_name=baby
+ /tmp/./script.sh fomu baby
hello, My name is fomu baby and today is Sun 21 Mar 2021 09:11:01 AM UTC
Finished: SUCCESS
________________________________________________________________________________
Using parameter with build
1.String Parameter
instead of hardcoding the parameter like above in the execute shell itself, we can just mention variables in the commands of the command shell of jenkins and use the option “This project is parameterized”
This will require us to configure the parameters, in every parameters there are 3 parts:
Name: enter the name of the variable you have mentioned such as “height” or “weight”
Default value: enter the default value that you want to be shown when you build the project. During build, it will ask to enter the value and since we have set some value as default, that default value will be preset. We can edit here also.
Description: Enter any des
After we have set the required number of parameters, instead of build you will observe “Build with Parameters” button
And in the shell, just enter the command like this:
echo “Hi My weight is $weight and height is $height”
Here the weight and height have already been set as parameters with default values 80 and 176. So output will be:
Started by user jai mishra
Running as SYSTEM
Building in workspace /var/jenkins_home/workspace/parameterised_build
[parameterised_build] $ /bin/sh -xe /tmp/jenkins766797687141550506.sh
+ echo Hi My weight is 80 and height is 175cm
Hi My weight is 80 and height is 175cm
Finished: SUCCESS
Now again is we click on “build with parameters” after first build, we are again asked for parameters, so just enter the parameters you want to try this time and build.
2. Choice Parameter
We we choose the “String Parameter” as parameter type in configure, we only get the option to set one default value and then enter manually during pre-build. However, we can use choice parameter which gives us a drop down list of all the choice we can have.
So choose “Choice Parameter” and then in choices section, enter all the choice you want.Ex:
Name: Environment
Choices:
dev
prod
test
Now in the command shell :
echo “today the engineer will deploy change to $environment”
Now upon “build with parameter”, you will get a drop down to select for ‘environment’ variable before we click finally on “build”
- Sending the parameter values during build inside script
To send the parameter value from Jenkins to inside a script, do below:
i. create below like script inside your jenkins and make it excutable:
#!/bin/bash
fname=$1
lname=$2
echo “Hello, $fname $lname”
save this script with name such as script2.sh
ii. Make it executable :
$chmod +x script2.sh
iii. Now if created on computer and jenkins is inside docker and it is running, use below command to send this inside jenkins
$docker cp script2.sh jenkins:/tmp/script2.sh
iv. Now script is inside jenkins container folder. Now configure jenkins job. So build new job and select “string parameter”:
Name: fname
Default value : jai
Name: lname
Default value : mishra
Now in the build execute shell enter below command to supply the above parameters during build:
./script2.sh $fname $lname
This will take the value from parameter during build and send it to jenkins script as inputs
- Using boolean values and condition keys
i. script:
_______________________
#!/bin/bash
fname=$1
lname=$2
show=$3
if [ “$show” = “true” ]; then
echo “Hello, $fname $lname”
else
echo “if you want to see the name, check the SHOW option”
fi
_______________________
*Please remember to leave space between [ “$show” = “true” ]; like this, else condition block doesnt work
ii. Now in the Jenkins jobs, make 2 string type parameter and 1 Boolean type, if. in boolean type the “Default Value” is chech box is ticker, then it will sen true for this variable and if this checkbox is not ticked, then it sends false and that is how we check the login inside script using if [ “$show” = “true” ]; then since ticked check box returns true and non ticker returns false
iii. In execute shell, :
/tmp/script2.sh $fname $lname $show
_____________________________________________________________________________________
To communicate between jenkins and docker
There can be multiple ways to do that, one would be to separately install jenkins on one VM and docker on other and make them communicate, however recommended way is to create 2 separate containers, one for jenkins and other
RUN lets you execute commands inside of your Docker image. These commands get executed once at build time and get written into your Docker image as a new layer. … CMD lets you define a default command to run when your container starts
We can directly use the SSH plugin on jenkins to and use the below ssh command to login to the remote_host network we created. SSH from inside the jenkins container to the docker container remote_host using command like this on jenkins container:
jenkins@fa66afa84dd0:/$ ssh remote_user@remote_host
remote_user@remote_host’s password:
Enter the password 1234 that is set for SSh and done.
Last failed login: Wed Mar 31 14:54:07 UTC 2021 from jenkins.jenkins_net on ssh:notty
There was 1 failed login attempt since the last successful login.
Last login: Fri Mar 26 15:04:47 2021 from jenkins.jenkins_net
[remote_user@1478456f6c9e ~]$
Downloading and Setting up SSH plugin
To communicate from the Jenkins container to another container ie remote_host, we need to install “SSH” plugin on Jenkins and then setup the hotname, port and credentials and then test connection. After this is setup, we can then go to configure and in the build step, we can choose “Execute Shell script using remote host” option. This will give us the option to choose the SSH connection we just configured and then after this is configured, once the job is build, jenkins will execute the shell script on the remote host.
_________________________________________________________________________________________
++++++++++ Jenkins Pipeline ++++++++++
We can create jenkins pipelines using below 2 ways. Jenkins Pipeline provides an extensible set of tools for modeling simple-to-complex delivery pipelines “as code”. The definition of a Jenkins Pipeline is typically written into a text file (called a Jenkinsfile) which in turn is checked into a project’s source control repository
i. Declarative (simple json type langauge)(this will be learnt)
ii. Advanced: groovy (tough)
Basically a pipeline script is a script that mentions how the different stages in the pipeline will work and what all steps and commands will be executed in all these stages.
There can be Single Step and Mutiple steps pipeline:
Single Step:
Ex:
______________________
pipeline {
agent any
stages {
stage(‘Build’) {
steps {
echo ‘Building..’
}
}
stage(‘Test’) {
steps {
echo ‘Testing..’
}
}
stage(‘Deploy’) {
steps {
echo ‘Deploying….’
}
}
}
}
______________________
agent: This is always mentioned as “any”
Muti-step Pipeline
______________________
pipeline {
agent any
stages {
stage(‘Build’) {
steps {
sh ‘echo “My first pipeline”‘
sh ”’
echo “By the way, I can do more stuff in here”
ls -lah
”’
}
}
}
}
______________________
The triple quotes sh ”’…..”’ allows us to write multiple commands within same step and same sh. Ex:
sh ”’
echo “By the way, I can do more stuff in here”
ls -lah
”’
From within a Jenkins pipeline you can execute any external program or command. If your pipeline will run on Unix/Linux you need to use the
‘ sh ‘ command. If your pipeline will run on MS Windows you’ll need to use the ‘bat’ command.
Basically sh will allow us to run that particular command on the machine that pipeline is being run on.
Retry
Basically there can be cases in which there are certain commands trying to be executed. They might be failing, so if our step fails, that process/step is skipped. But we can imple “Retry” to keep trying that step and retrying even though the step is failing:
______________
pipeline {
agent any
stages {
stage(‘Timeout’) {
steps {
retry(3) {
sh ‘I am not going to work :c’
}
}
}
}
}
_________________
Output:
Running on Jenkins in /var/jenkins_home/workspace/Retry
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Timeout)
[Pipeline] retry
[Pipeline] {
[Pipeline] sh
+ I am not going to work :c
/var/jenkins_home/workspace/Retry@tmp/durable-0160ba06/script.sh: 1: /var/jenkins_home/workspace/Retry@tmp/durable-0160ba06/script.sh: I: not found
[Pipeline] }
ERROR: script returned exit code 127
Retrying
[Pipeline] {
[Pipeline] sh
+ I am not going to work :c
/var/jenkins_home/workspace/Retry@tmp/durable-458af165/script.sh: 1: /var/jenkins_home/workspace/Retry@tmp/durable-458af165/script.sh: I: not found
[Pipeline] }
ERROR: script returned exit code 127
Retrying
[Pipeline] {
[Pipeline] sh
+ I am not going to work :c
/var/jenkins_home/workspace/Retry@tmp/durable-218b0aa9/script.sh: 1: /var/jenkins_home/workspace/Retry@tmp/durable-218b0aa9/script.sh: I: not found
[Pipeline] }
[Pipeline] // retry
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: script returned exit code 127
Finished: FAILURE
TimeOut
We can set TimeOut in steps to set a value to timeout that step if that step was not executed successfully. So basically lets say you have a step “sleep 5”, now since this step would take 5 secs to complete, if we set that step timeout as 3 secs, then jenkins timeout will wait for 3 secs to complete that step and then abort the whole step since the step was timedout and not completed since sleep 5 will take 5 secs to complete.:
_________________________
pipeline {
agent any
stages {
stage(‘Deploy’) {
steps {
retry(3) {
sh ‘echo hello’
}
timeout(time: 3, unit: ‘SECONDS’) {
sh ‘sleep 5’
}
}
}
}
}
_________________________
Output:
Started by user Jai Misra
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in /var/jenkins_home/workspace/timeout
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Deploy)
[Pipeline] retry
[Pipeline] {
[Pipeline] sh
+ echo hello
hello
[Pipeline] }
[Pipeline] // retry
[Pipeline] timeout
Timeout set to expire in 3 sec
[Pipeline] {
[Pipeline] sh
+ sleep 5
Cancelling nested steps due to timeout
Sending interrupt signal to process
Terminated
Terminated
script returned exit code 143
[Pipeline] }
[Pipeline] // timeout
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Timeout has been exceeded
Finished: ABORTED
see the timeout has caused this pipeline to abort and all the upcoming steps are also aborted, hence whole pipeline is terminated.
Enviroment Variables
We can set environment variables such as normal variable which will be used such as below:
_______________________
pipeline {
agent any
environment {
NAME = ‘ricardo’
LASTNAME = ‘gonzalez’
}
stages {
stage(‘Build’) {
steps {
sh ‘echo $NAME $LASTNAME’
}
}
}
}
_______________________
Output:
Started by user Jai Misra
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in /var/jenkins_home/workspace/env pipeline
[Pipeline] {
[Pipeline] withEnv
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Build)
[Pipeline] sh
+ echo ricardo gonzalez
ricardo gonzalez
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS
We how the values for variables are taken from the environment { } variables just like normally mentioning.
CREDENTIALS in Pipeline
We can provide our pipelines with credentials that we have created as global credentials in jenkins and use it through Credentials such as below:
_________________
pipeline {
agent any
environment {
secret = credentials(‘SECRET_TEXT’)
}
stages {
stage(‘Example stage 1’) {
steps {
sh ‘echo $secret’
}
}
}
}
_________________
The “environment” is used to mention the variables just as before, however this time instead of just mentioning the variables and its value, we mention secret = credentials(‘<Secret_ID>’).
Now in this the Secret_ID is obtained when we configure a global credential and choose credential type as “Secret Text”, then in that secret text, just mention the Secret as the variables values and the ID is the ID that will be referred in jenkins pipeline, so ID is like variable name. After the global credential has been created, you can use that credential referring to the ID using the secret = credentials(‘SECRET_TEXT’).
Started by user Jai Misra
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in /var/jenkins_home/workspace/creds pipeline
[Pipeline] {
[Pipeline] withCredentials
Masking supported pattern matches of $secret
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Example stage 1)
[Pipeline] sh
+ echo ****
****
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // withCredentials
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS
Since its a secret, the output is shown as **** but this can be used to pass the secret value to a job/function.
POST
POST is very imp and easy. POST is like if-else functionality inside the steps. Basically lets say that jenkins tries to execute a step, now it can either get executed, get failed also the job can be unstable in which the job was earlier failing but now its working. Now we can handle all the cases using POST which allows us to take further actions based on the state of the step. Ex:
_____________________
pipeline {
agent any
stages {
stage(‘Test’) {
steps {
sh ‘echo “Fail!”; exit 1’
}
}
}
post {
always {
echo ‘I will always get executed :D’
}
success {
echo ‘I will only get executed if this success’
}
failure {
echo ‘I will only get executed if this fails’
}
unstable {
echo ‘I will only get executed if this is unstable’
}
}
}
_____________________
Now if you observe the above script, you will observe that the step steps {sh ‘echo “Fail!”;exit 1’ } is followed by post with 4 different possibilities,
i. always: This section will always execute no matter if the step fails or not
ii. Success: this section is executed only if the step was executed successfully
iii. failure: executed when the step fails
iv. unstable: This section is executed when the step has been unstable which means it was failing earlier but now its working.
OUTPUT:
Started by user Jai Misra
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in /var/jenkins_home/workspace/POST (if-else) pipeline
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Test)
[Pipeline] sh
+ echo Fail!
Fail!
+ exit 1
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Declarative: Post Actions)
[Pipeline] echo
I will always get executed 😀
[Pipeline] echo
I will only get executed if this fails
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: script returned exit code 1
Finished: FAILURE
We can be selective with how many of these POSTs case we want to handle, we can even choose 1 of these 4 different conditions.