Creating the world's most capable DevOps Pipeline Agent

8 minutes

September 23, 2022

Lately, I’ve been working on supporting an awesome microservices project. For various reasons, we have code in Azure DevOps, an on-premise Kubernetes cluster and the project is written in Java utilising Apache Camel.

Microsoft provides some excellent cloud-based agents to utilise for Build and Release pipelines. However, if all your infrastructure is in an office and not up in the cloud then those agents don’t really do a great job of chatting to your systems behind your firewall.

So, began the task of creating a local agent. Microsoft again provides some great starting points for doing this, including a basic dockerfile. We have a Kubernetes cluster, so deploying our own DevOps Agent in one or more pods is perfect.

The MS code is a great starting point, but we need more. Dot Net compiler is pretty good, but it won’t compile Java (….yet). So we need Maven, plus we have firewalls with HTTPs inspection and a raft of internal services with internal SSL certs so we need to add some internal root certificates too.

Then we started on our Ansible journey and of course, we already have an on-premise build agent, so let’s just make it an all-in-one agent!

So now our agent does Ansible, including having the VMWare community modules, plus some Kerberos config for Windows deployments as well.

Plus for good measure, we have the Dotnet SDK and on top of that throw in some Powershell, because why not.

Down to the code…

So the general dockerfile from Microsoft looks like this:


FROM ubuntu:18.0

# To make it easier for build and release pipelines to run apt-get,
# configure apt to not require confirmation (assume the -y argument by default)
ENV DEBIAN_FRONTEND=noninteractive
RUN echo "APT::Get::Assume-Yes \"true\";" > /etc/apt/apt.conf.d/90assumeyes

RUN apt-get update && apt-get install -y --no-install-recommends \
    ca-certificates \
    curl \
    jq \
    git \
    iputils-ping \
    libcurl4 \
    libicu60 \
    libunwind8 \
    netcat \
    libssl1.0 \
  && rm -rf /var/lib/apt/lists/*

RUN curl -LsS https://aka.ms/InstallAzureCLIDeb | bash \
  && rm -rf /var/lib/apt/lists/*

# Can be 'linux-x64', 'linux-arm64', 'linux-arm', 'rhel.6-x64'.
ENV TARGETARCH=linux-x64

WORKDIR /azp

COPY ./start.sh .
RUN chmod +x start.sh

ENTRYPOINT ["./start.sh"]

Fairly straightforward. We install a bunch of packages, get AzureCLI, set some environmental variables and then set the start.sh file as our entry point (once we copy it and set it to be executable).

Start.sh is provided by MS and does the main job of connecting back to DevOps, registering against a specific instance and letting it know it’s ready to start running jobs.

Let’s amp this up a few notches:


FROM mcr.microsoft.com/openjdk/jdk:11-ubuntu


# To make it easier for build and release pipelines to run apt-get,
# configure apt to not require confirmation (assume the -y argument by default)
ENV DEBIAN_FRONTEND=noninteractive
RUN echo "APT::Get::Assume-Yes \"true\";" > /etc/apt/apt.conf.d/90assumeyes


RUN apt-get update && apt-get install -y --no-install-recommends \
    ca-certificates \
    curl \
    jq \
    git \
    iputils-ping \
    libcurl4 \
    libunwind8 \
    netcat \
    libssl1.0 \
  && rm -rf /var/lib/apt/lists/*


# Add custom root certs
ADD root.crt /usr/local/share/ca-certificates/root.crt
RUN chmod 644 /usr/local/share/ca-certificates/root.crt && \
  update-ca-certificates


# Add Certs to Keytool for Java/Maven
RUN keytool -importcert -file /usr/local/share/ca-certificates/root.crt -cacerts -keypass changeit -storepass changeit -noprompt -alias InternalRootCA


RUN curl -LsS https://aka.ms/InstallAzureCLIDeb | bash \
  && rm -rf /var/lib/apt/lists/*


RUN curl -LsS https://packages.microsoft.com/config/ubuntu/20.04/packages-microsoft-prod.deb -o packages-microsoft-prod.deb && \
 dpkg -i packages-microsoft-prod.deb && \
 rm packages-microsoft-prod.deb


# Install Compilers
RUN apt-get update && apt-get install -y --no-install-recommends \
    dotnet-sdk-6.0 \
    docker.io \
    apt-transport-https \
    maven


# Add Mavenrc file to set Certs for use
ADD .mavenrc /root/.mavenrc


# Add Maven settings
ADD settings.xml /root/.m2/settings.xml


# Install Powershell
RUN curl -LsS https://github.com/PowerShell/PowerShell/releases/download/v7.2.0/powershell-lts_7.2.0-1.deb_amd64.deb -o powershell-lts_7.2.0-1.deb_amd64.deb && \
 dpkg -i powershell-lts_7.2.0-1.deb_amd64.deb && \
 rm powershell-lts_7.2.0-1.deb_amd64.deb


# Install Ansible Components
RUN apt-get update && apt-get install -y gcc python-dev libkrb5-dev && \
    apt-get install python3-pip -y && \
    apt-get install openssh-client -y && \
    pip3 install --upgrade pip && \
    pip3 install --upgrade virtualenv && \
    pip3 install pywinrm[kerberos] && \
    apt install krb5-user -y && \ 
    pip3 install pywinrm && \
    pip3 install ansible && \
    pip3 install pyvmomi


# Add SSH Key
ADD ssh_priv_key /root/.ssh/id_rsa
ADD ssh_pub_key /root/.ssh/id_rsa.pub


RUN chmod 600 /root/.ssh/id_rsa && \
    chmod 600 /root/.ssh/id_rsa.pub


# Add Ansible.cfg
ADD ansible.cfg /root/.ansible.cfg


# Add Krb5.conf for Kerberos
ADD krb5.conf /etc/krb5.conf


# Install Ansible VMWare Community Collection
RUN ansible-galaxy collection install community.vmware
RUN pip install -r ~/.ansible/collections/ansible_collections/community/vmware/requirements.txt


ARG TARGETARCH=amd64
ARG AGENT_VERSION=2.194.0


WORKDIR /azp


RUN if [ "$TARGETARCH" = "amd64" ]; then \
      AZP_AGENTPACKAGE_URL=https://vstsagentpackage.azureedge.net/agent/${AGENT_VERSION}/vsts-agent-linux-x64-${AGENT_VERSION}.tar.gz; \
    else \
      AZP_AGENTPACKAGE_URL=https://vstsagentpackage.azureedge.net/agent/${AGENT_VERSION}/vsts-agent-linux-${TARGETARCH}-${AGENT_VERSION}.tar.gz; \
    fi; \
    curl -LsS "$AZP_AGENTPACKAGE_URL" | tar -xz



COPY ./start.sh .
RUN chmod +x *.sh


ENTRYPOINT [ "./start.sh" ]

That’s more like it!

How does it work?

Let’s go through a few steps…

FROM mcr.microsoft.com/openjdk/jdk:11-ubuntu

Instead of Ubuntu, we start from a different container source which is really still Ubuntu plus OpenJDK. This reference from MS provides a table with all the options: https://learn.microsoft.com/en-us/java/openjdk/containers

Then we can add a root cert.

ADD root.crt /usr/local/share/ca-certificates/root.cr
RUN chmod 644 /usr/local/share/ca-certificates/root.crt && \
  update-ca-certificatest

You can add as many certificates as you need here. You also need to create a keystore for Maven if your Java project wants to reference things that are internally signed.

RUN keytool -importcert -file /usr/local/share/ca-certificates/root.crt -cacerts -keypass changeit -storepass changeit -noprompt -alias InternalRootCA

Once we install Maven, we need to tell it to use the certs. Our .mavenrc looks like this:

MAVEN_OPTS="-Xmx512m -Djavax.net.ssl.trustStore=/usr/lib/jvm/msopenjdk-11/lib/security/cacerts 
                     -Djavax.net.ssl.trustStorePassword=changeit"\

The password here and in our keytool command probably wants to be different from “changeit”.

We also add a settings.xml for maven and that sits in the .m2 folder. This has our internal repo mirrors and other settings we need. Set yours as you require. Just remember in your Agent repo, you need to include these files so they can be composed into your mega agent.

# Install Compiler
RUN apt-get update && apt-get install -y --no-install-recommends \
    dotnet-sdk-6.0 \
    docker.io \
    apt-transport-https \
    mavens

Now time to get a few compilers in place. Don’t forget Docker as you want your Agent to compile your code and then compile your container.

RUN apt-get update && apt-get install -y gcc python-dev libkrb5-dev && 
    apt-get install python3-pip -y && \
    apt-get install openssh-client -y && \
    pip3 install --upgrade pip && \
    pip3 install --upgrade virtualenv && \
    pip3 install pywinrm[kerberos] && \
    apt install krb5-user -y && \ 
    pip3 install pywinrm && \
    pip3 install ansible && \
    pip3 install pyvmomi\

From there we shift gears to Ansible. Install all the components we need including python, and OpenSSH (remember ssh is NOT included in the default container…because why would it need to be).

ADD ssh_priv_key /root/.ssh/id_rs
ADD ssh_pub_key /root/.ssh/id_rsa.pub


RUN chmod 600 /root/.ssh/id_rsa && \
    chmod 600 /root/.ssh/id_rsa.puba

Then we can add SSH keys for passwordless ssh. For this example, we are using the root user and copying in a single key. You can add a specific ansible user and keys etc. This is just a basic setup.

# Add Krb5.conf for Kerbero
ADD krb5.conf /etc/krb5.confs

We run VMWare as our virtualisation platform, so we need to ensure we have the modules for that, otherwise, our provisioning is not going to work.

And that is everything in docker.

##Deploying into Kubernetes

To get this running in Kubernetes we need to setup a few secrets first. And before that you will need to create an Azure Token for authentication

kubectl create secret generic azdevops 
  --from-literal=AZP_URL=https://dev.azure.com/YOUR_INSTANCE/ \
  --from-literal=AZP_TOKEN=YOUR_TOKEN \
  --from-literal=AZP_POOL='YOUR POOL NAME'\

Get those secrets defined, then we can use them in our deployment file:

        env
          - name: AZP_URL
            valueFrom:
              secretKeyRef:
                name: azdevops
                key: AZP_URL
          - name: AZP_TOKEN
            valueFrom:
              secretKeyRef:
                name: azdevops
                key: AZP_TOKEN
          - name: AZP_POOL
            valueFrom:
              secretKeyRef:
                name: azdevops
                key: AZP_POOL:

MS has a good deployment spec to customize as you need

##Summing Up

That is about it. Docker-compose, kubectl apply and watch the agents pop up in DevOps and use to your heart’s content.

While your exact requirements probably don’t match mine, if you’re new to running containerization in enterprise environments, there’s always a LOT to consider. But if you’re anything like me and coming from years of general infrastructure support, then seeing examples is helpful to understand how to do things in this new-ish world of containers, docker, ansible etc. Most of the time it’s pretty simple, but you need to get your head around the syntax or the Kubernetes “method” of how it prefers to work.

So I hope this is helpful. For me, DevOps is all about starting small and building up over time. Every check-in you make to your infrastructure repo, is an improvement, a new feature and a step towards a better outcome. For each iteration, you gain a little more knowledge and add a little more functionality.

There’s lots of things to figure out. Docker, Kubernetes, DotNet, Maven, Ansible, Powershell etc are just some things we cover today. They all have their quirks and requirements and “ways”. It’s tough to learn 100% about all of them, but figuring out a little each time about some, means that after a while you can step back and find your infrastructure, your code and your pipelines are better in 1000 different ways after 1000 different changes.

References

Code on Github

Grab this file from my github here: https://github.com/robhogarth/DevopsPipelineAgent

Hi, I'm Rob Hogarth

Sysadmin, IT Guru and Author