Securing OpenShift Pipelines

Procedure

1. Set the namespace and name of the Kubernetes service account.

   $ export NAMESPACE=<namespace> 1
   $ export SERVICE_ACCOUNT_NAME=<service_account> 2

   1

   The namespace associated with the service account.

   2

   The name of the service account.

2. Create a Kubernetes secret.

   $ oc create secret registry-credentials \
     --from-file=.dockerconfigjson \ 1
     --type=kubernetes.io/dockerconfigjson \
     -n $NAMESPACE

   1

   Substitute with the path to your Docker config file. Default path is ~/.docker/config.json.

3. Give the service account access to the secret.

   $ oc patch serviceaccount $SERVICE_ACCOUNT_NAME \
     -p "{\"imagePullSecrets\": [{\"name\": \"registry-credentials\"}]}" -n $NAMESPACE

   If you patch the default pipeline service account that Red Hat OpenShift Pipelines assigns to all task runs, the Red Hat OpenShift Pipelines Operator will override the service account. As a best practice, you can perform the following steps:

   1. Create a separate service account to assign to user’s task runs.

      $ oc create serviceaccount <service_account_name>

   2. Associate the service account to the task runs by setting the value of the serviceaccountname field in the task run template.

      apiVersion: tekton.dev/v1
      kind: TaskRun
      metadata:
        name: build-push-task-run-2
      spec:
        taskRunTemplate:
          serviceAccountName: build-bot 1
        taskRef:
          name: build-push
      ...

      1

      Substitute with the name of the newly created service account.

Procedure

1. Create an encrypted x509 key pair and save it as a Kubernetes secret. For more information about creating a key pair and saving it as a secret, see "Signing secrets in Tekton Chains".

2. In the Tekton Chains configuration, disable the OCI storage, and set the task run storage and format to tekton. In the TektonConfig custom resource set the following values:

   apiVersion: operator.tekton.dev/v1alpha1
   kind: TektonConfig
   metadata:
     name: config
   spec:
   # ...
       chain:
         artifacts.oci.storage: ""
         artifacts.taskrun.format: tekton
         artifacts.taskrun.storage: tekton
   # ...

   For more information about configuring Tekton Chains using the TektonConfig custom resource, see "Configuring Tekton Chains".

3. To restart the Tekton Chains controller to ensure that the modified configuration is applied, enter the following command:

   $ oc delete po -n openshift-pipelines -l app=tekton-chains-controller

4. Create a task run by entering the following command:

   $ oc create -f https://raw.githubusercontent.com/tektoncd/chains/main/examples/taskruns/task-output-image.yaml 1

   1

   Replace the example URI with the URI or file path pointing to your task run.

   Example output

   taskrun.tekton.dev/build-push-run-output-image-qbjvh created

5. Check the status of the steps by entering the following command. Wait until the process finishes.

   $ tkn tr describe --last

   Example output

   [...truncated output...]
   NAME                            STATUS
   ∙ create-dir-builtimage-9467f   Completed
   ∙ git-source-sourcerepo-p2sk8   Completed
   ∙ build-and-push                Completed
   ∙ echo                          Completed
   ∙ image-digest-exporter-xlkn7   Completed

6. To retrieve the signature from the object stored as base64 encoded annotations, enter the following commands:

   $ tkn tr describe --last -o jsonpath="{.metadata.annotations.chains\.tekton\.dev/signature-taskrun-$TASKRUN_UID}" | base64 -d > sig

   $ export TASKRUN_UID=$(tkn tr describe --last -o  jsonpath='{.metadata.uid}')

7. To verify the signature using the public key that you created, enter the following command:

Procedure

1. Create an encrypted x509 key pair and save it as a Kubernetes secret:

   $ cosign generate-key-pair k8s://openshift-pipelines/signing-secrets

   Provide a password when prompted. Cosign stores the resulting private key as part of the signing-secrets Kubernetes secret in the openshift-pipelines namespace, and writes the public key to the cosign.pub local file.

2. Configure authentication for the image registry.

   1. To configure the Tekton Chains controller for pushing signature to an OCI registry, use the credentials associated with the service account of the task run. For detailed information, see the "Authenticating to an OCI registry" section.

   2. To configure authentication for a Kaniko task that builds and pushes image to the registry, create a Kubernetes secret of the docker config.json file containing the required credentials.

      $ oc create secret generic <docker_config_secret_name> \ 1
        --from-file <path_to_config.json> 2

      1

      Substitute with the name of the docker config secret.

      2

      Substitute with the path to docker config.json file.

3. Configure Tekton Chains by setting the artifacts.taskrun.format, artifacts.taskrun.storage, and transparency.enabled parameters in the chains-config object:

   $ oc patch configmap chains-config -n openshift-pipelines -p='{"data":{"artifacts.taskrun.format": "in-toto"}}'
   
   $ oc patch configmap chains-config -n openshift-pipelines -p='{"data":{"artifacts.taskrun.storage": "oci"}}'
   
   $ oc patch configmap chains-config -n openshift-pipelines -p='{"data":{"transparency.enabled": "true"}}'

4. Start the Kaniko task.

   1. Apply the Kaniko task to the cluster.

      $ oc apply -f examples/kaniko/kaniko.yaml 1

      1

      Substitute with the URI or file path to your Kaniko task.

   2. Set the appropriate environment variables.

      $ export REGISTRY=<url_of_registry> 1
      
      $ export DOCKERCONFIG_SECRET_NAME=<name_of_the_secret_in_docker_config_json> 2

      1

      Substitute with the URL of the registry where you want to push the image.

      2

      Substitute with the name of the secret in the docker config.json file.

   3. Start the Kaniko task.

      $ tkn task start --param IMAGE=$REGISTRY/kaniko-chains --use-param-defaults --workspace name=source,emptyDir="" --workspace name=dockerconfig,secret=$DOCKERCONFIG_SECRET_NAME kaniko-chains

      Observe the logs of this task until all steps are complete. On successful authentication, the final image will be pushed to $REGISTRY/kaniko-chains.

5. Wait for a minute to allow Tekton Chains to generate the provenance and sign it, and then check the availability of the chains.tekton.dev/signed=true annotation on the task run.

   $ oc get tr <task_run_name> \ 1
   -o json | jq -r .metadata.annotations
   
   {
     "chains.tekton.dev/signed": "true",
     ...
   }

   1

   Substitute with the name of the task run.

6. Verify the image and the attestation.

   $ cosign verify --key cosign.pub $REGISTRY/kaniko-chains
   
   $ cosign verify-attestation --key cosign.pub $REGISTRY/kaniko-chains

7. Find the provenance for the image in Rekor.

   1. Get the digest of the $REGISTRY/kaniko-chains image. You can search for it ing the task run, or pull the image to extract the digest.

   2. Search Rekor to find all entries that match the sha256 digest of the image.

      $ rekor-cli search --sha <image_digest> 1
      
      <uuid_1> 2
      <uuid_2> 3
      ...

      1

      Substitute with the sha256 digest of the image.

      2

      The first matching universally unique identifier (UUID).

      3

      The second matching UUID.

      The search result displays UUIDs of the matching entries. One of those UUIDs holds the attestation.

   3. Check the attestation.

      $ rekor-cli get --uuid <uuid> --format json | jq -r .Attestation | base64 --decode | jq

Procedure

1. Define a custom SCC with the fsGroup.type parameter set to RunAsAny:

   Example: Custom SCC

   apiVersion: security.openshift.io/v1
   kind: SecurityContextConstraints
   metadata:
     annotations:
       kubernetes.io/description: my-scc is a close replica of anyuid scc. pipelines-scc has fsGroup - RunAsAny.
     name: my-scc
   allowHostDirVolumePlugin: false
   allowHostIPC: false
   allowHostNetwork: false
   allowHostPID: false
   allowHostPorts: false
   allowPrivilegeEscalation: true
   allowPrivilegedContainer: false
   allowedCapabilities: null
   defaultAddCapabilities: null
   fsGroup:
     type: RunAsAny
   groups:
   - system:cluster-admins
   priority: 10
   readOnlyRootFilesystem: false
   requiredDropCapabilities:
   - MKNOD
   runAsUser:
     type: RunAsAny
   seLinuxContext:
     type: MustRunAs
   supplementalGroups:
     type: RunAsAny
   volumes:
   - configMap
   - downwardAPI
   - emptyDir
   - persistentVolumeClaim
   - projected
   - secret

2. Create the custom SCC:

   Example: Create the my-scc SCC

   $ oc create -f my-scc.yaml

3. Create a custom service account:

   Example: Create a fsgroup-runasany service account

   $ oc create serviceaccount fsgroup-runasany

4. Associate the custom SCC with the custom service account:

   Example: Associate the my-scc SCC with the fsgroup-runasany service account

   $ oc adm policy add-scc-to-user my-scc -z fsgroup-runasany

   If you want to use the custom service account for privileged tasks, you can associate the privileged SCC with the custom service account by running the following command:

   Example: Associate the privileged SCC with the fsgroup-runasany service account

   $ oc adm policy add-scc-to-user privileged -z fsgroup-runasany

5. Use the custom service account in the pipeline run and task run:

   Example: Pipeline run YAML with fsgroup-runasany custom service account

   apiVersion: tekton.dev/v1
   kind: PipelineRun
   metadata:
     name: <pipeline-run-name>
   spec:
     pipelineRef:
       name: <pipeline-cluster-task-name>
     taskRunTemplate:
       serviceAccountName: 'fsgroup-runasany'

   Example: Task run YAML with fsgroup-runasany custom service account

   apiVersion: tekton.dev/v1
   kind: TaskRun
   metadata:
     name: <task-run-name>
   spec:
     taskRef:
       name: <cluster-task-name>
     taskRunTemplate:
       serviceAccountName: 'fsgroup-runasany'

Procedure

1. Create the TriggerBinding resource from the YAML file available in the pipelines-tutorial repository:

   $ oc create -f https://raw.githubusercontent.com/openshift/pipelines-tutorial/master/03_triggers/01_binding.yaml

2. Create the TriggerTemplate resource from the YAML file available in the pipelines-tutorial repository:

   $ oc create -f https://raw.githubusercontent.com/openshift/pipelines-tutorial/master/03_triggers/02_template.yaml

3. Create the Trigger resource directly from the pipelines-tutorial repository:

   $ oc create -f https://raw.githubusercontent.com/openshift/pipelines-tutorial/master/03_triggers/03_trigger.yaml

4. Create an EventListener resource using a secure HTTPS connection:

   1. Add a label to enable the secure HTTPS connection to the Eventlistener resource:

      $ oc label namespace <ns-name> operator.tekton.dev/enable-annotation=enabled

   2. Create the EventListener resource from the YAML file available in the pipelines-tutorial repository:

      $ oc create -f https://raw.githubusercontent.com/openshift/pipelines-tutorial/master/03_triggers/04_event_listener.yaml

   3. Create a route with the re-encrypted TLS termination:

      $ oc create route reencrypt --service=<svc-name> --cert=tls.crt --key=tls.key --ca-cert=ca.crt --hostname=<hostname>

Procedure

1. Create a copy of the buildah cluster task as an ordinary task.

   $ oc get clustertask buildah -o yaml | yq '. |= (del .metadata |= with_entries(select(.key == "name" )))' | yq '.kind="Task"' | yq '.metadata.name="buildah-as-user"' | oc create -f -

2. Edit the copied buildah task.

   $ oc edit task buildah-as-user

   Example: Modified Buildah task with build user

   apiVersion: tekton.dev/v1
   kind: Task
   metadata:
     name: buildah-as-user
   spec:
     description: >-
       Buildah task builds source into a container image and
       then pushes it to a container registry.
       Buildah Task builds source into a container image using Project Atomic's
       Buildah build tool.It uses Buildah's support for building from Dockerfiles,
       using its buildah bud command.This command executes the directives in the
       Dockerfile to assemble a container image, then pushes that image to a
       container registry.
     params:
     - name: IMAGE
       description: Reference of the image buildah will produce.
     - name: BUILDER_IMAGE
       description: The location of the buildah builder image.
       default: registry.redhat.io/rhel8/buildah@sha256:99cae35f40c7ec050fed3765b2b27e0b8bbea2aa2da7c16408e2ca13c60ff8ee
     - name: STORAGE_DRIVER
       description: Set buildah storage driver
       default: vfs
     - name: DOCKERFILE
       description: Path to the Dockerfile to build.
       default: ./Dockerfile
     - name: CONTEXT
       description: Path to the directory to use as context.
       default: .
     - name: TLSVERIFY
       description: Verify the TLS on the registry endpoint (for push/pull to a non-TLS registry)
       default: "true"
     - name: FORMAT
       description: The format of the built container, oci or docker
       default: "oci"
     - name: BUILD_EXTRA_ARGS
       description: Extra parameters passed for the build command when building images.
       default: ""
     - description: Extra parameters passed for the push command when pushing images.
       name: PUSH_EXTRA_ARGS
       type: string
       default: ""
     - description: Skip pushing the built image
       name: SKIP_PUSH
       type: string
       default: "false"
     results:
     - description: Digest of the image just built.
       name: IMAGE_DIGEST
       type: string
     workspaces:
     - name: source
     steps:
     - name: build
       securityContext:
         runAsUser: 1000 1
       image: $(params.BUILDER_IMAGE)
       workingDir: $(workspaces.source.path)
       script: |
         echo "Running as USER ID `id`" 2
         buildah --storage-driver=$(params.STORAGE_DRIVER) bud \
           $(params.BUILD_EXTRA_ARGS) --format=$(params.FORMAT) \
           --tls-verify=$(params.TLSVERIFY) --no-cache \
           -f $(params.DOCKERFILE) -t $(params.IMAGE) $(params.CONTEXT)
         [[ "$(params.SKIP_PUSH)" == "true" ]] && echo "Push skipped" && exit 0
         buildah --storage-driver=$(params.STORAGE_DRIVER) push \
           $(params.PUSH_EXTRA_ARGS) --tls-verify=$(params.TLSVERIFY) \
           --digestfile $(workspaces.source.path)/image-digest $(params.IMAGE) \
           docker://$(params.IMAGE)
         cat $(workspaces.source.path)/image-digest | tee /tekton/results/IMAGE_DIGEST
       volumeMounts:
       - name: varlibcontainers
         mountPath: /home/build/.local/share/containers 3
     volumes:
     - name: varlibcontainers
       emptyDir: {}

   1

   Run the container explicitly as the user id 1000, which corresponds to the build user in the Buildah image.

   2

   Display the user id to confirm that the process is running as user id 1000.

   3

   You can change the path for the volume mount as necessary.

Procedure

1. Create a TaskRun object with a custom ConfigMap and Dockerfile objects.

   Example: A task run that runs Buildah as user id 1000

   apiVersion: v1
   data:
     Dockerfile: |
       ARG BASE_IMG=registry.access.redhat.com/ubi9/ubi
       FROM $BASE_IMG AS buildah-runner
       RUN dnf -y update && \
           dnf -y install git && \
           dnf clean all
       CMD git
   kind: ConfigMap
   metadata:
     name: dockerfile 1
   ---
   apiVersion: tekton.dev/v1
   kind: TaskRun
   metadata:
     name: buildah-as-user-1000
   spec:
     taskRunTemplate:
       serviceAccountName: pipelines-sa-userid-1000 2
     params:
     - name: IMAGE
       value: image-registry.openshift-image-registry.svc:5000/test/buildahuser
     taskRef:
       kind: Task
       name: buildah-as-user
     workspaces:
     - configMap:
         name: dockerfile 3
       name: source

   1

   Use a config map because the focus is on the task run, without any prior task that fetches some sources with a Dockerfile.

   2

   The name of the service account that you created.

   3

   Mount a config map as the source workspace for the buildah-as-user task.

2. (Optional) Create a pipeline and a corresponding pipeline run.

   Example: A pipeline and corresponding pipeline run

   apiVersion: tekton.dev/v1
   kind: Pipeline
   metadata:
     name: pipeline-buildah-as-user-1000
   spec:
     params:
     - name: IMAGE
     - name: URL
     workspaces:
     - name: shared-workspace
     - name: sslcertdir
       optional: true
     tasks:
     - name: fetch-repository 1
       taskRef:
         name: git-clone
         kind: ClusterTask
       workspaces:
       - name: output
         workspace: shared-workspace
       params:
       - name: url
         value: $(params.URL)
       - name: subdirectory
         value: ""
       - name: deleteExisting
         value: "true"
     - name: buildah
       taskRef:
         name: buildah-as-user 2
       runAfter:
       - fetch-repository
       workspaces:
       - name: source
         workspace: shared-workspace
       - name: sslcertdir
         workspace: sslcertdir
       params:
       - name: IMAGE
         value: $(params.IMAGE)
   ---
   apiVersion: tekton.dev/v1
   kind: PipelineRun
   metadata:
     name: pipelinerun-buildah-as-user-1000
   spec:
     taskRunSpecs:
       - pipelineTaskName: buildah
         taskServiceAccountName: pipelines-sa-userid-1000 3
     params:
     - name: URL
       value: https://github.com/openshift/pipelines-vote-api
     - name: IMAGE
       value: image-registry.openshift-image-registry.svc:5000/test/buildahuser
     pipelineRef:
       name: pipeline-buildah-as-user-1000
     workspaces:
     - name: shared-workspace 4
       volumeClaimTemplate:
         spec:
           accessModes:
             - ReadWriteOnce
           resources:
             requests:
               storage: 100Mi

   1

   Use the git-clone cluster task to fetch the source containing a Dockerfile and build it using the modified Buildah task.

   2

   Refer to the modified Buildah task.

   3

   Use the service account that you created for the Buildah task.

   4

   Share data between the git-clone task and the modified Buildah task using a persistent volume claim (PVC) created automatically by the controller.

3. Start the task run or the pipeline run.