Chapter 1. Using Tekton Chains for OpenShift Pipelines supply chain security
Tekton Chains is a Kubernetes Custom Resource Definition (CRD) controller. You can use it to manage the supply chain security of the tasks and pipelines created using Red Hat OpenShift Pipelines.
By default, Tekton Chains observes all task run executions in your OpenShift Container Platform cluster. When the task runs complete, Tekton Chains takes a snapshot of the task runs. It then converts the snapshot to one or more standard payload formats, and finally signs and stores all artifacts.
To capture information about task runs, Tekton Chains uses Result
objects. When the objects are unavailable, Tekton Chains the URLs and qualified digests of the OCI images.
1.1. Key features
-
You can sign task runs, task run results, and OCI registry images with cryptographic keys that are generated by tools such as
cosign
andskopeo
. -
You can use attestation formats such as
in-toto
. - You can securely store signatures and signed artifacts using OCI repository as a storage backend.
1.2. Configuring Tekton Chains
The Red Hat OpenShift Pipelines Operator installs Tekton Chains by default. You can configure Tekton Chains by modifying the TektonConfig
custom resource; the Operator automatically applies the changes that you make in this custom resource.
To edit the custom resource, use the following command:
$ oc edit TektonConfig config
The custom resource includes a chain:
array. You can add any supported configuration parameters to this array, as shown in the following example:
apiVersion: operator.tekton.dev/v1alpha1 kind: TektonConfig metadata: name: config spec: addon: {} chain: artifacts.taskrun.format: tekton config: {}
1.2.1. Supported parameters for Tekton Chains configuration
Cluster administrators can use various supported parameter keys and values to configure specifications about task runs, OCI images, and storage.
1.2.1.1. Supported parameters for task run artifacts
Key | Description | Supported values | Default value |
---|---|---|---|
| The format for storing task run payloads. |
|
|
|
The storage backend for task run signatures. You can specify multiple backends as a comma-separated list, such as |
|
|
| The signature backend for signing task run payloads. |
|
|
slsa/v1
is an alias of in-toto
for backwards compatibility.
1.2.1.2. Supported parameters for pipeline run artifacts
Parameter | Description | Supported values | Default value |
---|---|---|---|
| The format for storing pipeline run payloads. |
|
|
|
The storage backend for storing pipeline run signatures. You can specify multiple backends as a comma-separated list, such as |
|
|
| The signature backend for signing pipeline run payloads. |
|
|
|
When this parameter is |
|
|
-
slsa/v1
is an alias ofin-toto
for backwards compatibility. -
For the
grafeas
storage backend, only Container Analysis is supported. You can not configure thegrafeas
server address in the current version of Tekton Chains.
1.2.1.3. Supported parameters for OCI artifacts
Parameter | Description | Supported values | Default value |
---|---|---|---|
| The format for storing OCI payloads. |
|
|
|
The storage backend for storing OCI signatures. You can specify multiple backends as a comma-separated list, such as |
|
|
| The signature backend for signing OCI payloads. |
|
|
1.2.1.4. Supported parameters for KMS signers
Parameter | Description | Supported values | Default value |
---|---|---|---|
|
The URI reference to a KMS service to use in |
Supported schemes: |
1.2.1.5. Supported parameters for storage
Parameter | Description | Supported values | Default value |
---|---|---|---|
| The GCS bucket for storage | ||
| The OCI repository for storing OCI signatures and attestation. |
If you configure one of the artifact storage backends to | |
| The builder ID to set for in-toto attestations |
| |
|
The build type for in-toto attestation. When this parameter is |
|
If you enable the docdb
storage method is for any artifacts, configure docstore storage options. For more information about the go-cloud docstore URI format, see the docstore package documentation. Red Hat OpenShift Pipelines supports the following docstore services:
-
firestore
-
dynamodb
Parameter | Description | Supported values | Default value |
---|---|---|---|
|
The go-cloud URI reference to a |
| |
|
The value for the Mongo server URL to use for | ||
|
The directory where a file named |
Example value: |
If you enable the grafeas
storage method for any artifacts, configure Grafeas storage options. For more information about Grafeas notes and occurrences, see Grafeas concepts.
To create occurrences, Red Hat OpenShift Pipelines must first create notes that are used to link occurrences. Red Hat OpenShift Pipelines creates two types of occurrences: ATTESTATION
Occurrence and BUILD
Occurrence.
Red Hat OpenShift Pipelines uses the configurable noteid
as the prefix of the note name. It appends the suffix -simplesigning
for the ATTESTATION
note and the suffix -intoto
for the BUILD
note. If the noteid
field is not configured, Red Hat OpenShift Pipelines uses tekton-<NAMESPACE>
as the prefix.
Parameter | Description | Supported values | Default value |
---|---|---|---|
| The OpenShift Container Platform project in which the Grafeas server for storing occurrences is located. | ||
| Optional: the prefix to use for the name of all created notes. | A string without spaces. | |
|
Optional: the |
|
Optionally, you can enable additional uploads of binary transparency attestations.
Parameter | Description | Supported values | Default value |
---|---|---|---|
| Enable or disable automatic binary transparency uploads. |
|
|
| The URL for uploading binary transparency attestations, if enabled. |
|
If you set transparency.enabled
to manual
, only task runs and pipeline runs with the following annotation are uploaded to the transparency log:
chains.tekton.dev/transparency-upload: "true"
If you configure the x509
signature backend, you can optionally enable keyless signing with Fulcio.
Parameter | Description | Supported values | Default value |
---|---|---|---|
| Enable or disable requesting automatic certificates from Fulcio. |
|
|
| The Fulcio address for requesting certificates, if enabled. |
| |
| The expected OIDC issuer. |
| |
| The provider from which to request the ID Token. |
| Red Hat OpenShift Pipelines attempts to use every provider |
| Path to the file containing the ID Token. | ||
|
The URL for the TUF server. |
|
If you configure the kms
signature backend, set the KMS configuration, including OIDC and Spire, as necessary.
Parameter | Description | Supported values | Default value |
---|---|---|---|
|
URI of the KMS server (the value of | ||
|
Authentication token for the KMS server (the value of | ||
|
The full pathname of the file that contains the authentication token for the KMS server (the value of |
Example value: | |
|
The path for OIDC authentication (for example, | ||
| The role for OIDC authentication. | ||
|
The URI of the Spire socket for the KMS token (for example, | ||
| The audience for requesting a SVID from Spire. |
1.2.2. Creating and mounting the Mongo server URL secret
You can provide the value of the Mongo server URL to use for docdb
storage (MONGO_SERVER_URL
) using a secret. You must create this secret, mount it on the Tekton Chains controller, and set the storage.docdb.mongo-server-url-dir
parameter to the directory where the secret is mounted.
Prerequisites
-
You installed the OpenShift CLI (
oc
) utility. -
You are logged in to your OpenShift Container Platform cluster with administrative rights for the
openshift-pipelines
namespace.
Procedure
Create a secret named
mongo-url
with theMONGO_SERVER_URL
file that contains the the Mongo server URL value by entering the following command:$ oc create secret generic mongo-url -n tekton-chains \ --from-file=MONGO_SERVER_URL=<path>/MONGO_SERVER_URL 1
- 1
- The full path and name of the
MONGO_SERVER_URL
file that contains the the Mongo server URL value.
In the
TektonConfig
custom resource (CR), in thechain
section, configure mounting the secret on the Tekton Chains controller and set thestorage.docdb.mongo-server-url-dir
parameter to the directory where the secret is mounted, as shown in the following example:Example configuration for mounting the
mongo-url
secretapiVersion: operator.tekton.dev/v1 kind: TektonConfig metadata: name: config spec: # ... chain: disabled: false storage.docdb.mongo-server-url-dir: /tmp/mongo-url options: deployments: tekton-chains-controller: spec: template: spec: containers: - name: tekton-chains-controller volumeMounts: - mountPath: /tmp/mongo-url name: mongo-url volumes: - name: mongo-url secret: secretName: mongo-url # ...
1.2.3. Creating and mounting the KMS authentication token secret
You can provide the authentication token for the KMS server using a secret. For example, if the KMS provider is Hashicorp Vault, the secret must contain the value of VAULT_TOKEN
.
You must create this secret, mount it on the Tekton Chains controller, and set the signers.kms.auth.token-path
parameter to the full pathname of the authentication token file.
Prerequisites
-
You installed the OpenShift CLI (
oc
) utility. -
You are logged in to your OpenShift Container Platform cluster with administrative rights for the
openshift-pipelines
namespace.
Procedure
Create a secret named
kms-secrets
with theKMS_AUTH_TOKEN
file that contains the authentication token for the KMS server by entering the following command:$ oc create secret generic kms-secrets -n tekton-chains \ --from-file=KMS_AUTH_TOKEN=<path_and_name> 1
- 1
- The full path and name of the file that contains the authentication token for the KMS server, for example,
/home/user/KMS_AUTH_TOKEN
. You can use another file name instead ofKMS_AUTH_TOKEN
.
In the
TektonConfig
custom resource (CR), in thechain
section, configure mounting the secret on the Tekton Chains controller and set thesigners.kms.auth.token-path
parameter to the full pathname of the authentication token file, as shown in the following example:Example configuration for mounting the
kms-secrets
secretapiVersion: operator.tekton.dev/v1 kind: TektonConfig metadata: name: config spec: # ... chain: disabled: false signers.kms.auth.token-path: /etc/kms-secrets/KMS_AUTH_TOKEN options: deployments: tekton-chains-controller: spec: template: spec: containers: - name: tekton-chains-controller volumeMounts: - mountPath: /etc/kms-secrets name: kms-secrets volumes: - name: kms-secrets secret: secretName: kms-secrets # ...
1.2.4. Enabling Tekton Chains to operate only in selected namespaces
By default, the Tekton Chains controller monitors resources in all namespaces. You can customize Tekton Chains to run only in specific namespaces, which provides granular control over its operation.
Prerequisites
-
You are logged in to your OpenShift Container Platform cluster with
cluster-admin
privileges.
Procedure
In the
TektonConfig
CR, in thechain
section, add the--namespace=
argument to contain the namespaces that the controller should monitor.The following example shows the configuration for the Tekton Chains controller to only monitor resources within the
dev
andtest
namespaces, filteringPipelineRun
andTaskRun
objects accordingly:apiVersion: operator.tekton.dev/v1alpha1 kind: TektonConfig metadata: name: config spec: chain: disabled: false options: deployments: tekton-chains-controller: spec: template: spec: containers: - args: - --namespace=dev, test 1 name: tekton-chains-controller
- 1
- If the
--namespace
argument is not provided or is left empty, the controller watches all namespaces by default.
1.3. Secrets for signing data in Tekton Chains
Cluster administrators can generate a key pair and use Tekton Chains to sign artifacts using a Kubernetes secret. For Tekton Chains to work, a private key and a password for encrypted keys must exist as part of the signing-secrets
secret in the openshift-pipelines
namespace.
Currently, Tekton Chains supports the x509
and cosign
signature schemes.
Use only one of the supported signature schemes.
The x509 signing scheme
To use the x509
signing scheme with Tekton Chains, you must fulfill the following requirements:
-
Store the private key in the
signing-secrets
with thex509.pem
structure. -
Store the private key as an unencrypted
PKCS #8
PEM file. -
The key is of
ed25519
orecdsa
type.
The cosign signing scheme
To use the cosign
signing scheme with Tekton Chains, you must fulfill the following requirements:
-
Store the private key in the
signing-secrets
with thecosign.key
structure. -
Store the password in the
signing-secrets
with thecosign.password
structure. -
Store the private key as an encrypted PEM file of type
ENCRYPTED COSIGN PRIVATE KEY
.
1.3.1. Generating the x509 key pair by using the TektonConfig CR
To use the x509
signing scheme for Tekton Chains secrets, you must generate the x509
key pair.
You can generate the x509
key pair by setting the generateSigningSecret
field in the TektonConfig
custom resource (CR) to true
. The Red Hat OpenShift Pipelines Operator generates an ecdsa
type key pair: an x509.pem
private key and an x509-pub.pem
public key. The Operator stores the keys in the signing-secrets
secret in the openshift-pipelines
namespace.
If you set the generateSigningSecret
field from true
to false
, the Red Hat OpenShift Pipelines Operator overrides and empties any value in the signing-secrets
secret. Ensure that you store the x509-pub.pem
public key outside of the secret to protect the key from the deletion. The Operator can use the key at a later stage to verify artifact attestations.
The Red Hat OpenShift Pipelines Operator does not provide the following functions to limit potential security issues:
- Key rotation
- Auditing key usage
- Proper access control to the key
Prerequisites
-
You installed the OpenShift CLI (
oc
) utility. -
You are logged in to your OpenShift Container Platform cluster with administrative rights for the
openshift-pipelines
namespace.
Procedure
Edit the
TektonConfig
CR by running the following command:$ oc edit TektonConfig config
In the
TektonConfig
CR, set thegenerateSigningSecret
value totrue
:Example of creating an ecdsa key pair by using the TektonConfig CR
apiVersion: operator.tekton.dev/v1 kind: TektonConfig metadata: name: config spec: # ... chain: disabled: false generateSigningSecret: true 1 # ...
- 1
- The default value is
false
. Setting the value totrue
generates theecdsa
key pair.
1.3.2. Signing with the cosign tool
You can use the cosign
signing scheme with Tekton Chains using the cosign
tool.
Prerequisites
- You installed the cosign tool.
Procedure
Generate the
cosign.key
andcosign.pub
key pairs by running the following command:$ cosign generate-key-pair k8s://openshift-pipelines/signing-secrets
Cosign prompts you for a password and then creates a Kubernetes secret.
-
Store the encrypted
cosign.key
private key and thecosign.password
decryption password in thesigning-secrets
Kubernetes secret. Ensure that the private key is stored as an encrypted PEM file of theENCRYPTED COSIGN PRIVATE KEY
type.
1.3.3. Signing with the skopeo tool
You can generate keys using the skopeo
tool and use them in the cosign
signing scheme with Tekton Chains.
Prerequisites
- You installed the skopeo tool.
Procedure
Generate a public/private key pair by running the following command:
$ skopeo generate-sigstore-key --output-prefix <mykey> 1
- 1
- Replace
<mykey>
with a key name of your choice.
Skopeo prompts you for a passphrase for the private key and then creates the key files named
<mykey>.private
and<mykey>.pub
.Encode the
<mykey>.pub
file using thebase64
tool by running the following command:$ base64 -w 0 <mykey>.pub > b64.pub
Encode the
<mykey>.private
file using thebase64
tool by running the following command:$ base64 -w 0 <mykey>.private > b64.private
Encode the passhprase using the
base64
tool by running the following command:$ echo -n '<passphrase>' | base64 -w 0 > b64.passphrase 1
- 1
- Replace
<passphrase>
with the passphrase that you used for the key pair.
Create the
signing-secrets
secret in theopenshift-pipelines
namespace by running the following command:$ oc create secret generic signing-secrets -n openshift-pipelines
Edit the
signing-secrets
secret by running the following command:$ oc edit secret -n openshift-pipelines signing-secrets
Add the encoded keys in the data of the secret in the following way:
apiVersion: v1 data: cosign.key: <Encoded <mykey>.private> 1 cosign.password: <Encoded passphrase> 2 cosign.pub: <Encoded <mykey>.pub> 3 immutable: true kind: Secret metadata: name: signing-secrets # ... type: Opaque
1.3.4. Resolving the "secret already exists" error
If the signing-secret
secret is already populated, the command to create this secret might output the following error message:
Error from server (AlreadyExists): secrets "signing-secrets" already exists
You can resolve this error by deleting the secret.
Procedure
Delete the
signing-secret
secret by running the following command:$ oc delete secret signing-secrets -n openshift-pipelines
- Re-create the key pairs and store them in the secret using your preferred signing scheme.
1.4. Authenticating to an OCI registry
Before pushing signatures to an OCI registry, cluster administrators must configure Tekton Chains to authenticate with the registry. The Tekton Chains controller uses the same service account under which the task runs execute. To set up a service account with the necessary credentials for pushing signatures to an OCI registry, perform the following steps:
Procedure
Set the namespace and name of the Kubernetes service account.
$ export NAMESPACE=<namespace> 1 $ export SERVICE_ACCOUNT_NAME=<service_account> 2
Create a Kubernetes secret.
$ oc create secret registry-credentials \ --from-file=.dockerconfigjson \ 1 --type=kubernetes.io/dockerconfigjson \ -n $NAMESPACE
- 1
- Substitute with the path to your Docker config file. Default path is
~/.docker/config.json
.
Give the service account access to the secret.
$ oc patch serviceaccount $SERVICE_ACCOUNT_NAME \ -p "{\"imagePullSecrets\": [{\"name\": \"registry-credentials\"}]}" -n $NAMESPACE
If you patch the default
pipeline
service account that Red Hat OpenShift Pipelines assigns to all task runs, the Red Hat OpenShift Pipelines Operator will override the service account. As a best practice, you can perform the following steps:Create a separate service account to assign to user’s task runs.
$ oc create serviceaccount <service_account_name>
Associate the service account to the task runs by setting the value of the
serviceaccountname
field in the task run template.apiVersion: tekton.dev/v1 kind: TaskRun metadata: name: build-push-task-run-2 spec: taskRunTemplate: serviceAccountName: build-bot 1 taskRef: name: build-push ...
- 1
- Substitute with the name of the newly created service account.
1.5. Creating and verifying task run signatures without any additional authentication
To verify signatures of task runs using Tekton Chains with any additional authentication, perform the following tasks:
-
Generate an encrypted
x509
orcosign
key pair and store it as a Kubernetes secret. - Configure the Tekton Chains backend storage.
- Create a task run, sign it, and store the signature and the payload as annotations on the task run itself.
- Retrieve the signature and payload from the signed task run.
- Verify the signature of the task run.
Prerequisites
Ensure that the following components are installed on the cluster:
- Red Hat OpenShift Pipelines Operator
- Tekton Chains
- Cosign
Procedure
-
Generate an encrypted
x509
orcosign
key pair. For more information about creating a key pair and saving it as a secret, see "Secrets for signing data in Tekton Chains". In the Tekton Chains configuration, disable the OCI storage, and set the task run storage and format to
tekton
. In theTektonConfig
custom resource set the following values:apiVersion: operator.tekton.dev/v1alpha1 kind: TektonConfig metadata: name: config spec: # ... chain: artifacts.oci.storage: "" artifacts.taskrun.format: tekton artifacts.taskrun.storage: tekton # ...
For more information about configuring Tekton Chains using the
TektonConfig
custom resource, see "Configuring Tekton Chains".To restart the Tekton Chains controller to ensure that the modified configuration is applied, enter the following command:
$ oc delete po -n openshift-pipelines -l app=tekton-chains-controller
Create a task run by entering the following command:
$ oc create -f https://raw.githubusercontent.com/tektoncd/chains/main/examples/taskruns/task-output-image.yaml 1
- 1
- Replace the example URI with the URI or file path pointing to your task run.
Example output
taskrun.tekton.dev/build-push-run-output-image-qbjvh created
Check the status of the steps by entering the following command. Wait until the process finishes.
$ tkn tr describe --last
Example output
[...truncated output...] NAME STATUS ∙ create-dir-builtimage-9467f Completed ∙ git-source-sourcerepo-p2sk8 Completed ∙ build-and-push Completed ∙ echo Completed ∙ image-digest-exporter-xlkn7 Completed
To retrieve the signature from the object stored as
base64
encoded annotations, enter the following commands:$ tkn tr describe --last -o jsonpath="{.metadata.annotations.chains\.tekton\.dev/signature-taskrun-$TASKRUN_UID}" | base64 -d > sig
$ export TASKRUN_UID=$(tkn tr describe --last -o jsonpath='{.metadata.uid}')
- To verify the signature using the public key that you created, enter the following command:
$ cosign verify-blob-attestation --insecure-ignore-tlog --key path/to/cosign.pub --signature sig --type slsaprovenance --check-claims=false /dev/null 1
- 1
- Replace
path/to/cosign.pub
with the path name of the public key file.Example output
Verified OK
Additional resources
1.6. Using Tekton Chains to sign and verify image and provenance
Cluster administrators can use Tekton Chains to sign and verify images and provenances, by performing the following tasks:
-
Generate an encrypted
x509
orcosign
key pair and store it as a Kubernetes secret. - Set up authentication for the OCI registry to store images, image signatures, and signed image attestations.
- Configure Tekton Chains to generate and sign provenance.
- Create an image with Kaniko in a task run.
- Verify the signed image and the signed provenance.
Prerequisites
Ensure that the following are installed on the cluster:
Procedure
-
Generate an encrypted
x509
orcosign
key pair. For more information about creating a key pair and saving it as a secret, see "Secrets for signing data in Tekton Chains". Configure authentication for the image registry.
- To configure the Tekton Chains controller for pushing signature to an OCI registry, use the credentials associated with the service account of the task run. For detailed information, see the "Authenticating to an OCI registry" section.
To configure authentication for a Kaniko task that builds and pushes image to the registry, create a Kubernetes secret of the docker
config.json
file containing the required credentials.$ oc create secret generic <docker_config_secret_name> \ 1 --from-file <path_to_config.json> 2
Configure Tekton Chains by setting the
artifacts.taskrun.format
,artifacts.taskrun.storage
, andtransparency.enabled
parameters in thechains-config
object:$ oc patch configmap chains-config -n openshift-pipelines -p='{"data":{"artifacts.taskrun.format": "in-toto"}}' $ oc patch configmap chains-config -n openshift-pipelines -p='{"data":{"artifacts.taskrun.storage": "oci"}}' $ oc patch configmap chains-config -n openshift-pipelines -p='{"data":{"transparency.enabled": "true"}}'
Start the Kaniko task.
Apply the Kaniko task to the cluster.
$ oc apply -f examples/kaniko/kaniko.yaml 1
- 1
- Substitute with the URI or file path to your Kaniko task.
Set the appropriate environment variables.
$ export REGISTRY=<url_of_registry> 1 $ export DOCKERCONFIG_SECRET_NAME=<name_of_the_secret_in_docker_config_json> 2
Start the Kaniko task.
$ tkn task start --param IMAGE=$REGISTRY/kaniko-chains --use-param-defaults --workspace name=source,emptyDir="" --workspace name=dockerconfig,secret=$DOCKERCONFIG_SECRET_NAME kaniko-chains
Observe the logs of this task until all steps are complete. On successful authentication, the final image will be pushed to
$REGISTRY/kaniko-chains
.
Wait for a minute to allow Tekton Chains to generate the provenance and sign it, and then check the availability of the
chains.tekton.dev/signed=true
annotation on the task run.$ oc get tr <task_run_name> \ 1 -o json | jq -r .metadata.annotations { "chains.tekton.dev/signed": "true", ... }
- 1
- Substitute with the name of the task run.
Verify the image and the attestation.
$ cosign verify --key cosign.pub $REGISTRY/kaniko-chains $ cosign verify-attestation --key cosign.pub $REGISTRY/kaniko-chains
Find the provenance for the image in Rekor.
- Get the digest of the $REGISTRY/kaniko-chains image. You can search for it ing the task run, or pull the image to extract the digest.
Search Rekor to find all entries that match the
sha256
digest of the image.$ rekor-cli search --sha <image_digest> 1 <uuid_1> 2 <uuid_2> 3 ...
The search result displays UUIDs of the matching entries. One of those UUIDs holds the attestation.
Check the attestation.
$ rekor-cli get --uuid <uuid> --format json | jq -r .Attestation | base64 --decode | jq