Red Hat SummitRelease Notes
Release notes for Red Hat's Trusted Artifact Signer 1.4
Abstract
Chapter 1. Introduction
Red Hat’s Trusted Artifact Signer (RHTAS) service enhances software supply chain security by simplifying cryptographic signing and verifying of software artifacts, such as container images, binaries, and Git commits. Trusted Artifact Signer provides a production ready deployment of the SecureSign community project.
The Trusted Artifact Signer software Release Notes documents new features and enhancements, bug fixes, and known issues for the latest version, 1.4. We add the newest items to the top in each chapter, as we build upon the official release notes over the lifecycle of the major, and minor releases.
- New for this release
- Generally Available : The ability to configure the RHTAS services for high-availability environments.
- Generally Available : Implementation of the Sigstore Policy Controller admission controller for Red Hat OpenShift Container Platform.
- Generally Available : Support for using a PostgreSQL database for Trillian.
- Technology Preview : Signing and verifying AI/ML models.
- Technology Preview : RHTAS Console, a web-based user interface where users can search the transparency log, and view signing events.
- Technology Preview : Key Management Service (KMS) support for managing Rekor signing keys.
-
New
cosignversion 3.0.4, helping to streamline the initialization process. - Various Conforma improvements and fixes.
Chapter 2. Technology previews
An overview of the Technology Preview features introduced or updated in this release of Red Hat Trusted Artifact Signer (RHTAS).
Technology Preview features are not supported with Red Hat production service level agreements (SLAs), might not be functionally complete, and Red Hat does not recommend to use them for production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about Red Hat Technology Preview features support scope, see https://access.redhat.com/support/offerings/techpreview/.
- KMS support for managing Rekor signing keys
- With this release, you can now use your existing Key Management Service (KMS) infrastructure for managing the Rekor signing keys. This enhancement removes the requirement for manually configuring and managing the signing keys.
- Support added for signing and verifying AI/ML models
With this release, we introduced a new Model Validation Operator, and a command-line procedure to do pod validation for artificial intelligence (AI) and machine learning (ML) models before running your workloads. The Model Validation Operator integrates a webhook to automatically inject an initialization container to do validation, ensuring that only verified models are used. This validation process works with the RHTAS service to validate the authenticity of the AI/ML models. You can also sign and verify AI/ML models by using the command-line interface, without needing to install a separate binary.
For more information, see the RHTAS Administration Guide.
- Trusted Artifact Signer Console
- In this release, we added the RHTAS Console, a user-friendly graphical interface for viewing read-only information such as The Update Framework (TUF) root details and certificates. This improves the user’s quality of life when managing a RTHAS environment. Before this update, security details were distributed across multiple tools, causing a less user-friendly experience. With this release, users have a unified console, with more features being added in later RHTAS releases, such as: functionality for Rekor Search UI, Rekor keys, along with artifact retrieval and verification actions. For deployment instructions, please see the project’s repository on GitHub for more details.
Chapter 3. New features and enhancements
A list of all major enhancements, and new features introduced in this release of Red Hat Trusted Artifact Signer (RHTAS). Some of these features and enhancements were introduced as Technology Previews in earlier releases, and are now generally available (GA) and fully supported in this release.
The features and enhancements added by this release are:
- Including Cosign version 3.0.4
-
With this release, we have fully migrated to Cosign version 3.0.4, to help streamline our infrastructure. The modernized cache layout of Sigstore and the simplified initialization patterns of the latest major version are now in use. This shift reduces boilerplate code by adopting the
signingConfigapproach, generated directly throughcosign initialize. Additionally, Gitsign and Conforma have been updated to their latest versions, ensuring compatibility with updated Cosign packages andsigstore-golibraries. You can view the latest RHTAS component versions in the appendix.
- Automated job to update the TUF repository after upgrading the RHTAS Operator
With this release, the RHTAS Operator introduces an integrated migration job that automates the update of The Update Framework (TUF) repository during the upgrade from version 1.3.x to 1.4.0. This enhancement streamlines the upgrade process by automatically adopting the new TUF repository structure necessary for Cosign version 3, thereby reducing the risk of a broken "trusted root" state. Users can now do Operator Lifecycle Manager (OLM) upgrades without manual intervention, and both Cosign v2 and v3, and the RHTAS Console, function correctly post-upgrade. This ensures an enterprise-grade experience by guaranteeing that the update is seamless.
If you store your signer keys outside of Red Hat OpenShift, then you must manually update the TUF repository by following this procedure, before you can use the new features of Cosign version 3.
- Command line binaries for RHTAS are available on the Red Hat Developer Portal
- With this release, the command-line interface (CLI) binaries used with RHTAS are now accessible on the Red Hat Developer Portal. Initially, these binaries were exclusively available within OpenShift or Red Hat Enterprise Linux deployments of RHTAS, but customers can now download the binaries for all supported architectures on the Developer Portal, without the need for deploying the RHTAS service.
- Added the Sigstore Policy Controller admission controller
- With this release, users can deploy the Sigstore Policy Controller admission controller alongside RHTAS deployments running on Red Hat OpenShift. This integration offers users a method to manage the container images that are permitted to operate on their OpenShift clusters, based on signatures or attestations generated by RHTAS. Users can install and manage the Sigstore Policy Controller admission controller by installing an Operator that reconciles the upstream Helm chart. This Operator ensures that cluster workloads are only allowed if they comply with the specified cluster image policies. For more information about the Sigstore Policy Controller, you can refer to the RHTAS Administration Guide.
- High availability support added for Trusted Artifact Signer on Red Hat OpenShift
- With this release, users can now configure RHTAS for High Availability (HA) in single cluster deployments, enhancing service reliability and performance. The RHTAS deployment now keeps key components replicated to eliminate single points of failure, provides load balancing, fail over mechanisms, and health checks. This allows the system to manage workloads effectively, ensuring the uptime required for continuous CI/CD pipelines that rely on the Trusted Artifact Signer service, and maintaining operational continuity. For more information about configuring RHTAS for High Availability, you can refer to the RHTAS Administration Guide.
- New configuration options for scaling Trusted Artifact Signer’s services
-
With this release, we implemented enhanced pod scheduling and resource management for RHTAS. This enhancement provides granular control over scaling, scheduling, and resource allocation through a new
PodRequirementsspecification. This addresses the need for fine-grained deployment options by offering control over Custom Resources (CR) such as: Fulcio, Certificate Transparency log (CTlog), Rekor, Trillian, Timestamp Authority (TSA), and The Update Framework (TUF) Trust Root. Users can now manage pod affinity rules, define a matching toleration for node taints, specify the number of replicas for high availability, and set compute resource requests and limits. These new configuration options are also exposed in the OpenShift console UI for easier management.
- New configuration options for Rekor external search index
- With this release, users can use their own Redis database to serve as the search index for Rekor. This integration enables connection with external, highly-available, and managed database or caching services. For production environments that demand greater scalability, reliability, and the ability to use existing infrastructure is essential. When an external search index is configured, the RHTAS Operator will not deploy the embedded Redis instance. Instead, the Rekor service actively uses the specified external connection configuration, which includes support for TLS-enabled connections. This gives users more flexibility, along with an enterprise-ready deployment of RHTAS, simplifying management and enhancing overall performance.
- New configuration options for Rekor attestation storage
- With this release, you can now configure external storage for Rekor attestations. This new feature enhances scalability and flexibility when managing Rekor attestations. This allows for the use of many Rekor replicas simultaneously. We expanded Rekor’s Custom Resource Definition (CRD) with a new attestations section. In this section you can specify a storage URL from storage providers such as: Amazon Web Services (AWS) S3, Google Cloud Storage (GCS), or a file-based persistent volume claim (PVC).
- Added the
--show-warningsoption to Conforma commands -
With this release, we added the
--show-warningsoption to Conforma commands. By default, displaying warning lines are enabled, but they can be disabled when necessary, by setting--show-warnings=false. This enhancement aims to improve logs for users primarily concerned with failures and specific messages.
- Human-readable output for the
ec validate inputcommand -
With this release, users can now validate input by using the
ec validate input --output textcommand, which provides readable, line-oriented results similar to theec validate imagecommand. This improves interactive use, and aligns with the default friendly-text mode commonly used when validating images. As a result, validate-input runs now support showing compact human-readable output instead of JSON or YAML.
- Support for using component names as exemptions
-
With this release, users can now refine exemptions on the
EnterpriseContractPolicyby using specific component names. This enhancement allows for a more precise policy, as it does not relax the policy for unrelated images, and improves the effectiveness of the policy.
- Support for skipping image signature verification in key-less mode
-
With this release, the
ec validate imagecommand now includes the--skip-image-sig-checkoption. By default, image signature verification is enabled. However, when using this option, the command will bypass the image signature verification step, while still performing attestation checks. In public-key mode, both image and attestation signature checks remain unchanged.
- Conforma support for Linux
ppc64leands390xarchitectures -
With this release, Conforma is now available for
ppc64leands390xarchitectures.
- Support for ANSI colors for Conforma command outputs
-
With this release, the
ec validate inputcommand now employs American National Standards Institute (ANSI) colors for a more uniform visual experience, aligning with the visual cues provided byec validate imageand related commands. This change makes it easier to distinguish pass or fail, and emphasis within a standard terminal, enhancing overall usability.
- One-step validation for exported snapshot resources
-
With this release, you can validate exported snapshot resources in a single step by using the following command,
ec validate image. This command reads the snapshot body from.specwhen present. As a result, this minimizes potential errors, and streamlines the process for users by allowing them to pipe theoc get snapshotcommand directly into theec validate imagecommand, therefore eliminating the need for an extrajq .specstep.
- The Conforma CLI properly supports SLSA version 1
- With this release, the Conforma command-line interface (CLI) now consistently handles Supply-chain Levels for Software Artifacts (SLSA) provenance version 1 in both validation and inspection paths. This aligns with the format’s structure and expectations. This enhancement ensures that users receive first-class handling comparable to earlier iterations. As a result, policies can now be evaluated, and tools can inspect attestations reliably, as SLSA version 1 provenance is handled consistently.
- Support for fetching files inside a Tekton task bundle
-
With this release, we defined a new Open Policy Agent (OPA) rego function named
ec.oci.blob_files. This new rego function helps policy authors to directly access and view files from Open Container Initiative (OCI) blob layer tarballs within Tekton task bundles and similar artifacts. As a result, this eliminates the need for manually extracting the contents of the tar file, and streamlines the process.
Chapter 4. Bug fixes
In this release of Red Hat Trusted Artifact Signer (RHTAS), we fixed the following bugs. In addition to these fixes, we also list the descriptions of previously known issues found in earlier versions that we fixed.
- Helm chart disables
PodDisruptionBudgetby default -
The Helm chart included a
PodDisruptionBudget(PDB) that previously defaulted tominAvailable=1, which coincided with the defaultreplicaCountof1for RHTAS components. This arrangement led to deadlocks during node draining, and OpenShift cluster upgrades. With this release, the PDB is disabled by default within the Helm chart, ensuring smooth progression of node draining, and when performing OpenShift cluster upgrades. Users running multiple replicas can re-enable the PDB for added availability guarantees during disruptions.
- Cosign does not respect individual TSA certificate chains during rotation
With this release, we updated
cosignto version 3. This update fixes the issue wherecosignexpects only one single Timestamp Authority (TSA) certificate chain. You can rotating the TSA certificate chain by giving the whole TSA certificate chain to The Update Framework (TUF) as an individual target. During the rotation process, setting the new TSA certificate chain as the new TUF target, and expiring the old TSA certificate chain no longer displays the following error message.main.go:74: error during command execution: unable to load TSA certificates: TSA certificate chain must contain exactly one leaf certificateFor information about rotating the TSA signer key and certificate chain see our procedure for Red Hat OpenShift Container Platform, or Red Hat Enterprise Linux.
Chapter 5. Known issues
Resolved known issues for this release of Red Hat Trusted Artifact Signer (RHTAS):
A list of unresolved known issues found in this release, and earlier releases of RHTAS:
- Client-side load balancing does not balance traffic properly
-
In RHTAS high availability (HA) deployments, gRPC services, such as
trillian-logserver, uses the defaultpick_firstload balancing policy. Combined with theClusterIPOpenShift service, this results in all requests being directed to only one or two back-end replicas. This issue makes horizontal scaling ineffective, as the workload is not evenly distributed, thus limiting overall throughput to the capacity of a single instance. Currently there is no workaround available.
- Cosign OIDC provider selection is ignored when using the
--oidc-issuerflag We found an issue on how Cosign selects an OpenID Connect (OIDC) provider when using Cosign version 3 or later. Cosign adheres to the
signing_config.v0.2.jsonprovided by The Update Framework (TUF) to determine the service URLs for RHTAS components. Cosign follows the "winner-takes-all" logic for selecting which OIDC provider to use for authentication by identifying which OIDC entry has the highest compatible API version within a certain period of time. If there are multiple OIDC providers that meet this criteria, then the first OIDC provider is selected, and the others are ignored. This logic prevents the user from manually selecting which OIDC provider they want to used, even if using the--oidc-issuerflag, because thesigning_config.v0.2.jsonis active. This can result in a hard failure, with the following message:Error: cannot specify service URLs and use signing configTo workaround this issue, you must bypass the automated signing configuration by setting the
--use-signing-configtofalse. This allows you to explicitly set the OIDC provider of your choice when using the--oidc-issuerflag.
- Restoring RHTAS data to new OpenShift cluster
When restoring RHTAS data to a new Red Hat OpenShift cluster, you must regenerate the TLS certificates due to a change of the CA authority for the cluster. This change disrupts secure communication between components, leading some pods to halt during the restoration process. To resolve this issue, initiate the restoration, then before scaling the operator, delete all the TLS certificates by running the following commands:
$ oc delete secret securesign-sample-ctlog-tls securesign-sample-rekor-redis-tls securesign-sample-trillian-db-tls securesign-sample-trillian-logserver-tls securesign-sample-trillian-logsigner-tls $ oc scale deploy rhtas-operator-controller-manager --replicas=1 -n openshift-operatorsAs a result, all pods will start, and communication between components will be re-established.
- The Trillian CR status update fails
-
The Trillian custom resource (CR) fails to update the
status.replicasfield within the CR after a user specifies a custom number of replicas. This results in a mismatch between the number of replicas defined and the number reported in the CR status. Although the correct number of pods are deployed, the status field incorrectly displays the default value, which might cause confusion during monitoring. To work around this issue, manually update thestatus.replicasfield in the CR to match the actual number of replicas. As a result of this workaround, the status field accurately reflects the number of replicas.
- Rekor Search UI does not show records after upgrade
After upgrading the RHTAS Operator to the latest version, the existing Rekor data is not found when searching by email address. The
backfill-redisCron job, which ensures that Rekor Search UI can query the transparency log only runs once per day, at midnight. To workaround this issue, you can trigger thebackfill-redisjob manually, instead of waiting until midnight.To trigger the
backfill-redisjob from the command-line interface, run the following command:$ oc create job --from=cronjob/backfill-redis backfill-redis -n trusted-artifact-signerDoing this adds the missing data back to the Rekor Search UI.
Appendix A. Trusted Artifact Signer components and version numbers
The following tables list Red Hat’s Trusted Artifact Signer (RHTAS) software components and their corresponding version numbers for the 1.4 release.
| Binary | Version |
|---|---|
|
| 3.0.4 |
|
| 0.14.0 |
|
| 1.4.3 |
|
| 0.7 |
|
| 1.7.2 |
|
| 1.7.2 |
|
| 0.12.0 |
|
| 0.19.0 |
|
| 1.2.8 |
| Component | Version |
|---|---|
| logserver | 1.7.2 |
| logsigner | 1.7.2 |
| database | 1.7.2 |
| redis | 1.7.2 |
| Component | Version |
|---|---|
| rekor-server | 1.4.3 |
| backfill-redis | 1.4.3 |
| rekor-search-ui | Unversioned |
| rekor-monitor | Unversioned |
| Component | Version |
|---|---|
| fulcio-server | 1.8.4 |
| Component | Version |
|---|---|
| certificate-transparency-go | 1.3.2 |
| Component | Version |
|---|---|
| timestamp-authority | 2.0.4 |
| Component | Version |
|---|---|
| policy-controller | 0.12.0 |
| policy-controller-operator | 0.12.0 |
| policy-controller-operator-bundle | 0.12.0 |
| Component | Version |
|---|---|
| model-transparency | 1.1.1 |
| model-validation-operator | 1.1.1 |
| model-validation-operator-bundle | 1.1.1 |
'%20d='M201.5%20305.5c-13.8%200-24.9-11.1-24.9-24.6%200-13.8%2011.1-24.9%2024.9-24.9%2013.6%200%2024.6%2011.1%2024.6%2024.9%200%2013.6-11.1%2024.6-24.6%2024.6zM504%20256c0%20137-111%20248-248%20248S8%20393%208%20256%20119%208%20256%208s248%20111%20248%20248zm-132.3-41.2c-9.4%200-17.7%203.9-23.8%2010-22.4-15.5-52.6-25.5-86.1-26.6l17.4-78.3%2055.4%2012.5c0%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.3%2024.9-24.9s-11.1-24.9-24.9-24.9c-9.7%200-18%205.8-22.1%2013.8l-61.2-13.6c-3-.8-6.1%201.4-6.9%204.4l-19.1%2086.4c-33.2%201.4-63.1%2011.3-85.5%2026.8-6.1-6.4-14.7-10.2-24.1-10.2-34.9%200-46.3%2046.9-14.4%2062.8-1.1%205-1.7%2010.2-1.7%2015.5%200%2052.6%2059.2%2095.2%20132%2095.2%2073.1%200%20132.3-42.6%20132.3-95.2%200-5.3-.6-10.8-1.9-15.8%2031.3-16%2019.8-62.5-14.9-62.5zM302.8%20331c-18.2%2018.2-76.1%2017.9-93.6%200-2.2-2.2-6.1-2.2-8.3%200-2.5%202.5-2.5%206.4%200%208.6%2022.8%2022.8%2087.3%2022.8%20110.2%200%202.5-2.2%202.5-6.1%200-8.6-2.2-2.2-6.1-2.2-8.3%200zm7.7-75c-13.6%200-24.6%2011.1-24.6%2024.9%200%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.1%2024.9-24.6%200-13.8-11-24.9-24.9-24.9z'/%3e%3c/svg%3e){kind=small}