Chapter 8. Repository mirroring


8.1. Repository mirroring

Red Hat Quay repository mirroring lets you mirror images from external container registries, or another local registry, into your Red Hat Quay cluster. Using repository mirroring, you can synchronize images to Red Hat Quay based on repository names and tags.

From your Red Hat Quay cluster with repository mirroring enabled, you can perform the following:

  • Choose a repository from an external registry to mirror
  • Add credentials to access the external registry
  • Identify specific container image repository names and tags to sync
  • Set intervals at which a repository is synced
  • Check the current state of synchronization

To use the mirroring functionality, you need to perform the following actions:

  • Enable repository mirroring in the Red Hat Quay configuration file
  • Run a repository mirroring worker
  • Create mirrored repositories

All repository mirroring configurations can be performed using the configuration tool UI or by the Red Hat Quay API.

8.2. Repository mirroring compared to geo-replication

Red Hat Quay geo-replication mirrors the entire image storage backend data between 2 or more different storage backends while the database is shared, for example, one Red Hat Quay registry with two different blob storage endpoints. The primary use cases for geo-replication include the following:

  • Speeding up access to the binary blobs for geographically dispersed setups
  • Guaranteeing that the image content is the same across regions

Repository mirroring synchronizes selected repositories, or subsets of repositories, from one registry to another. The registries are distinct, with each registry having a separate database and separate image storage.

The primary use cases for mirroring include the following:

  • Independent registry deployments in different data centers or regions, where a certain subset of the overall content is supposed to be shared across the data centers and regions
  • Automatic synchronization or mirroring of selected (allowlisted) upstream repositories from external registries into a local Red Hat Quay deployment
Note

Repository mirroring and geo-replication can be used simultaneously.

Table 8.1. Red Hat Quay Repository mirroring and geo-replication comparison
Feature / CapabilityGeo-replicationRepository mirroring

What is the feature designed to do?

A shared, global registry

Distinct, different registries

What happens if replication or mirroring has not been completed yet?

The remote copy is used (slower)

No image is served

Is access to all storage backends in both regions required?

Yes (all Red Hat Quay nodes)

No (distinct storage)

Can users push images from both sites to the same repository?

Yes

No

Is all registry content and configuration identical across all regions (shared database)?

Yes

No

Can users select individual namespaces or repositories to be mirrored?

No

Yes

Can users apply filters to synchronization rules?

No

Yes

Are individual / different role-base access control configurations allowed in each region

No

Yes

8.3. Using repository mirroring

The following list shows features and limitations of Red Hat Quay repository mirroring:

  • With repository mirroring, you can mirror an entire repository or selectively limit which images are synced. Filters can be based on a comma-separated list of tags, a range of tags, or other means of identifying tags through Unix shell-style wildcards. For more information, see the documentation for wildcards.
  • When a repository is set as mirrored, you cannot manually add other images to that repository.
  • Because the mirrored repository is based on the repository and tags you set, it will hold only the content represented by the repository and tag pair. For example if you change the tag so that some images in the repository no longer match, those images will be deleted.
  • Only the designated robot can push images to a mirrored repository, superseding any role-based access control permissions set on the repository.
  • Mirroring can be configured to rollback on failure, or to run on a best-effort basis.
  • With a mirrored repository, a user with read permissions can pull images from the repository but cannot push images to the repository.
  • Changing settings on your mirrored repository can be performed in the Red Hat Quay user interface, using the Repositories Mirrors tab for the mirrored repository you create.
  • Images are synced at set intervals, but can also be synced on demand.

8.4. Mirroring configuration UI

  1. Start the Quay container in configuration mode and select the Enable Repository Mirroring check box. If you want to require HTTPS communications and verify certificates during mirroring, select the HTTPS and cert verification check box.

    Enable mirroring and require HTTPS and verified certificates

  2. Validate and download the configuration file, and then restart Quay in registry mode using the updated config file.

8.5. Mirroring configuration fields

Table 8.2. Mirroring configuration
FieldTypeDescription

FEATURE_REPO_MIRROR

Boolean

Enable or disable repository mirroring

Default: false

REPO_MIRROR_INTERVAL

Number

The number of seconds between checking for repository mirror candidates

Default: 30

REPO_MIRROR_SERVER_HOSTNAME

String

Replaces the SERVER_HOSTNAME as the destination for mirroring.

Default: None

Example:
openshift-quay-service

REPO_MIRROR_TLS_VERIFY

Boolean

Require HTTPS and verify certificates of Quay registry during mirror.

Default: true

REPO_MIRROR_ROLLBACK

Boolean

When set to true, the repository rolls back after a failed mirror attempt.

Default: false

8.6. Mirroring worker

Use the following procedure to start the repository mirroring worker.

Procedure

  • If you have not configured TLS communications using a /root/ca.crt certificate, enter the following command to start a Quay pod with the repomirror option:

    $ sudo podman run -d --name mirroring-worker \
      -v $QUAY/config:/conf/stack:Z \
      registry.redhat.io/quay/quay-rhel8:v3.13.1 repomirror
  • If you have configured TLS communications using a /root/ca.crt certificate, enter the following command to start the repository mirroring worker:

    $ sudo podman run -d --name mirroring-worker \
      -v $QUAY/config:/conf/stack:Z \
      -v /root/ca.crt:/etc/pki/ca-trust/source/anchors/ca.crt:Z \
      registry.redhat.io/quay/quay-rhel8:v3.13.1 repomirror

8.7. Creating a mirrored repository

When mirroring a repository from an external container registry, you must create a new private repository. Typically, the same name is used as the target repository, for example, quay-rhel8.

Create new Red Hat Quay repo

8.7.1. Repository mirroring settings

Use the following procedure to adjust the settings of your mirrored repository.

Prerequisites

  • You have enabled repository mirroring in your Red Hat Quay configuration file.
  • You have deployed a mirroring worker.

Procedure

  1. In the Settings tab, set the Repository State to Mirror:

    Create a new Red Hat Quay repo mirror

  2. In the Mirror tab, enter the details for connecting to the external registry, along with the tags, scheduling and access information:

    Repository mirroring

  3. Enter the details as required in the following fields:

    • Registry Location: The external repository you want to mirror, for example, registry.redhat.io/quay/quay-rhel8
    • Tags: This field is required. You may enter a comma-separated list of individual tags or tag patterns. (See Tag Patterns section for details.)
    • Start Date: The date on which mirroring begins. The current date and time is used by default.
    • Sync Interval: Defaults to syncing every 24 hours. You can change that based on hours or days.
    • Robot User: Create a new robot account or choose an existing robot account to do the mirroring.
    • Username: The username for accessing the external registry holding the repository you are mirroring.
    • Password: The password associated with the Username. Note that the password cannot include characters that require an escape character (\).

8.7.2. Advanced settings

In the Advanced Settings section, you can configure SSL/TLS and proxy with the following options:

  • Verify TLS: Select this option if you want to require HTTPS and to verify certificates when communicating with the target remote registry.
  • Accept Unsigned Images: Selecting this option allows unsigned images to be mirrored.
  • HTTP Proxy: Select this option if you want to require HTTPS and to verify certificates when communicating with the target remote registry.
  • HTTPS PROXY: Identify the HTTPS proxy server needed to access the remote site, if a proxy server is needed.
  • No Proxy: List of locations that do not require proxy.

8.7.3. Synchronize now

Use the following procedure to initiate the mirroring operation.

Procedure

  • To perform an immediate mirroring operation, press the Sync Now button on the repository’s Mirroring tab. The logs are available on the Usage Logs tab:

    Usage logs

    When the mirroring is complete, the images will appear in the Tags tab:

    Repository mirroring tags

    Below is an example of a completed Repository Mirroring screen:

    Repository mirroring details

8.8. Event notifications for mirroring

There are three notification events for repository mirroring:

  • Repository Mirror Started
  • Repository Mirror Success
  • Repository Mirror Unsuccessful

The events can be configured inside of the Settings tab for each repository, and all existing notification methods such as email, Slack, Quay UI, and webhooks are supported.

8.9. Mirroring tag patterns

At least one tag must be entered. The following table references possible image tag patterns.

8.9.1. Pattern syntax

Pattern

Description

*

Matches all characters

?

Matches any single character

[seq]

Matches any character in seq

[!seq]

Matches any character not in seq

8.9.2. Example tag patterns

Example Pattern

Example Matches

v3*

v32, v3.1, v3.2, v3.2-4beta, v3.3

v3.*

v3.1, v3.2, v3.2-4beta

v3.?

v3.1, v3.2, v3.3

v3.[12]

v3.1, v3.2

v3.[12]*

v3.1, v3.2, v3.2-4beta

v3.[!1]*

v3.2, v3.2-4beta, v3.3

8.10. Working with mirrored repositories

Once you have created a mirrored repository, there are several ways you can work with that repository. Select your mirrored repository from the Repositories page and do any of the following:

  • Enable/disable the repository: Select the Mirroring button in the left column, then toggle the Enabled check box to enable or disable the repository temporarily.
  • Check mirror logs: To make sure the mirrored repository is working properly, you can check the mirror logs. To do that, select the Usage Logs button in the left column. Here’s an example:

    View logs for your Red Hat Quay repo mirror

  • Sync mirror now: To immediately sync the images in your repository, select the Sync Now button.
  • Change credentials: To change the username and password, select DELETE from the Credentials line. Then select None and add the username and password needed to log into the external registry when prompted.
  • Cancel mirroring: To stop mirroring, which keeps the current images available but stops new ones from being synced, select the CANCEL button.
  • Set robot permissions: Red Hat Quay robot accounts are named tokens that hold credentials for accessing external repositories. By assigning credentials to a robot, that robot can be used across multiple mirrored repositories that need to access the same external registry.

    You can assign an existing robot to a repository by going to Account Settings, then selecting the Robot Accounts icon in the left column. For the robot account, choose the link under the REPOSITORIES column. From the pop-up window, you can:

    • Check which repositories are assigned to that robot.
    • Assign read, write or Admin privileges to that robot from the PERMISSION field shown in this figure: Assign a robot to mirrored repo
  • Change robot credentials: Robots can hold credentials such as Kubernetes secrets, Docker login information, and Mesos bundles. To change robot credentials, select the Options gear on the robot’s account line on the Robot Accounts window and choose View Credentials. Add the appropriate credentials for the external repository the robot needs to access.

    Assign permission to a robot

  • Check and change general setting: Select the Settings button (gear icon) from the left column on the mirrored repository page. On the resulting page, you can change settings associated with the mirrored repository. In particular, you can change User and Robot Permissions, to specify exactly which users and robots can read from or write to the repo.

8.11. Repository mirroring recommendations

Best practices for repository mirroring include the following:

  • Repository mirroring pods can run on any node. This means that you can run mirroring on nodes where Red Hat Quay is already running.
  • Repository mirroring is scheduled in the database and runs in batches. As a result, repository workers check each repository mirror configuration file and reads when the next sync needs to be. More mirror workers means more repositories can be mirrored at the same time. For example, running 10 mirror workers means that a user can run 10 mirroring operators in parallel. If a user only has 2 workers with 10 mirror configurations, only 2 operators can be performed.
  • The optimal number of mirroring pods depends on the following conditions:

    • The total number of repositories to be mirrored
    • The number of images and tags in the repositories and the frequency of changes
    • Parallel batching

      For example, if a user is mirroring a repository that has 100 tags, the mirror will be completed by one worker. Users must consider how many repositories one wants to mirror in parallel, and base the number of workers around that.

      Multiple tags in the same repository cannot be mirrored in parallel.

Red Hat logoGithubRedditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

© 2024 Red Hat, Inc.