Red Hat Summit Red Hat SummitSupportConsoleDevelopersStart a trialContact

Chapter 8. Bulk importing in Red Hat Developer Hub

Important

These features are for Technology Preview only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs), might not be functionally complete, and Red Hat does not recommend using them for production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information on Red Hat Technology Preview features, see Technology Preview Features Scope.

Red Hat Developer Hub can automate the onboarding of GitHub repositories and GitLab projects, and track their import status.

You can enable the Bulk Import feature for users and give them the necessary permissions to access it. This feature is available for GitHub repositories and GitLab projects.

Prerequisites

Procedure

  1. The Bulk Import plugins are installed but disabled by default. To enable the ./dynamic-plugins/dist/red-hat-developer-hub-backstage-plugin-bulk-import-backend-dynamic and ./dynamic-plugins/dist/red-hat-developer-hub-backstage-plugin-bulk-import plugins, edit your dynamic-plugins.yaml with the following content:

    dynamic-plugins.yaml fragment

    plugins:
  - package: ./dynamic-plugins/dist/red-hat-developer-hub-backstage-plugin-bulk-import-backend-dynamic
    disabled: false
  - package: ./dynamic-plugins/dist/red-hat-developer-hub-backstage-plugin-bulk-import
    disabled: false
    Copy to Clipboard Toggle word wrap

    See Installing and viewing plugins in Red Hat Developer Hub.

  2. Configure the required bulk.import RBAC permission for the users who are not administrators as follows:

    rbac-policy.csv fragment

    p, role:default/bulk-import, bulk.import, use, allow
g, user:default/<your_user>, role:default/bulk-import
    Copy to Clipboard Toggle word wrap

    Note that only Developer Hub administrators or users with the bulk.import permission can use the Bulk Import feature. See Permission policies in Red Hat Developer Hub.

Verification

  • The sidebar displays a Bulk Import option.
  • The Bulk Import page shows a list of added GitHub repositories and GitLab projects.

8.2. Importing multiple GitHub repositories
Copy link

In Red Hat Developer Hub, you can select your GitHub repositories and automate their onboarding to the Developer Hub catalog.

Prerequisites

Procedure

  1. Click Bulk Import in the left sidebar.

  2. Click the Add button in the top-right corner to see the list of all repositories accessible from the configured GitHub integrations.

    1. From the Repositories view, you can select any repository, or search for any accessible repositories. For each repository selected, a catalog-info.yaml is generated.
    2. From the Organizations view, you can select any organization by clicking Select in the third column. This option allows you to select one or more repositories from the selected organization.

  3. Click Preview file to view or edit the details of the pull request for each repository.

    1. Review the pull request description and the catalog-info.yaml file content.
    2. Optional: when the repository has a .github/CODEOWNERS file, you can select the Use CODEOWNERS file as Entity Owner checkbox to use it, rather than having the content-info.yaml contain a specific entity owner.
    3. Click Save.

  4. Click Create pull requests. At this point, a set of dry-run checks runs against the selected repositories to ensure they meet the requirements for import, such as:

    1. Verifying that there is no entity in the Developer Hub catalog with the name specified in the repository catalog-info.yaml
    2. Verifying that the repository is not empty

    3. Verifying that the repository contains a .github/CODEOWNERS file if the Use CODEOWNERS file as Entity Owner checkbox is selected for that repository

      • If any errors occur, the pull requests are not created, and you see a Failed to create PR error message detailing the issues. To view more details about the reasons, click Edit.
      • If there are no errors, the pull requests are created, and you are redirected to the list of added repositories.
  5. Review and merge each pull request that creates a catalog-info.yml file.

Verification

  • The Added entities list displays the repositories you imported, each with an appropriate status: either Waiting for approval or Added.
  • For each Waiting for approval import job listed, there is a corresponding pull request adding the catalog-info.yaml file in the corresponding repository.

8.3. Importing multiple GitLab repositories
Copy link

In Red Hat Developer Hub, you can select your GitLab projects and automate their onboarding to the Developer Hub catalog. This feature is a Technology preview.

Important

Technology Preview features provide early access to upcoming product innovations, enabling you to test functionality and provide feedback during the development process. However, these features are not fully supported under Red Hat Subscription Level Agreements, may not be functionally complete, and are not intended for production use. As Red Hat considers making future iterations of Technology Preview features generally available, we will attempt to resolve any issues that customers experience when using these features. See: Technology Preview support scope.

Prerequisites

  • You have include::modules/streamline-software-development-and-management/proc-enabling-and-authorizing-bulk-import-capabilities.adoc[enabled the Bulk Import feature and given access to it].
  • You have set up a GitLab personal access token (PAT).

Procedure

  1. In RHDH, click Bulk Import
  2. Click Import.
  3. Select GitLab as your Approval tool option.

  4. Use the Project and Group views to see the list of all available GitLab projects and groups:

    • Use the Project view to select GitLab projects for importing.
    • Use the Group view to select GitLab groups and their associated projects for importing.
  5. In GitLab, review the automatically created "Ad catalog-info.yaml file" merge request for each project you selected for Bulk Import.
  6. Merge the merge request.

Verification

  1. In RHDH, click Bulk Import

  2. In the Imported entities list, each imported GitLab project has the appropriate status: either Waiting for approval or Added.

    • For each Waiting for approval import job listed, there is a corresponding merge request adding the catalog-info.yaml file in the corresponding project.

The Bulk Import backend plugin adds the following events to the Developer Hub audit logs. See Audit logs in Red Hat Developer Hub for more information on how to configure and view audit logs.

Bulk Import Events:

BulkImportUnknownEndpoint
Tracks requests to unknown endpoints.
BulkImportPing
Tracks GET requests to the /ping endpoint, which allows us to make sure the bulk import backend is up and running.
BulkImportFindAllOrganizations
Tracks GET requests to the /organizations endpoint, which returns the list of organizations accessible from all configured GitHub Integrations.
BulkImportFindRepositoriesByOrganization
Tracks GET requests to the /organizations/:orgName/repositories endpoint, which returns the list of repositories for the specified organization (accessible from any of the configured GitHub Integrations).
BulkImportFindAllRepositories
Tracks GET requests to the /repositories endpoint, which returns the list of repositories accessible from all configured GitHub Integrations.
BulkImportFindAllImports
Tracks GET requests to the /imports endpoint, which returns the list of existing import jobs along with their statuses.
BulkImportCreateImportJobs
Tracks POST requests to the /imports endpoint, which allows to submit requests to bulk-import one or many repositories into the Developer Hub catalog, by eventually creating import pull requests in the target repositories.
BulkImportFindImportStatusByRepo
Tracks GET requests to the /import/by-repo endpoint, which fetches details about the import job for the specified repository.
BulkImportDeleteImportByRepo
Tracks DELETE requests to the /import/by-repo endpoint, which deletes any existing import job for the specified repository, by closing any open import pull request that could have been created.

Example bulk import audit logs

{
  "actor": {
    "actorId": "user:default/myuser",
    "hostname": "localhost",
    "ip": "::1",
    "userAgent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36"
  },
  "eventName": "BulkImportFindAllOrganizations",
  "isAuditLog": true,
  "level": "info",
  "message": "'get /organizations' endpoint hit by user:default/myuser",
  "meta": {},
  "plugin": "bulk-import",
  "request": {
    "body": {},
    "method": "GET",
    "params": {},
    "query": {
      "pagePerIntegration": "1",
      "sizePerIntegration": "5"
    },
    "url": "/api/bulk-import/organizations?pagePerIntegration=1&sizePerIntegration=5"
  },
  "response": {
    "status": 200
  },
  "service": "backstage",
  "stage": "completion",
  "status": "succeeded",
  "timestamp": "2024-08-26 16:41:02"
}
Copy to Clipboard Toggle word wrap

As an administrator, you can use the Bulk Import plugin to run a Scaffolder template task with specified parameters, which you must define within the template.

The Bulk Import plugin analyzes Git repository information and provides the following parameters for the Scaffolder template task:

repoUrl

Normalized repository URL in the following format: 

  ${gitProviderHost}?owner=${owner}&repo=${repository-name}
Copy to Clipboard Toggle word wrap
name
The repository name.
organization
The repository owner, which can be a user nickname or organization name.
branchName
The proposed repository branch. By default, the proposed repository branch is bulk-import-catalog-entity.
targetBranchName
The default branch of the Git repository.
gitProviderHost
The Git provider host parsed from the repository URL. You can use this parameter to write Git-provider-agnostic templates.

Example of a Scaffolder template: 

parameters:
  - title: Repository details
    required:
      - repoUrl
      - branchName
      - targetBranchName
      - name
      - organization
    properties:
      repoUrl:
        type: string
        title: Repository URL (Backstage format)
        description: github.com?owner=Org&repo=repoName
      organization:
        type: string
        title: Owner of the repository
      name:
        type: string
        title: Name of the repository
      branchName:
        type: string
        title: Branch to add the catalog entity to
      targetBranchName:
        type: string
        title: Branch to target the PR/MR to
      gitProviderHost:
        type: string
        title: Git provider host
Copy to Clipboard Toggle word wrap

As an administrator, you can create a custom Scaffolder template in line with the repository conventions of your organization and add the template into the Red Hat Developer Hub catalog for use by the Bulk Import plugin on multiple selected repositories.

You can define various custom tasks, including, but not limited to the following:

  • Importing existing catalog entities from a repository
  • Creating pull requests for cleanup
  • Calling webhooks for external system integration

Prerequisites

  • You created a custom Scaffolder template for the Bulk Import plugin.

  • You have run your RHDH instance with the following environment variable enabled to allow the use of the Scaffolder functionality: 

    export NODE_OPTIONS=--no-node-snapshot
    Copy to Clipboard Toggle word wrap

Procedure

  • Configure your app-config.yaml configuration to instruct the Bulk Import plugin to use your custom template as shown in the following example: 

    bulkImport:
  importTemplate: <your_template_entity_reference_or_template_name>
  importAPI: `open-pull-requests` | `scaffolder`;
    Copy to Clipboard Toggle word wrap

    where:

    importTemplate:
    Enter your Scaffolder template entity reference.
    importAPI
    Set the API to 'scaffolder' to trigger the defined workflow for high-fidelity automation. This field defines the import workflow and currently supports two following options:
    open-pull-requests
    This is the default import workflow, which includes the logic for creating pull requests for every selected repository.
    scaffolder

    This workflow uses an import scenario defined in the Scaffolder template to create import jobs. Select this option to use the custom import scenario defined in your Scaffolder template.

    Optional: You can direct the Bulk Import plugin to hand off the entire list of selected repositories to a custom Orchestrator workflow.

    Important

    The Scaffolder template must be generic and not specific to a single repository if you want your custom Scaffolder template to run successfully for every repository in the bulk list.

Verification

  • The Bulk Import plugin runs the custom Scaffolder template for the list of repositories using the /task-imports API endpoint.

8.7. Data handoff and custom workflow design
Copy link

When you configure the Bulk Import plugin by setting the importAPI field to scaffolder, the Bulk Import Backend passes all necessary context directly to the Scaffolder API.

As an administrator, you can define the Scaffolder template workflow and structure the workflow to do the following:

Define template parameters to consume input
Structure the Scaffolder template to receive the repository data as template parameters for the current workflow run. The template must be generic, and not specific to a single repository, so that it can successfully run for every repository in the bulk list.
Automate processing for each repository
Implement the custom logic needed for a single repository within the template. The Orchestrator iterates through the repository list, launching the template once for each repository and passes only the data for that single repository to the template run. This allows you to automate tasks such as creating the catalog-info.yaml, running compliance checks, or registering the entity with the catalog.
Back to top
Red Hat logoGithubredditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Theme

© 2025 Red Hat