Migrating 3scale

1. Depending on the database used with 3scale, set ${SYSTEM_DB} with one of the following values:

   • If the database is MySQL, SYSTEM_DB=system-mysql.
   • If the database is PostgreSQL, SYSTEM_DB=system-postgresql.

2. Create a back-up file with the existing DeploymentConfigs:

   $ THREESCALE_DC_NAMES="apicast-production apicast-staging backend-cron backend-listener backend-redis backend-worker system-app system-memcache ${SYSTEM_DB} system-redis system-sidekiq system-sphinx zync zync-database zync-que"
   for component in ${THREESCALE_DC_NAMES}; do oc get --export -o yaml dc ${component} > ${component}_dc.yml ; done

   Copy to Clipboard Toggle word wrap

3. Backup all existing OpenShift resources in the project that are exported through the export all command:

   $ oc get -o yaml --export all > threescale-project-elements.yaml

   Copy to Clipboard Toggle word wrap

4. Create a back-up file with the additional elements that are not exported with the export all command:

   $ for object in rolebindings serviceaccounts secrets imagestreamtags cm rolebindingrestrictions limitranges resourcequotas pvc templates cronjobs statefulsets hpa deployments replicasets poddisruptionbudget endpoints
   do
     oc get -o yaml --export $object > $object.yaml
   done

   Copy to Clipboard Toggle word wrap
5. Verify that all generated files are not empty and that all of them have the expected content.

1. Get the current MASTER_ACCESS_TOKEN value:

   $ MASTER_ACCESS_TOKEN=$(oc get secret system-seed -o json | jq -r .data.MASTER_ACCESS_TOKEN | base64 -d)

   Copy to Clipboard Toggle word wrap

2. Verify that MASTER_ACCESS_TOKEN is not empty and has the existing value:

   $ echo ${MASTER_ACCESS_TOKEN}

   Copy to Clipboard Toggle word wrap

3. Update the pre-hook pod command from system-app DeploymentConfig to the new command needed for this release:

   $ oc patch dc/system-app -p "{\"spec\":{\"strategy\":{\"rollingParams\":{\"pre\":{\"execNewPod\":{\"command\":[\"bash\",\"-c\",\"bundle exec rake boot openshift:deploy MASTER_ACCESS_TOKEN=\\\"${MASTER_ACCESS_TOKEN}\\\" && bundle exec rake services:create_backend_apis services:update_metric_owners proxy:update_proxy_rule_owners\"]}}}}}}"

   Copy to Clipboard Toggle word wrap

4. Verify that pre-hook pod command has changed to the new value:

   $ oc get dc system-app -o json | jq .spec.strategy.rollingParams.pre.execNewPod.command

   Copy to Clipboard Toggle word wrap

   • The result of the previous command should be:

     [
       "bash",
       "-c",
       "bundle exec rake boot openshift:deploy MASTER_ACCESS_TOKEN=\"<your-master-access-token>\" && bundle exec rake services:create_backend_apis services:update_metric_owners proxy:update_proxy_rule_owners"
     ]

     Copy to Clipboard Toggle word wrap

Procedure

1. Given that Zync does not manage the routes of external gateways, you can modify the deployment option of each service not managed by Zync, by following the steps under one of the proposed alternatives:

   • In 3scale:

     1. Go to the Integration page, and click edit integration settings.
     2. Choose the correct deployment option, and save your changes if any.

   • Using the API:

     1. Update the service with a service identifier (ID) with an access token (ACCESS_TOKEN) and the tenant endpoint (TENANT_URL):

        $ curl -XPUT "${TENANT_URL}/admin/api/services/${ID}.json" -d deployment_option=self_managed -d access_token="${ACCESS_TOKEN}"

        Copy to Clipboard Toggle word wrap

        Alternatively, you can use the command below if you are using APIcast hosted:

        $ curl -XPUT "${TENANT_URL}/admin/api/services/${ID}.json" -d deployment_option=hosted -d access_token="${ACCESS_TOKEN}"

        Copy to Clipboard Toggle word wrap

     2. For each service of each tenant, modify the deployment_option field via 3scale or the API:

        • These are the cases where you can set deployment_option to self_managed:

          • APIcast is linked to a custom route in OpenShift.
          • APIcast is hosted outside of OpenShift.
          • APICAST_PATH_ROUTING is set to true.
        • In other cases, set deployment_option to hosted.

2. Among the potentially existing routes, some default routes were automatically created in 2.5 by 3scale. Start by removing them:

   $ oc delete route system-master
   $ oc delete route system-provider-admin
   $ oc delete route system-developer
   $ oc delete route api-apicast-production
   $ oc delete route api-apicast-staging

   Copy to Clipboard Toggle word wrap

   • In case you deployed 3scale 2.5 with WILDCARD_POLICY=Subdomain you must remove the wildcard route with:

     $ oc delete route apicast-wildcard-router

     Copy to Clipboard Toggle word wrap
   • Otherwise, if you deployed 3scale 2.5 without WILDCARD_POLICY=Subdomain, you must remove the routes you manually created for the 3scale tenants and services, to avoid having duplications of the routes that Zync will create.

1. Force the resync of all 3scale services and tenants OpenShift routes with Zync:

   $ SYSTEM_SIDEKIQ_POD=$(oc get pods | grep sidekiq | awk '{print $1}')

   Copy to Clipboard Toggle word wrap

2. Check that SYSTEM_SIDEKIQ_POD environment variable result is not empty:

   $ echo ${SYSTEM_SIDEKIQ_POD}

   Copy to Clipboard Toggle word wrap

3. Finally, perform the resynchronization:

   $ oc exec -it ${SYSTEM_SIDEKIQ_POD} -- bash -c 'bundle exec rake zync:resync:domains'

   Copy to Clipboard Toggle word wrap

   You will see output of this style with information about notifications to system:

   $ No valid API key has been set, notifications will not be sent
   ActiveMerchant MODE set to 'production'
   [Core] Using http://backend-listener:3000/internal/ as URL
   OpenIdAuthentication.store is nil. Using in-memory store.
   [EventBroker] notifying subscribers of Domains::ProviderDomainsChangedEvent 59a554f6-7b3f-4246-9c36-24da988ca800
   [EventBroker] notifying subscribers of ZyncEvent caa8e941-b734-4192-acb0-0b12cbaab9ca
   Enqueued ZyncWorker#d92db46bdba7a299f3e88f14 with args: ["caa8e941-b734-4192-acb0-0b12cbaab9ca", {:type=>"Provider", :id=>1, :parent_event_id=>"59a554f6-7b3f-4246-9c36-24da988ca800", :parent_event_type=>"Domains::ProviderDomainsChangedEvent", :tenant_id=>1}]
   [EventBroker] notifying subscribers of Domains::ProviderDomainsChangedEvent 9010a199-2af1-4023-9b8d-297bd618096f
   …

   Copy to Clipboard Toggle word wrap

   New routes are created for all the existing tenants and services, after forcing Zync to reevaluate them. Route creation might take some minutes depending on the number of services and tenants.

   By the end of the process, you will see created:

   • One Master Admin Portal route.

     For every 3scale tenant two routes are created:

   • Tenant’s Admin Portal route.

   • Tenant’s Developer Portal route.

     For every 3scale service two routes are created:

   • APIcast staging Route corresponding to the service.
   • APIcast production Route corresponding to the service.

4. Verify that all the expected routes explained above have been created for all your existing services and tenants. You can see all the routes by running:

   $ oc get routes

   Copy to Clipboard Toggle word wrap

   The host/port field shown as the output of the previous command must be the URL of the routes.

   • In case you deployed 3scale 2.5 with the WILDCARD_POLICY set to Subdomain, all of the new routes must have the same base WildcardDomain as the old OpenShift wildcard Route.
   • Otherwise, in case you deployed 3scale 2.5 without WILDCARD_POLICY=Subdomain the new routes must have the same host as the old routes that you have removed, including the ones that were automatically created by 3scale in the 2.5 release.
5. Finally, in case you were using custom SSL certificates for the old wildcard route, or the old manually created routes, install them into the new ones created by Zync. You can do so by editing the routes via the OpenShift web panel and adding the certificate/s into them.

6. Verify that Services and Tenants that existed before this migration are still resolvable using the new routes. To do so perform the following tests:

   1. Resolve the route of an existing APIcast production URL associated to a 3scale service that already existed before this migration.
   2. Resolve the route of an existing APIcast staging URL associated to a 3scale service that already existed before this migration.
   3. Resolve the route of an existing Tenant that already existed before this migration.

7. When verifying that the new Zync functionality is working, confirm that new routes are generated when creating new tenants and services. To do so perform the following tests:

   1. Create a new tenant from the ‘master’ panel and verify that after some seconds the new Routes associated to it appear in OpenShift.
   2. Create a new Service in one of your existing tenants and verify that after some seconds the new Routes associated to it appear in OpenShift.

8. Remove the apicast-wildcard-router service:

   $ oc delete service apicast-wildcard-router

   Copy to Clipboard Toggle word wrap

9. Remove the deprecated WildcardRouter subsystem:

   $ oc delete ImageStream amp-wildcard-router
   $ oc delete DeploymentConfig apicast-wildcard-router

   Copy to Clipboard Toggle word wrap

   After you have performed all the listed steps, 3scale upgrade from 2.6 to 2.7 in a template-based deployment is now complete.

Procedure

1. Edit the APIManager object by adding the following annotations:

   Note

   The APIManager object is your 3scale installation that you intend to upgrade.

   apps.3scale.net/apimanager-threescale-version: "2.6"
   apps.3scale.net/threescale-operator-version: "0.3.0"

   Copy to Clipboard Toggle word wrap

2. Get all existing APIManagers in an existing OpenShift project:

   oc get apimanagers

   Copy to Clipboard Toggle word wrap

3. Edit the APIManager:

   oc annotate apimanager <apimanager-name>

   Copy to Clipboard Toggle word wrap
4. Add the previous tags in the annotations section when editing the object.
5. If the annotations section is not available, create this section.
6. Save the changes to finish editing.

Procedure

1. Open the OpenShift user interface.
2. Select the project where the APIManager, that is, your operator-based 3scale installation and 3scale operator are deployed.

3. The menu structure depends on the OpenShift version you are using:

   • For OCP 4.1, click Catalog > Operator Management.
   • For OCP 4.2, click Operators > Installed Operators
4. Select the 3scale Operator Subscription.

5. Edit the channel of that subscription, select the threescale-2.7 channel and save the changes.

   • This should start the upgrade process.
   • Wait until the upgrade procedure finishes for the APIManager.

6. Query the pods status on the project:

   oc get pods

   Copy to Clipboard Toggle word wrap
   • Wait until all the new versions are running and ready without errors.

   • They might have errors temporarily during the upgrade process.

     Note

     Times can vary from 5-10 minutes approximately. Be sure to keep checking the state of the pods until all of them are running, ready, and without errors.

7. To confirm that the upgrade process has been successful, log in to the 3scale Admin Portal and check that it works as expected.

   After you have performed all the listed steps, the 3scale upgrade via the 3scale operator from 2.6 to 2.7 is now complete.