Chapter 2. Upgrading 3scale 2.6 to 2.7 using templates

1. Depending on the database used with 3scale, set ${SYSTEM_DB} with one of the following values:

   • If the database is MySQL, SYSTEM_DB=system-mysql.
   • If the database is PostgreSQL, SYSTEM_DB=system-postgresql.

2. Create a back-up file with the existing DeploymentConfigs:

   $ THREESCALE_DC_NAMES="apicast-production apicast-staging backend-cron backend-listener backend-redis backend-worker system-app system-memcache ${SYSTEM_DB} system-redis system-sidekiq system-sphinx zync zync-database zync-que"
   for component in ${THREESCALE_DC_NAMES}; do oc get --export -o yaml dc ${component} > ${component}_dc.yml ; done

3. Backup all existing OpenShift resources in the project that are exported through the export all command:

   $ oc get -o yaml --export all > threescale-project-elements.yaml

4. Create a back-up file with the additional elements that are not exported with the export all command:

   $ for object in rolebindings serviceaccounts secrets imagestreamtags cm rolebindingrestrictions limitranges resourcequotas pvc templates cronjobs statefulsets hpa deployments replicasets poddisruptionbudget endpoints
   do
     oc get -o yaml --export $object > $object.yaml
   done

5. Verify that all generated files are not empty and that all of them have the expected content.

1. Get the current MASTER_ACCESS_TOKEN value:

   $ MASTER_ACCESS_TOKEN=$(oc get secret system-seed -o json | jq -r .data.MASTER_ACCESS_TOKEN | base64 -d)

2. Verify that MASTER_ACCESS_TOKEN is not empty and has the existing value:

   $ echo ${MASTER_ACCESS_TOKEN}

3. Update the pre-hook pod command from system-app DeploymentConfig to the new command needed for this release:

   $ oc patch dc/system-app -p "{\"spec\":{\"strategy\":{\"rollingParams\":{\"pre\":{\"execNewPod\":{\"command\":[\"bash\",\"-c\",\"bundle exec rake boot openshift:deploy MASTER_ACCESS_TOKEN=\\\"${MASTER_ACCESS_TOKEN}\\\" && bundle exec rake services:create_backend_apis services:update_metric_owners proxy:update_proxy_rule_owners\"]}}}}}}"

4. Verify that pre-hook pod command has changed to the new value:

   $ oc get dc system-app -o json | jq .spec.strategy.rollingParams.pre.execNewPod.command

   • The result of the previous command should be:

     [
       "bash",
       "-c",
       "bundle exec rake boot openshift:deploy MASTER_ACCESS_TOKEN=\"<your-master-access-token>\" && bundle exec rake services:create_backend_apis services:update_metric_owners proxy:update_proxy_rule_owners"
     ]

Procedure

1. Given that Zync does not manage the routes of external gateways, you can modify the deployment option of each service not managed by Zync, by following the steps under one of the proposed alternatives:

   • In 3scale:

     1. Go to the Integration page, and click edit integration settings.
     2. Choose the correct deployment option, and save your changes if any.

   • Using the API:

     1. Update the service with a service identifier (ID) with an access token (ACCESS_TOKEN) and the tenant endpoint (TENANT_URL):

        $ curl -XPUT "${TENANT_URL}/admin/api/services/${ID}.json" -d deployment_option=self_managed -d access_token="${ACCESS_TOKEN}"

        Alternatively, you can use the command below if you are using APIcast hosted:

        $ curl -XPUT "${TENANT_URL}/admin/api/services/${ID}.json" -d deployment_option=hosted -d access_token="${ACCESS_TOKEN}"

     2. For each service of each tenant, modify the deployment_option field via 3scale or the API:

        • These are the cases where you can set deployment_option to self_managed:

          • APIcast is linked to a custom route in OpenShift.
          • APIcast is hosted outside of OpenShift.
          • APICAST_PATH_ROUTING is set to true.
        • In other cases, set deployment_option to hosted.

2. Among the potentially existing routes, some default routes were automatically created in 2.5 by 3scale. Start by removing them:

   $ oc delete route system-master
   $ oc delete route system-provider-admin
   $ oc delete route system-developer
   $ oc delete route api-apicast-production
   $ oc delete route api-apicast-staging

   • In case you deployed 3scale 2.5 with WILDCARD_POLICY=Subdomain you must remove the wildcard route with:

     $ oc delete route apicast-wildcard-router

   • Otherwise, if you deployed 3scale 2.5 without WILDCARD_POLICY=Subdomain, you must remove the routes you manually created for the 3scale tenants and services, to avoid having duplications of the routes that Zync will create.

1. Force the resync of all 3scale services and tenants OpenShift routes with Zync:

   $ SYSTEM_SIDEKIQ_POD=$(oc get pods | grep sidekiq | awk '{print $1}')

2. Check that SYSTEM_SIDEKIQ_POD environment variable result is not empty:

   $ echo ${SYSTEM_SIDEKIQ_POD}

3. Finally, perform the resynchronization:

   $ oc exec -it ${SYSTEM_SIDEKIQ_POD} -- bash -c 'bundle exec rake zync:resync:domains'

   You will see output of this style with information about notifications to system:

   $ No valid API key has been set, notifications will not be sent
   ActiveMerchant MODE set to 'production'
   [Core] Using http://backend-listener:3000/internal/ as URL
   OpenIdAuthentication.store is nil. Using in-memory store.
   [EventBroker] notifying subscribers of Domains::ProviderDomainsChangedEvent 59a554f6-7b3f-4246-9c36-24da988ca800
   [EventBroker] notifying subscribers of ZyncEvent caa8e941-b734-4192-acb0-0b12cbaab9ca
   Enqueued ZyncWorker#d92db46bdba7a299f3e88f14 with args: ["caa8e941-b734-4192-acb0-0b12cbaab9ca", {:type=>"Provider", :id=>1, :parent_event_id=>"59a554f6-7b3f-4246-9c36-24da988ca800", :parent_event_type=>"Domains::ProviderDomainsChangedEvent", :tenant_id=>1}]
   [EventBroker] notifying subscribers of Domains::ProviderDomainsChangedEvent 9010a199-2af1-4023-9b8d-297bd618096f
   …

   New routes are created for all the existing tenants and services, after forcing Zync to reevaluate them. Route creation might take some minutes depending on the number of services and tenants.

   By the end of the process, you will see created:

   • One Master Admin Portal route.

     For every 3scale tenant two routes are created:

   • Tenant’s Admin Portal route.

   • Tenant’s Developer Portal route.

     For every 3scale service two routes are created:

   • APIcast staging Route corresponding to the service.
   • APIcast production Route corresponding to the service.

4. Verify that all the expected routes explained above have been created for all your existing services and tenants. You can see all the routes by running:

   $ oc get routes

   The host/port field shown as the output of the previous command must be the URL of the routes.

   • In case you deployed 3scale 2.5 with the WILDCARD_POLICY set to Subdomain, all of the new routes must have the same base WildcardDomain as the old OpenShift wildcard Route.
   • Otherwise, in case you deployed 3scale 2.5 without WILDCARD_POLICY=Subdomain the new routes must have the same host as the old routes that you have removed, including the ones that were automatically created by 3scale in the 2.5 release.
5. Finally, in case you were using custom SSL certificates for the old wildcard route, or the old manually created routes, install them into the new ones created by Zync. You can do so by editing the routes via the OpenShift web panel and adding the certificate/s into them.

6. Verify that Services and Tenants that existed before this migration are still resolvable using the new routes. To do so perform the following tests:

   1. Resolve the route of an existing APIcast production URL associated to a 3scale service that already existed before this migration.
   2. Resolve the route of an existing APIcast staging URL associated to a 3scale service that already existed before this migration.
   3. Resolve the route of an existing Tenant that already existed before this migration.

7. When verifying that the new Zync functionality is working, confirm that new routes are generated when creating new tenants and services. To do so perform the following tests:

   1. Create a new tenant from the ‘master’ panel and verify that after some seconds the new Routes associated to it appear in OpenShift.
   2. Create a new Service in one of your existing tenants and verify that after some seconds the new Routes associated to it appear in OpenShift.

8. Remove the apicast-wildcard-router service:

   $ oc delete service apicast-wildcard-router

9. Remove the deprecated WildcardRouter subsystem:

   $ oc delete ImageStream amp-wildcard-router
   $ oc delete DeploymentConfig apicast-wildcard-router

   After you have performed all the listed steps, 3scale upgrade from 2.6 to 2.7 in a template-based deployment is now complete.