Chapter 2. Installing 3scale on OpenShift

1. Create an Identity and Access Management (IAM) policy with the following minimum permissions:

   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Effect": "Allow",
               "Action": "s3:ListAllMyBuckets",
               "Resource": "arn:aws:s3:::*"
           },
           {
               "Effect": "Allow",
               "Action": "s3:*",
               "Resource": [
                   "arn:aws:s3:::targetBucketName",
                   "arn:aws:s3:::targetBucketName/*"
               ]
           }
       ]
   }

2. Create a CORS configuration with the following rules:

   <?xml version="1.0" encoding="UTF-8"?>
   <CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
   <CORSRule>
       <AllowedOrigin>https://*</AllowedOrigin>
       <AllowedMethod>GET</AllowedMethod>
   </CORSRule>
   </CORSConfiguration>

Procedure

1. Click Operators > Installed Operators.

   1. From the list of Installed Operators, click 3scale Operator.
2. Click the API Manager tab.
3. Click Create APIManager.

4. Clear the sample content and add the following YAML definitions to the editor, then click Create.

   • Before 3scale 2.8, you could configure the automatic addition of replicas by setting the highAvailability field to true. From 3scale 2.8, the addition of replicas is controlled through the replicas field in the APIManager CR as shown in the following example.

     Note

     The wildcardDomain parameter can be any desired name you wish to give that resolves to an IP address, which is a valid DNS domain.

   • APIManager CR with minimum requirements:

     apiVersion: apps.3scale.net/v1alpha1
     kind: APIManager
     metadata:
       name: apimanager-sample
     spec:
       wildcardDomain: example.com

   • APIManager CR with replicas configured:

     apiVersion: apps.3scale.net/v1alpha1
     kind: APIManager
     metadata:
       name: apimanager-sample
     spec:
       system:
         appSpec:
           replicas: 1
         sidekiqSpec:
           replicas: 1
       zync:
         appSpec:
           replicas: 1
         queSpec:
           replicas: 1
       backend:
         cronSpec:
           replicas: 1
         listenerSpec:
           replicas: 1
         workerSpec:
           replicas: 1
       apicast:
         productionSpec:
           replicas: 1
         stagingSpec:
           replicas: 1
       wildcardDomain: example.com

1. Run the following commands to get the Admin Portal credentials:

   oc get secret system-seed -o json | jq -r .data.ADMIN_USER | base64 -d
   oc get secret system-seed -o json | jq -r .data.ADMIN_PASSWORD | base64 -d

   1. Log in as the Admin Portal administrator to verify these credentials are working.

2. Run the following commands to get the Master Admin Portal credentials:

   oc get secret system-seed -o json | jq -r .data.MASTER_USER | base64 -d
   oc get secret system-seed -o json | jq -r .data.MASTER_PASSWORD | base64 -d

   1. Log in as the Master Admin Portal administrator to verify these credentials are working.

Procedure

1. Click Operators > Installed Operators.
2. Click 3scale API Management from the list of Installed Operators.
3. Click the Subscription tab. Under the Subscription Details heading you will see the subheading Approval.
4. Click the link below Approval. The link is set to Automatic by default. A modal with the heading, Change Update Approval Strategy will pop up.
5. Choose the option of your preference: Automatic (default) or Manual, and then click Save.

Procedure

1. Write Lua code that defines the custom environment that you want to inject. For example, the following env1.lua file shows a custom logging policy that the 3scale operator loads for all services.

   local cjson = require('cjson')
   local PolicyChain = require('apicast.policy_chain')
   local policy_chain = context.policy_chain
   
   local logging_policy_config = cjson.decode([[
   {
     "enable_access_logs": false,
     "custom_logging": "\"{{request}}\" to service {{service.id}} and {{service.name}}"
   }
   ]])
   
   policy_chain:insert( PolicyChain.load_policy('logging', 'builtin', logging_policy_config), 1)
   
   return {
     policy_chain = policy_chain,
     port = { metrics = 9421 },
   }

2. Create a secret from the Lua file that defines the custom environment. For example:

   oc create secret generic custom-env-1 --from-file=./env1.lua

   A secret can contain multiple custom environments. Specify the `–from-file option for each file that defines a custom environment. The operator loads each custom environment.

3. Define an APIManager custom resource that references the secret you just created. The following example shows only content relative to referencing the secret that defines the custom environment.

   apiVersion: apps.3scale.net/v1alpha1
   kind: APIManager
   metadata:
     name: apimanager-apicast-custom-environment
   spec:
     wildcardDomain: <desired-domain>
     apicast:
       productionSpec:
         customEnvironments:
           - secretRef:
               name: custom-env-1
       stagingSpec:
         customEnvironments:
           - secretRef:
               name: custom-env-1

   An APIManager custom resource can reference multiple secrets that define custom environments. The operator loads each custom environment.

4. Create the APIManager custom resource that adds the custom environment. For example:

   oc apply -f apimanager.yaml

Procedure

1. Create a secret from the files that define one custom policy. For example:

   oc create secret generic my-first-custom-policy-secret \
    --from-file=./apicast-policy.json \
    --from-file=./init.lua \
    --from-file=./my-first-custom-policy.lua

   If you have more than one custom policy, create a secret for each custom policy. A secret can contain only one custom policy.

2. Define an APIManager custom resource that references each secret that contains a custom policy. You can specify the same secret for APIcast staging and APIcast production. The following example shows only content relative to referencing secrets that contain custom policies.

   apiVersion: apps.3scale.net/v1alpha1
   kind: APIManager
   metadata:
     name: apimanager-apicast-custom-policy
   spec:
     apicast:
       stagingSpec:
         customPolicies:
           - name: my-first-custom-policy
             version: "0.1"
             secretRef:
               name: my-first-custom-policy-secret
           - name: my-second-custom-policy
             version: "0.1"
             secretRef:
               name: my-second-custom-policy-secret
       productionSpec:
         customPolicies:
           - name: my-first-custom-policy
             version: "0.1"
             secretRef:
               name: my-first-custom-policy-secret
           - name: my-second-custom-policy
             version: "0.1"
             secretRef:
               name: my-second-custom-policy-secret

   An APIManager custom resource can reference multiple secrets that define different custom policies. The operator loads each custom policy.

3. Create the APIManager custom resource that references the secrets that contain the custom policies. For example:

   oc apply -f apimanager.yaml

Procedure

1. Define a secret that contains your OpenTracing configuration details in stringData.config. This is the only valid value for the attribute that contains your OpenTracing configuration details. Any other specification prevents APIcast from receiving your OpenTracing configuration details. The folowing example shows a valid secret definition:

   apiVersion: v1
   kind: Secret
   metadata:
     name: myjaeger
   stringData:
     config: |-
         {
         "service_name": "apicast",
         "disabled": false,
         "sampler": {
           "type": "const",
           "param": 1
         },
         "reporter": {
           "queueSize": 100,
           "bufferFlushInterval": 10,
           "logSpans": false,
           "localAgentHostPort": "jaeger-all-in-one-inmemory-agent:6831"
         },
         "headers": {
           "jaegerDebugHeader": "debug-id",
           "jaegerBaggageHeader": "baggage",
           "TraceContextHeaderName": "uber-trace-id",
           "traceBaggageHeaderPrefix": "testctx-"
         },
         "baggage_restrictions": {
             "denyBaggageOnInitializationFailure": false,
             "hostPort": "127.0.0.1:5778",
             "refreshInterval": 60
         }
         }
   type: Opaque

2. Create the secret. For example, if you saved the previous secret definition in the myjaeger.yaml file, you would run the following command:

   oc create secret generic myjaeger --from-file myjaeger.yaml

3. Define an APIManager custom resource that specifies OpenTracing attributes. In the CR definition, set the openTracing.tracingConfigSecretRef.name attribute to the name of the secret that contains your OpenTracing configuration details. The following example shows only content relative to configuring OpenTracing.

   apiVersion: apps.3scale.net/v1alpha1
   kind: APIManager
   metadata:
     name: apimanager1
   spec:
     apicast:
       stagingSpec:
         ...
         openTracing:
           enabled: true
           tracingLibrary: jaeger
           tracingConfigSecretRef:
             name: myjaeger
       productionSpec:
         ...
           openTracing:
             enabled: true
             tracingLibrary: jaeger
             tracingConfigSecretRef:
               name: myjaeger

4. Create the APIManager custom resource that configures OpenTracing. For example, if you saved the APIManager custom resource in the apimanager1.yaml file, you would run the following command:

   oc apply -f apimanager1.yaml

Procedure

1. Create a secret from your valid certificate, for example:

   oc create secret tls mycertsecret --cert=server.crt --key=server.key

   The configuration exposes secret references in the APIManager CRD. You create the secret and then reference the name of the secret in the APIManager custom resource as follows:

   • Production: The APIManager CR exposes the certificate in the .spec.apicast.productionSpec.httpsCertificateSecretRef field.

   • Staging: The APIManager CR exposes the certificate in the .spec.apicast.stagingSpec.httpsCertificateSecretRef field.

     Optionally, you can configure the following:

   • httpsPort indicates which port APIcast should start listening on for HTTPS connections. If this clashes with the HTTP port APIcast uses this port for HTTPS only.

   • httpsVerifyDepth defines the maximum length of the client certificate chain.

     Note

     Provide a valid certificate and reference from the APImanager CR. If the configuration can access httpsPort but not httpsCertificateSecretRef, APIcast uses an embedded self-signed certificate. This is not recommended.

2. Click Operators > Installed Operators.
3. From the list of Installed Operators, click 3scale Operator.
4. Click the API Manager tab.
5. Click Create APIManager.

6. Add the following YAML definitions to the editor.

   1. If enabling for production, configure the following YAML defintions:

      spec:
        apicast:
          productionSpec:
            httpsPort: 8443
            httpsVerifyDepth: 1
            httpsCertificateSecretRef:
              name: mycertsecret

   2. If enabling for staging, configure the following YAML defintions:

      spec:
        apicast:
          stagingSpec:
            httpsPort: 8443
            httpsVerifyDepth: 1
            httpsCertificateSecretRef:
              name: mycertsecret

7. Click Create.

Procedure

1. Create a new database.

2. Apply the following settings:

   ALTER SYSTEM SET max_string_size=extended SCOPE=SPFILE;

3. Create a regular non-system user before the Oracle Database installation with 3scale.

   1. The password for a regular Oracle Database non-system user must be unique and not match the system password.

   2. The Oracle Database initialization script runs the following SQL command:

      ALTER USER {DB_USER} IDENTIFIED BY {DB_PASSWORD}

      • Make sure this SQL command executes correctly before starting the 3scale installation.
      • {DB_USER} and {DB_PASSWORD} are the username and password of the regular non-system user.
      • Your database configuration might prevent this command from completing successfully if the parameters PASSWORD_REUSE_TIME and PASSWORD_REUSE_MAX are set in a way that restricts reusing the same password.

      • The Oracle Database system user executes commands with system privileges.

        • Some are detailed in this GitHub repository.
        • The latest can be executed in the Oracle Database initializer when the tables are initialized in the database. There may be other commands executed not listed in these links.
      • The system user is also required for upgrades when there are any schema migrations to run, so other commands not included in the previous links may be executed.

Procedure

1. Download 3scale OpenShift templates from the GitHub repository and extract the archive:

   tar -xzf 3scale-amp-openshift-templates-3scale-2.12.0-GA.tar.gz

2. From the Instant Client Downloads page, download:

   • A client: It can be either basic-lite or basic.
   • The ODBC driver.

   • The SDK for Oracle Database 19c.

     • For 3scale, use Instant Client Downloads for Linux x86-64 (64-bit)
     • For ppc64le and 3scale, use Oracle Instant Client Downloads for Linux on Power Little Endian (64-bit)

3. Check the Red Hat 3scale API Management Supported Configurations for the following Oracle software components:

   • Oracle Instant Client Package: Basic or Basic Light
   • Oracle Instant Client Package: SDK

   • Oracle Instant Client Package: ODBC

     Table 2.3. Oracle 19c example packages for 3scale

     Oracle 19c package name	Compressed file name
     Basic	instantclient-basic-linux.x64-19.8.0.0.0dbru.zip
     Basic Light	instantclient-basiclite-linux.x64-19.8.0.0.0dbru.zip
     SDK	instantclient-sdk-linux.x64-19.8.0.0.0dbru.zip
     ODBC	instantclient-odbc-linux.x64-19.8.0.0.0dbru.zip

     Table 2.4. Oracle 19c example packages for ppc64le and 3scale

     Oracle 19c package name	Compressed file name
     Basic	instantclient-basic-linux.leppc64.c64-19.3.0.0.0dbru.zip
     Basic Light	instantclient-basiclite-linux.leppc64.c64-19.3.0.0.0dbru.zip
     SDK	instantclient-sdk-linux.leppc64.c64-19.3.0.0.0dbru.zip
     ODBC	instantclient-odbc-linux.leppc64.c64-19.3.0.0.0dbru.zip

   Note

   If the client packages versions downloaded and stored locally do not match with the ones 3scale expects, 3scale will automatically download and use the appropriate ones in the following steps.

4. Place your Oracle Database Instant Client Package files into the 3scale-amp-openshift-templates-3scale-2.12-GA/amp/system-oracle/oracle-client-files directory.

5. Login to your registry.redhat.io account using the credentials you created in Creating a Registry Service Account.

   $ docker login registry.redhat.io

6. Build the custom system Oracle-based image. The image tag must be a fixed image tag as in the following example:

   $ docker build . --tag myregistry.example.com/system-oracle:2.12.0-1

7. Push the system Oracle-based image to a container registry accessible by the OCP cluster. This container registry is where your 3scale solution is going to be installed:

   $ docker push myregistry.example.com/system-oracle:2.12.0-1

Procedure

1. Set up the Oracle Database URL connection string and Oracle Database system password by creating the system-database secret with the corresponding fields. See, External databases installation for the Oracle Database.

2. Install your 3scale solution by creating an APIManager custom resource. Follow the instructions in Deploying 3scale using the operator.

   • The APIManager custom resource must specify the .spec.system.image field set to the system’s Oracle-based image you previous built:

     apiVersion: apps.3scale.net/v1alpha1
     kind: APIManager
     metadata:
       name: example-apimanager
     spec:
       imagePullSecrets:
       - name: threescale-registry-auth
       - name: custom-registry-auth
       system:
         image: "myregistry.example.com/system-oracle:2.12.0-1"
       externalComponents:
         system:
           database: true

Procedure

1. Find the PVC containing the erroneous MySQL data with the oc get pvc command:

   # oc get pvc
   NAME                    STATUS    VOLUME    CAPACITY   ACCESSMODES   AGE
   backend-redis-storage   Bound     vol003    100Gi      RWO,RWX       4d
   mysql-storage           Bound     vol006    100Gi      RWO,RWX       4d
   system-redis-storage    Bound     vol008    100Gi      RWO,RWX       4d
   system-storage          Bound     vol004    100Gi      RWO,RWX       4d

2. Stop the deployment of the system-mysql pod by clicking cancel deployment in the OpenShift UI.
3. Delete everything under the MySQL path to clean the volume.
4. Start a new system-mysql deployment.

Procedure

1. Type the following command to verify the configuration of your container registry authentication:

   $ oc get secret

   • If your secret exists, you will see the following output in the terminal:

     threescale-registry-auth          kubernetes.io/dockerconfigjson        1         4m9s

   • However, if you do not see the output, you must do the following:
2. Use the credentials you previously set up while Creating a registry service account to create your secret.
3. Use the steps in Configuring registry authentication in OpenShift, replacing <your-registry-service-account-username> and <your-registry-service-account-password> in the oc create secret command provided.

4. Generate the threescale-registry-auth secret in the same namespace as the APIManager resource. You must run the following inside the <project-name>:

   oc project <project-name>
   oc create secret docker-registry threescale-registry-auth \
     --docker-server=registry.redhat.io \
     --docker-username="<your-registry-service-account-username>" \
     --docker-password="<your-registry-service-account-password>"
     --docker-email="<email-address>"

5. Delete and recreate the APIManager resource:

   $ oc delete -f apimanager.yaml
   apimanager.apps.3scale.net "example-apimanager" deleted
   
   $ oc create -f apimanager.yaml
   apimanager.apps.3scale.net/example-apimanager created

Verification

1. Type the following command to confirm that deployments have a status of Starting or Ready. The pods then begin to spawn:

   $ oc describe apimanager
   (...)
   Status:
     Deployments:
       Ready:
         apicast-staging
         system-memcache
         system-mysql
         system-redis
         zync
         zync-database
         zync-que
       Starting:
         apicast-production
         backend-cron
         backend-worker
         system-sidekiq
         system-sphinx
       Stopped:
         backend-listener
         backend-redis
         system-app

2. Type the following command to see the status of each pod:

   $ oc get pods
   NAME                               READY   STATUS             RESTARTS   AGE
   3scale-operator-66cc6d857b-sxhgm   1/1     Running            0          17h
   apicast-production-1-deploy        1/1     Running            0          17m
   apicast-production-1-pxkqm         0/1     Pending            0          17m
   apicast-staging-1-dbwcw            1/1     Running            0          17m
   apicast-staging-1-deploy           0/1     Completed          0          17m
   backend-cron-1-deploy              1/1     Running            0          17m

Procedure

1. The directories used for the persistent volumes MUST have the write permissions for the root group. Having read-write permissions for the root user is not enough as the MySQL service runs as a different user in the root group. Execute the following command as the root user:

   chmod -R g+w /path/for/pvs

2. Execute the following command to prevent SElinux from blocking access:

   chcon -Rt svirt_sandbox_file_t /path/for/pvs