Chapter 1. Installing 3scale API Management on OpenShift

Procedure

1. In the OpenShift Container Platform console, log in using an account with administrator privileges.
2. Click Operators > OperatorHub
3. In the Filter by keyword box, type 3scale operator to find Red Hat Integration - 3scale.
4. Click Red Hat Integration - 3scale. Information about the operator is displayed.
5. Read the information about the operator and click Install. The Install Operator page opens.
6. On the Install Operator page, select the desired channel to update in the Update channel section.

7. In the Installation mode section, select where to install the operator.

   1. All namespaces on the cluster (default) - The operator will be available in all namespaces on the cluster.
   2. A specific namespace on the cluster - The operator will only be available in the specific single namespace on the cluster that you have selected.
8. Click Install.
9. After the installation is complete, the system displays a confirmation message indicating that the operator is ready for use.

10. Verify that the 3scale operator ClusterServiceVersion (CSV) is correctly installed. Also check if it reports that the installation of the operator has been successful:

    • Click Operators > Installed Operators.
    • Click on the Red Hat Integration - 3scale operator.
    • In the Details tab, scroll down to the Conditions section, where the Succeeded condition should read InstallSucceeded under the Reason column.

Procedure

1. In the OpenShift Container Platform console, log in using an account with administrator privileges.
2. Click Operators > OperatorHub. The installed Operators page is displayed.
3. Enter 3scale into the Filter by name to find the operator and click on it.
4. On the Operator Details page, select Uninstall Operator from the Actions drop-down menu to remove it from a specific namespace.

5. An Uninstall Operator? dialog box is displayed, reminding you that:

   Removing the operator will not remove any of its custom resource definitions or managed resources. If your operator has deployed applications on the cluster or configured off-cluster resources, these will continue to run and need to be cleaned up manually.
   This action removes the operator as well as the Operator deployments and pods, if any. Any operands and resources managed by the operator, including CRDs and CRs, are not removed. The web console enables dashboards and navigation items for some operators. To remove these after uninstalling the operator, you might need to manually delete the operator CRDs.

   Copy to Clipboard Toggle word wrap
6. Select Uninstall. This operator stops running and no longer receives updates.
7. In the OpenShift Container Platform console click Operators > OperatorHub.
8. In the Filter by keyword box, type 3scale operator to find Red Hat Integration - 3scale.
9. Click Red Hat Integration - 3scale. Information about the operator is displayed.
10. Click Install. The Install Operator page opens.
11. On the Install Operator page, select the desired channel to update in the Update channel section.
12. In the Installation mode section, select All namespaces on the cluster (default). The operator will be available in all namespaces on the cluster.
13. Click Subscribe. The 3scale operator details page is displayed and you can see the Subscription Overview.
14. Confirm that the subscription Upgrade Status is displayed as Up to date.
15. Verify that the 3scale operator ClusterServiceVersion (CSV) is displayed.

Procedure

1. Click Operators > Installed Operators.

   1. From the list of Installed Operators, click Red Hat Integration - 3scale.
2. Click the API Manager tab.
3. Click Create APIManager.

4. Clear the sample content and add the following YAML definitions to the editor, then click Create.

   • Before 3scale 2.8, you could configure the automatic addition of replicas by setting the highAvailability field to true. From 3scale 2.8, the addition of replicas is controlled through the replicas field in the APIManager CR as shown in the following example.

     Note

     The value of the wildcardDomain parameter must be a valid domain name that resolves to the address of your OpenShift Container Platform (OCP) router. For example, apps.mycluster.example.com.

   • APIManager CR with minimum requirements:

     apiVersion: apps.3scale.net/v1alpha1
     kind: APIManager
     metadata:
       name: apimanager-sample
     spec:
       wildcardDomain: example.com

     Copy to Clipboard Toggle word wrap

   • APIManager CR with replicas configured:

     apiVersion: apps.3scale.net/v1alpha1
     kind: APIManager
     metadata:
       name: apimanager-sample
     spec:
       system:
         appSpec:
           replicas: 1
         sidekiqSpec:
           replicas: 1
       zync:
         appSpec:
           replicas: 1
         queSpec:
           replicas: 1
       backend:
         cronSpec:
           replicas: 1
         listenerSpec:
           replicas: 1
         workerSpec:
           replicas: 1
       apicast:
         productionSpec:
           replicas: 1
         stagingSpec:
           replicas: 1
       wildcardDomain: example.com

     Copy to Clipboard Toggle word wrap

1. Run the following commands to get the Admin Portal credentials:

   $ oc get secret system-seed -n <namespace> -o json | jq -r .data.ADMIN_USER | base64 -d
   $ oc get secret system-seed -n <namespace> -o json | jq -r .data.ADMIN_PASSWORD | base64 -d

   Copy to Clipboard Toggle word wrap
   1. Log in as the Admin Portal administrator to verify these credentials are working.

2. Run the following commands to get the Master Admin Portal credentials:

   $ oc get secret system-seed -n <namespace> -o json | jq -r .data.MASTER_USER | base64 -d
   $ oc get secret system-seed -n <namespace> -o json | jq -r .data.MASTER_PASSWORD | base64 -d

   Copy to Clipboard Toggle word wrap
   1. Log in as the Master Admin Portal administrator to verify these credentials are working.

Procedure

1. Write Lua code that defines the custom environment that you want to inject. For example, the following env1.lua file shows a custom logging policy that the 3scale operator loads for all services.

   local cjson = require('cjson')
   local PolicyChain = require('apicast.policy_chain')
   local policy_chain = context.policy_chain
   
   local logging_policy_config = cjson.decode([[
   {
     "enable_access_logs": false,
     "custom_logging": "\"{{request}}\" to service {{service.id}} and {{service.name}}"
   }
   ]])
   
   policy_chain:insert( PolicyChain.load_policy('logging', 'builtin', logging_policy_config), 1)
   
   return {
     policy_chain = policy_chain,
     port = { metrics = 9421 },
   }

   Copy to Clipboard Toggle word wrap

2. Create a secret from the Lua file that defines the custom environment. For example:

   $ oc create secret generic custom-env-1 --from-file=./env1.lua

   Copy to Clipboard Toggle word wrap

   A secret can contain multiple custom environments. Specify the –from-file option for each file that defines a custom environment. The operator loads each custom environment.

3. Define an APIManager custom resource (CR) that references the secret you just created. The following example shows only content relative to referencing the secret that defines the custom environment.

   apiVersion: apps.3scale.net/v1alpha1
   kind: APIManager
   metadata:
     name: apimanager-apicast-custom-environment
   spec:
     wildcardDomain: <desired-domain>
     apicast:
       productionSpec:
         customEnvironments:
           - secretRef:
               name: custom-env-1
       stagingSpec:
         customEnvironments:
           - secretRef:
               name: custom-env-1

   Copy to Clipboard Toggle word wrap

   An APIManager CR can reference multiple secrets that define custom environments. The operator loads each custom environment.

4. Create the APIManager CR that adds the custom environment. For example:

   $ oc apply -f apimanager.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Create a secret from the files that define one custom policy. For example:

   $ oc create secret generic my-first-custom-policy-secret \
    --from-file=./apicast-policy.json \
    --from-file=./init.lua \
    --from-file=./my-first-custom-policy.lua

   Copy to Clipboard Toggle word wrap

   If you have more than one custom policy, create a secret for each custom policy. A secret can contain only one custom policy.

2. Use the 3scale operator to monitor secret changes. Add the apimanager.apps.3scale.net/watched-by=apimanager label to begin the 3scale operator secret changes monitoring:

   $ oc label secret my-first-custom-policy-secret apimanager.apps.3scale.net/watched-by=apimanager

   Copy to Clipboard Toggle word wrap
   Note

   By default, changes to the secret are not tracked by the 3scale operator. With the label in place, the 3scale operator automatically updates the APIcast deployment whenever you make changes to the secret. This happens in both staging and production environments where the secret is in use. The 3scale operator will not take ownership of the secret in any way.

3. Define an APIManager CR that references each secret that contains a custom policy. You can specify the same secret for APIcast staging and APIcast production. The following example shows only content relative to referencing secrets that contain custom policies.

   apiVersion: apps.3scale.net/v1alpha1
   kind: APIManager
   metadata:
     name: apimanager-apicast-custom-policy
   spec:
     apicast:
       stagingSpec:
         customPolicies:
           - name: my-first-custom-policy
             version: "0.1"
             secretRef:
               name: my-first-custom-policy-secret
           - name: my-second-custom-policy
             version: "0.1"
             secretRef:
               name: my-second-custom-policy-secret
       productionSpec:
         customPolicies:
           - name: my-first-custom-policy
             version: "0.1"
             secretRef:
               name: my-first-custom-policy-secret
           - name: my-second-custom-policy
             version: "0.1"
             secretRef:
               name: my-second-custom-policy-secret

   Copy to Clipboard Toggle word wrap

   An APIManager CR can reference multiple secrets that define different custom policies. The operator loads each custom policy.

4. Create the APIManager CR that references the secrets that contain the custom policies. For example:

   $ oc apply -f apimanager.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Define a secret that contains your OpenTracing configuration details in stringData.config. This is the only valid value for the attribute that contains your OpenTracing configuration details. Any other specification prevents APIcast from receiving your OpenTracing configuration details. The following example shows a valid secret definition:

   apiVersion: v1
   kind: Secret
   metadata:
     name: myjaeger
   stringData:
     config: |-
         {
         "service_name": "apicast",
         "disabled": false,
         "sampler": {
           "type": "const",
           "param": 1
         },
         "reporter": {
           "queueSize": 100,
           "bufferFlushInterval": 10,
           "logSpans": false,
           "localAgentHostPort": "jaeger-all-in-one-inmemory-agent:6831"
         },
         "headers": {
           "jaegerDebugHeader": "debug-id",
           "jaegerBaggageHeader": "baggage",
           "TraceContextHeaderName": "uber-trace-id",
           "traceBaggageHeaderPrefix": "testctx-"
         },
         "baggage_restrictions": {
             "denyBaggageOnInitializationFailure": false,
             "hostPort": "127.0.0.1:5778",
             "refreshInterval": 60
         }
         }
   type: Opaque

   Copy to Clipboard Toggle word wrap

2. Create the secret. For example, if you saved the previous secret definition in the myjaeger.yaml file, you would run the following command:

   $ oc create -f myjaeger.yaml

   Copy to Clipboard Toggle word wrap

3. Define an APIManager custom resource (CR) that specifies OpenTracing attributes. In the CR definition, set the openTracing.tracingConfigSecretRef.name attribute to the name of the secret that contains your OpenTracing configuration details. The following example shows only content relative to configuring OpenTracing:

   apiVersion: apps.3scale.net/v1alpha1
   kind: APIManager
   metadata:
     name: apimanager1
   spec:
     apicast:
       stagingSpec:
         ...
         openTracing:
           enabled: true
           tracingLibrary: jaeger
           tracingConfigSecretRef:
             name: myjaeger
       productionSpec:
         ...
           openTracing:
             enabled: true
             tracingLibrary: jaeger
             tracingConfigSecretRef:
               name: myjaeger

   Copy to Clipboard Toggle word wrap

4. Create the APIManager CR that configures OpenTracing. For example, if you saved the APIManager CR in the apimanager1.yaml file, you would run the following command:

   $ oc apply -f apimanager1.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Create a secret from your valid certificate, for example:

   $ oc create secret tls mycertsecret --cert=server.crt --key=server.key

   Copy to Clipboard Toggle word wrap

   The configuration exposes secret references in the APIManager custom resource (CR). You create the secret and then reference the name of the secret in the APIManager CR as follows:

   • Production: The APIManager CR exposes the certificate in the .spec.apicast.productionSpec.httpsCertificateSecretRef field.

   • Staging: The APIManager CR exposes the certificate in the .spec.apicast.stagingSpec.httpsCertificateSecretRef field.

     Optionally, you can configure the following:

   • httpsPort indicates which port APIcast should start listening on for HTTPS connections. If this clashes with the HTTP port APIcast uses this port for HTTPS only.

   • httpsVerifyDepth defines the maximum length of the client certificate chain.

     Note

     Provide a valid certificate and reference from the APIManager CR. If the configuration can access httpsPort but not httpsCertificateSecretRef, APIcast uses an embedded self-signed certificate. This is not recommended.

2. Click Operators > Installed Operators.
3. From the list of Installed Operators, click 3scale Operator.
4. Click the API Manager tab.
5. Click Create APIManager.

6. Add the following YAML definitions to the editor.

   1. If enabling for production, configure the following YAML definitions:

      spec:
        apicast:
          productionSpec:
            httpsPort: 8443
            httpsVerifyDepth: 1
            httpsCertificateSecretRef:
              name: mycertsecret

      Copy to Clipboard Toggle word wrap

   2. If enabling for staging, configure the following YAML defintions:

      spec:
        apicast:
          stagingSpec:
            httpsPort: 8443
            httpsVerifyDepth: 1
            httpsCertificateSecretRef:
              name: mycertsecret

      Copy to Clipboard Toggle word wrap
7. Click Create.

Procedure

1. If you are not already logged in, log in to OpenShift and select the project where your 3scale On-premises instance is installed using the oc command:

   $ oc login -u <user> <url>
   $ oc project <3scale-project>

   Copy to Clipboard Toggle word wrap

2. Use the oc patch command to update the secret system-smtp. Use the option -p with the following JSON: {"stringData":{"<field>":"<value>"}}, where <field> is the field of the secret, and <value> is the desired value. The complete list of available fields is listed in the table below.

   Expand

   Table 1.4. system-smtp

   Field	Description	Default value in the secret	Default value in the application mailer
   address	The address of a remote mail server.	""	nil
   port	The port of the remote mail server to use.	""	0
   username	Username if the mail server requires authentication and the authentication type requires it.	""	nil
   password	Password if the mail server requires authentication and the authentication type requires it.	""	nil
   authentication	Use if the mail server requires authentication. Set the authentication types: plain to send the password in the clear, login to send password Base64 encoded, or cram_md5 to combine a challenge/response mechanism based on the HMAC-MD5 algorithm.	""	nil
   openssl.verify.mode	When using TLS, you can set how OpenSSL checks the certificate. This is useful if you need to validate a self-signed and/or a wildcard certificate. You can use the name of an OpenSSL verify constant: none or peer.	""	nil
   from_address	from address value for the no-reply mail.	""	"no-reply@{wildcardDomain}"

   Show more
   Note
   • HELO domain used for the mail server configuration is the value of {wildcardDomain}.
   • {wildcardDomain} is a placeholder for the value that is set in spec.wildcardDomain in the APIManager CR.

   Examples

   $ oc patch secret system-smtp -p '{"stringData":{"address":"<your_address>"}}'
   $ oc patch secret system-smtp -p '{"stringData":{"port":"587"}}'
   $ oc patch secret system-smtp -p '{"stringData":{"username":"<your_username>"}}'
   $ oc patch secret system-smtp -p '{"stringData":{"password":"<your_password>"}}'
   $ oc patch secret system-smtp -p '{"stringData":{"authentication":"plain"}}'
   $ oc patch secret system-smtp -p '{"stringData":{"openssl.verify.mode":"none"}}'

   Copy to Clipboard Toggle word wrap

3. After you have set the secret variables, redeploy the system-app and system-sidekiq pods:

   $ oc rollout restart deployment/system-app
   $ oc rollout restart deployment/system-sidekiq

   Copy to Clipboard Toggle word wrap

4. Check the status of the rollout to ensure it has finished:

   $ oc rollout status deployment/system-app
   $ oc rollout status deployment/system-sidekiq

   Copy to Clipboard Toggle word wrap

Verification

1. Type the following commands to check the deployment:

   $ oc get dc/apicast-staging -o yaml

   Copy to Clipboard Toggle word wrap

   $ oc get dc/apicast-production -o yaml

   Copy to Clipboard Toggle word wrap

2. Verify inclusion of the environment variables:

   # apicast-staging
   - name: APICAST_SERVICE_CACHE_SIZE
      value: '10'

   Copy to Clipboard Toggle word wrap

   # apicast-production
   - name: APICAST_SERVICE_CACHE_SIZE
      value: '20'

   Copy to Clipboard Toggle word wrap

Procedure

1. Create a new database.

2. Apply the following settings:

   ALTER SYSTEM SET max_string_size=extended SCOPE=SPFILE;

   Copy to Clipboard Toggle word wrap

3. Configure a database user

   There are two options for setting up Oracle Database integration in 3scale: with or without providing the Oracle SYSTEM user password.

   3scale uses the SYSTEM user only for the initial setup, which consist in creating a regular user and granting it the required privileges. The following SQL commands will set up a regular user with proper permissions. ({DB_USER} and {DB_PASSWORD} are placeholders that need to be replaced with actual values):

   CREATE USER {DB_USER} IDENTIFIED BY {DB_PASSWORD};
   GRANT unlimited tablespace TO {DB_USER};
   GRANT create session TO {DB_USER};
   GRANT create table TO {DB_USER};
   GRANT create view TO {DB_USER};
   GRANT create sequence TO {DB_USER};
   GRANT create trigger TO {DB_USER};
   GRANT create procedure TO {DB_USER};

   Copy to Clipboard Toggle word wrap

   1. Using the SYSTEM user:

      • Provide the SYSTEM user password in ORACLE_SYSTEM_PASSWORD field of the system-database secret.
      • The regular user does not need to exist before the installation. It will be created by the 3scale initialization script.
      • Provide the desired username and password for the regular user in the connection string, for example, oracle-enhanced://{DB_USER}:{DB_PASSWORD}@{DB_HOST}:{DB_PORT}/{DB_NAME} in the URL field of the system-database secret.
      • The password for the regular Oracle Database non-system user must be unique and not match the SYSTEM user password.

      • If the user with the specified username already exists, the 3scale initialization script will attempt to update the password using the following command:

        ALTER USER {DB_USER} IDENTIFIED BY {DB_PASSWORD}

        Copy to Clipboard Toggle word wrap

        Your database configuration might prevent this command from completing successfully if the parameters PASSWORD_REUSE_TIME and PASSWORD_REUSE_MAX are set in a way that restricts reusing the same password.

   2. Manual setup of the regular database user:

      • You do not need to provide the ORACLE_SYSTEM_PASSWORD in the system-database secret.
      • The regular database user (not SYSTEM) specified in the connection string in the URL field of the system-database secret needs to exist prior to the 3scale installation.
      • The regular user used for the installation must have all the privileges listed above.

Procedure

1. Download 3scale OpenShift templates from the GitHub repository and extract the archive:

   curl -L -o system-oracle-3scale-2.15.0-GA.tar.gz https://github.com/3scale/system-oracle/archive/refs/tags/3scale-2.15.0-GA.tar.gz
   tar -xzf system-oracle-3scale-2.15.0-GA.tar.gz

   Copy to Clipboard Toggle word wrap

2. From the Instant Client Downloads page, download:

   • A client: It can be either basic-lite or basic.
   • The ODBC driver.

   • The SDK for Oracle Database 19c.

     • For 3scale, use Instant Client Downloads for Linux x86-64 (64-bit)
     • For ppc64le and 3scale, use Oracle Instant Client Downloads for Linux on Power Little Endian (64-bit)

3. Check the table for the following Oracle software component versions:

   • Oracle Instant Client Package: Basic or Basic Light
   • Oracle Instant Client Package: SDK

   • Oracle Instant Client Package: ODBC

     Expand

     Table 1.6. Oracle 19c example packages for 3scale

     Oracle 19c package name	Compressed file name
     Basic	instantclient-basic-linux.x64-19.18.0.0.0dbru.zip
     Basic Light	instantclient-basiclite-linux.x64-19.18.0.0.0dbru.zip
     SDK	instantclient-sdk-linux.x64-19.18.0.0.0dbru.zip
     ODBC	instantclient-odbc-linux.x64-19.18.0.0.0dbru.zip

     Show more
     Expand

     Table 1.7. Oracle 19c example packages for ppc64le and 3scale

     Oracle 19c package name	Compressed file name
     Basic	instantclient-basic-linux.leppc64.c64-19.3.0.0.0dbru.zip
     Basic Light	instantclient-basiclite-linux.leppc64.c64-19.3.0.0.0dbru.zip
     SDK	instantclient-sdk-linux.leppc64.c64-19.3.0.0.0dbru.zip
     ODBC	instantclient-odbc-linux.leppc64.c64-19.3.0.0.0dbru.zip

     Show more
   Note

   If the client packages versions downloaded and stored locally do not match with the ones 3scale expects, 3scale will automatically download and use the appropriate ones in the following steps.

4. Place your Oracle Database Instant Client Package files into the system-oracle-3scale-2.15.0-GA/oracle-client-files directory.

5. Build the custom system Oracle-based image. The image tag must be a fixed image tag as in the following example:

   $ cd system-oracle-3scale-2.15.0-GA
   $ docker build . --tag myregistry.example.com/system-oracle:2.15.0-1

   Copy to Clipboard Toggle word wrap

6. Push the system Oracle-based image to a container registry accessible by the OCP cluster. This container registry is where your 3scale solution is going to be installed:

   $ docker push myregistry.example.com/system-oracle:2.15.0-1

   Copy to Clipboard Toggle word wrap

Procedure

1. Set up the Oracle Database URL connection string and Oracle Database system password by creating the system-database secret with the corresponding fields. See, External databases installation for the Oracle Database.

2. Install your 3scale solution by creating an APIManager CR. Follow the instructions in Deploying 3scale API Management using the operator.

   • The APIManager CR must specify the .spec.system.image field set to the system’s Oracle-based image you previous built:

     apiVersion: apps.3scale.net/v1alpha1
     kind: APIManager
     metadata:
       name: example-apimanager
     spec:
       imagePullSecrets:
       - name: threescale-registry-auth
       - name: custom-registry-auth
       system:
         image: "myregistry.example.com/system-oracle:2.15.0-1"
       externalComponents:
         system:
           database: true

     Copy to Clipboard Toggle word wrap

Procedure

1. Find the PVC containing the erroneous MySQL data with the oc get pvc command:

   # oc get pvc
   NAME                    STATUS    VOLUME    CAPACITY   ACCESSMODES   AGE
   backend-redis-storage   Bound     vol003    100Gi      RWO,RWX       4d
   mysql-storage           Bound     vol006    100Gi      RWO,RWX       4d
   system-redis-storage    Bound     vol008    100Gi      RWO,RWX       4d
   system-storage          Bound     vol004    100Gi      RWO,RWX       4d

   Copy to Clipboard Toggle word wrap
2. Stop the deployment of the system-mysql pod by clicking cancel deployment in the OpenShift Container Platform (OCP) console.
3. Delete everything under the MySQL path to clean the volume.
4. Start a new system-mysql deployment.

Procedure

1. The directories used for the persistent volumes MUST have the write permissions for the root group. Having read-write permissions for the root user is not enough as the MySQL service runs as a different user in the root group. Execute the following command as the root user:

   $ chmod -R g+w /path/for/pvs

   Copy to Clipboard Toggle word wrap

2. Execute the following command to prevent SElinux from blocking access:

   $ chcon -Rt svirt_sandbox_file_t /path/for/pvs

   Copy to Clipboard Toggle word wrap