Operating 3scale

An installed and initially configured AMP On-Premises instance on a supported OpenShift version.

This document is not intended for local installations on laptops or similar end user equipment.

As your APIcast deployment grows, you may need to increase the amount of storage available. How you scale up storage depends on which type of file system you are using for your persistent storage.

If you are using a network file system (NFS), you can scale up your persistent volume using the oc edit pv command:

oc edit pv <pv_name>

If you are using any other storage method, you must scale up your persistent volume manually using one of the methods listes in the following sections.

When 3scale is deployed on-premises, you can configure audit logging to stdout to forward all application logs to standard OpenShift pod logs. This enables all logs to be in one place for querying by Elasticsearch, Fluentd, and Kibana (EFK) logging tools. These tools provide increased visibility on changes made to your 3scale configuration, who made these changes, and when. For example, this includes changes to billing, application plans, API configuration, and so on.

Each component’s deployment configuration contains logs for access and exceptions. If you encounter issues with your deployment, check these logs for details.

Follow these steps to access logs in 3scale:

Job queues contain logs of information sent from the system-sidekiq pods. Use these logs to check if your cluster is processing data. You can query the logs using the OpenShift CLI:

oc get jobs

oc logs <job>

To prevent monotonic growth, 3scale schedules by default, automatic purging of the following tables:

With the exception of the above listed tables, the alerts table requires manual purging by the database administrator.

DELETE FROM alerts WHERE timestamp < NOW() - INTERVAL 14 DAY;

DELETE FROM alerts WHERE timestamp < NOW() - INTERVAL '14 day';

DELETE FROM alerts WHERE timestamp <= TRUNC(SYSDATE) - 14;

For other tables not specified in this section, the database administrator must manually clean the tables that the system does not automatically purge.

You can install the toolbox on Red Hat Enterprise Linux using yum or rpm.

For example:

$ sudo subscription-manager repos --enable=rhel-7-server-3scale-amp-2.6-rpms --enable rhel-server-rhscl-7-rpms

You can however use unsupported versions on Fedora Linux, Ubuntu Linux, Windows, or macOS by downloading and installing the latest .rpm, .deb, .msi or .pkg file from GitHub.

3scale remote list [--config-file <config_file>]

Shows the list of existing remotes (name, URL and authentication key).

$ 3scale remote list
instance_a https://example_a.net 123456789
instance_b https://example_b.net 987654321

3scale remote add [--config-file <config_file>] <name> <url>

Adds a remote with short name <name> at <url>.

3scale remote add instance_a https://123456789@example_a.net

3scale remote remove [--config-file <config_file>] <name>

Removes the remote woth short name <name>.

3scale remote remove instance_a

3scale remote rename [--config-file <config_file>] <old_name> <new_name>

Renames remote with short name <old_name> to <new_name>.

3scale remote rename instance_a instance_b

Use the following steps to create a new application plan:

The following example creates a new application plan:

3scale application-plan create [opts] <remote> <service> <plan-name>

Use the following options while creating application plans:

Options
       --approval-required=<value>      The application requires approval: true or false
       --cost-per-month=<value>         Cost per month
    -d --default                        This will make the default application plan
       --disabled                       This will disable all methods and metrics in
                                        the application plan
       --end-user-required=<value>      End user required: true or false
    -p --published                      This will publish the application plan
       --setup-fee=<value>              Set-up fee
    -t --system-name=<value>            This will set application plan system name
       --trial-period-days=<value>      The trial period in days

Options for application-plan
    -c --config-file=<value>            3scale toolbox configuration file
                                        (default: $HOME/.3scalerc.yaml)
    -h --help                           show help for this command
    -k --insecure                       Proceed and operate even for server
                                        connections otherwise considered insecure
    -v --version                        This will print the version of this command
       --verbose                        Verbose mode

Use the following steps to create a new application plan if it does not exist, or to update an existing one:

The following example updates the application plan:

3scale application-plan apply [opts] <remote> <service> <plan>

Use the following options while updating application plans:

Options
       --approval-required=<value>      The application requires approval: true or false
       --cost-per-month=<value>         Cost per month
       --default                        This will make the default application plan
       --disabled                       This will disable all methods and metrics in
                                        the application plan
       --enabled                        This will enable the application plan
       --end-user-required=<value>      End user required: true or false
       --hide                           This will hide the application plan
    -n --name=<value>                   This will set the plan name
    -p --publish                        This will publish the application plan
       --setup-fee=<value>              Set-up fee
       --trial-period-days=<value>      The trial period in days

Options for application-plan
    -c --config-file=<value>            3scale toolbox configuration file
                                        (default: $HOME/.3scalerc.yaml)
    -h --help                           show help for this command
    -k --insecure                       Proceed and operate even for server
                                        connections otherwise considered
                                        insecure
    -v --version                        This will print the version of this command
       --verbose                        Verbose mode

The following example lists the application plan:

3scale application-plan list [opts] <remote> <service>

Use the following options while listing application plans:

Options for application-plan
    -c --config-file=<value>      3scale toolbox configuration file
                                  (default:$HOME/.3scalerc.yaml)
    -h --help                     show help for this command
    -k --insecure                 Proceed and operate even for server
                                  connections otherwise considered insecure
    -v --version                  This will print the version of this command
       --verbose                  Verbose mode

The following example shows the application plan:

3scale application-plan show [opts] <remote> <service> <plan>

Use the following options while showing application plans:

Options for application-plan
    -c --config-file=<value>      3scale toolbox configuration file
                                  (default: $HOME/.3scalerc.yaml)
    -h --help                     show help for this command
    -k --insecure                 Proceed and operate even for server
                                  connections otherwise considered insecure
    -v --version                  This will print the version of this command
       --verbose                  Verbose mode

The following example deletes the application plan:

3scale application-plan delete [opts] <remote> <service> <plan>

Use the following options while deleting application plans:

Options for application-plan
    -c --config-file=<value>      3scale toolbox configuration file
                                  (default: $HOME/.3scalerc.yaml)
    -h --help                     show help for this command
    -k --insecure                 Proceed and operate even for server
                                  connections otherwise considered insecure
    -v --version                  This will print the version of this command
       --verbose                  Verbose mode

You can export or import a single application plan to or from yaml content.

Note the following: * Limits defined in the application plan are included. * Pricing rules defined in the application plan are included. * Metrics/methods referenced by limits and pricing rules are included. * Features defined in the application plan are included. * Service can be referenced by id or system_name. * Application Plan can be referenced by id or system_name.

3scale application-plan export [opts] <remote> <service_system_name> <plan_system_name>

3scale application-plan export -f plan.yaml remote_name service_name plan_name

Options
    -f --file=<value>             Write to file instead of stdout

Options for application-plan
    -c --config-file=<value>      3scale toolbox configuration file
                                  (default: $HOME/.3scalerc.yaml)
    -h --help                     show help for this command
    -k --insecure                 Proceed and operate even for server
                                  connections otherwise considered insecure
    -v --version                  Prints the version of this command
       --verbose                  Verbose mode

3scale application-plan import [opts] <remote> <service_system_name>

3scale application-plan import -f plan.yaml remote_name service_name

3scale application-plan import -f http[s]://domain/resource/path.yaml remote_name service_name

Options
    -f --file=<value>                  Read from file or url instead of stdin
    -p --plan=<value>                  Override application plan reference

Options for application-plan
    -c --config-file=<value>           3scale toolbox configuration file
                                       (default: $HOME/.3scalerc.yaml)
    -h --help                          show help for this command
    -k --insecure                      Proceed and operate even for server
                                       connections otherwise considered
                                       insecure
    -v --version                       Prints the version of this command
       --verbose                       Verbose mode

Use the following steps for creating metrics:

The following example creates metrics:

3scale metric create [opts] <remote> <service> <metric-name>

Use the following options while creating metrics:

Options
       --description=<value>      This will set a metric description
       --disabled                 This will disable this metric in all application plans
    -t --system-name=<value>      This will set the application plan system name
       --unit=<value>             Metric unit: default hit

Option for metric
    -c --config-file=<value>      3scale toolbox configuration file
                                  (default: $HOME/.3scalerc.yaml)
    -h --help                     show help for this command
    -k --insecure                 Proceed and operate even for server
                                  connections otherwise considered insecure
    -v --version                  This will print the version of this command
       --verbose                  Verbose mode

Use the following steps to create new metrics if they do not exist, or to update an existing one:

The following example updates metrics:

3scale metric apply [opts] <remote> <service> <metric>

Use the following options while updating metrics:

Options
       --description=<value>      This will set a metric description
       --disabled                 This will disable this metric in all application plans
       --enabled                  This will enable this metric in all application plans
    -n --name=<value>             This will set the metric name
       --unit=<value>             Metric unit: default hit

Options for metric
    -c --config-file=<value>      3scale toolbox configuration file
                                  (default: $HOME/.3scalerc.yaml)
    -h --help                     show help for this command
    -k --insecure                 Proceed and operate even for server
                                  connections otherwise considered insecure
    -v --version                  This will print the version of this command
       --verbose                  Verbose mode

The following example lists metrics:

3scale metric list [opts] <remote> <service>

Use the following options while listing metrics:

Options for metric
    -c --config-file=<value>      3scale toolbox configuration file
                                  (default: $HOME/.3scalerc.yaml)
    -h --help                     show help for this command
    -k --insecure                 Proceed and operate even for server
                                  connections otherwise considered insecure
    -v --version                  This will print the version of this command
       --verbose                  Verbose mode

The following example deletes metrics:

3scale metric delete [opts] <remote> <service> <metric>

Use the following options while deleting metrics:

Options for metric
    -c --config-file=<value>      3scale toolbox configuration file
                                  (default: $HOME/.3scalerc.yaml)
    -h --help                     show help for this command
    -k --insecure                 Proceed and operate even for server
                                  connections otherwise considered insecure
    -v --version                  This will print the version of this command
       --verbose                  Verbose mode

The following example creates a method:

3scale method create [opts] <remote> <service> <method-name>

Use the following options while creating methods:

Option
       --description=<value>      This will set a method description
       --disabled                 This will disable this method in all
                                  application plans
    -t --system-name=<value>      This will set the method system name

Options for method
    -c --config-file=<value>      3scale toolbox configuration file (default:
                                  $HOME/.3scalerc.yaml)
    -h --help                     show help for this command
    -k --insecure                 Proceed and operate even for server
                                  connections otherwise considered insecure
    -v --version                  This will print the version of this command
       --verbose                  Verbose mode

Use the following steps for creating new methods if they do not exist, or to update an existing ones:

The following example updates a method:

3scale method apply [opts] <remote> <service> <method>

Use the following options while updating methods:

Options
       --description=<value>      This will set a method description
       --disabled                 This will disable this method in all
                                  application plans
       --enabled                  This will enable this method in all
                                  application plans
    -n --name=<value>             This will set the method name

Options for method
    -c --config-file=<value>      3scale toolbox configuration file
                                  (default: $HOME/.3scalerc.yaml)
    -h --help                     This will show help for this command
    -k --insecure                 Proceed and operate even for server
                                  connections otherwise considered insecure
    -v --version                  This will print the version of this command
       --verbose                  Verbose mode

The following example lists methods:

3scale method list [opts] <remote> <service>

Use the following options while listing methods:

Options for method
    -c --config-file=<value>      3scale toolbox configuration file
                                  (default: $HOME/.3scalerc.yaml)
    -h --help                     show help for this command
    -k --insecure                 Proceed and operate even for server
                                  connections otherwise considered insecure
    -v --version                  This will print the version of this command
       --verbose                  Verbose mode

The following example deletes methods:

3scale method delete [opts] <remote> <service> <metric>

Use the following options while deleting methods:

Options for method
    -c --config-file=<value>      3scale toolbox configuration file
                                  (default: $HOME/.3scalerc.yaml)
    -h --help                     show help for this command
    -k --insecure                 Proceed and operate even for server
                                  connections otherwise considered insecure
    -v --version                  This will print the version of this command
       --verbose                  Verbose mode

The following example creates a new service:

3scale service create [options] <remote> <service-name>

Use the following options while creating services:

Options
    -a --authentication-mode=<value>      Specify authentication mode of the
                                          service ('1' for API key, '2' for
                                          App Id / App Key, 'oauth' for OAuth
                                          mode, 'oidc' for OpenID Connect)
    -d --deployment-mode=<value>          Specify the deployment mode of the
                                          service
       --description=<value>              Specify the description of the
                                          service
    -s --system-name=<value>              Specify the system-name of the
                                          service
       --support-email=<value>            Specify the support email of the
                                          service

Options for service
    -c --config-file=<value>              3scale toolbox configuration file
                                          (default:
                                          $HOME/.3scalerc.yaml)
    -h --help                             show help for this command
    -k --insecure                         Proceed and operate even for server
                                          connections otherwise considered
                                          insecure
    -v --version                          Prints the version of this command
       --verbose                          Verbose mode

Use the following to create new services if they do not exist, or to update an existing one:

The following example updates services:

3scale service apply <remote> <service-id_or_system-name>

Use the following options while updating services:

Options
    -a --authentication-mode=<value>      Specify authentication mode of the
                                          service ('1' for API key, '2' for
                                          App Id / App Key, 'oauth' for OAuth
                                          mode, 'oidc' for OpenID Connect)
    -d --deployment-mode=<value>          Specify the deployment mode of the
                                          service
       --description=<value>              Specify the description of the
                                          service
    -n --name=<value>                     Specify the name of the metric
       --support-email=<value>            Specify the support email of the
                                          service

Options for services
    -c --config-file=<value>              3scale toolbox configuration file
                                          (default: $HOME/.3scalerc.yaml)
    -h --help                             show help for this command
    -k --insecure                         Proceed and operate even for server
                                          connections otherwise considered
                                          insecure
    -v --version                          Prints the version of this command
       --verbose                          Verbose mode

The following example lists services:

3scale service list <remote>

Use the following options while listing services:

Options for services
    -c --config-file=<value>      3scale toolbox configuration file (default:
                                  $HOME/.3scalerc.yaml)
    -h --help                     show help for this command
    -k --insecure                 Proceed and operate even for server
                                  connections otherwise considered insecure
    -v --version                  Prints the version of this command
       --verbose                  Verbose mode

The following example shows services:

3scale service show <remote> <service-id_or_system-name>

Use the following options while showing services:

Options for services
    -c --config-file=<value>      3scale toolbox configuration file
                                  (default: $HOME/.3scalerc.yaml)
    -h --help                     show help for this command
    -k --insecure                 Proceed and operate even for server
                                  connections otherwise considered insecure
    -v --version                  Prints the version of this command
       --verbose                  Verbose mode

The following example deletes services:

3scale service delete <remote> <service-id_or_system-name>

Use the following options while deleting services:

Options for services
    -c --config-file=<value>      3scale toolbox configuration file (default:
                                  $HOME/.3scalerc.yaml)
    -h --help                     show help for this command
    -k --insecure                 Proceed and operate even for server
                                  connections otherwise considered insecure
    -v --version                  Prints the version of this command
       --verbose                  Verbose mode

To create a new ActiveDocs from your OpenAPI / Swagger compliant API defintion:

Options
    -d --description=<value>           Specify the description of the
                                       ActiveDocs
    -i --service-id=<value>            Specify the Service ID associated to
                                       the ActiveDocs
    -p --published                     Specify it to publish the ActiveDoc on
                                       the Developer Portal. Otherwise it
                                       will be hidden
    -s --system-name=<value>           Specify the system-name of the
                                       ActiveDocs
       --skip-swagger-validations      Specify it to skip validation of the
                                       Swagger specification

Options for ActiveDocs
    -c --config-file=<value>           3scale toolbox configuration file
                                       (default: $HOME/.3scalerc.yaml)
    -h --help                          show help for this command
    -k --insecure                      Proceed and operate even for server
                                       connections otherwise considered
                                       insecure
    -v --version                       Prints the version of this command
       --verbose                       Verbose mode

Use the following to create new ActiveDoc if they do not exist, or to update existing ActiveDocs with a new API definition:

3scale activedocs apply <remote> <activedocs_id_or_system_name>

Use the following options while updating ActiveDocs:

Options
    -d --description=<value>           Specify the description of the
                                       ActiveDocs
       --hide                          Specify it to hide the ActiveDocs on
                                       the Developer Portal
    -i --service-id=<value>            Specify the Service ID associated to
                                       the ActiveDocs
       --openapi-spec=<value>          Specify the swagger spec. Can be a
                                       file, an URL or '-' to read from
                                       stdin. This option is mandatory when
                                       applying the ActiveDoc for the first
                                       time
    -p --publish                       Specify it to publish the ActiveDocs
                                       on the Developer Portal. Otherwise it
                                       will be hidden
    -s --name=<value>                  Specify the name of the ActiveDocs
       --skip-swagger-validations      Specify it to skip validation of the
                                       Swagger specification

Options for ActiveDocs
    -c --config-file=<value>           3scale toolbox configuration file
                                       (default: $HOME/.3scalerc.yaml)
    -h --help                          show help for this command
    -k --insecure                      Proceed and operate even for server
                                       connections otherwise considered
                                       insecure
    -v --version                       Prints the version of this command
       --verbose                       Verbose mode

To get information about all ActiveDocs in the developer portal, including

The following example lists all defined ActiveDocs:

3scale activedocs list <remote>

Use the following options while listing ActiveDocs:

Options for ActiveDocs
    -c --config-file=<value>      3scale toolbox configuration file
                                  (default: $HOME/.3scalerc.yaml)
    -h --help                     show help for this command
    -k --insecure                 Proceed and operate even for server
                                  connections otherwise considered insecure
    -v --version                  Prints the version of this command
       --verbose                  Verbose mode

The following example removes ActiveDocs:

3scale activedocs delete <remote> <activedocs-id_or-system-name>

Use the following options while deleting ActiveDocs:

Options for ActiveDocs
    -c --config-file=<value>      3scale toolbox configuration file
                                  (default: $HOME/.3scalerc.yaml)
    -h --help                     show help for this command
    -k --insecure                 Proceed and operate even for server
                                  connections otherwise considered insecure
    -v --version                  Prints the version of this command
       --verbose                  Verbose mode

The following example lists proxy configurations:

3scale proxy-config list <remote> <service> <environment>

Use the following options while listing proxy configurations:

Options for proxy-config
    -c --config-file=<value>      3scale toolbox configuration file (default:
                                  /home/msoriano/.3scalerc.yaml)
    -h --help                     show help for this command
    -k --insecure                 Proceed and operate even for server
                                  connections otherwise considered insecure
    -v --version                  Prints the version of this command
       --verbose                  Verbose mode

The following example shows proxy configurations:

3scale proxy-config show <remote> <service> <environment>

Use the following options while showing proxy configurations:

Options for proxy-config
    -c --config-file=<value>         3scale toolbox configuration file
                                     (default: /home/msoriano/.3scalerc.yaml)
    -h --help                        show help for this command
    -k --insecure                    Proceed and operate even for server
                                     connections otherwise considered
                                     insecure
    -v --version                     Prints the version of this command
       --verbose                     Verbose mode

The following example promotes the latest staging proxy configuration to the production environment:

3scale proxy-config promote <remote> <service>

Use the following options while promoting the latest staging proxy configurations to the production environment:

Options for proxy-config
    -c --config-file=<value>      3scale toolbox configuration file (default:
                                  /home/msoriano/.3scalerc.yaml)
    -h --help                     show help for this command
    -k --insecure                 Proceed and operate even for server
                                  connections otherwise considered insecure
    -v --version                  Prints the version of this command
       --verbose                  Verbose mode

The following example lists applications:

3scale application list [opts] <remote>

Use the following options while listing applications:

OPTIONS
       --account=<value>          Filter by account
       --plan=<value>             Filter by application plan.
                                  Service option required.
       --service=<value>          Filter by service

Use the create command to create one application linked to a given 3scale account and application plan.

The following example creates applications:

3scale application create [opts] <remote> <account> <service> <application-plan> <name>

Use the following options while creating applications:

Options
       --application-id=<value>       App ID or Client ID (for OAuth and
                                      OpenID Connect authentication modes) of
                                      the application to be created.
       --application-key=<value>      App Key(s) or Client Secret (for OAuth
                                      and OpenID Connect authentication
                                      modes) of the application to be
                                      created.
       --description=<value>          Application description
       --redirect-url=<value>         OpenID Connect redirect url
       --user-key=<value>             User Key (API Key) of the application
                                      to be created.

The following example shows applications:

3scale application show [opts] <remote> <application>

Use the following to create new applications if they do not exist, or to update existing applications:

3scale application apply [opts] <remote> <application>

Use the following options while updating applications:

OPTIONS
       --account=<value>              Application's account. Required when
                                      creating
       --application-key=<value>      App Key(s) or Client Secret (for OAuth
                                      and OpenID Connect authentication
                                      modes) of the application to be
                                      created. Only used when application
                                      does not exist.
       --description=<value>          Application description
       --name=<value>                 Application name
       --plan=<value>                 Application's plan. Required when
                                      creating
       --redirect-url=<value>         OpenID Connect redirect url
       --resume                       Resume a suspended application
       --service=<value>              Application's service. Required when
                                      creating
       --suspend                      Suspends an application (changes the
                                      state to suspended)
       --user-key=<value>             User Key (API Key) of the application
                                      to be created.

The following example deletes an application:

3scale application delete [opts] <remote> <application>

The API provider cycle stages are based on specifying, developing, and deploying your APIs. The following describes the goal and outcome of each stage:

The API consumer cycle stages are based on promoting, distributing, and refining your APIs for consumption. The following describes the goal and outcome of each stage:

The following samples are provided in the Red Hat Integration repository as examples of how to create and deploy your Jenkins pipelines for API lifecycle automation:

These samples use a 3scale Jenkins shared library that calls the 3scale toolbox to demonstrate key API management capabilities. After you have performed the setup steps in this topic, you can install the pipelines using the OpenShift templates provided for each of the sample use cases in the Red Hat Integration repository.

Setting up a 3scale hosted environment is required by all of the sample Jenkins CI/CD pipelines.

Setting up a 3scale on-premises environment is required by the Hybrid - open and Hybrid - OIDC sample Jenkins CI/CD pipelines only.

If you are using the Hybrid - OpenID Connect (OIDC) or Semantic versioning sample pipelines, perform the steps in this section to deploy Red Hat Single Sign-On (RH-SSO) with 3scale. This is required for OIDC authentication, which is used in both samples.

This section describes how to install the toolbox, create your remote 3scale instance, and provision the secret used to access the Admin Portal.

This section shows how to deploy the example API backends provided with the sample pipelines. You can substitute your own API backends as needed when creating and deploying your own pipelines

This section is for use with APIcast self-managed instances in 3scale hosted environments. It applies to all of the sample pipelines except SaaS - API key.

After you have set up the required environments, you can install and deploy the sample pipelines using the OpenShift templates provided for each of the sample use cases in the Red Hat Integration repository. For example, this section shows the SaaS - API Key sample only.

The following limitations apply in this release:

In this approach a separate 3scale instance is deployed for each API back-end environment. The benefit of this architecture is that each environment will be isolated from one another, therefore there are no shared databases or other resources. For example, any load testing being done in one environment will not impact the resources in other environments.

In this approach a single 3scale instance is used but the multitenancy feature is used to support multiple API back ends.

When APIcast built-in gateways are used, the API back end configured using the above approaches described in Section 4.4, “APIcast gateway” will be handled automatically. When a tenant is added by a 3scale master admin, a route is created for the tenant in production and staging built-in APIcast gateways. See Understanding multitenancy subdomains

Therefore, each API back-end environment mapped to a different tenant would get its own route. For example:

Additional APIcast gateways are those deployed on a different OpenShift cluster than the one on which 3scale instance is running. There is more than one way to set up and use additional APIcast gateways. The value of environment variable THREESCALE_PORTAL_ENDPOINT used when starting APIcast depends how the additional APIcast gateways are set up.

A separate APIcast gateway can be used for each API back-end environment. For example:

DEV_APICAST -> DEV_TENANT ; DEV_APICAST started with THREESCALE_PORTAL_ENDPOINT = admin portal for DEV_TENANT
QA_APICAST -> QA_TENANT ; QA_APICAST started with THREESCALE_PORTAL_ENDPOINT = admin portal for QA_APICAST
PROD_APICAST -> PROD_TENANT ; PROD_APICAST started with THREESCALE_PORTAL_ENDPOINT = admin portal for PROD_APICAST

The THREESCALE_PORTAL_ENDPOINT is used by APIcast to download the configuration. Given that a separate APIcast gateway is used for each tenant that maps to an API back-end environment, the THREESCALE_PORTAL_ENDPOINT is set to the Admin Portal for the tenant that contains all the service configurations specific to that API back-end environment.

A single APIcast gateway can be used with multiple API back-end environments. In this case THREESCALE_PORTAL_ENDPOINT is set to the Master Admin Portal.

The following procedure creates an API with the label api: api01.

The following procedure adds a plan with the label api: api01.

The following procedure adds a metric called metric01.

The following procedure sets a limit with a limit of 10 hits per day for the metric.

The following procedure adds a MappingRule to increment metric01.

Use the following procedure to bind using the binding object.

system-mysql is a relational database which stores information about users, accounts, APIs, plans, and more, in the 3scale Admin Console.

A subset of this information related to services is synchronized to the Backend component and stored in backend-redis. system-mysql is the source of truth for this information.

system-storage stores files to be read and written by the System component.

They fall into two categories:

A zync-database is a relational database which stores information related to the synchronization of identities between 3scale and an Identity provider (IdP). This information is not duplicated in other components and is the sole source of truth.

backend-redis contains multiple data sets used by the Backend component:

system-redis contains queues for jobs to be processed in background. These are ephemeral and are deleted once processed.

Execute MySQL Backup Command:

oc rsh $(oc get pods -l 'deploymentConfig=system-mysql' -o json | jq -r '.items[0].metadata.name') bash -c 'export MYSQL_PWD=${MYSQL_ROOT_PASSWORD}; mysqldump --single-transaction -hsystem-mysql -uroot system' | gzip > system-mysql-backup.gz

Archive the system-storage files to another storage:

oc rsync $(oc get pods -l 'deploymentConfig=system-app' -o json | jq '.items[0].metadata.name' -r):/opt/system/public/system ./local/dir

Execute Postgres Backup Command:

 oc rsh $(oc get pods -l 'deploymentConfig=zync-database' -o json | jq '.items[0].metadata.name' -r) bash -c 'pg_dumpall -c --if-exists' | gzip > zync-database-backup.gz

Backup the dump.rb file from redis:

oc cp $(oc get pods -l 'deploymentConfig=backend-redis' -o json | jq '.items[0].metadata.name' -r):/var/lib/redis/data/dump.rdb ./backend-redis-dump.rdb

Backup the dump.rb file from redis:

oc cp $(oc get pods -l 'deploymentConfig=system-redis' -o json | jq '.items[0].metadata.name' -r):/var/lib/redis/data/dump.rdb ./system-redis-dump.rdb

Restore the Backup file to system-storage:

oc rsync ./local/dir $(oc get pods -l 'deploymentConfig=system-app' -o json | jq '.items[0].metadata.name' -r):/opt/system/public/system

After restoring backend-redis a sync of the Config information from System should be forced to ensure the information in Backend is consistent with that in System, which is the source of truth.

The following sections attempt to outline some common issues you may see in the APIcast error log during the initial phases of your integration with 3scale: at the beginning using APIcast Hosted and prior to go-live, running the self-managed APIcast.

It is rare to run into issues with your API gateway after you have fully tested your setup and have been live with your API for a while. However, here are some of the issues you might encounter in a live production environment.

upstream timed out (110: Connection timed out) while connecting to upstream, client: X.X.X.X, server: api.example.com, request: "GET /RESOURCE?CREDENTIALS HTTP/1.1", upstream: "http://Y.Y.Y.Y:80/RESOURCE?CREDENTIALS", host: "api.example.com"

If you make changes to your API such as adding a new endpoint, you must ensure that you add a new method and URL mapping before downloading a new set of configuration files for your API gateway.

The most common problem when you have modified the configuration downloaded from 3scale will be code errors in the Lua, which will result in a 500 - Internal server error such as:

curl -v -X GET "http://localhost/"
* About to connect() to localhost port 80 (#0)
*   Trying 127.0.0.1... connected
> GET / HTTP/1.1
> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
> Host: localhost
> Accept: */*
>
< HTTP/1.1 500 Internal Server Error
< Server: openresty/1.5.12.1
< Date: Thu, 04 Feb 2016 10:22:25 GMT
< Content-Type: text/html
< Content-Length: 199
< Connection: close
<

<head><title>500 Internal Server Error</title></head>

<center><h1>500 Internal Server Error</h1></center>
<hr><center>openresty/1.5.12.1</center>


* Closing connection #0

You can see the nginx error.log to know the cause, such as:

2016/02/04 11:22:25 [error] 8980#0: *1 lua entry thread aborted: runtime error: /home/pili/NGINX/troubleshooting/nginx.lua:66: bad argument #3 to '_newindex' (number expected, got nil)
stack traceback:
coroutine 0:
  [C]: in function '_newindex'
  /home/pili/NGINX/troubleshooting/nginx.lua:66: in function 'error_authorization_failed'
  /home/pili/NGINX/troubleshooting/nginx.lua:330: in function 'authrep'
  /home/pili/NGINX/troubleshooting/nginx.lua:283: in function 'authorize'
  /home/pili/NGINX/troubleshooting/nginx.lua:392: in function  while sending to client, client: 127.0.0.1, server: api-2445581381726.staging.apicast.io, request: "GET / HTTP/1.1", host: "localhost"

In the access.log this will look like the following:

127.0.0.1 - - [04/Feb/2016:11:22:25 +0100] "GET / HTTP/1.1" 500 199 "-" "curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3"

The above section gives you a an overview of the most common, well-known issues that you might encounter at any stage of your 3scale journey.

If all of these have been checked and you are still unable to find the cause and solution for your issue, you should proceed to the more detailed [operational troubleshooting](#troubleshooting-checklists) sections below. Start at your API and work your way back to the client in order to try to identify the point of failure.

Use telnet to check the basic TCP/IP connectivity telnet api.example.com 443

telnet echo-api.3scale.net 80
Trying 52.21.167.109...
Connected to tf-lb-i2t5pgt2cfdnbdfh2c6qqoartm-829217110.us-east-1.elb.amazonaws.com.
Escape character is '^]'.
Connection closed by foreign host.

telnet su1.3scale.net 443
Trying 174.129.235.69...
telnet: Unable to connect to remote host: Connection timed out

Try to connect to the same server from different network locations, devices, and directions. For example, if your client is unable to reach your API, try to connect to your API from a machine that should have access such as the API gateway.

If any of the attempted connections succeed, you can rule out any problems with the actual server and concentrate your troubleshooting on the network between them, as this is where the problem will most likely be.

Try to connect to the server by using its IP address instead of its hostname e.g. telnet 94.125.104.17 80 instead of telnet apis.io 80

This will rule out any problems with the DNS.

You can get the IP address for a server using dig for example for 3scale dig su1.3scale.net or dig any su1.3scale.net if you suspect there may be multiple IPs that a host may resolve to.

NB: Some hosts block `dig any`

You can use OpenSSL to test:

CONNECTED(00000003)
140735196860496:error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:s3_pkt.c:1456:SSL alert number 40
140735196860496:error:1409E0E5:SSL routines:ssl3_write_bytes:ssl handshake failure:s3_pkt.c:644:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 0 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : SSLv3
    Cipher    : 0000
    Session-ID:
    Session-ID-ctx:
    Master-Key:
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1454932872
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
---

For more details, see the OpenSSL man pages.

To confirm that the API is up and responding to requests, make the same request directly to your API (not going through the API gateway). You should ensure that you are sending the same parameters and headers as the request that goes through the API gateway. If you are unsure of the exact request that is failing, capture the traffic between the API gateway and your API.

If the call succeeds, you can rule out any problems with the API, otherwise you should troubleshoot your API further.

To rule out any network issues between the API gateway and the API, make the same call as before — directly to your API — from your API gateway server.

If the call succeeds, you can move on to troubleshooting the API gateway itself.

There are a number of steps to go through to check that the API gateway is working correctly.

Once you are sure the API gateway is running correctly, the next step is troubleshooting the connection between the API gateway and 3scale.

2015/11/20 11:33:51 [error] 3578#0: *1 upstream timed out (110: Connection timed out) while connecting to upstream, client: 127.0.0.1, server: , request: "GET /api/activities.json?user_key=USER_KEY HTTP/1.1", subrequest: "/threescale_authrep", upstream: "https://54.83.62.186:443/transactions/authrep.xml?provider_key=YOUR_PROVIDER_KEY&service_id=SERVICE_ID&usage[hits]=1&user_key=USER_KEY&log%5Bcode%5D=", host: "localhost"

http {
  lua_shared_dict api_keys 10m;
  server_names_hash_bucket_size 128;
  lua_package_path ";;$prefix/?.lua;";
  init_by_lua 'math.randomseed(ngx.time()) ; cjson = require("cjson")';

  resolver 8.8.8.8 8.8.4.4;

nslookup su1.3scale.net 8.8.8.8
;; connection timed out; no servers could be reached

2016/05/09 14:15:15 [alert] 9391#0: send() failed (1: Operation not permitted) while resolving, resolver: 8.8.8.8:53

body_filter_by_lua_block{
  if ngx.req.get_headers()["X-3scale-debug"] == ngx.var.provider_key then
    local resp = ""
    ngx.ctx.buffered = (ngx.ctx.buffered or "") .. string.sub(ngx.arg[1], 1, 1000)
    if ngx.arg[2] then
      resp = ngx.ctx.buffered
    end

    ngx.log(0, ngx.req.raw_header())
    ngx.log(0, resp)
  end
}

2016/05/05 14:24:33 [] 7238#0: *57 [lua] body_filter_by_lua:7: GET /api/contacts.json?access_token=7c6f24f5 HTTP/1.1
Host: 726e3b99.ngrok.io
User-Agent: curl/7.43.0
Accept: */*
X-Forwarded-Proto: https
X-Forwarded-For: 2.139.235.79

 while sending to client, client: 127.0.0.1, server: pili-virtualbox, request: "GET /api/contacts.json?access_token=7c6f24f5 HTTP/1.1", subrequest: "/threescale_authrep", upstream: "https://54.83.62.94:443/transactions/oauth_authrep.xml?provider_key=REDACTED&service_id=REDACTED&usage[hits]=1&access_token=7c6f24f5", host: "726e3b99.ngrok.io"
2016/05/05 14:24:33 [] 7238#0: *57 [lua] body_filter_by_lua:8: <?xml version="1.0" encoding="UTF-8"?><error code="access_token_invalid">access_token "7c6f24f5" is invalid: expired or never defined</error> while sending to client, client: 127.0.0.1, server: pili-virtualbox, request: "GET /api/contacts.json?access_token=7c6f24f5 HTTP/1.1", subrequest: "/threescale_authrep", upstream: "https://54.83.62.94:443/transactions/oauth_authrep.xml?provider_key=REDACTED&service_id=REDACTED&usage[hits]=1&access_token=7c6f24f5", host: "726e3b99.ngrok.io"

Sometimes calls that work when you call the API from the command line fail when going through ActiveDocs.

To enable ActiveDocs calls to work, we send these out through a proxy on our side. This proxy will add certain headers that can sometimes cause issues on the API if they are not expected. To identify if this is the case, try the following steps:

For a comprehensive guide on this, see the NGINX Logging and Monitoring docs.

To double-check the error codes that are returned by the 3scale Service Management API endpoints, see the 3scale API Documentation page by following these steps:

The following is a list HTTP response codes returned by 3scale, and the conditions under which they are returned:

Most of these error responses will also contain an XML body with a machine readable error category and a human readable explanation.

When using the standard API gateway configuration, any return code different from 200 provided by 3scale can result in a response to the client with one of the following codes: