Chapter 1. Troubleshooting and support for MCP gateway

Procedure

1. Check the pod status by running the following command:

   $ oc get pods -n <mcp_system>

   Replace <mcp_system> with the name of the MCP gateway deployment that you are checking.

2. Describe problem pods by running the following command:

   $ oc describe pod -n <mcp_system> <pod_name>

   • Replace <mcp_system> with the name of the MCP gateway deployment that you are checking.
   • Replace <pod_name> with the name of the pod that you are checking.

3. Check the pod logs by running the following command:

   $ oc logs -n <mcp_system> <pod_name>

   • Replace <mcp_system> with the name of the MCP gateway deployment that you are checking.
   • Replace <pod_name> with the name of the pod that you are checking.

Procedure

1. Check the status of the specific MCPGatewayExtension CR by running the following command:

   $ oc get mcpgatewayextension -n <namespace>

   Replace <namespace> with the namespace where your MCPGatewayExtension CR is applied.

2. Check for conflicting MCPGatewayExtension CRs by running the following command:

   $ oc get mcpgatewayextension -A

   There must be only one extension per namespace.

3. Verify that the target Gateway object exists by running the following command:

   $ oc get gateway -n <gateway_namespace>

   Replace <gateway_namespace> with the namespace where your Gateway object is applied.

4. Check the MCPServerRegistration CR namespace and route status by running the following command:

   $ oc describe mcpsr <mcpsr_name> -n <mcpsr_namespace>

   • Replace <mcpsr_name> with the name of your MCPServerRegistration CR.
   • Replace <mcpsr_namespace> with the namespace where your MCPServerRegistration CR is applied.
5. Ensure that the MCPGatewayExtension CR targets the Gateway object that the HTTPRoute CR is attached to.

Procedure

1. If tools from your on-premise MCP server do not display in tools/list, check that the MCP server is properly discovered by MCP gateway components by running the following command:

   • List the MCPServerRegistration CRs by running the following command:

     $ oc get mcpsr -A

   • Get the detailed status and configuration of your MCPServerRegistration CR by running the following command:

     $ oc describe mcpserverregistration <mcpsr_name> -n <mcpsr_namespace>

     • Replace <mcpsr_name> with the name of your MCP server.
     • Replace <mcpsr_namespace> with the namespace where your MCPServerRegistration CR is applied.

2. Verify that the MCPServerRegistration CR targetRef points to correct HTTPRoute name and namespace by running the following commands:

   1. Get the name and namespace of the HTTPRoute CR that the MCPServerRegistration CR is attempting to attach to by running the following command:

      $ oc get mcpserverregistration <mcpsr_name> -n <mcpsr_namespace> -o jsonpath='{.spec.targetRef.name}{"\n"}{.spec.targetRef.namespace}{"\n"}'

      • Replace <mcpsr_name> with the name of your MCP server.
      • Replace <mcpsr_namespace> with the namespace where your MCPServerRegistration CR is applied.
   2. If the MCPServerRegistration CR targets the HTTPRoute CR that is applied in the same namespace,the .spec.targetRef.namespace is not needed. No output is expected.

   3. Cross-reference the HTTPRoute details from of the previous command with your HTTPRoute CRs by verifying that a route actually exists with those exact details by running the following command:

      $ oc get httproute <httproute_name> -n <httproute_namespace>

      • Replace <httproute_name> with the value from the output of the earlier step.
      • Replace <httproute_namespace> with the value from the output of the earlier step.
   4. If you get a NotFound server Error from the earlier step, correct your MCPServerRegistration CR.

3. Verify that the MCP gateway controller component successfully bound the extension and route together by checking the status with the following command:

   $ oc get mcpserverregistration <mcpsr_name> -n <mcpsr_namespace> -o jsonpath='{.status.conditions[?(@.type=="Accepted")]}'

   • Replace <mcpsr_name> with the name of your MCP server.

   • Replace <mcpsr_namespace> with the namespace where your MCPServerRegistration CR is applied.

     1. If you get a status: False output, the MCP gateway controller component found the route but could not use it. Check that you created a ReferenceGrant CR if the MCPGatewayExtension and HTTPRoute CRs are in different namespaces.

4. Check that the backend MCP server is running by entering the following command:

   $ oc get pods -n <mcpsr_namespace>

   Replace <mcpsr_namespace> with the namespace where your MCPServerRegistration CR is applied.

5. If your MCP server pods are crashing, check for the following conditions:

   • CrashLoopBackOff: This generally means that your MCP server is starting but then crashing. This problem can be caused by a missing environment variable or by permission issues.
   • Pending: This might mean that the pod has not started because of a resource issue in the cluster.
   • ErrImagePull: Either the cluster cannot reach the registry, or you do not have permission to pull the image. Check your OpenShift Container Platform settings and network configuration.
   • If your pod has a Running status with a high restart count, your MCP server might be unstable because of memory constraints or connectivity issues. Check your OpenShift Container Platform settings and network configuration.

6. Verify that the targeted application of your MCPServerRegistration CR actually exists and has a valid entry point by running the following command:

   $ oc get svc -n <namespace> <service_name>

   • Replace <namespace> with the namespace where your application runs.
   • Replace <service_name> with the name of your application.
7. Check that your application has a standard ClusterIP assigned to handle load balancing across pods.
8. Compare the ports to your MCPServerRegistration CR to make sure that they match and that either the TCP or SCTP protocol is used.

9. Check that the selectors on the Service CR for your application match the labels on your pods by running the following command:

   $ oc get endpoints <service_name> -n <namespace>

   • Replace <service_name> with the name of your application.
   • Replace <namespace> with the namespace where your application runs.

10. Check that the attached HTTPRoute CR has valid backend reference by running the following command:

    $ oc describe httproute <route_name>

11. If your MCPServerRegistration CR is active, but tools are missing, test the backend server directly by running the following commands:

    1. Start a debug session based on your existing MCP server deployment by running the following command:

       $ oc debug deployment/<mcpsr_name> -it

       Replace <mcpsr_name> with the name of your MCP server.

    2. Test whether your backend MCP server is functional by running the following command:

       sh-4.4# curl -X POST https://localhost:<port>/mcp \
         -H "Content-Type: application/json" \
         -d '{"jsonrpc": "2.0", "id": 1, "method": "tools/list"}'

       If the command returns a list of tools, your backend is healthy.

12. If your backend is healthy but tools are missing in the Gateway CR, your MCPServerRegistration CR is not correctly mapping the service. Check the logs to see why the backend rejected the request by running the following command:

    $ oc logs -l app=<mcp_server_label>

    Replace <mcp_server_label> with the application defined in the metadata.labels.app: section of your MCP server Pod or Deployment CR.

13. If you created an MCPServerRegistration CR, but your tools are not displaying, check the MCP broker component router logs for errors by running the following command:

    $ oc logs -n <mcp_system> -l app.kubernetes.io/name=<mcp_gateway>

    • Replace <mcp_system> with the namespace of your MCP gateway deployment.

    • Replace <mcp_gateway> with the name of your MCP gateway deployment.

      1. If you see errors, check for typos in the targetRef parameter value and ensure that you are pointing to an existing Service CR.
14. Verify that the backend MCP server is implementing the tools/list method correctly. Using the MCP Inspector is the easiest way to check.
15. Check your backend MCP server logs for errors.
16. Ensure that your backend MCP server is returning valid MCP protocol responses. Using the MCP Conformance Test Framework is the easiest way to catch protocol errors.
17. Verify that the toolPrefix entry in the MCPServerRegistration CR is valid, meaning that there are no spaces or special characters.

18. If tools appear without the configured prefix, check the MCPServerRegistration CR by running the following command:

    $ oc get mcpsr <mcpsr_name> -n <mcpsr_namespace> -o yaml | grep toolPrefix

    • Replace <mcpsr_name> with the name of your MCP server.

    • Replace <mcpsr_namespace> with the namespace where your MCPServerRegistration CR is applied.

      1. Ensure that a toolPrefix is set correctly in MCPServerRegistration CR.

19. Check the MCP gateway controller component logs for problems with your MCPServerRegistration CR by running the following command:

    $ oc logs -n <mcp_system> deployment/mcp-gateway-controller | grep prefix

    Replace <mcp_system> with the namespace of your MCP gateway deployment.

    1. If you see failed to generate prefix, then there is an error in your MCPServerRegistration CR metadata or there is a conflict.
    2. If there is no output at all, then either the controller is not detecting your MCPServerRegistration CR, or the request is failing before it gets to the routing stage.

20. Ensure that you restart the MCP gateway broker component after MCPServerRegistration CR changes by running the following command:

    $ oc rollout restart deployment/<mcp_gateway> -n <mcp_system>

    • * Replace <mcp_gateway> with the name of your MCP gateway deployment.
    • Replace <mcp_system> with the namespace of your MCP gateway deployment.

Procedure

1. If you are seeing errors such as 502 Bad Gateway and 403 Forbidden, check the ServiceEntry CR by running the following command:

   $ oc get serviceentry -n <namespace>

   Replace with the <namespace> where the ServiceEntry CR is applied.

   1. If the command returns an empty list, create a ServiceEntry CR to allow egress.

2. For detailed troubleshooting, such as to checks ports and the TLS setting, examine the ServiceEntry CR activity by running the following command:

   $ oc describe serviceentry <se_name> -n <namespace>

   • Replace with the <se_name> with the name the ServiceEntry CR.
   • Replace with the <namespace> where the ServiceEntry CR is applied.
3. Verify that ServiceEntry CR hosts values are exact matches external hostname values.

4. If you are seeing TLS errors, verify that a DestinationRule CR exists by running the following command:

   $ oc get destinationrule -n <namespace>

   Replace with the <namespace> where the DestinationRule CR is applied.

   1. If the previous command returns No resources found in <namespace> namespace, create a DestinationRule CR.
   2. Ensure that DestinationRule CR host values exactly match ServiceEntry CR host values.
   3. Check the TLS configuration in DestinationRule CR.

5. For detailed troubleshooting of actual connection details, such as encryption, load balancing, and connection pooling, examine the DestinationRule CR by running the following command:

   $ oc describe destinationrule <dr_name> -n <namespace>

   • Replace with the <dr_name> with the name the DestinationRule CR.

   • Replace with the <namespace> where the DestinationRule CR is applied.

     1. Ensure that your network egress policies allow external traffic from the pod.

6. Test the DNS resolution by running the following command:

   $ oc run -it --rm debug --image=<image_path/name> --restart=Never -- \
     nslookup <external_hostname>

   • Replace <image_path/name> with the path to your application image.
   • Replace <external_hostname> with your external MCP server hostname parameter value.

7. Test the external connectivity by running the following command:

   $ oc run -it --rm debug --image=<image_path/name> --restart=Never -- \
     curl -v https://<external_hostname_url>

   • Replace <image_path/name> with the path to your application image.
   • Replace <external_hostname_url> with your external MCP server URL.
8. Verify that the HTTPRoute CR backendRef value is configured with the correct external hostname.
9. Ensure that the HTTPRoute CR has the URLRewrite filter applied to rewrite to the external hostname.

Procedure

1. Check that a Secret custom resource (CR) exists and that it has the correct label by running the following command:

   $ oc get secret <secret_name> -n <namespace> --show-labels

   • Replace <secret_name> with the name of your Secret CR.
   • Replace <namespace> with the namespace where your Secret CR is applied.

2. Verify the Secret CR contents by running the following command:

   $ oc get secret <secret_name> -n <namespace> -o yaml

   • Replace <secret_name> with the name of your Secret CR.
   • Replace <namespace> with the namespace where your Secret CR is applied.

3. Check the MCPServerRegistration CR credentialRef parameter value by running the following command:

   $ oc get mcpsr <mcpsr_name> -n <namespace> -o yaml | grep -A 3 credentialRef

   • Replace <mcpsr_name> with the name of your MCPServerRegistration CR.
   • Replace <namespace> with the namespace where your MCPServerRegistration CR is applied.
4. Ensure that the Secret CR has the label mcp.kuadrant.io/secret: "true".
5. Verify that the Secret CR data key matches the credentialRef.key in the MCPServerRegistration CR.
6. Check the credential format.
7. Verify that the credential you are using has the necessary permissions for the external service.

8. Check the MCP gateway broker component logs for credential errors by running the following command:

   $ oc logs -n <mcp_system> deployment/mcp-gateway | grep -i auth

   Replace <mcp_system> with the name of your MCP gateway deployment.

Procedure

1. When your clients cannot discover OAuth configuration, discovery is not working. Use the following steps to troubleshoot this situation:

   1. Retrieve the JSON object that lists the security requirements for your backend MCP server by running the following command:

      $ curl https://<mcp_hostname>/.well-known/oauth-protected-resource

      • Replace <mcp_hostname> with the name of the host for your MCP server.

        Example output

        {
          "resource_name": "MCP Server",
          "resource": "https://mcp.example.com/mcp",
          "authorization_servers": [
            "https://auth.provider.com/realms/mcp"
          ],
          "bearer_methods_supported": ["header"],
          "scopes_supported": [
            "basic",
            "groups",
            "roles",
            "profile"
          ]
        }

   2. Check that your associated HTTPRoute CR includes a path for /.well-known/oauth-protected-resource by running the following command:

      $ oc get httproute <route_name> -n <mcp_system> -o json | jq -r '.spec.rules[].matches[].path.value | select(. == "/.well-known/oauth-protected-resource")'

      • Replace <route_name> with the associated HTTPRoute CR.
      • Replace <mcp_system> with the name of your MCP gateway deployment.
   3. If there is no output, your HTTPRoute CR does not have a path for /.well-known/oauth-protected-resource. You must add one.

   4. Check the specific AuthPolicy CR configuration by running the following command:

      $ oc describe authpolicy <authn_name> -n <namespace>

      • Replace <authn_name> with the name of your AuthPolicy CR.
      • Replace <namespace> with the namespace where your AuthPolicy CR is applied.

   5. Check that you excluded all /.well-known/ paths from your AuthPolicy CR by trying to reach the endpoint without any credentials by using the following command:

      $ curl -o /dev/null -s -w "%{https_code}\n" https://<mcp_hostname>/.well-known/oauth-protected-resource

      Replace <mcp_hostname> with the name of the host for your MCP server.

      Note

      The following codes are examples of possible outputs:

      • 200: Means that the exclusion exists and matches.
      • 401: Means that your AuthPolicy CR is still demanding a token for this path. The exclusion is either not present or not working.
      • 404: The exclusion might be present and working, but the HTTPRoute CR does not point to that path or to a valid backend.

   6. Optional. Check all MCP broker component environment variables by running the following command:

      $ oc get deployment <mcp_gateway> -n <mcp_system> -o yaml | grep -A 10 env

      • Replace <mcp_gateway> with the name of your MCP gateway deployment.
      • Replace <mcp_system> with the namespace where your MCP gateway deployment is applied.
   7. Verify that the MCP broker component pod restarted after any environment variable changes.

2. If your valid tokens are being rejected with 401 errors, your JWT token validation is failing. Use the following steps to troubleshoot this situation:

   1. List the AuthPolicy CRs by running the following command:

      $ oc get authpolicy -A

   2. Check the specific AuthPolicy CR configuration by running the following command:

      $ oc describe authpolicy <authn_name> -n <namespace>

      • Replace <authn_name> with the name of your AuthPolicy CR.
      • Replace <namespace> with the namespace where your AuthPolicy CR is applied.
   3. Verify that the issuerUrl in the AuthPolicy CR matches your identity provider’s realm.

   4. Check the Authorino Operator logs by running the following command:

      $ oc logs -n <kuadrant_system> -l authorino-resource=authorino

      Replace <kuadrant_system> with the namespace where Connectivity Link is installed.

   5. Decode JWT to verify claims by running the following command:

      $ echo "<example_token>" | cut -d. -f2 | base64 -d | jq

      Replace <example_token> with your token.

   6. Ensure that your issuer URL is reachable from the cluster by running the following command:

      $ oc exec -n <kuadrant_system> deployment/authorino -- curl -s https://auth.provider.com/realms/mcp/.well-known/openid-configuration

      • Replace <kuadrant_system> with the name of your Connectivity Link deployment.
   7. Check the token expiration time by examining the exp claim.
   8. Verify the audience, if required, by using an aud claim.
   9. Ensure that the token includes all required claims such as groups, email, and so on.

3. If your 401 Unauthorized responses do not include OAuth discovery information, the WWW-Authenticate header is missing. This usually means that your AuthPolicy CR is not properly configured. Use the following steps to troubleshoot this situation:

   1. Isolate the failure point by using verbose output which lists the TLS handshake, the HTTP headers, and the server response code by running the following command:

      $ curl -v https://<mcp_hostname>/mcp \
        -H "Content-Type: application/json" \
        -d '{"jsonrpc": "2.0", "id": 1, "method": "tools/list"}'

      Replace <mcp_hostname> with the hostname for your backend MCP server.

   2. Verify that your AuthPolicy CR spec.response.unauthenticated.headers: list includes response.unauthenticated.headers.WWW-Authenticate.
   3. Check that your AuthPolicy CR response.unauthenticated.headers.WWW-Authenticate.value includes the correct metadata.
   4. Ensure that your AuthPolicy CR is applied to the correct Gateway object and listener.

Procedure

1. Check your AuthPolicy custom resource (CR) authorization rules by running the following command:

   $ oc get authpolicy <policy_name> -n <namespace> -o yaml | grep -A 20 authorization

   • Replace <policy_name> with the name of your AuthPolicy CR.
   • Replace <namespace> `with the namespace where your `AuthPolicy CR is applied.

2. Check the Authorino Operator logs for CEL evaluation by running the following command:

   $ oc logs -n <kuadrant_system> -l authorino-resource=authorino | grep -i authz

   Replace <kuadrant_system> with the namespace where Connectivity Link is installed.

3. Ensure that the Authorino Operator can communicate with your identity server.
4. Verify that your JWT token includes resource_access[server-name].roles claims.

5. When your authorization checks are not enforced, first check your AuthPolicy CR status by running the following command:

   $ oc describe authpolicy <policy_name> -n <namespace>

   • Replace <policy_name> with the name of your AuthPolicy CR.
   • Replace <namespace> `with the namespace where your `AuthPolicy CR is applied.

6. Next, verify your AuthPolicy CR targets the correct resource by running the following command:

   $ oc get authpolicy <policy_name> -n <namespace> -o yaml | grep -A 5 targetRef

   • Replace <policy_name> with the name of your AuthPolicy CR.
   • Replace <namespace> `with the namespace where your `AuthPolicy CR is applied.

7. Ensure that your AuthPolicy CR targetRef matches your Gateway object name and namespace by running the following command:

   $ echo "--- AuthPolicy Targets ---" && \
   oc get authpolicy -n <mcp_system> -o jsonpath='{range .items[*]}{.metadata.name}{"\t targets -> \t"}{.spec.targetRef.kind}{"/"}{.spec.targetRef.name}{"\n"}{end}' && \
   echo "--- Actual Gateways ---" && \
   oc get gateway -n <mcp_system> -o custom-columns=NAME:.metadata.name

   Replace <mcp_system> with the name of your MCP gateway deployment.

8. If your AuthPolicy CR and your Gateway object are in different namespaces, you must either move the AuthPolicy CR into the same namespace as the Gateway object, or target the HTTPRoute CR instead.

9. Check that your AuthPolicy CR sectionName matches your Gateway object listener name.

   $ oc describe authpolicy <auth_policy_name> -n <mcp_system>

   Replace <auth_policy_name> with the name of your AuthPolicy CR. Replace <mcp_system> with the name of your MCP gateway deployment.

10. Examine the Status block for an entry about your listener. If the sectionName is wrong, the policy shows "Accepted", but the policy does not affect the intended traffic path.

11. Check that Kuadrant Operator is working by running the following command:

    $ oc get pods -n <kuadrant_system>

    Replace <kuadrant_system> with the namespace where your Connectivity Link is installed.

    Example output

    NAME                                                  READY   STATUS    RESTARTS   AGE
    authorino-78c5679f94-abc12                            1/1     Running   0          5d
    dns-operator-controller-manager-5d4789f6-x1y2z        1/1     Running   0          5d
    kuadrant-operator-controller-manager-8495bc4d-98765   1/1     Running   0          5d
    limitador-67f89bc5d4-z9w8v                            1/1     Running   0          5d

    • If the authorino-* pod shows CrashLoopBackOff, it either cannot reach your OIDC issuer or has an invalid configuration.
    • If the kuadrant-operator-controller-manager-* pod is down, any changes you make to your AuthPolicy CR cannot be applied to the Gateway object because the controller pod does not reconcile your AuthPolicy CR.
12. Remedy pod issues as required.

13. Check the Authorino Operator logs for CEL errors by running the following command:

    $ oc logs -n <kuadrant_system> -l authorino-resource=authorino | grep -i cel

    Replace <kuadrant_system> with the namespace where your Connectivity Link is installed.

14. Verify the CEL syntax in your authorization rules.
15. Check that referenced fields exist, such as auth.identity.groups.
16. Verify that the JSON or data object coming back from the source matches the exact fields you referenced in your Authorino configuration.
17. Test CEL expression syntax using online validators.
18. Add persistent logging to understand the CEL evaluation context.