Networking

1. Create a Secret resource containing the custom certificate in the openshift-ingress namespace using the tls.crt and tls.key files.

   $ oc --namespace openshift-ingress create secret tls custom-certs-default --cert=tls.crt --key=tls.key

2. Update the IngressController CR to reference the new certificate secret:

   $ oc patch --type=merge --namespace openshift-ingress-operator ingresscontrollers/default \
     --patch '{"spec":{"defaultCertificate":{"name":"custom-certs-default"}}}'

3. Verify the update was effective:

   $ echo Q |\
     openssl s_client -connect console-openshift-console.apps.<domain>:443 -showcerts 2>/dev/null |\
     openssl x509 -noout -subject -issuer -enddate

   where:

   <domain>

   Specifies the base domain name for your cluster.

   Example output

   subject=C = US, ST = NC, L = Raleigh, O = RH, OU = OCP4, CN = *.apps.example.com
   issuer=C = US, ST = NC, L = Raleigh, O = RH, OU = OCP4, CN = example.com
   notAfter=May 10 08:32:45 2022 GM

   Tip

   You can alternatively apply the following YAML to set a custom default certificate:

   apiVersion: operator.openshift.io/v1
   kind: IngressController
   metadata:
     name: default
     namespace: openshift-ingress-operator
   spec:
     defaultCertificate:
       name: custom-certs-default

   The certificate secret name should match the value used to update the CR.

Procedure

1. View the current number of available replicas for the default IngressController:

   $ oc get -n openshift-ingress-operator ingresscontrollers/default -o jsonpath='{$.status.availableReplicas}'

   Example output

   2

2. Scale the default IngressController to the desired number of replicas using the oc patch command. The following example scales the default IngressController to 3 replicas:

   $ oc patch -n openshift-ingress-operator ingresscontroller/default --patch '{"spec":{"replicas": 3}}' --type=merge

   Example output

   ingresscontroller.operator.openshift.io/default patched

3. Verify that the default IngressController scaled to the number of replicas that you specified:

   $ oc get -n openshift-ingress-operator ingresscontrollers/default -o jsonpath='{$.status.availableReplicas}'

   Example output

   3

   Tip

   You can alternatively apply the following YAML to scale an Ingress Controller to three replicas:

   apiVersion: operator.openshift.io/v1
   kind: IngressController
   metadata:
     name: default
     namespace: openshift-ingress-operator
   spec:
     replicas: 3               1

   1

   If you need a different amount of replicas, change the replicas value.

Procedure

1. Create an IngressController custom resource (CR) in a file named <name>-ingress-controller.yaml, such as in the following example:

   apiVersion: operator.openshift.io/v1
   kind: IngressController
   metadata:
     namespace: openshift-ingress-operator
     name: <name> 1
   spec:
     domain: <domain> 2
     endpointPublishingStrategy:
       type: LoadBalancerService
       loadBalancer:
         scope: Internal 3

   1

   Replace <name> with a name for the IngressController object.

   2

   Specify the domain for the application published by the controller.

   3

   Specify a value of Internal to use an internal load balancer.

2. Create the Ingress Controller defined in the previous step by running the following command:

   $ oc create -f <name>-ingress-controller.yaml 1

   1

   Replace <name> with the name of the IngressController object.

3. Optional: Confirm that the Ingress Controller was created by running the following command:

   $ oc --all-namespaces=true get ingresscontrollers

Procedure

1. Configure the Ingress Controller resource to allow global access.

   Note

   You can also create an Ingress Controller and specify the global access option.

   1. Configure the Ingress Controller resource:

      $ oc -n openshift-ingress-operator edit ingresscontroller/default

   2. Edit the YAML file:

      Sample clientAccess configuration to Global

        spec:
          endpointPublishingStrategy:
            loadBalancer:
              providerParameters:
                gcp:
                  clientAccess: Global 1
                type: GCP
              scope: Internal
            type: LoadBalancerService

      1

      Set gcp.clientAccess to Global.

   3. Save the file to apply the changes.

2. Run the following command to verify that the service allows global access:

   $ oc -n openshift-ingress edit svc/router-default -o yaml

   The output shows that global access is enabled for GCP with the annotation, networking.gke.io/internal-load-balancer-allow-global-access.

Procedure

1. Configure the default Ingress Controller for your cluster to be internal by deleting and recreating it.

   $ oc replace --force --wait --filename - <<EOF
   apiVersion: operator.openshift.io/v1
   kind: IngressController
   metadata:
     namespace: openshift-ingress-operator
     name: default
   spec:
     endpointPublishingStrategy:
       type: LoadBalancerService
       loadBalancer:
         scope: Internal
   EOF

Procedure

1. Configure the wildcard policy.

   1. Use the following command to edit the IngressController resource:

      $ oc edit IngressController

   2. Under spec, set the wildcardPolicy field to WildcardsDisallowed or WildcardsAllowed:

      spec:
        routeAdmission:
          wildcardPolicy: WildcardsDisallowed # or WildcardsAllowed

Procedure

1. Configure the HTTPHeaders field for the Ingress Controller.

   1. Use the following command to edit the IngressController resource:

      $ oc edit IngressController

   2. Under spec, set the HTTPHeaders policy field to Append, Replace, IfNone, or Never:

      apiVersion: operator.openshift.io/v1
      kind: IngressController
      metadata:
        name: default
        namespace: openshift-ingress-operator
      spec:
        httpHeaders:
          forwardedHeaderPolicy: Append

Procedure

1. Edit the Ingress Controller resource:

   $ oc -n openshift-ingress-operator edit ingresscontroller/default

2. Set the PROXY configuration:

   • If your Ingress Controller uses the hostNetwork endpoint publishing strategy type, set the spec.endpointPublishingStrategy.hostNetwork.protocol subfield to PROXY:

     Sample hostNetwork configuration to PROXY

       spec:
         endpointPublishingStrategy:
           hostNetwork:
             protocol: PROXY
           type: HostNetwork

   • If your Ingress Controller uses the NodePortService endpoint publishing strategy type, set the spec.endpointPublishingStrategy.nodePort.protocol subfield to PROXY:

     Sample nodePort configuration to PROXY

       spec:
         endpointPublishingStrategy:
           nodePort:
             protocol: PROXY
           type: NodePortService

Procedure

1. Configure the appsDomain field by specifying an alternative default domain for user-created routes.

   1. Edit the ingress cluster resource:

      $ oc edit ingresses.config/cluster -o yaml

   2. Edit the YAML file:

      Sample appsDomain configuration to test.example.com

      apiVersion: config.openshift.io/v1
      kind: Ingress
      metadata:
        name: cluster
      spec:
        domain: apps.example.com            1
        appsDomain: <test.example.com>      2

      1

      Specifies the default domain. You cannot modify the default domain after installation.

      2

      Optional: Domain for OpenShift Container Platform infrastructure to use for application routes. Instead of the default prefix, apps, you can use an alternative prefix like test.

2. Verify that an existing route contains the domain name specified in the appsDomain field by exposing the route and verifying the route domain change:

   Note

   Wait for the openshift-apiserver finish rolling updates before exposing the route.

   1. Expose the route:

      $ oc expose service hello-openshift
      route.route.openshift.io/hello-openshift exposed

      Example output:

      $ oc get routes
      NAME              HOST/PORT                                   PATH   SERVICES          PORT       TERMINATION   WILDCARD
      hello-openshift   hello_openshift-<my_project>.test.example.com
      hello-openshift   8080-tcp                 None

Procedure

1. Configure the httpCompression field for the Ingress Controller.

   1. Use the following command to edit the IngressController resource:

      $ oc edit -n openshift-ingress-operator ingresscontrollers/default

   2. Under spec, set the httpCompression policy field to mimeTypes and specify a list of MIME types that should have compression applied:

      apiVersion: operator.openshift.io/v1
      kind: IngressController
      metadata:
        name: default
        namespace: openshift-ingress-operator
      spec:
        httpCompression:
          mimeTypes:
          - "text/html"
          - "text/css; charset=utf-8"
          - "application/json"
         ...

Procedure

1. Get the router pod name by running the following command:

   $ oc get pods -n openshift-ingress

   Example output

   NAME                              READY   STATUS    RESTARTS   AGE
   router-default-76bfffb66c-46qwp   1/1     Running   0          11h

2. Get the router’s username and password, which the router pod stores in the /var/lib/haproxy/conf/metrics-auth/statsUsername and /var/lib/haproxy/conf/metrics-auth/statsPassword files:

   1. Get the username by running the following command:

      $ oc rsh <router_pod_name> cat metrics-auth/statsUsername

   2. Get the password by running the following command:

      $ oc rsh <router_pod_name> cat metrics-auth/statsPassword

3. Get the router IP and metrics certificates by running the following command:

   $ oc describe pod <router_pod>

4. Get the raw statistics in Prometheus format by running the following command:

   $ curl -u <user>:<password> http://<router_IP>:<stats_port>/metrics

5. Access the metrics securely by running the following command:

   $ curl -u user:password https://<router_IP>:<stats_port>/metrics -k

6. Access the default stats port, 1936, by running the following command:

   $ curl -u <user>:<password> http://<router_IP>:<stats_port>/metrics

   Example 6.1. Example output

   …​ # HELP haproxy_backend_connections_total Total number of connections. # TYPE haproxy_backend_connections_total gauge haproxy_backend_connections_total{backend="http",namespace="default",route="hello-route"} 0 haproxy_backend_connections_total{backend="http",namespace="default",route="hello-route-alt"} 0 haproxy_backend_connections_total{backend="http",namespace="default",route="hello-route01"} 0 …​ # HELP haproxy_exporter_server_threshold Number of servers tracked and the current threshold value. # TYPE haproxy_exporter_server_threshold gauge haproxy_exporter_server_threshold{type="current"} 11 haproxy_exporter_server_threshold{type="limit"} 500 …​ # HELP haproxy_frontend_bytes_in_total Current total of incoming bytes. # TYPE haproxy_frontend_bytes_in_total gauge haproxy_frontend_bytes_in_total{frontend="fe_no_sni"} 0 haproxy_frontend_bytes_in_total{frontend="fe_sni"} 0 haproxy_frontend_bytes_in_total{frontend="public"} 119070 …​ # HELP haproxy_server_bytes_in_total Current total of incoming bytes. # TYPE haproxy_server_bytes_in_total gauge haproxy_server_bytes_in_total{namespace="",pod="",route="",server="fe_no_sni",service=""} 0 haproxy_server_bytes_in_total{namespace="",pod="",route="",server="fe_sni",service=""} 0 haproxy_server_bytes_in_total{namespace="default",pod="docker-registry-5-nk5fz",route="docker-registry",server="10.130.0.89:5000",service="docker-registry"} 0 haproxy_server_bytes_in_total{namespace="default",pod="hello-rc-vkjqx",route="hello-route",server="10.130.0.90:8080",service="hello-svc-1"} 0 …​

7. Launch the stats window by entering the following URL in a browser:

   http://<user>:<password>@<router_IP>:<stats_port>

8. Optional: Get the stats in CSV format by entering the following URL in a browser:

   http://<user>:<password>@<router_ip>:1936/metrics;csv

Procedure

1. Create a config map named my-custom-error-code-pages in the openshift-config namespace:

   $ oc -n openshift-config create configmap my-custom-error-code-pages \
   --from-file=error-page-503.http \
   --from-file=error-page-404.http

   Important

   If you do not specify the correct format for the custom error code response page, a router pod outage occurs. To resolve this outage, you must delete or correct the config map and delete the affected router pods so they can be recreated with the correct information.

2. Patch the Ingress Controller to reference the my-custom-error-code-pages config map by name:

   $ oc patch -n openshift-ingress-operator ingresscontroller/default --patch '{"spec":{"httpErrorCodePages":{"name":"my-custom-error-code-pages"}}}' --type=merge

   The Ingress Operator copies the my-custom-error-code-pages config map from the openshift-config namespace to the openshift-ingress namespace. The Operator names the config map according to the pattern, <your_ingresscontroller_name>-errorpages, in the openshift-ingress namespace.

3. Display the copy:

   $ oc get cm default-errorpages -n openshift-ingress

   Example output

   NAME                       DATA   AGE
   default-errorpages         2      25s  1

   1

   The example config map name is default-errorpages because the default Ingress Controller custom resource (CR) was patched.

4. Confirm that the config map containing the custom error response page mounts on the router volume where the config map key is the filename that has the custom HTTP error code response:

   • For 503 custom HTTP custom error code response:

     $ oc -n openshift-ingress rsh <router_pod> cat /var/lib/haproxy/conf/error_code_pages/error-page-503.http

   • For 404 custom HTTP custom error code response:

     $ oc -n openshift-ingress rsh <router_pod> cat /var/lib/haproxy/conf/error_code_pages/error-page-404.http

1. Create a test project and application:

    $ oc new-project test-ingress

   $ oc new-app django-psql-example

2. For 503 custom http error code response:

   1. Stop all the pods for the application.

   2. Run the following curl command or visit the route hostname in the browser:

      $ curl -vk <route_hostname>

3. For 404 custom http error code response:

   1. Visit a non-existent route or an incorrect route.

   2. Run the following curl command or visit the route hostname in the browser:

      $ curl -vk <route_hostname>

4. Check if the errorfile attribute is properly in the haproxy.config file:

   $ oc -n openshift-ingress rsh <router> cat /var/lib/haproxy/conf/haproxy.config | grep errorfile

Procedure

1. Create the PtpConfig resource. For example:

   1. Save the following YAML in the grandmaster-clock-ptp-config.yaml file:

      Example PTP grandmaster clock configuration

      apiVersion: ptp.openshift.io/v1
      kind: PtpConfig
      metadata:
        name: grandmaster-clock
        namespace: openshift-ptp
        annotations: {}
      spec:
        profile:
          - name: grandmaster-clock
            # The interface name is hardware-specific
            interface: $interface
            ptp4lOpts: "-2"
            phc2sysOpts: "-a -r -r -n 24"
            ptpSchedulingPolicy: SCHED_FIFO
            ptpSchedulingPriority: 10
            ptpSettings:
              logReduce: "true"
            ptp4lConf: |
              [global]
              #
              # Default Data Set
              #
              twoStepFlag 1
              slaveOnly 0
              priority1 128
              priority2 128
              domainNumber 24
              #utc_offset 37
              clockClass 255
              clockAccuracy 0xFE
              offsetScaledLogVariance 0xFFFF
              free_running 0
              freq_est_interval 1
              dscp_event 0
              dscp_general 0
              dataset_comparison G.8275.x
              G.8275.defaultDS.localPriority 128
              #
              # Port Data Set
              #
              logAnnounceInterval -3
              logSyncInterval -4
              logMinDelayReqInterval -4
              logMinPdelayReqInterval -4
              announceReceiptTimeout 3
              syncReceiptTimeout 0
              delayAsymmetry 0
              fault_reset_interval -4
              neighborPropDelayThresh 20000000
              masterOnly 0
              G.8275.portDS.localPriority 128
              #
              # Run time options
              #
              assume_two_step 0
              logging_level 6
              path_trace_enabled 0
              follow_up_info 0
              hybrid_e2e 0
              inhibit_multicast_service 0
              net_sync_monitor 0
              tc_spanning_tree 0
              tx_timestamp_timeout 50
              unicast_listen 0
              unicast_master_table 0
              unicast_req_duration 3600
              use_syslog 1
              verbose 0
              summary_interval 0
              kernel_leap 1
              check_fup_sync 0
              clock_class_threshold 7
              #
              # Servo Options
              #
              pi_proportional_const 0.0
              pi_integral_const 0.0
              pi_proportional_scale 0.0
              pi_proportional_exponent -0.3
              pi_proportional_norm_max 0.7
              pi_integral_scale 0.0
              pi_integral_exponent 0.4
              pi_integral_norm_max 0.3
              step_threshold 2.0
              first_step_threshold 0.00002
              max_frequency 900000000
              clock_servo pi
              sanity_freq_limit 200000000
              ntpshm_segment 0
              #
              # Transport options
              #
              transportSpecific 0x0
              ptp_dst_mac 01:1B:19:00:00:00
              p2p_dst_mac 01:80:C2:00:00:0E
              udp_ttl 1
              udp6_scope 0x0E
              uds_address /var/run/ptp4l
              #
              # Default interface options
              #
              clock_type OC
              network_transport L2
              delay_mechanism E2E
              time_stamping hardware
              tsproc_mode filter
              delay_filter moving_median
              delay_filter_length 10
              egressLatency 0
              ingressLatency 0
              boundary_clock_jbod 0
              #
              # Clock description
              #
              productDescription ;;
              revisionData ;;
              manufacturerIdentity 00:00:00
              userDescription ;
              timeSource 0xA0
        recommend:
          - profile: grandmaster-clock
            priority: 4
            match:
              - nodeLabel: "node-role.kubernetes.io/$mcp"

   2. Create the CR by running the following command:

      $ oc create -f grandmaster-clock-ptp-config.yaml

Verification

1. Check that the PtpConfig profile is applied to the node.

   1. Get the list of pods in the openshift-ptp namespace by running the following command:

      $ oc get pods -n openshift-ptp -o wide

      Example output

      NAME                          READY   STATUS    RESTARTS   AGE     IP             NODE
      linuxptp-daemon-74m2g         3/3     Running   3          4d15h   10.16.230.7    compute-1.example.com
      ptp-operator-5f4f48d7c-x7zkf  1/1     Running   1          4d15h   10.128.1.145   compute-1.example.com

   2. Check that the profile is correct. Examine the logs of the linuxptp daemon that corresponds to the node you specified in the PtpConfig profile. Run the following command:

      $ oc logs linuxptp-daemon-74m2g -n openshift-ptp -c linuxptp-daemon-container

      Example output

      ts2phc[94980.334]: [ts2phc.0.config] nmea delay: 98690975 ns
      ts2phc[94980.334]: [ts2phc.0.config] ens3f0 extts index 0 at 1676577329.999999999 corr 0 src 1676577330.901342528 diff -1
      ts2phc[94980.334]: [ts2phc.0.config] ens3f0 master offset         -1 s2 freq      -1
      ts2phc[94980.441]: [ts2phc.0.config] nmea sentence: GNRMC,195453.00,A,4233.24427,N,07126.64420,W,0.008,,160223,,,A,V
      phc2sys[94980.450]: [ptp4l.0.config] CLOCK_REALTIME phc offset       943 s2 freq  -89604 delay    504
      phc2sys[94980.512]: [ptp4l.0.config] CLOCK_REALTIME phc offset      1000 s2 freq  -89264 delay    474

Procedure

1. Create the following PtpConfig CR, and then save the YAML in the ordinary-clock-ptp-config.yaml file.

   Example PTP ordinary clock configuration

   apiVersion: ptp.openshift.io/v1
   kind: PtpConfig
   metadata:
     name: ordinary-clock
     namespace: openshift-ptp
     annotations: {}
   spec:
     profile:
       - name: ordinary-clock
         # The interface name is hardware-specific
         interface: $interface
         ptp4lOpts: "-2 -s"
         phc2sysOpts: "-a -r -n 24"
         ptpSchedulingPolicy: SCHED_FIFO
         ptpSchedulingPriority: 10
         ptpSettings:
           logReduce: "true"
         ptp4lConf: |
           [global]
           #
           # Default Data Set
           #
           twoStepFlag 1
           slaveOnly 1
           priority1 128
           priority2 128
           domainNumber 24
           #utc_offset 37
           clockClass 255
           clockAccuracy 0xFE
           offsetScaledLogVariance 0xFFFF
           free_running 0
           freq_est_interval 1
           dscp_event 0
           dscp_general 0
           dataset_comparison G.8275.x
           G.8275.defaultDS.localPriority 128
           #
           # Port Data Set
           #
           logAnnounceInterval -3
           logSyncInterval -4
           logMinDelayReqInterval -4
           logMinPdelayReqInterval -4
           announceReceiptTimeout 3
           syncReceiptTimeout 0
           delayAsymmetry 0
           fault_reset_interval -4
           neighborPropDelayThresh 20000000
           masterOnly 0
           G.8275.portDS.localPriority 128
           #
           # Run time options
           #
           assume_two_step 0
           logging_level 6
           path_trace_enabled 0
           follow_up_info 0
           hybrid_e2e 0
           inhibit_multicast_service 0
           net_sync_monitor 0
           tc_spanning_tree 0
           tx_timestamp_timeout 50
           unicast_listen 0
           unicast_master_table 0
           unicast_req_duration 3600
           use_syslog 1
           verbose 0
           summary_interval 0
           kernel_leap 1
           check_fup_sync 0
           clock_class_threshold 7
           #
           # Servo Options
           #
           pi_proportional_const 0.0
           pi_integral_const 0.0
           pi_proportional_scale 0.0
           pi_proportional_exponent -0.3
           pi_proportional_norm_max 0.7
           pi_integral_scale 0.0
           pi_integral_exponent 0.4
           pi_integral_norm_max 0.3
           step_threshold 2.0
           first_step_threshold 0.00002
           max_frequency 900000000
           clock_servo pi
           sanity_freq_limit 200000000
           ntpshm_segment 0
           #
           # Transport options
           #
           transportSpecific 0x0
           ptp_dst_mac 01:1B:19:00:00:00
           p2p_dst_mac 01:80:C2:00:00:0E
           udp_ttl 1
           udp6_scope 0x0E
           uds_address /var/run/ptp4l
           #
           # Default interface options
           #
           clock_type OC
           network_transport L2
           delay_mechanism E2E
           time_stamping hardware
           tsproc_mode filter
           delay_filter moving_median
           delay_filter_length 10
           egressLatency 0
           ingressLatency 0
           boundary_clock_jbod 0
           #
           # Clock description
           #
           productDescription ;;
           revisionData ;;
           manufacturerIdentity 00:00:00
           userDescription ;
           timeSource 0xA0
     recommend:
       - profile: ordinary-clock
         priority: 4
         match:
           - nodeLabel: "node-role.kubernetes.io/$mcp"

   Table 13.1. PTP ordinary clock CR configuration options

   Custom resource field	Description
   name	The name of the PtpConfig CR.
   profile	Specify an array of one or more profile objects. Each profile must be uniquely named.
   interface	Specify the network interface to be used by the ptp4l service, for example ens787f1.
   ptp4lOpts	Specify system config options for the ptp4l service, for example -2 to select the IEEE 802.3 network transport. The options should not include the network interface name -i <interface> and service config file -f /etc/ptp4l.conf because the network interface name and the service config file are automatically appended. Append --summary_interval -4 to use PTP fast events with this interface.
   phc2sysOpts	Specify system config options for the phc2sys service. If this field is empty, the PTP Operator does not start the phc2sys service. For Intel Columbiaville 800 Series NICs, set phc2sysOpts options to -a -r -m -n 24 -N 8 -R 16. -m prints messages to stdout. The linuxptp-daemon DaemonSet parses the logs and generates Prometheus metrics.
   ptp4lConf	Specify a string that contains the configuration to replace the default /etc/ptp4l.conf file. To use the default configuration, leave the field empty.
   tx_timestamp_timeout	For Intel Columbiaville 800 Series NICs, set tx_timestamp_timeout to 50.
   boundary_clock_jbod	For Intel Columbiaville 800 Series NICs, set boundary_clock_jbod to 0.
   ptpSchedulingPolicy	Scheduling policy for ptp4l and phc2sys processes. Default value is SCHED_OTHER. Use SCHED_FIFO on systems that support FIFO scheduling.
   ptpSchedulingPriority	Integer value from 1-65 used to set FIFO priority for ptp4l and phc2sys processes when ptpSchedulingPolicy is set to SCHED_FIFO. The ptpSchedulingPriority field is not used when ptpSchedulingPolicy is set to SCHED_OTHER.
   ptpClockThreshold	Optional. If ptpClockThreshold is not present, default values are used for the ptpClockThreshold fields. ptpClockThreshold configures how long after the PTP master clock is disconnected before PTP events are triggered. holdOverTimeout is the time value in seconds before the PTP clock event state changes to FREERUN when the PTP master clock is disconnected. The maxOffsetThreshold and minOffsetThreshold settings configure offset values in nanoseconds that compare against the values for CLOCK_REALTIME (phc2sys) or master offset (ptp4l). When the ptp4l or phc2sys offset value is outside this range, the PTP clock state is set to FREERUN. When the offset value is within this range, the PTP clock state is set to LOCKED.
   recommend	Specify an array of one or more recommend objects that define rules on how the profile should be applied to nodes.
   .recommend.profile	Specify the .recommend.profile object name defined in the profile section.
   .recommend.priority	Set .recommend.priority to 0 for ordinary clock.
   .recommend.match	Specify .recommend.match rules with nodeLabel or nodeName.
   .recommend.match.nodeLabel	Update nodeLabel with the key of node.Labels from the node object by using the oc get nodes --show-labels command. For example: node-role.kubernetes.io/worker.
   .recommend.match.nodeLabel	Update nodeName with value of node.Name from the node object by using the oc get nodes command. For example: compute-0.example.com.

2. Create the PtpConfig CR by running the following command:

   $ oc create -f ordinary-clock-ptp-config.yaml

Verification

1. Check that the PtpConfig profile is applied to the node.

   1. Get the list of pods in the openshift-ptp namespace by running the following command:

      $ oc get pods -n openshift-ptp -o wide

      Example output

      NAME                            READY   STATUS    RESTARTS   AGE   IP               NODE
      linuxptp-daemon-4xkbb           1/1     Running   0          43m   10.1.196.24      compute-0.example.com
      linuxptp-daemon-tdspf           1/1     Running   0          43m   10.1.196.25      compute-1.example.com
      ptp-operator-657bbb64c8-2f8sj   1/1     Running   0          43m   10.129.0.61      control-plane-1.example.com

   2. Check that the profile is correct. Examine the logs of the linuxptp daemon that corresponds to the node you specified in the PtpConfig profile. Run the following command:

      $ oc logs linuxptp-daemon-4xkbb -n openshift-ptp -c linuxptp-daemon-container

      Example output

      I1115 09:41:17.117596 4143292 daemon.go:107] in applyNodePTPProfile
      I1115 09:41:17.117604 4143292 daemon.go:109] updating NodePTPProfile to:
      I1115 09:41:17.117607 4143292 daemon.go:110] ------------------------------------
      I1115 09:41:17.117612 4143292 daemon.go:102] Profile Name: profile1
      I1115 09:41:17.117616 4143292 daemon.go:102] Interface: ens787f1
      I1115 09:41:17.117620 4143292 daemon.go:102] Ptp4lOpts: -2 -s
      I1115 09:41:17.117623 4143292 daemon.go:102] Phc2sysOpts: -a -r -n 24
      I1115 09:41:17.117626 4143292 daemon.go:116] ------------------------------------

Procedure

1. Create the following PtpConfig CR, and then save the YAML in the boundary-clock-ptp-config.yaml file.

   Example PTP boundary clock configuration

   apiVersion: ptp.openshift.io/v1
   kind: PtpConfig
   metadata:
     name: boundary-clock
     namespace: openshift-ptp
     annotations: {}
   spec:
     profile:
       - name: boundary-clock
         ptp4lOpts: "-2"
         phc2sysOpts: "-a -r -n 24"
         ptpSchedulingPolicy: SCHED_FIFO
         ptpSchedulingPriority: 10
         ptpSettings:
           logReduce: "true"
         ptp4lConf: |
           # The interface name is hardware-specific
           [$iface_slave]
           masterOnly 0
           [$iface_master_1]
           masterOnly 1
           [$iface_master_2]
           masterOnly 1
           [$iface_master_3]
           masterOnly 1
           [global]
           #
           # Default Data Set
           #
           twoStepFlag 1
           slaveOnly 0
           priority1 128
           priority2 128
           domainNumber 24
           #utc_offset 37
           clockClass 248
           clockAccuracy 0xFE
           offsetScaledLogVariance 0xFFFF
           free_running 0
           freq_est_interval 1
           dscp_event 0
           dscp_general 0
           dataset_comparison G.8275.x
           G.8275.defaultDS.localPriority 128
           #
           # Port Data Set
           #
           logAnnounceInterval -3
           logSyncInterval -4
           logMinDelayReqInterval -4
           logMinPdelayReqInterval -4
           announceReceiptTimeout 3
           syncReceiptTimeout 0
           delayAsymmetry 0
           fault_reset_interval -4
           neighborPropDelayThresh 20000000
           masterOnly 0
           G.8275.portDS.localPriority 128
           #
           # Run time options
           #
           assume_two_step 0
           logging_level 6
           path_trace_enabled 0
           follow_up_info 0
           hybrid_e2e 0
           inhibit_multicast_service 0
           net_sync_monitor 0
           tc_spanning_tree 0
           tx_timestamp_timeout 50
           unicast_listen 0
           unicast_master_table 0
           unicast_req_duration 3600
           use_syslog 1
           verbose 0
           summary_interval 0
           kernel_leap 1
           check_fup_sync 0
           clock_class_threshold 135
           #
           # Servo Options
           #
           pi_proportional_const 0.0
           pi_integral_const 0.0
           pi_proportional_scale 0.0
           pi_proportional_exponent -0.3
           pi_proportional_norm_max 0.7
           pi_integral_scale 0.0
           pi_integral_exponent 0.4
           pi_integral_norm_max 0.3
           step_threshold 2.0
           first_step_threshold 0.00002
           max_frequency 900000000
           clock_servo pi
           sanity_freq_limit 200000000
           ntpshm_segment 0
           #
           # Transport options
           #
           transportSpecific 0x0
           ptp_dst_mac 01:1B:19:00:00:00
           p2p_dst_mac 01:80:C2:00:00:0E
           udp_ttl 1
           udp6_scope 0x0E
           uds_address /var/run/ptp4l
           #
           # Default interface options
           #
           clock_type BC
           network_transport L2
           delay_mechanism E2E
           time_stamping hardware
           tsproc_mode filter
           delay_filter moving_median
           delay_filter_length 10
           egressLatency 0
           ingressLatency 0
           boundary_clock_jbod 0
           #
           # Clock description
           #
           productDescription ;;
           revisionData ;;
           manufacturerIdentity 00:00:00
           userDescription ;
           timeSource 0xA0
     recommend:
       - profile: boundary-clock
         priority: 4
         match:
           - nodeLabel: "node-role.kubernetes.io/$mcp"

   Table 13.2. PTP boundary clock CR configuration options

   Custom resource field	Description
   name	The name of the PtpConfig CR.
   profile	Specify an array of one or more profile objects.
   name	Specify the name of a profile object which uniquely identifies a profile object.
   ptp4lOpts	Specify system config options for the ptp4l service. The options should not include the network interface name -i <interface> and service config file -f /etc/ptp4l.conf because the network interface name and the service config file are automatically appended.
   ptp4lConf	Specify the required configuration to start ptp4l as boundary clock. For example, ens1f0 synchronizes from a grandmaster clock and ens1f3 synchronizes connected devices.
   <interface_1>	The interface that receives the synchronization clock.
   <interface_2>	The interface that sends the synchronization clock.
   tx_timestamp_timeout	For Intel Columbiaville 800 Series NICs, set tx_timestamp_timeout to 50.
   boundary_clock_jbod	For Intel Columbiaville 800 Series NICs, ensure boundary_clock_jbod is set to 0. For Intel Fortville X710 Series NICs, ensure boundary_clock_jbod is set to 1.
   phc2sysOpts	Specify system config options for the phc2sys service. If this field is empty, the PTP Operator does not start the phc2sys service.
   ptpSchedulingPolicy	Scheduling policy for ptp4l and phc2sys processes. Default value is SCHED_OTHER. Use SCHED_FIFO on systems that support FIFO scheduling.
   ptpSchedulingPriority	Integer value from 1-65 used to set FIFO priority for ptp4l and phc2sys processes when ptpSchedulingPolicy is set to SCHED_FIFO. The ptpSchedulingPriority field is not used when ptpSchedulingPolicy is set to SCHED_OTHER.
   ptpClockThreshold	Optional. If ptpClockThreshold is not present, default values are used for the ptpClockThreshold fields. ptpClockThreshold configures how long after the PTP master clock is disconnected before PTP events are triggered. holdOverTimeout is the time value in seconds before the PTP clock event state changes to FREERUN when the PTP master clock is disconnected. The maxOffsetThreshold and minOffsetThreshold settings configure offset values in nanoseconds that compare against the values for CLOCK_REALTIME (phc2sys) or master offset (ptp4l). When the ptp4l or phc2sys offset value is outside this range, the PTP clock state is set to FREERUN. When the offset value is within this range, the PTP clock state is set to LOCKED.
   recommend	Specify an array of one or more recommend objects that define rules on how the profile should be applied to nodes.
   .recommend.profile	Specify the .recommend.profile object name defined in the profile section.
   .recommend.priority	Specify the priority with an integer value between 0 and 99. A larger number gets lower priority, so a priority of 99 is lower than a priority of 10. If a node can be matched with multiple profiles according to rules defined in the match field, the profile with the higher priority is applied to that node.
   .recommend.match	Specify .recommend.match rules with nodeLabel or nodeName.
   .recommend.match.nodeLabel	Update nodeLabel with the key of node.Labels from the node object by using the oc get nodes --show-labels command. For example: node-role.kubernetes.io/worker.
   .recommend.match.nodeLabel	Update nodeName with value of node.Name from the node object by using the oc get nodes command. For example: compute-0.example.com.

2. Create the CR by running the following command:

   $ oc create -f boundary-clock-ptp-config.yaml

Verification

1. Check that the PtpConfig profile is applied to the node.

   1. Get the list of pods in the openshift-ptp namespace by running the following command:

      $ oc get pods -n openshift-ptp -o wide

      Example output

      NAME                            READY   STATUS    RESTARTS   AGE   IP               NODE
      linuxptp-daemon-4xkbb           1/1     Running   0          43m   10.1.196.24      compute-0.example.com
      linuxptp-daemon-tdspf           1/1     Running   0          43m   10.1.196.25      compute-1.example.com
      ptp-operator-657bbb64c8-2f8sj   1/1     Running   0          43m   10.129.0.61      control-plane-1.example.com

   2. Check that the profile is correct. Examine the logs of the linuxptp daemon that corresponds to the node you specified in the PtpConfig profile. Run the following command:

      $ oc logs linuxptp-daemon-4xkbb -n openshift-ptp -c linuxptp-daemon-container

      Example output

      I1115 09:41:17.117596 4143292 daemon.go:107] in applyNodePTPProfile
      I1115 09:41:17.117604 4143292 daemon.go:109] updating NodePTPProfile to:
      I1115 09:41:17.117607 4143292 daemon.go:110] ------------------------------------
      I1115 09:41:17.117612 4143292 daemon.go:102] Profile Name: profile1
      I1115 09:41:17.117616 4143292 daemon.go:102] Interface:
      I1115 09:41:17.117620 4143292 daemon.go:102] Ptp4lOpts: -2
      I1115 09:41:17.117623 4143292 daemon.go:102] Phc2sysOpts: -a -r -n 24
      I1115 09:41:17.117626 4143292 daemon.go:116] ------------------------------------

Procedure

1. Edit the PtpConfig CR profile:

   $ oc edit PtpConfig -n openshift-ptp

2. Change the ptpSchedulingPolicy and ptpSchedulingPriority fields:

   apiVersion: ptp.openshift.io/v1
   kind: PtpConfig
   metadata:
     name: <ptp_config_name>
     namespace: openshift-ptp
   ...
   spec:
     profile:
     - name: "profile1"
   ...
       ptpSchedulingPolicy: SCHED_FIFO 1
       ptpSchedulingPriority: 10 2

   1

   Scheduling policy for ptp4l and phc2sys processes. Use SCHED_FIFO on systems that support FIFO scheduling.

   2

   Required. Sets the integer value 1-65 used to configure FIFO priority for ptp4l and phc2sys processes.

3. Save and exit to apply the changes to the PtpConfig CR.

Verification

1. Get the name of the linuxptp-daemon pod and corresponding node where the PtpConfig CR has been applied:

   $ oc get pods -n openshift-ptp -o wide

   Example output

   NAME                            READY   STATUS    RESTARTS   AGE     IP            NODE
   linuxptp-daemon-gmv2n           3/3     Running   0          1d17h   10.1.196.24   compute-0.example.com
   linuxptp-daemon-lgm55           3/3     Running   0          1d17h   10.1.196.25   compute-1.example.com
   ptp-operator-3r4dcvf7f4-zndk7   1/1     Running   0          1d7h    10.129.0.61   control-plane-1.example.com

2. Check that the ptp4l process is running with the updated chrt FIFO priority:

   $ oc -n openshift-ptp logs linuxptp-daemon-lgm55 -c linuxptp-daemon-container|grep chrt

   Example output

   I1216 19:24:57.091872 1600715 daemon.go:285] /bin/chrt -f 65 /usr/sbin/ptp4l -f /var/run/ptp4l.0.config -2  --summary_interval -4 -m

1. DU application requests a subscription: The DU sends an API request to the cloud-event-proxy sidecar to create a PTP events subscription. The cloud-event-proxy sidecar creates a subscription resource.
2. cloud-event-proxy sidecar creates the subscription: The event resource is persisted by the cloud-event-proxy sidecar. The cloud-event-proxy sidecar container sends an acknowledgment with an ID and URL location to access the stored subscription resource. The sidecar creates an AMQ messaging listener protocol for the resource specified in the subscription.
3. DU application receives the PTP event notification: The cloud-event-proxy sidecar container listens to the address specified in the resource qualifier. The DU events consumer processes the message and passes it to the return URL specified in the subscription.
4. cloud-event-proxy sidecar validates the PTP event and posts it to the DU application: The cloud-event-proxy sidecar receives the event, unwraps the cloud events object to retrieve the data, and fetches the return URL to post the event back to the DU consumer application.
5. DU application uses the PTP event: The DU application events consumer receives and processes the PTP event.

Verification

1. Check that the AMQ Interconnect Operator is available and the required pods are running:

   $ oc get pods -n amq-interconnect

   Example output

   NAME                                    READY   STATUS    RESTARTS   AGE
   amq-interconnect-645db76c76-k8ghs       1/1     Running   0          23h
   interconnect-operator-5cb5fc7cc-4v7qm   1/1     Running   0          23h

2. Check that the required linuxptp-daemon PTP event producer pods are running in the openshift-ptp namespace.

   $ oc get pods -n openshift-ptp

   Example output

   NAME                     READY   STATUS    RESTARTS       AGE
   linuxptp-daemon-2t78p    3/3     Running   0              12h
   linuxptp-daemon-k8n88    3/3     Running   0              12h

Procedure

1. Modify the default PTP Operator config to enable PTP fast events.

   1. Save the following YAML in the ptp-operatorconfig.yaml file:

      apiVersion: ptp.openshift.io/v1
      kind: PtpOperatorConfig
      metadata:
        name: default
        namespace: openshift-ptp
      spec:
        daemonNodeSelector:
          node-role.kubernetes.io/worker: ""
        ptpEventConfig:
          enableEventPublisher: true 1
          transportHost: amqp://<instance_name>.<namespace>.svc.cluster.local 2

      1

      Set enableEventPublisher to true to enable PTP fast event notifications.

      2

      Set transportHost to the AMQ router that you configured where <instance_name> and <namespace> correspond to the AMQ Interconnect router instance name and namespace, for example, amqp://amq-interconnect.amq-interconnect.svc.cluster.local

   2. Update the PtpOperatorConfig CR:

      $ oc apply -f ptp-operatorconfig.yaml

2. Create a PtpConfig custom resource (CR) for the PTP enabled interface, and set the required values for ptpClockThreshold and ptp4lOpts. The following YAML illustrates the required values that you must set in the PtpConfig CR:

   spec:
     profile:
     - name: "profile1"
       interface: "enp5s0f0"
       ptp4lOpts: "-2 -s --summary_interval -4" 1
       phc2sysOpts: "-a -r -m -n 24 -N 8 -R 16" 2
       ptp4lConf: "" 3
     ptpClockThreshold: 4
       holdOverTimeout: 5
       maxOffsetThreshold: 100
       minOffsetThreshold: -100

   1

   Append --summary_interval -4 to use PTP fast events.

   2

   Required phc2sysOpts values. -m prints messages to stdout. The linuxptp-daemon DaemonSet parses the logs and generates Prometheus metrics.

   3

   Specify a string that contains the configuration to replace the default /etc/ptp4l.conf file. To use the default configuration, leave the field empty.

   4

   Optional. If the ptpClockThreshold stanza is not present, default values are used for the ptpClockThreshold fields. The stanza shows default ptpClockThreshold values. The ptpClockThreshold values configure how long after the PTP master clock is disconnected before PTP events are triggered. holdOverTimeout is the time value in seconds before the PTP clock event state changes to FREERUN when the PTP master clock is disconnected. The maxOffsetThreshold and minOffsetThreshold settings configure offset values in nanoseconds that compare against the values for CLOCK_REALTIME (phc2sys) or master offset (ptp4l). When the ptp4l or phc2sys offset value is outside this range, the PTP clock state is set to FREERUN. When the offset value is within this range, the PTP clock state is set to LOCKED.

Procedure

1. Get the list of active linuxptp-daemon pods.

   $ oc get pods -n openshift-ptp

   Example output

   NAME                    READY   STATUS    RESTARTS   AGE
   linuxptp-daemon-2t78p   3/3     Running   0          8h
   linuxptp-daemon-k8n88   3/3     Running   0          8h

2. Access the metrics for the required cloud-event-proxy container by running the following command:

   $ oc exec -it <linuxptp-daemon> -n openshift-ptp -c cloud-event-proxy -- curl 127.0.0.1:9091/metrics

   where:

   <linuxptp-daemon>

   Specifies the pod you want to query, for example, linuxptp-daemon-2t78p.

   Example output

   # HELP cne_amqp_events_published Metric to get number of events published by the transport
   # TYPE cne_amqp_events_published gauge
   cne_amqp_events_published{address="/cluster/node/compute-1.example.com/ptp/status",status="success"} 1041
   # HELP cne_amqp_events_received Metric to get number of events received  by the transport
   # TYPE cne_amqp_events_received gauge
   cne_amqp_events_received{address="/cluster/node/compute-1.example.com/ptp",status="success"} 1019
   # HELP cne_amqp_receiver Metric to get number of receiver created
   # TYPE cne_amqp_receiver gauge
   cne_amqp_receiver{address="/cluster/node/mock",status="active"} 1
   cne_amqp_receiver{address="/cluster/node/compute-1.example.com/ptp",status="active"} 1
   cne_amqp_receiver{address="/cluster/node/compute-1.example.com/redfish/event",status="active"}
   ...

Procedure

1. Enter the following command to return the list of available PTP metrics from the cloud-event-proxy sidecar container:

   $ oc exec -it <linuxptp_daemon_pod> -n openshift-ptp -c cloud-event-proxy -- curl 127.0.0.1:9091/metrics

   where:

   <linuxptp_daemon_pod>

   Specifies the pod you want to query, for example, linuxptp-daemon-2t78p.

2. Copy the name of the PTP metric you want to query from the list of returned metrics, for example, cne_amqp_events_received.
3. In the OpenShift Container Platform web console, click Observe → Metrics.
4. Paste the PTP metric into the Expression field, and click Run queries.

1. Check the name of an install plan:

   $ oc -n external-dns-operator get sub external-dns-operator -o yaml | yq '.status.installplan.name'

   Example output

   install-zcvlr

2. Check the status of an install plan, the status of an install plan must be Complete:

   $ oc -n external-dns-operator get ip <install_plan_name> -o yaml | yq .status.phase'

   Example output

   Complete

3. Use the oc get command to view the Deployment status:

   $ oc get -n external-dns-operator deployment/external-dns-operator

   Example output

   NAME                    READY     UP-TO-DATE   AVAILABLE   AGE
   external-dns-operator   1/1       1            1           23h

Procedure

1. View the logs of the External DNS Operator:

   $ oc logs -n external-dns-operator deployment/external-dns-operator -c external-dns-operator

Procedure

1. Click Operators → OperatorHub in the OpenShift Container Platform Web Console.
2. Click External DNS Operator. You can use the Filter by keyword text box or the filter list to search for External DNS Operator from the list of Operators.
3. Select the external-dns-operator namespace.
4. On the External DNS Operator page, click Install.

5. On the Install Operator page, ensure that you selected the following options:

   1. Update the channel as stable-v1.0.
   2. Installation mode as A specific name on the cluster.
   3. Installed namespace as external-dns-operator. If namespace external-dns-operator does not exist, it gets created during the Operator installation.
   4. Select Approval Strategy as Automatic or Manual. Approval Strategy is set to Automatic by default.
   5. Click Install.

Procedure

1. Check the user. The user must have access to the kube-system namespace. If you don’t have the credentials, as you can fetch the credentials from the kube-system namespace to use the cloud provider client:

   $ oc whoami

   Example output

   system:admin

2. Fetch the values from aws-creds secret present in kube-system namespace.

   $ export AWS_ACCESS_KEY_ID=$(oc get secrets aws-creds -n kube-system  --template={{.data.aws_access_key_id}} | base64 -d)
   $ export AWS_SECRET_ACCESS_KEY=$(oc get secrets aws-creds -n kube-system  --template={{.data.aws_secret_access_key}} | base64 -d)

3. Get the routes to check the domain:

   $ oc get routes --all-namespaces | grep console

   Example output

   openshift-console          console             console-openshift-console.apps.testextdnsoperator.apacshift.support                       console             https   reencrypt/Redirect     None
   openshift-console          downloads           downloads-openshift-console.apps.testextdnsoperator.apacshift.support                     downloads           http    edge/Redirect          None

4. Get the list of dns zones to find the one which corresponds to the previously found route’s domain:

   $ aws route53 list-hosted-zones | grep testextdnsoperator.apacshift.support

   Example output

   HOSTEDZONES	terraform	/hostedzone/Z02355203TNN1XXXX1J6O	testextdnsoperator.apacshift.support.	5

5. Create ExternalDNS resource for route source:

   $ cat <<EOF | oc create -f -
   apiVersion: externaldns.olm.openshift.io/v1alpha1
   kind: ExternalDNS
   metadata:
     name: sample-aws 1
   spec:
     domains:
     - filterType: Include   2
       matchType: Exact   3
       name: testextdnsoperator.apacshift.support 4
     provider:
       type: AWS 5
     source:  6
       type: OpenShiftRoute 7
       openshiftRouteOptions:
         routerName: default 8
   EOF

   1

   Defines the name of external DNS resource.

   2

   By default all hosted zones are selected as potential targets. You can include a hosted zone that you need.

   3

   The matching of the target zone’s domain has to be exact (as opposed to regular expression match).

   4

   Specify the exact domain of the zone you want to update. The hostname of the routes must be subdomains of the specified domain.

   5

   Defines the AWS Route53 DNS provider.

   6

   Defines options for the source of DNS records.

   7

   Defines OpenShift route resource as the source for the DNS records which gets created in the previously specified DNS provider.

   8

   If the source is OpenShiftRoute, then you can pass the OpenShift Ingress Controller name. External DNS Operator selects the canonical hostname of that router as the target while creating CNAME record.

6. Check the records created for OCP routes using the following command:

   $ aws route53 list-resource-record-sets --hosted-zone-id Z02355203TNN1XXXX1J6O --query "ResourceRecordSets[?Type == 'CNAME']" | grep console

Procedure

1. Check the user. The user must have access to the kube-system namespace. If you don’t have the credentials, as you can fetch the credentials from the kube-system namespace to use the cloud provider client:

   $ oc whoami

   Example output

   system:admin

2. Fetch the values from azure-credentials secret present in kube-system namespace.

   $ CLIENT_ID=$(oc get secrets azure-credentials  -n kube-system  --template={{.data.azure_client_id}} | base64 -d)
   $ CLIENT_SECRET=$(oc get secrets azure-credentials  -n kube-system  --template={{.data.azure_client_secret}} | base64 -d)
   $ RESOURCE_GROUP=$(oc get secrets azure-credentials  -n kube-system  --template={{.data.azure_resourcegroup}} | base64 -d)
   $ SUBSCRIPTION_ID=$(oc get secrets azure-credentials  -n kube-system  --template={{.data.azure_subscription_id}} | base64 -d)
   $ TENANT_ID=$(oc get secrets azure-credentials  -n kube-system  --template={{.data.azure_tenant_id}} | base64 -d)

3. Login to azure with base64 decoded values:

   $ az login --service-principal -u "${CLIENT_ID}" -p "${CLIENT_SECRET}" --tenant "${TENANT_ID}"

4. Get the routes to check the domain:

   $ oc get routes --all-namespaces | grep console

   Example output

   openshift-console          console             console-openshift-console.apps.test.azure.example.com                       console             https   reencrypt/Redirect     None
   openshift-console          downloads           downloads-openshift-console.apps.test.azure.example.com                     downloads           http    edge/Redirect          None

5. Get the list of dns zones to find the one which corresponds to the previously found route’s domain:

   $ az network dns zone list --resource-group "${RESOURCE_GROUP}"

6. Create ExternalDNS resource for route source:

   apiVersion: externaldns.olm.openshift.io/v1alpha1
   kind: ExternalDNS
   metadata:
     name: sample-azure 1
   spec:
     zones:
     - "/subscriptions/1234567890/resourceGroups/test-azure-xxxxx-rg/providers/Microsoft.Network/dnszones/test.azure.example.com" 2
     provider:
       type: Azure 3
     source:
       openshiftRouteOptions: 4
         routerName: default 5
       type: OpenShiftRoute 6
   EOF

   1

   Specifies the name of External DNS CR.

   2

   Define the zone ID.

   3

   Defines the Azure DNS provider.

   4

   You can define options for the source of DNS records.

   5

   If the source is OpenShiftRoute then you can pass the OpenShift Ingress Controller name. External DNS selects the canonical hostname of that router as the target while creating CNAME record.

   6

   Defines OpenShift route resource as the source for the DNS records which gets created in the previously specified DNS provider.

7. Check the records created for OCP routes using the following command:

   $ az network dns record-set list -g "${RESOURCE_GROUP}"  -z test.azure.example.com | grep console

   Note

   To create records on private hosted zones on private Azure dns, you need to specify the private zone under zones which populates the provider type to azure-private-dns in the ExternalDNS container args.

Procedure

1. Check the user. The user must have access to the kube-system namespace. If you don’t have the credentials, as you can fetch the credentials from the kube-system namespace to use the cloud provider client:

   $ oc whoami

   Example output

   system:admin

2. Copy the value of service_account.json in gcp-credentials secret in a file encoded-gcloud.json by running the following command:

   $ oc get secret gcp-credentials -n kube-system --template='{{$v := index .data "service_account.json"}}{{$v}}' | base64 -d - > decoded-gcloud.json

3. Export Google credentials:

   $ export GOOGLE_CREDENTIALS=decoded-gcloud.json

4. Activate your account by using the following command:

   $ gcloud auth activate-service-account  <client_email as per decoded-gcloud.json> --key-file=decoded-gcloud.json

5. Set your project:

   $ gcloud config set project <project_id as per decoded-gcloud.json>

6. Get the routes to check the domain:

   $ oc get routes --all-namespaces | grep console

   Example output

   openshift-console          console             console-openshift-console.apps.test.gcp.example.com                       console             https   reencrypt/Redirect     None
   openshift-console          downloads           downloads-openshift-console.apps.test.gcp.example.com                     downloads           http    edge/Redirect          None

7. Get the list of managed zones to find the zone which corresponds to the previously found route’s domain:

   $ gcloud dns managed-zones list | grep test.gcp.example.com
   qe-cvs4g-private-zone test.gcp.example.com

8. Create ExternalDNS resource for route source:

   apiVersion: externaldns.olm.openshift.io/v1alpha1
   kind: ExternalDNS
   metadata:
     name: sample-gcp 1
   spec:
     domains:
       - filterType: Include 2
         matchType: Exact 3
         name: test.gcp.example.com 4
     provider:
       type: GCP 5
     source:
       openshiftRouteOptions: 6
         routerName: default 7
       type: OpenShiftRoute 8
   EOF

   1

   Specifies the name of External DNS CR.

   2

   By default all hosted zones are selected as potential targets. You can include a hosted zone that you need.

   3

   The matching of the target zone’s domain has to be exact (as opposed to regular expression match).

   4

   Specify the exact domain of the zone you want to update. The hostname of the routes must be subdomains of the specified domain.

   5

   Defines Google Cloud DNS provider.

   6

   You can define options for the source of DNS records.

   7

   If the source is OpenShiftRoute then you can pass the OpenShift Ingress Controller name. External DNS selects the canonical hostname of that router as the target while creating CNAME record.

   8

   Defines OpenShift route resource as the source for the DNS records which gets created in the previously specified DNS provider.

9. Check the records created for OCP routes using the following command:

   $ gcloud dns record-sets list --zone=qe-cvs4g-private-zone | grep console

Procedure

1. Create the config map to contain the CA bundle in the external-dns-operator namespace by running the following command:

   $ oc -n external-dns-operator create configmap trusted-ca

2. To inject the trusted CA bundle into the config map, add the config.openshift.io/inject-trusted-cabundle=true label to the config map by running the following command:

   $ oc -n external-dns-operator label cm trusted-ca config.openshift.io/inject-trusted-cabundle=true

3. Update the subscription of the External DNS Operator by running the following command:

   $ oc -n external-dns-operator patch subscription external-dns-operator --type='json' -p='[{"op": "add", "path": "/spec/config", "value":{"env":[{"name":"TRUSTED_CA_CONFIGMAP_NAME","value":"trusted-ca"}]}}]'

Verification

1. To create a namespace with network policies complete the following steps:

   1. Create a namespace for verification:

      $ cat <<EOF| oc create -f -
      kind: Namespace
      apiVersion: v1
      metadata:
        name: verify-audit-logging
        annotations:
          k8s.ovn.org/acl-logging: '{ "deny": "alert", "allow": "alert" }'
      EOF

      Example output

      namespace/verify-audit-logging created

   2. Enable audit logging:

      $ oc annotate namespace verify-audit-logging k8s.ovn.org/acl-logging='{ "deny": "alert", "allow": "alert" }'

      namespace/verify-audit-logging annotated

   3. Create network policies for the namespace:

      $ cat <<EOF| oc create -n verify-audit-logging -f -
      apiVersion: networking.k8s.io/v1
      kind: NetworkPolicy
      metadata:
        name: deny-all
      spec:
        podSelector:
          matchLabels:
        policyTypes:
        - Ingress
        - Egress
      ---
      apiVersion: networking.k8s.io/v1
      kind: NetworkPolicy
      metadata:
        name: allow-from-same-namespace
      spec:
        podSelector: {}
        policyTypes:
         - Ingress
         - Egress
        ingress:
          - from:
              - podSelector: {}
        egress:
          - to:
             - namespaceSelector:
                matchLabels:
                  namespace: verify-audit-logging
      EOF

      Example output

      networkpolicy.networking.k8s.io/deny-all created
      networkpolicy.networking.k8s.io/allow-from-same-namespace created

2. Create a pod for source traffic in the default namespace:

   $ cat <<EOF| oc create -n default -f -
   apiVersion: v1
   kind: Pod
   metadata:
     name: client
   spec:
     containers:
       - name: client
         image: registry.access.redhat.com/rhel7/rhel-tools
         command: ["/bin/sh", "-c"]
         args:
           ["sleep inf"]
   EOF

3. Create two pods in the verify-audit-logging namespace:

   $ for name in client server; do
   cat <<EOF| oc create -n verify-audit-logging -f -
   apiVersion: v1
   kind: Pod
   metadata:
     name: ${name}
   spec:
     containers:
       - name: ${name}
         image: registry.access.redhat.com/rhel7/rhel-tools
         command: ["/bin/sh", "-c"]
         args:
           ["sleep inf"]
   EOF
   done

   Example output

   pod/client created
   pod/server created

4. To generate traffic and produce network policy audit log entries, complete the following steps:

   1. Obtain the IP address for pod named server in the verify-audit-logging namespace:

      $ POD_IP=$(oc get pods server -n verify-audit-logging -o jsonpath='{.status.podIP}')

   2. Ping the IP address from the previous command from the pod named client in the default namespace and confirm that all packets are dropped:

      $ oc exec -it client -n default -- /bin/ping -c 2 $POD_IP

      Example output

      PING 10.128.2.55 (10.128.2.55) 56(84) bytes of data.
      
      --- 10.128.2.55 ping statistics ---
      2 packets transmitted, 0 received, 100% packet loss, time 2041ms

   3. Ping the IP address saved in the POD_IP shell environment variable from the pod named client in the verify-audit-logging namespace and confirm that all packets are allowed:

      $ oc exec -it client -n verify-audit-logging -- /bin/ping -c 2 $POD_IP

      Example output

      PING 10.128.0.86 (10.128.0.86) 56(84) bytes of data.
      64 bytes from 10.128.0.86: icmp_seq=1 ttl=64 time=2.21 ms
      64 bytes from 10.128.0.86: icmp_seq=2 ttl=64 time=0.440 ms
      
      --- 10.128.0.86 ping statistics ---
      2 packets transmitted, 2 received, 0% packet loss, time 1001ms
      rtt min/avg/max/mdev = 0.440/1.329/2.219/0.890 ms

5. Display the latest entries in the network policy audit log:

   $ for pod in $(oc get pods -n openshift-ovn-kubernetes -l app=ovnkube-node --no-headers=true | awk '{ print $1 }') ; do
       oc exec -it $pod -n openshift-ovn-kubernetes -- tail -4 /var/log/ovn/acl-audit-log.log
     done

   Example output

   Defaulting container name to ovn-controller.
   Use 'oc describe pod/ovnkube-node-hdb8v -n openshift-ovn-kubernetes' to see all of the containers in this pod.
   2021-06-13T19:33:11.590Z|00005|acl_log(ovn_pinctrl0)|INFO|name="verify-audit-logging_deny-all", verdict=drop, severity=alert: icmp,vlan_tci=0x0000,dl_src=0a:58:0a:80:02:39,dl_dst=0a:58:0a:80:02:37,nw_src=10.128.2.57,nw_dst=10.128.2.55,nw_tos=0,nw_ecn=0,nw_ttl=64,icmp_type=8,icmp_code=0
   2021-06-13T19:33:12.614Z|00006|acl_log(ovn_pinctrl0)|INFO|name="verify-audit-logging_deny-all", verdict=drop, severity=alert: icmp,vlan_tci=0x0000,dl_src=0a:58:0a:80:02:39,dl_dst=0a:58:0a:80:02:37,nw_src=10.128.2.57,nw_dst=10.128.2.55,nw_tos=0,nw_ecn=0,nw_ttl=64,icmp_type=8,icmp_code=0
   2021-06-13T19:44:10.037Z|00007|acl_log(ovn_pinctrl0)|INFO|name="verify-audit-logging_allow-from-same-namespace_0", verdict=allow, severity=alert: icmp,vlan_tci=0x0000,dl_src=0a:58:0a:80:02:3b,dl_dst=0a:58:0a:80:02:3a,nw_src=10.128.2.59,nw_dst=10.128.2.58,nw_tos=0,nw_ecn=0,nw_ttl=64,icmp_type=8,icmp_code=0
   2021-06-13T19:44:11.037Z|00008|acl_log(ovn_pinctrl0)|INFO|name="verify-audit-logging_allow-from-same-namespace_0", verdict=allow, severity=alert: icmp,vlan_tci=0x0000,dl_src=0a:58:0a:80:02:3b,dl_dst=0a:58:0a:80:02:3a,nw_src=10.128.2.59,nw_dst=10.128.2.58,nw_tos=0,nw_ecn=0,nw_ttl=64,icmp_type=8,icmp_code=0