Chapter 2. OpenShift Container Platform 3.5 Release Notes

When running more than one container in the same pod, those containers can now share memory segments because /dev/shm is shared across containers in a pod. /dev/shm is limited to 64 MB and charged under the memcg cgroup.

It is very common to add an external service to your application. This is a service that does not live on the OpenShift Container Platform. Before this feature, you had to add that endpoint to your application via IP address. These services often had pools of IPs or required access via fully qualified domain name (FQDN), which is similar to SNI in OpenShift Container Platform’s load balancer. Now, OpenShift Container Platform has a new ExternalName type that will accept an FQDN.

OpenShift Container Platform 3.5 now supports setting a nodeSelector on a namespace, with the ability to whitelist labels during the selection. In cpodNodeSelectorPluginConfig, there are four added levels of logic to allow for various control of the selection process:

Previously, the kubelet directory had to be labeled svirt_sandbox_file_t. This widens the attack surface available to containers escaping namespace isolation. Now, the kubelet directory has an SELinux context and the kubelet cannot share SELinux labels that are known, preventing the intruder from doing more damage. With OpenShift Container Platform 3.5, the use of svirt_sandbox_file_t is removed on the kublet directory and openshift.local.volumes directory. The need to carry a matching SElinux MAC is removed through the use of a :Z bind mount flag for container directories.

The Horizontal Pod Autoscaler (HPA) now ignores the CPU usage for pods that are not marked as ready, so that any extra CPU usage caused by initialization is not the cause for a scale-up. Pods in the unready state will have zero CPU usage when scaling up, and the HPA ignores them when scaling down. Pods without known metrics will have a 0% CPU usage when scaling up, and 100% CPU when scaling down. This allows for more stability during the HPA decision.

Many applications require that permissions on the mounted file (that hold their configuration) be only owner readable. Users can now specify the mode for the mount point in the downward API annotation or the configMap.

Previously, when a pod terminated, volumes used by emptyDir, configMaps, and secrets on the node were kept. Now, the kubelet deletes memory-backed volumes when the associated pods enter into terminated mode. This makes the vectors less likely to be used as an attack vector.

Controllers on the master can become unstable or cause resource issues when they continuously retry on unresponsive endpoints in the cluster. OpenShift Container Platform 3.5 introduces logic to the controllers that controls how many times they retry calls, and intelligently backs them off so that they do not hang.

These controllers now contain the logic:

The kubelet is now able to collect any attribute on the node for scheduling considerations. This feature is currently in Technology Preview.

The cluster operator must advertise a per-node opaque resource on one or more nodes. Users must request the opaque resource in pods. To advertise a new opaque integer resource, the cluster operator should submit a PATCH HTTP request to the API server to specify the available quantity in the status.capacity for a node in the cluster. After this operation, the node’s status.capacity will include a new resource. The status.allocatable field is updated automatically with the new resource asynchronously by the kubelet.

See Opaque Integer Resources for more information.

StatefulSets (currently in Technology Preview and formerly known as PetSets) offer more control over scale, network naming, handling of PVs, and deployment sequencing.

This new controller allows for the deployment of application types that require changes to their configuration or deployment count (instances) to be done in a specific and ordered manner.

Supported:

Not Supported:

See more information about web console enhancements related to this feature for OpenShift Container Platform 3.5.

Application service certificate regeneration is currently in Technology Preview.

The controller will now look over the expiry of application certificates that have used the service.alpha.openshift.io/serving-cert-secret-name API and regenerate them.

Set the service.alpha.openshift.io/serving-cert-secret-name to the name you want to use for your secret. Then, your PodSpec can mount that secret. When it is available, your pod will run. The certificate will be good for the internal service DNS name, <service.name>.<service.namespace>.svc. The certificate and key are in PEM format, stored in tls.crt and tls.key, respectively.

$ oc get secret ssl-key -o yaml

kind: Secret
metadata:
  annotations:
    service.alpha.openshift.io/expiry: 2017-03-19T08:07:07Z

When the regenerator finds a certificate that does not have the expiry annotation, it will regenerate as well. However, the existing secret is not invalidated. Therefore, no manual intervention is required to get the regeneration behavior.

See Service Serving Certificate Secrets for more information.

By default, the certificates used to govern the etcd, master, and kubelet expire after two to five years. There is now an oc command to change this expiry to be end-user configurable. This has not been implemented in the Ansible installer yet.

Use the oc adm ca command, specifying a validity period greater than two years:

# oc adm ca create-master-certs --hostnames=example.org --signer-expire-days=$[365*2+1]`

See Creating New Configuration Files for more information.

The can-i and scc-review command options allow users to better understand their permissions and security context constraints (SCC) setting in their projects. Users see a list of the commands they are allowed to execute.

The can-i command option tests scopes in terms of the user and role. The scc-review command option checks which ServiceAccount can create a pod.

scc-subject-review can check whether a user or a ServiceAccount can create a pod.

List which permissions a particular user or group has in the project by project administrator:

$ oc policy can-i --list --user=**
$ oc policy can-i --list --groups=**

List which permissions a particular user or group has in the project by system administrator role:

$ oc policy can-i --list --user=** -n <project>
$ oc policy can-i --list --groups=**  -n <project>

Determine if users can have all the combination of verbs and resources from oc policy can-i --list [--user|--groups]

$ oc policy can-i <verb> <resource> --[--user|--groups]

Test the SCCs with scopes: oc policy can-i [--user|--groups]

$ oc policy can-i <verb> <resource> [--user|--groups] --scopes=user:info
$ oc policy can-i <verb> <resource> [--user|--groups] --scopes=user:info,role:admin:<namespace>
$ oc policy can-i <verb> <resource> [--user|--groups] --scopes=role:view:*
$ oc policy can-i <verb> <resource> [--user|--groups] --scopes=role:edit:*
$ oc policy can-i <verb> <resource> [--user|--groups] --scopes=role:admin:*
$ oc policy can-i <verb> <resource> [--user|--groups] --scopes=role:admin:*:!

Test with the ignore-scopes flag in the oc policy can-i [--user|--groups] command:

$ oc policy can-i <verb> <resource> [--user|--groups] --ignore-scopes=true

The lower-level user cannot list project administrator or system administrator roles:

$ oc policy can-i --list --user project admin
$ oc policy can-i --list --user system:admin

Check whether a user or a ServiceAccount can create a pod:

$ oc policy scc-subject-review -f examples/hello-openshift/hello-pod.json
RESOURCE ALLOWED BY
Pod/hello-openshift restricted

See Authorization for more information.

Users can now test for GitHub team membership at log in.

There is now a list of one or more GitHub teams to which a user must have membership in order to authenticate. If specified, only GitHub users that are members of at least one of the listed teams will be allowed to log in. If this is not specified, then any person with a valid GitHub account can log in.

See Authentication for more information.

In OpenShift Container Platform 3.5, there is now the qualification of the Kubernetes interface for an external dynamic provisioner so that Red Hat can support a customer using a third-party storage solution such as NetApp Trident.

There is a concept of in-tree and out-of-tree with Kubernetes storage. Out-of-Tree means that it is not in the Kubernetes source tree and does not ship in Kubernetes or OpenShift Container Platform. The ability is provided post-installation. Many of the third-party storage vendors gravitate towards out-of-tree because it allows them to ship on their own schedule and own the distribution of their code.

See Available Dynamically Provisioned Plug-ins for more information.

Dynamic provisioning is now available for Azure block storage. Just like AWS and GCE, you declare the Azure cloud provider in the cloud-config file, and then create StorageClasses with the Azure block storage options and connection information.

kubernetesMasterConfig:
  ...
  apiServerArguments:
    cloud-provider:
      - "azure"
    cloud-config:
      - "/etc/azure/azure.conf"
  controllerArguments:
    cloud-provider:
      - "azure"
    cloud-config:
      - "/etc/azure/azure.conf"

kind: StorageClass
apiVersion: storage.k8s.io/v1beta1
metadata:
  name: slow
provisioner: kubernetes.io/azure-disk
parameters:
  skuName: Standard_LRS
  location: eastus
  storageAccount: azure_storage_account_name

See Dynamic Provisioning and Creating Storage Classes for more information.

With OpenShift Container Platform 3.5, the default value of the METRICS_RESOLUTION parameter is now 30 (seconds). This change was introduced to better match the cAdvisor housekeeping interval of 30 seconds (BZ#1421834).

Increasing the METRICS_RESOLUTION interval helped achieve better results in relation to how many pods can be monitored by one set of metrics pods. In OpenShift Container Platform 3.5, tests showed that OpenShift metrics collection was stable for test cases up to 25,000 monitored pods in a OpenShift Container Platform cluster.

See Scaling Cluster Metrics for more information.

Currently, up to 100 container native storage (CNS) volumes on one trusted storage pool (TSP) is supported. For more information, see the Persistent Storage Using GlusterFS.

OpenShift Container Platform 3.5 introduces multicast support. Pods can now send or receive traffic with other pods subscribed to the same multicast group.

This requires the ovs-multitenant plug-in and only works with annotated namespaces:

netnamespace.network.openshift.io/multicast-enabled: "true"

Pods in different tenants can subscribe to same multicast group, but cannot see each other’s traffic. Administrator tenant (default project) multicast traffic does not appear in other projects. Overlay (OVS and tenants) and underlay (virtual machine and a physical server) multicast traffic never mix.

See Managing Networking for more information.

In OpenShift Container Platform 3.5, there is the added ability to see the subdomain wildcard routes added in OpenShift Container Platform 3.4, create them, and edit them using the CLI.

Add the wildcard support. Enable this on the router. The default is off:

$ oc env dc/router ROUTER_ALLOW_WILDCARD_ROUTES=true

Create an application or service, then create the wildcard route:

$ oc expose svc service-unsecure --wildcard-policy=Subdomain --name=app --hostname=app.example.com

Create an edge, passthrough, or reencrypt route, for example:

$ oc create route edge edgeroute --service=service-secure --wildcard-policy=Subdomain --hostname=edge.edgeroute.com

Test the route:

$ curl --resolve edge2.edgeroute.com:443:$router_ip https://edge2.edgeroute.com -k

Support was also added to the web console.

This new feature provides the ability to create claims from different namespaces on the first directory of the path. The goal is to be able to split an application into different pods running in different namespaces.

This works by providing a way to disable the host claims is sufficient (initially). The administrator handles the routes and forbids projects from manipulating them.

For example:

Create a route in namespace 1 with:

Create a route in namespace 2 with:

namespace 2 →/bar      1
namespace 2 →/         2
namespace 2 →/bar/test 3

Network Policy (currently in Technology Preview) is an optional plug-in specification of how selections of pods are allowed to communicate with each other and other network endpoints.

Network Policy works by way of namespace isolation at the network layer using defined labels. You can also limit connections to specific ports (e.g., only TCP ports 80 and 443).

kind: NetworkPolicy
apiVersion: extensions/v1beta1
metadata:
  name: allow-http-and-https
spec:
podSelector:
ingress:
- ports:
  - protocol: TCP
    - port: 80
    - port: 443

After installing the Network Policy plug-in, an annotation must first be set on the namespace, which flips the namespace from allow all traffic to deny all traffic. At that point, you can create NetworkPolicies that define what traffic to allow. The annotation is as follows:

$ oc annotate namespace ${ns}
'net.beta.kubernetes.io/network-policy={"ingress":{"isolation":"DefaultDeny"}}'

With Network Policy in Technology Preview, not all features are available. Multi-tenant isolation is not available by default. Currently, it must be configured by creating default isolation policies for each namespace, and there is currently no clean path to upgrade or migrate from the multi-tenant plug-in.

See Managing Networking for more information.

In OpenShift Container Platform 3.5, there is added support for the K8s Ingress object, a set of rules that allow inbound connections to reach cluster services.

Ingress is disabled in the router, by default. When enabled, Ingress objects are handled equivalently to routes. The precedence rules apply to both if they claim the same host name.

Example Testing Ingress Object with TLS

apiVersion: v1
kind: Secret
metadata:
  name: test-secret
data:
  tls.crt: `base64 -w 0 /some/path/tls.crt`
  tls.key: `base64 -w 0 /some/path/tls.key`

$ cat ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: test-ingress
spec:
  tls:
  - secretName: test-secret
  backend:
    serviceName: test-service
    servicePort: 8080

See Ingress Resources for more information.

Previously, oc start-build only allowed a local file to be specified, but did not allow a URL to a remote resource. Now, users can pull in artifacts via oc start-build --from-file=<some URL>.

This feature only works against GET-based endpoints that do not require authentication and use either no transport layer security (TLS), or TLS with a certificate trusted by the client. This feature does not reinvent curl. The file is downloaded by the CLI, then uploaded to the binary build endpoint.

Users now also have the ability to set environment variable when creating an object (for example, an application) from a template. Previously, this was a separate step following template creation.

Both oc new-app and oc process now support -p for parameter values. The -v flag is deprecated.

In OpenShift Container Platform 3.5, enablement materials regarding use of CI/CD pipelines with OpenShift Container Platform are improved. The complexity and number of pipeline samples provided is increased.

Support is added to oc new-app and oc new-build so that the commands are pipeline aware.

Figure 2.1. Pipelines Page

See Promoting Applications Across Environments and Creating New Applications for more information.

OpenShift Container Platform uses the following default configuration for eviction-hard.

...
kubeletArguments:
  eviction-hard:
  - memory.available<100Mi
  - nodefs.available<10%
  - nodefs.inodesFree<5%
  - imagefs.available<15%
...

See Handling Out of Resource Errors for more information.

In OpenShift Container Platform 3.5, there is now a "Run on OpenShift" experience that allows you to provide external links in the web console to deploy templates.

Use the URL pattern to select a template or image. You can customize it to have it come from separate project. The end-user is prompted for the project.

See Create From URL for more information.

There are now added service details on configuration, traffic, routes, and pods.

There is a new section highlighting routes, service and target ports, host name, and TLS. There is also a section iterating pods and their status.

Figure 2.2. Service Details View

In OpenShift Container Platform 3.5, there is now the ability to easily work with configuration data decoupled from the container image. You can:

Figure 2.3. Create a ConfigMap

Figure 2.4. Add Config Files

Users no longer have to search logs to gain a better understanding of why build failed. Individual build status messages are now updated with details that are available via the web console and the CLI.

Figure 2.5. Build Failures as Seen in the Web Console

Figure 2.6. Build Failures as Seen in the CLI

Custom resource listing and details pages for StatefulSets (formerly known as PetSets) is now available. Users can get details of all StatefulSets, including deployments and replica sets.

See the Web Console documentation for more information.

To apply this update, run the following on all hosts where you intend to initiate Ansible-based installation or upgrade procedures:

# yum update atomic-openshift-utils

To upgrade an existing OpenShift Container Platform 3.4 or 3.5 cluster to this latest release, use the automated upgrade playbook. See Performing Automated In-place Cluster Upgrades for instructions.

To upgrade an existing OpenShift Container Platform 3.4 or 3.5 cluster to this latest release, use the automated upgrade playbook. See Performing Automated In-place Cluster Upgrades for instructions.

To upgrade an existing OpenShift Container Platform 3.4 or 3.5 cluster to this latest release, use the automated upgrade playbook. See Performing Automated In-place Cluster Upgrades for instructions.

This release updates the Red Hat Container Registry (registry.access.redhat.com) with the following images:

openshift3/ose-pod:v3.5.5.24-2
rhel7/pod-infrastructure:v3.5.5.24-2
openshift3/ose:v3.5.5.24-2
openshift3/ose-docker-registry:v3.5.5.24-2
openshift3/ose-egress-router:v3.5.5.24-2
openshift3/ose-keepalived-ipfailover:v3.5.5.24-2
openshift3/ose-f5-router:v3.5.5.24-2
openshift3/ose-deployer:v3.5.5.24-2
openshift3/ose-haproxy-router:v3.5.5.24-2
openshift3/node:v3.5.5.24-2
openshift3/ose-recycler:v3.5.5.24-2
openshift3/ose-sti-builder:v3.5.5.24-2
openshift3/ose-docker-builder:v3.5.5.24-2
openshift3/logging-deployer:v3.5.5.24-2
openshift3/metrics-deployer:v3.5.5.24-2
openshift3/openvswitch:v3.5.5.24-2
openshift3/logging-auth-proxy:3.5.0-15
openshift3/logging-curator:3.5.0-13
openshift3/logging-elasticsearch:3.5.0-23
openshift3/logging-fluentd:3.5.0-15
openshift3/logging-kibana:3.5.0-14
openshift3/metrics-cassandra:3.5.0-18
openshift3/metrics-hawkular-metrics:3.5.0-21
openshift3/metrics-hawkular-openshift-agent:3.5.0-15
openshift3/metrics-heapster:3.5.0-15
openshift3/registry-console:3.5-13

To upgrade an existing OpenShift Container Platform 3.4 or 3.5 cluster to this latest release, use the automated upgrade playbook. See Performing Automated In-place Cluster Upgrades for instructions.

To apply this update, run the following on all hosts where you intend to initiate Ansible-based installation or upgrade procedures:

# yum update atomic-openshift-utils

To upgrade an existing OpenShift Container Platform 3.4 or 3.5 cluster to this latest release, use the automated upgrade playbook. See Performing Automated In-place Cluster Upgrades for instructions.

This release updates the Red Hat Container Registry (registry.access.redhat.com) with the following images:

openshift3/ose-pod:v3.5.5.31.19-2
rhel7/pod-infrastructure:v3.5.5.31.19-2
openshift3/ose-ansible:v3.5.5.31.19-2
openshift3/ose:v3.5.5.31.19-2
openshift3/ose-docker-registry:v3.5.5.31.19-2
openshift3/ose-egress-router:v3.5.5.31.19-2
openshift3/ose-keepalived-ipfailover:v3.5.5.31.19-2
openshift3/ose-f5-router:v3.5.5.31.19-2
openshift3/ose-deployer:v3.5.5.31.19-2
openshift3/ose-haproxy-router:v3.5.5.31.19-2
openshift3/node:v3.5.5.31.19-2
openshift3/ose-recycler:v3.5.5.31.19-2
openshift3/ose-sti-builder:v3.5.5.31.19-2
openshift3/ose-docker-builder:v3.5.5.31.19-2
openshift3/logging-deployer:v3.5.5.31.19-2
openshift3/logging-curator:v3.5.5.31.23-2
openshift3/metrics-deployer:v3.5.5.31.19-2
openshift3/openvswitch:v3.5.5.31.19-2
openshift3/logging-auth-proxy:3.5.0-28
openshift3/logging-elasticsearch:3.5.0-37
openshift3/logging-fluentd:3.5.0-26
openshift3/logging-kibana:3.5.0-30
openshift3/metrics-cassandra:3.5.0-33
openshift3/metrics-hawkular-metrics:3.5.0-36
openshift3/metrics-hawkular-openshift-agent:3.5.0-26
openshift3/metrics-heapster:3.5.0-26
openshift3/registry-console:3.5-26

To upgrade an existing OpenShift Container Platform 3.4 or 3.5 cluster to this latest release, use the automated upgrade playbook. See Performing Automated In-place Cluster Upgrades for instructions.

To upgrade an existing OpenShift Container Platform 3.4 or 3.5 cluster to this latest release, use the automated upgrade playbook. See Performing Automated In-place Cluster Upgrades for instructions.

Image Registry

Logging

Web Console

Master

Networking

Pod

Storage

This release updates the Red Hat Container Registry (registry.access.redhat.com) with the following images:

openshift3/logging-curator:v3.5.5.31.36-4
openshift3/logging-elasticsearch:3.5.0-46
openshift3/logging-kibana:3.5.0-43
openshift3/metrics-deployer:v3.5.5.31.36-5
openshift3/metrics-heapster:3.5.0-33
openshift3/ose-deployer:v3.5.5.31.36-4
openshift3/ose:v3.5.5.31.36-4
openshift3/ose-egress-router:v3.5.5.31.36-4
openshift3/ose-haproxy-router:v3.5.5.31.36-4
openshift3/openvswitch:v3.5.5.31.36-5
openshift3/ose-recycler:v3.5.5.31.36-4
openshift3/ose-sti-builder:v3.5.5.31.36-4
openshift3/registry-console:3.5.0-33
openshift3/ose-f5-router:v3.5.5.31.36-4
openshift3/logging-auth-proxy:3.5.0-37
openshift3/logging-deployer:v3.5.5.31.36-5
openshift3/logging-fluentd:3.5.0-38
openshift3/metrics-cassandra:3.5.0-41
openshift3/metrics-hawkular-metrics:3.5.0-49
openshift3/metrics-hawkular-openshift-agent:3.5.0-33
openshift3/ose-docker-builder:v3.5.5.31.36-4
openshift3/ose-docker-registry:v3.5.5.31.36-4
openshift3/ose-keepalived-ipfailover:v3.5.5.31.36-4
openshift3/node:v3.5.5.31.36-5
openshift3/ose-pod:v3.5.5.31.36-4

To upgrade an existing OpenShift Container Platform 3.4 or 3.5 cluster to this latest release, use the automated upgrade playbook. See Performing Automated In-place Cluster Upgrades for instructions.

This release updates the Red Hat Container Registry (registry.access.redhat.com) with the following images:

openshift3/logging-curator:v3.5.5.31.47-10
openshift3/logging-deployer:v3.5.5.31.47-10
openshift3/metrics-deployer:v3.5.5.31.47-10
openshift3/node:v3.5.5.31.47-10
openshift3/openvswitch:v3.5.5.31.47-10
openshift3/ose-ansible:v3.5.5.31.47-10
openshift3/ose-base:v3.5.5.31.47-10
openshift3/ose-deployer:v3.5.5.31.47-10
openshift3/ose-docker-builder:v3.5.5.31.47-10
openshift3/ose-docker-registry:v3.5.5.31.47-10
openshift3/ose-egress-router:v3.5.5.31.47-10
openshift3/ose-f5-router:v3.5.5.31.47-10
openshift3/ose-haproxy-router:v3.5.5.31.47-10
openshift3/ose-keepalived-ipfailover:v3.5.5.31.47-10
openshift3/ose-pod:v3.5.5.31.47-10
openshift3/ose-recycler:v3.5.5.31.47-10
openshift3/ose-sti-builder:v3.5.5.31.47-10
openshift3/ose:v3.5.5.31.47-10

Authentication

Image Registry

Logging

Management Console

Metrics

Networking

Pod

Storage

Security

To upgrade an existing OpenShift Container Platform 3.4 or 3.5 cluster to this latest release, use the automated upgrade playbook. See Performing Automated In-place Cluster Upgrades for instructions.

This release updates the Red Hat Container Registry (registry.access.redhat.com) with the following images:

openshift3/logging-curator:v3.5.5.31.48-2
openshift3/metrics-deployer:v3.5.5.31.48-3
openshift3/node:v3.5.5.31.48-2
openshift3/openvswitch:v3.5.5.31.48-2
openshift3/ose-base:v3.5.5.31.48-2
openshift3/ose-deployer:v3.5.5.31.48-2
openshift3/ose-docker-builder:v3.5.5.31.48-2
openshift3/ose-docker-registry:v3.5.5.31.48-2
openshift3/ose-egress-router:v3.5.5.31.48-2
openshift3/ose-f5-router:v3.5.5.31.48-2
openshift3/ose-haproxy-router:v3.5.5.31.48-3
openshift3/ose-keepalived-ipfailover:v3.5.5.31.48-2
openshift3/ose-pod:v3.5.5.31.48-2
openshift3/ose-sti-builder:v3.5.5.31.48-2
openshift3/ose:v3.5.5.31.48-2

To upgrade an existing OpenShift Container Platform 3.4 or 3.5 cluster to this latest release, use the automated upgrade playbook. See Performing Automated In-place Cluster Upgrades for instructions.

This release updates the Red Hat Container Registry (registry.access.redhat.com) with the following images:

openshift3/logging-kibana:3.5.0-54
openshift3/node:v3.5.5.31.48-10
openshift3/openvswitch:v3.5.5.31.48-11

To upgrade an existing OpenShift Container Platform 3.4 or 3.5 cluster to this latest release, use the automated upgrade playbook. See Performing Automated In-place Cluster Upgrades for instructions.

Command Line Interface

Installer

Logging

Web Console

To upgrade an existing OpenShift Container Platform 3.4 or 3.5 cluster to this latest release, use the automated upgrade playbook. See Performing Automated In-place Cluster Upgrades for instructions.

To upgrade an existing OpenShift Container Platform 3.4 or 3.5 cluster to this latest release, use the automated upgrade playbook. See Performing Automated In-place Cluster Upgrades for instructions.