Administrator Guide

The openshift start command is used to launch OpenShift servers. The command and its subcommands (master to launch a master server and node to launch a node server) all take a limited set of arguments that are sufficient for launching servers in a development or experimental environment.

However, these arguments are insufficient to describe and control the full set of configuration and security options that are necessary in a production environment. To provide those options, it is necessary to use the dedicated master and node configuration files.

Master configuration files and node configuration files are fully specified with no default values. Therefore, any empty value indicates that you want to start up with an empty value for that parameter. This makes it easy to reason about exactly what your configuration is, but it also makes it difficult to remember all of the options to specify. To make this easier, the configuration files can be created with the --write-config option and then used with the --config option.

For masters, the openshift start command accepts options that indicate that it should simply write the configuration files that it would have used, then terminate. For nodes, a configuration file can be written using the oadm create-node-config command. Creating new configuration files is useful to get a starting point for defining your configuration.

The following commands write the relevant launch configuration file(s), certificate files, and any other necessary files to the specified --write-config or --node-dir directory.

To create configuration files for an all-in-one server (a master and a node on the same host) in the specified directory:

$ openshift start --write-config=/openshift.local.config

To create a master configuration file and other required files in the specified directory:

$ openshift start master --write-config=/openshift.local.config/master

To create a node configuration file and other related files in the specified directory:

$ oadm create-node-config --node-dir=/openshift.local.config/node-<node_hostname> --node=<node_hostname> --hostnames=<hostname>,<ip_address>

For the --hostnames option in the above command, use a comma-delimited list of every host name or IP address you want server certificates to be valid for. The above command also assumes that certificate files are located in an openshift.local.config/master/ directory. If they are not, you can include options to specify their location. Run the command with the -h option to see details.

Once you have modified the master and/or node configuration files to your specifications, you can use them when launching servers by specifying them as an argument. Keep in mind that if you specify a configuration file, none of the other command line options you pass are respected.

To launch an all-in-one server using a master configuration and a node configuration file:

$ openshift start --master-config=/openshift.local.config/master/master-config.yaml --node-config=/openshift.local.config/node-<node_hostname>/node-config.yaml

To launch a master server using a master configuration file:

$ openshift start master --config=/openshift.local.config/master/master-config.yaml

To launch a node server using a node configuration file:

$ openshift start node --config=/openshift.local.config/node-<node_hostname>/node-config.yaml

The following master-config.yaml file is a sample master configuration file taken at a point in time. You can create a new master configuration file to see the valid options for your installed version of OpenShift.

apiLevels:
- v1beta3
- v1
apiVersion: v1
assetConfig:
  logoutURL: ""
  masterPublicURL: https://10.0.2.15:8443
  publicURL: https://10.0.2.15:8443/console/
  servingInfo:
    bindAddress: 0.0.0.0:8443
    certFile: master.server.crt
    clientCA: ""
    keyFile: master.server.key
    maxRequestsInFlight: 0
    requestTimeoutSeconds: 0
controllers: '*'
corsAllowedOrigins:
- 10.0.2.15:8443
- 127.0.0.1
- localhost
dnsConfig:
  bindAddress: 0.0.0.0:53
etcdClientInfo:
  ca: ca.crt
  certFile: master.etcd-client.crt
  keyFile: master.etcd-client.key
  urls:
  - https://10.0.2.15:4001
etcdConfig:
  address: 10.0.2.15:4001
  peerAddress: 10.0.2.15:7001
  peerServingInfo:
    bindAddress: 0.0.0.0:7001
    certFile: etcd.server.crt
    clientCA: ca.crt
    keyFile: etcd.server.key
  servingInfo:
    bindAddress: 0.0.0.0:4001
    certFile: etcd.server.crt
    clientCA: ca.crt
    keyFile: etcd.server.key
  storageDirectory: /root/openshift.local.etcd
etcdStorageConfig:
  kubernetesStoragePrefix: kubernetes.io
  kubernetesStorageVersion: v1
  openShiftStoragePrefix: openshift.io
  openShiftStorageVersion: v1
imageConfig:
  format: openshift/origin-${component}:${version}
  latest: false
kind: MasterConfig
kubeletClientInfo:
  ca: ca.crt
  certFile: master.kubelet-client.crt
  keyFile: master.kubelet-client.key
  port: 10250
kubernetesMasterConfig:
  apiLevels:
  - v1beta3
  - v1
  apiServerArguments: null
  controllerArguments: null
  masterCount: 1
  masterIP: 10.0.2.15
  podEvictionTimeout: 5m
  schedulerConfigFile: ""
  servicesNodePortRange: 30000-32767
  servicesSubnet: 172.30.0.0/16
  staticNodeNames: []
masterClients:
  externalKubernetesKubeConfig: ""
  openshiftLoopbackKubeConfig: openshift-master.kubeconfig
masterPublicURL: https://10.0.2.15:8443
networkConfig:
  clusterNetworkCIDR: 10.1.0.0/16
  hostSubnetLength: 8
  networkPluginName: ""
  serviceNetworkCIDR: 172.30.0.0/16
oauthConfig:
  assetPublicURL: https://10.0.2.15:8443/console/
  grantConfig:
    method: auto
  identityProviders:
  - challenge: true
    login: true
    name: anypassword
    provider:
      apiVersion: v1
      kind: AllowAllPasswordIdentityProvider
  masterPublicURL: https://10.0.2.15:8443
  masterURL: https://10.0.2.15:8443
  sessionConfig:
    sessionMaxAgeSeconds: 300
    sessionName: ssn
    sessionSecretsFile: ""
  tokenConfig:
    accessTokenMaxAgeSeconds: 86400
    authorizeTokenMaxAgeSeconds: 300
policyConfig:
  bootstrapPolicyFile: policy.json
  openshiftInfrastructureNamespace: openshift-infra
  openshiftSharedResourcesNamespace: openshift
projectConfig:
  defaultNodeSelector: ""
  projectRequestMessage: ""
  projectRequestTemplate: ""
  securityAllocator:
    mcsAllocatorRange: s0:/2
    mcsLabelsPerProject: 5
    uidAllocatorRange: 1000000000-1999999999/10000
routingConfig:
  subdomain: router.default.svc.cluster.local
serviceAccountConfig:
  managedNames:
  - default
  - builder
  - deployer
  masterCA: ca.crt
  privateKeyFile: serviceaccounts.private.key
  publicKeyFiles:
  - serviceaccounts.public.key
servingInfo:
  bindAddress: 0.0.0.0:8443
  certFile: master.server.crt
  clientCA: ca.crt
  keyFile: master.server.key
  maxRequestsInFlight: 0
  requestTimeoutSeconds: 3600

The following node-config.yaml file is a sample node configuration file taken at a point in time. You can create a new node configuration file to see the valid options for your installed version of OpenShift.

allowDisabledDocker: true
apiVersion: v1
dnsDomain: cluster.local
dnsIP: 10.0.2.15
dockerConfig:
  execHandlerName: native
imageConfig:
  format: openshift/origin-${component}:${version}
  latest: false
kind: NodeConfig
masterKubeConfig: node.kubeconfig
networkConfig:
  mtu: 1450
  networkPluginName: ""
nodeIP: ""
nodeName: node1.example.com
podManifestConfig: 1
  path: "/path/to/pod-manifest-file" 2
  fileCheckIntervalSeconds: 30 3
servingInfo:
  bindAddress: 0.0.0.0:10250
  certFile: server.crt
  clientCA: node-client-ca.crt
  keyFile: server.key
volumeDirectory: /root/openshift.local.volumes

You can manage nodes in your instance using the CLI.

When you perform node management operations, the CLI interacts with node objects that are representations of actual node hosts. The master uses the information from node objects to validate nodes with health checks.

To list all nodes that are known to the master:

$ oc get nodes
NAME                 LABELS                                        STATUS
node1.example.com    kubernetes.io/hostname=node1.example.com      Ready
node2.example.com    kubernetes.io/hostname=node2.example.com      Ready

To only list information about a single node, replace <node> with the full node name:

$ oc get node <node>

The STATUS column in the output of these commands can show nodes with the following conditions:

To get more detailed information about a specific node, including the reason for the current condition:

$ oc describe node <node>

For example:

$ oc describe node node1.example.com
Name:			node1.example.com
Labels:			kubernetes.io/hostname=node1.example.com
CreationTimestamp:	Wed, 10 Jun 2015 17:22:34 +0000
Conditions:
  Type		Status	LastHeartbeatTime			LastTransitionTime			Reason					Message
  Ready 	True 	Wed, 10 Jun 2015 19:56:16 +0000 	Wed, 10 Jun 2015 17:22:34 +0000 	kubelet is posting ready status
Addresses:	127.0.0.1
Capacity:
 memory:	1017552Ki
 pods:		100
 cpu:		2
Version:
 Kernel Version:		3.17.4-301.fc21.x86_64
 OS Image:			Fedora 21 (Twenty One)
 Container Runtime Version:	docker://1.6.0
 Kubelet Version:		v0.17.1-804-g496be63
 Kube-Proxy Version:		v0.17.1-804-g496be63
ExternalID:			node1.example.com
Pods:				(2 in total)
  docker-registry-1-9yyw5
  router-1-maytv
No events.

To add nodes to your existing OpenShift cluster, you can run an Ansible playbook that handles installing the node components, generating the required certificates, and other important steps. See the advanced installation method for instructions on running the playbook directly.

Alternatively, if you used the quick installation method, you can re-run the installer to add nodes, which performs the same steps.

When you delete a node with the CLI, although the node object is deleted in Kubernetes, the pods that exist on the node itself are not deleted. However, the pods cannot be accessed by OpenShift. The behavior around deleting nodes and pods with the CLI is under active development.

To delete a node:

$ oc delete node <node>

To add or update labels on a node:

$ oc label node <node> <key_1>=<value_1> ... <key_n>=<value_n>

To see more detailed usage:

$ oc label -h

To list all or selected pods on one or more nodes:

$ oadm manage-node <node1> <node2> \
    --list-pods [--pod-selector=<pod_selector>] [-o json|yaml]

To list all or selected pods on selected nodes:

$ oadm manage-node --selector=<node_selector> \
    --list-pods [--pod-selector=<pod_selector>] [-o json|yaml]

By default, healthy nodes with a Ready status are marked as schedulable, meaning that new pods are allowed for placement on the node. Manually marking a node as unschedulable blocks any new pods from being scheduled on the node. Existing pods on the node are not affected.

To mark a node or nodes as unschedulable:

$ oadm manage-node <node1> <node2> --schedulable=false

For example:

$ oadm manage-node node1.example.com --schedulable=false
NAME                 LABELS                                        STATUS
node1.example.com    kubernetes.io/hostname=node1.example.com      Ready,SchedulingDisabled

To mark a currently unschedulable node or nodes as schedulable:

$ oadm manage-node <node1> <node2> --schedulable

Alternatively, instead of specifying specific node names (e.g., <node1><node2>), you can use the --selector=<node_selector> option to mark selected nodes as schedulable or unschedulable.

Evacuating pods allows you to migrate all or selected pods from a given node or nodes. Nodes must first be marked unschedulable to perform pod evacuation.

Only pods backed by a replication controller can be evacuated; the replication controllers create new pods on other nodes and remove the existing pods from the specified node(s). Bare pods, meaning those not backed by a replication controller, are unaffected by default. You can evacuate a subset of pods by specifying a pod-selector. Pod selector is based on labels, so all the pods with the specified label will be evacuated.

To list pods that will be migrated without actually performing the evacuation, use the --dry-run option:

$ oadm manage-node <node1> <node2> \
    --evacuate --dry-run [--pod-selector=<pod_selector>]

To actually evacuate all or selected pods on one or more nodes:

$ oadm manage-node <node1> <node2> \
    --evacuate [--pod-selector=<pod_selector>]

You can force deletion of bare pods by using the --force option:

$ oadm manage-node <node1> <node2> \
    --evacuate --force [--pod-selector=<pod_selector>]

Alternatively, instead of specifying specific node names (e.g., <node1><node2>), you can use the --selector=<node_selector> option to evacuate pods on selected nodes.

Pods inside of an OpenShift cluster are only reachable via their IP addresses on the cluster network. An edge load balancer can be used to accept traffic from outside networks and proxy the traffic to pods inside the OpenShift cluster. In cases where the load balancer is not part of the cluster network, routing becomes a hurdle as the internal cluster network is not accessible to the edge load balancer.

To solve this problem where the OpenShift cluster is using OpenShift SDN as the cluster networking solution, there are two ways to achieve network access to the pods.

If possible, run an OpenShift node instance on the load balancer itself that uses OpenShift SDN as the networking plug-in. This way, the edge machine gets its own Open vSwitch bridge that the SDN automatically configures to provide access to the pods and nodes that reside in the cluster. The routing table is dynamically configured by the SDN as pods are created and deleted, and thus the routing software is able to reach the pods.

Mark the load balancer machine as an unschedulable node so that no pods end up on the load balancer itself:

$ oadm manage-node <load_balancer_hostname> --schedulable=false

If the load balancer comes packaged as a Docker container, then it is even easier to integrate with OpenShift: Simply run the load balancer as a pod with the host port exposed. The pre-packaged HAProxy router in OpenShift runs in precisely this fashion.

In some cases, the previous solution is not possible. For example, an F5 BIG-IP® host cannot run an OpenShift node instance or the OpenShift SDN because F5® uses a custom, incompatible Linux kernel and distribution.

Instead, to enable F5 BIG-IP® to reach pods, you can choose an existing node within the cluster network as a ramp node and establish a tunnel between the F5 BIG-IP® host and the designated ramp node. Because it is otherwise an ordinary OpenShift node, the ramp node has the necessary configuration to route traffic to any pod on any node in the cluster network. The ramp node thus assumes the role of a gateway through which the F5 BIG-IP® host has access to the entire cluster network.

Following is an example of establishing an ipip tunnel between an F5 BIG-IP® host and a designated ramp node.

On the F5 BIG-IP® host:

On the ramp node:

$ oc label node ramp-node-1 f5rampnode=true
$ oc label node ramp-node-2 f5rampnode=true

$ echo '
    { "kind": "ServiceAccount",
      "apiVersion": "v1",
      "metadata": { "name": "f5ipfailover" }
    }
  ' | oc create -f -

$ oc edit scc privileged

...
users:
- system:serviceaccount:openshift-infra:build-controller
- system:serviceaccount:default:router
- system:serviceaccount:default:f5ipfailover

$ oc get scc privileged -o json |
    jq '.users |= .+ ["system:serviceaccount:default:f5ipfailover"]' |
      oc replace scc -f -

# RAMP_IP=10.20.30.4
# IFNAME=eth0 1
# oadm ipfailover <name-tag> \
    --virtual-ips=$RAMP_IP \
    --interface=$IFNAME \
    --watch-port=0 \
    --replicas=2 \
    --service-account=f5ipfailover  \
    --selector='f5rampnode=true'

As an OpenShift administrator, you may want to view the logs from all containers in one user interface. The currently supported method for aggregating container logs in OpenShift Enterprise is using a centralized file system. Additional supported methods are planned for inclusion in future releases.

This method reads all container logs and forwards them to a central server for storage on the file system.

# systemctl enable td-agent
# systemctl start td-agent

The OpenShift master includes a built-in OAuth server. Developers and administrators obtain OAuth access tokens to authenticate themselves to the API.

As an administrator, you can configure OAuth using a master configuration file to specify an identity provider. If you installed OpenShift using the Quick Installation or Advanced Installation method, the Deny All identity provider is used by default, which denies access for all user names and passwords. To allow access, you must choose a different identity provider and configure the master configuration file appropriately (located at /etc/openshift/master/master-config.yaml by default).

When running a master without a configuration file, the Allow All identity provider is used by default, which allows any non-empty user name and password to log in. This is useful for testing purposes. To use other identity providers, or to modify any token, grant, or session options, you must run the master from a configuration file.

You can configure the master for authentication using your desired identity provider by modifying the master configuration file. The following sections detail the identity providers supported by OpenShift.

There are three parameters common to all identity providers:

oauthConfig:
  ...
  identityProviders:
  - name: my_allow_provider 1
    challenge: true 2
    login: true 3
    provider:
      apiVersion: v1
      kind: AllowAllPasswordIdentityProvider

oauthConfig:
  ...
  identityProviders:
  - name: my_deny_provider 1
    challenge: true 2
    login: true 3
    provider:
      apiVersion: v1
      kind: DenyAllPasswordIdentityProvider

# yum install httpd-tools

$ htpasswd -c </path/to/users.htpasswd> <user_name>

$ htpasswd </path/to/users.htpasswd> <user_name>

$ htpasswd -D </path/to/users.htpasswd> <user_name>

oauthConfig:
  ...
  identityProviders:
  - name: my_htpasswd_provider 1
    challenge: true 2
    login: true 3
    provider:
      apiVersion: v1
      kind: HTPasswdPasswordIdentityProvider
      file: /path/to/users.htpasswd 4

ldap://host:port/basedn?attribute?scope?filter

(&(<filter>)(<attribute>=<username>))

ldap://ldap.example.com/o=Acme?cn?sub?(enabled=true)

oauthConfig:
  ...
  identityProviders:
  - name: "my_ldap_provider" 1
    challenge: true 2
    login: true 3
    provider:
      apiVersion: v1
      kind: LDAPPasswordIdentityProvider
      attributes:
        id: 4
        - dn
        email: 5
        - mail
        name: 6
        - cn
        preferredUsername: 7
        - uid
      bindDN: "" 8
      bindPassword: "" 9
      ca: my-ldap-ca-bundle.crt 10
      insecure: false 11
      url: "ldap://ldap.example.com/ou=users,dc=acme,dc=com?uid" 12

{"error":"Error message"}

{"sub":"userid"} 1

oauthConfig:
  ...
  identityProviders:
  - name: my_remote_basic_auth_provider 1
    challenge: true 2
    login: true 3
    provider:
      apiVersion: v1
      kind: BasicAuthPasswordIdentityProvider
      url: https://www.example.com/remote-idp 4
      ca: /path/to/ca.file 5
      certFile: /path/to/client.crt 6
      keyFile: /path/to/client.key 7

oauthConfig:
  ...
  identityProviders:
  - name: my_request_header_provider 1
    challenge: true 2
    login: true 3
    provider:
      apiVersion: v1
      kind: RequestHeaderIdentityProvider
      challengeURL: "https://www.example.com/challenging-proxy/oauth/authorize?${query}" 4
      loginURL: "https://www.example.com/login-proxy/oauth/authorize?${query}" 5
      clientCA: /path/to/client-ca.file 6
      headers: 7
      - X-Remote-User
      - SSO-User

# yum install -y httpd mod_ssl mod_session apr-util-openssl

# oadm ca create-signer-cert \
  --cert='/etc/openshift/master/proxyca.crt' \
  --key='/etc/openshift/master/proxyca.key' \
  --name='openshift-proxy-signer@1432232228' \
  --serial='/etc/openshift/master/proxyca.serial.txt'

# oadm create-api-client-config \
  --certificate-authority='/etc/openshift/master/proxyca.crt' \
  --client-dir='/etc/openshift/master/proxy' \
  --signer-cert='/etc/openshift/master/proxyca.crt' \
  --signer-key='/etc/openshift/master/proxyca.key' \
  --signer-serial='/etc/openshift/master/proxyca.serial.txt' \
  --user='system:proxy' 1

# pushd /etc/openshift/master
# cp master.server.crt /etc/pki/tls/certs/localhost.crt 2
# cp master.server.key /etc/pki/tls/private/localhost.key
# cp ca.crt /etc/pki/CA/certs/ca.crt
# cat proxy/system\:proxy.crt \
  proxy/system\:proxy.key > \
  /etc/pki/tls/certs/authproxy.pem
# popd

LoadModule auth_form_module modules/mod_auth_form.so
LoadModule session_module modules/mod_session.so
LoadModule request_module modules/mod_request.so

# Nothing needs to be served over HTTP.  This virtual host simply redirects to
# HTTPS.
<VirtualHost *:80>
  DocumentRoot /var/www.adoc
  RewriteEngine              On
  RewriteRule     ^(.*)$     https://%{HTTP_HOST}$1 [R,L]
</VirtualHost>

<VirtualHost *:443>
  # This needs to match the certificates you generated.  See the CN and X509v3
  # Subject Alternative Name in the output of:
  # openssl x509 -text -in /etc/pki/tls/certs/localhost.crt
  ServerName www.example.com

  DocumentRoot /var/www.adoc
  SSLEngine on
  SSLCertificateFile /etc/pki/tls/certs/localhost.crt
  SSLCertificateKeyFile /etc/pki/tls/private/localhost.key
  SSLCACertificateFile /etc/pki/CA/certs/ca.crt

  SSLProxyEngine on
  SSLProxyCACertificateFile /etc/pki/CA/certs/ca.crt
  # It's critical to enforce client certificates on the Master.  Otherwise
  # requests could spoof the X-Remote-User header by accessing the Master's
  # /oauth/authorize endpoint directly.
  SSLProxyMachineCertificateFile /etc/pki/tls/certs/authproxy.pem

  # Send all requests to the console
  RewriteEngine              On
  RewriteRule     ^/console(.*)$     https://%{HTTP_HOST}:8443/console$1 [R,L]

  # In order to using the challenging-proxy an X-Csrf-Token must be present.
  RewriteCond %{REQUEST_URI} ^/challenging-proxy
  RewriteCond %{HTTP:X-Csrf-Token} ^$ [NC]
  RewriteRule ^.* - [F,L]

  <Location /challenging-proxy/oauth/authorize>
    # Insert your backend server name/ip here.
    ProxyPass https://[MASTER]:8443/oauth/authorize
    AuthType basic
  </Location>

  <Location /login-proxy/oauth/authorize>
    # Insert your backend server name/ip here.
    ProxyPass https://[MASTER]:8443/oauth/authorize

    # mod_auth_form providers are implemented by mod_authn_dbm, mod_authn_file,
    # mod_authn_dbd, mod_authnz_ldap and mod_authn_socache.
    AuthFormProvider file
    AuthType form
    AuthName openshift
    ErrorDocument 401 /login.adoc
  </Location>

  <ProxyMatch /oauth/authorize>
    AuthUserFile /etc/openshift/htpasswd
    AuthName openshift
    Require valid-user
    RequestHeader set X-Remote-User %{REMOTE_USER}s

    # For ldap:
    # AuthBasicProvider ldap
    # AuthLDAPURL "ldap://ldap.example.com:389/ou=People,dc=my-domain,dc=com?uid?sub?(objectClass=*)"

    # It's possible to remove the mod_auth_form usage and replace it with
    # something like mod_auth_kerb, mod_auth_gsspai or even mod_auth_mellon.
    # The former would be able to support both the login and challenge flows
    # from the Master.  Mellon would likely only support the login flow.

    # For Kerberos
    # yum install mod_auth_gssapi
    # AuthType GSSAPI
    # GssapiCredStore keytab:/etc/httpd.keytab
  </ProxyMatch>

</VirtualHost>

RequestHeader unset X-Remote-User

# yum -y install httpd-tools
# touch /etc/openshift/htpasswd
# htpasswd -c /etc/openshift/htpasswd <user_name>

  identityProviders:
  - name: requestheader
    challenge: true
    login: true
    provider:
      apiVersion: v1
      kind: RequestHeaderIdentityProvider
      challengeURL: "https://[MASTER]/challenging-proxy/oauth/authorize?${query}"
      loginURL: "https://[MASTER]/login-proxy/oauth/authorize?${query}"
      clientCA: /etc/openshift/master/proxyca.crt
      headers:
      - X-Remote-User

# systemctl restart httpd
# systemctl restart openshift-master

oauthConfig:
  ...
  identityProviders:
  - name: github 1
    challenge: false 2
    login: true 3
    provider:
      apiVersion: v1
      kind: GitHubIdentityProvider
      clientID: ... 4
      clientSecret: ... 5

oauthConfig:
  ...
  identityProviders:
  - name: google 1
    challenge: false 2
    login: true 3
    provider:
      apiVersion: v1
      kind: GoogleIdentityProvider
      clientID: ... 4
      clientSecret: ... 5
      hostedDomain: "" 6

oauthConfig:
  ...
  identityProviders:
  - name: my_openid_connect 1
    challenge: false 2
    login: true 3
    provider:
      apiVersion: v1
      kind: OpenIDIdentityProvider
      clientID: ... 4
      clientSecret: ... 5
      claims:
        id:
        - sub 6
        preferredUsername:
        - preferred_username
        name:
        - name
        email:
        - email
      urls:
        authorize: https://myidp.example.com/oauth2/authorize 7
        token: https://myidp.example.com/oauth2/token 8

oauthConfig:
  ...
  identityProviders:
  - name: my_openid_connect
    challenge: false
    login: true
    provider:
      apiVersion: v1
      kind: OpenIDIdentityProvider
      clientID: ...
      clientSecret: ...
      ca: my-openid-ca-bundle.crt 1
      extraScopes: 2
      - email
      - profile
      extraAuthorizeParameters: 3
        include_granted_scopes: "true"
      claims:
        id: 4
        - custom_id_claim
        - sub
        preferredUsername: 5
        - preferred_username
        - email
        name: 6
        - nickname
        - given_name
        - name
        email: 7
        - custom_email_claim
        - email
      urls:
        authorize: https://myidp.example.com/oauth2/authorize
        token: https://myidp.example.com/oauth2/token
        userInfo: https://myidp.example.com/oauth2/userinfo 8

The OAuth server generates two kinds of tokens:

Use the tokenConfig stanza to set token options:

oauthConfig:
  ...
  tokenConfig:
    accessTokenMaxAgeSeconds: 86400 1
    authorizeTokenMaxAgeSeconds: 300 2

To configure how the OAuth server responds to token requests for a client the user has not previously granted permission, set the method value in the grantConfig stanza. Valid values for method are:

oauthConfig:
  ...
  grantConfig:
    method: auto

The OAuth server uses a signed and encrypted cookie-based session during login and redirect flows.

Use the sessionConfig stanza to set session options:

oauthConfig:
  ...
  sessionConfig:
    sessionMaxAgeSeconds: 300 1
    sessionName: ssn 2
    sessionSecretsFile: "..." 3

If no sessionSecretsFile is specified, a random signing and encryption secret is generated at each start of the master server. This means that any logins in progress will have their sessions invalidated if the master is restarted. It also means that if multiple masters are configured, they will not be able to decode sessions generated by one of the other masters.

To specify the signing and encryption secret to use, specify a sessionSecretsFile. This allows you separate secret values from the configuration file and keep the configuration file distributable, for example for debugging purposes.

Multiple secrets can be specified in the sessionSecretsFile to enable rotation. New sessions are signed and encrypted using the first secret in the list. Existing sessions are decrypted and authenticated by each secret until one succeeds.

apiVersion: v1
kind: SessionSecrets
secrets: 1
- authentication: "..." 2
  encryption: "..." 3
- authentication: "..."
  encryption: "..."
...

When a person uses the command line or web console, their API token authenticates them to the OpenShift API. However, when a regular user’s credentials are not available, it is common for components to make API calls independently. For example:

Service accounts provide a flexible way to control API access without sharing a regular user’s credentials.

Every service account has an associated username that can be granted roles, just like a regular user. The username is derived from its project and name: system:serviceaccount:<project>:<name>

For example, to add the view role to the robot service account in the top-secret project:

$ oc policy add-role-to-user view system:serviceaccount:top-secret:robot

Every service account is also a member of two groups:

For example, to allow all service accounts in all projects to view resources in the top-secret project:

$ oc policy add-role-to-group view system:serviceaccounts -n top-secret

To allow all service accounts in the managers project to edit resources in the top-secret project:

$ oc policy add-role-to-group edit system:serviceaccounts:managers -n top-secret

Service accounts authenticate to the API using tokens signed by a private RSA key. The authentication layer verifies the signature using a matching public RSA key.

To enable service account token generation, update the master configurationserviceAccountConfig stanza to specify a privateKeyFile (for signing), and a matching public key file in the publicKeyFiles list:

serviceAccountConfig:
  ...
  masterCA: ca.crt 1
  privateKeyFile: serviceaccounts.private.key 2
  publicKeyFiles:
  - serviceaccounts.public.key 3
  - ...

Service accounts are required in each project to run builds, deployments, and other pods. The managedNames setting in the master configuration file controls which service accounts are automatically created in every project:

serviceAccountConfig:
  ...
  managedNames: 1
  - builder 2
  - deployer 3
  - default 4
  - ...

All service accounts in a project are given the system:image-puller role, which allows pulling images from any image stream in the project using the internal Docker registry.

Several infrastructure controllers run using service account credentials. The following service accounts are created in the OpenShift infrastructure namespace at server start, and given the following roles cluster-wide:

To configure the namespace where those service accounts are created, set the openshiftInfrastructureNamespace field in the master configuration file:

policyConfig:
  ...
  openshiftInfrastructureNamespace: openshift-infra

Set limitSecretReferences field in master configuration file to true to require pod secret references to be whitelisted by their service accounts. Set its value to false to allow pods to reference any secret in the namespace.

serviceAccountConfig:
  ...
  limitSecretReferences: false

You can use the CLI to view authorization policies and the administrator CLI to manage the roles and bindings within a policy.

Roles grant various levels of access in the system-wide cluster policy as well as project-scoped local policies. Users and groups can be associated with, or bound to, multiple roles at the same time. You can view details about the roles and their bindings using the oc describe command.

Users with the cluster-admindefault role in the cluster policy can view cluster policy and all local policies. Users with the admindefault role in a given local policy can view that project-scoped policy.

$ oc describe clusterPolicy default

$ oc describe clusterPolicy default
Name:				default
Created:			4 hours ago
Labels:				<none>
Last Modified:			2015-06-10 17:22:25 +0000 UTC
admin				Verbs					Resources																Resource Names	Non-Resource URLs				Extension
				[create delete get list update watch]	[pods/proxy projects resourcegroup:exposedkube resourcegroup:exposedopenshift resourcegroup:granter secrets]				[][]
				[get list watch]			[pods/exec pods/portforward resourcegroup:allkube resourcegroup:allkube-status resourcegroup:allopenshift-status resourcegroup:policy]	[][]
				[get update]				[imagestreams/layers]															[][]
basic-user			Verbs					Resources																Resource Names	Non-Resource URLs				Extension
				[get]					[users]																	[~][]
				[list]					[projectrequests]															[][]
				[get list]				[clusterroles]																[][]
				[list]					[projects]																[][]
				[create]				[subjectaccessreviews]															[][]						IsPersonalSubjectAccessReview
cluster-admin			Verbs					Resources																Resource Names	Non-Resource URLs				Extension
				[*]					[*]																	[][]
				[*]					[]																	[][*]
cluster-reader			Verbs					Resources																Resource Names	Non-Resource URLs				Extension
				[get list watch]			[*]																	[][]
				[get]					[]																	[][*]
cluster-status			Verbs					Resources																Resource Names	Non-Resource URLs				Extension
				[get]					[]																	[][/api /healthz /healthz/* /osapi /version]
edit				Verbs					Resources																Resource Names	Non-Resource URLs				Extension
				[create delete get list update watch]	[pods/proxy resourcegroup:exposedkube resourcegroup:exposedopenshift secrets]								[][]
				[get list watch]			[pods/exec pods/portforward projects resourcegroup:allkube resourcegroup:allkube-status resourcegroup:allopenshift-status]		[][]
self-provisioner		Verbs					Resources																Resource Names	Non-Resource URLs				Extension
				[create]				[projectrequests]															[][]
system:build-controller		Verbs					Resources																Resource Names	Non-Resource URLs				Extension
				[get list watch]			[builds]																[][]
				[update]				[builds]																[][]
				[get]					[imagestreams]																[][]
				[create delete get list]		[pods]																	[][]
				[create update]				[events]																[][]
system:component		Verbs					Resources																Resource Names	Non-Resource URLs				Extension
				[*]					[*]																	[][]
system:deployer			Verbs					Resources																Resource Names	Non-Resource URLs				Extension
				[get list]				[replicationcontrollers]														[][]
				[get update]				[replicationcontrollers]														[][]
				[create get list watch]			[pods]																	[][]
system:deployment-controller	Verbs					Resources																Resource Names	Non-Resource URLs				Extension
				[list watch]				[replicationcontrollers]														[][]
				[get update]				[replicationcontrollers]														[][]
				[create delete get list update]		[pods]																	[][]
				[create update]				[events]																[][]
system:image-builder		Verbs					Resources																Resource Names	Non-Resource URLs				Extension
				[get update]				[imagestreams/layers]															[][]
system:image-pruner		Verbs					Resources																Resource Names	Non-Resource URLs				Extension
				[delete]				[images]																[][]
				[get list]				[buildconfigs builds deploymentconfigs images imagestreams pods replicationcontrollers]							[][]
				[update]				[imagestreams/status]															[][]
system:image-puller		Verbs					Resources																Resource Names	Non-Resource URLs				Extension
				[get]					[imagestreams/layers]															[][]
system:node			Verbs					Resources																Resource Names	Non-Resource URLs				Extension
				[get list watch]			[services]																[][]
				[create get list watch]			[nodes]																	[][]
				[update]				[nodes/status]																[][]
				[create update]				[events]																[][]
				[get list watch]			[pods]																	[][]
				[create delete get]			[pods]																	[][]
				[update]				[pods/status]																[][]
				[get]					[secrets]																[][]
				[get]					[persistentvolumeclaims persistentvolumes]												[][]
				[get]					[endpoints]																[][]
system:node-proxier		Verbs					Resources																Resource Names	Non-Resource URLs				Extension
				[list watch]				[endpoints services]															[][]
system:oauth-token-deleter	Verbs					Resources																Resource Names	Non-Resource URLs				Extension
				[delete]				[oauthaccesstokens oauthauthorizetokens]												[][]
system:registry			Verbs					Resources																Resource Names	Non-Resource URLs				Extension
				[delete get]				[images]																[][]
				[get]					[imagestreamimages imagestreams imagestreamtags]											[][]
				[update]				[imagestreams]																[][]
				[create]				[imagestreammappings]															[][]
system:replication-controller	Verbs					Resources																Resource Names	Non-Resource URLs				Extension
				[list watch]				[replicationcontrollers]														[][]
				[get update]				[replicationcontrollers]														[][]
				[list watch]				[pods]																	[][]
				[create delete]				[pods]																	[][]
				[create update]				[events]																[][]
system:router			Verbs					Resources																Resource Names	Non-Resource URLs				Extension
				[list watch]				[endpoints routes]															[][]
system:sdn-manager		Verbs					Resources																Resource Names	Non-Resource URLs				Extension
				[create delete get list watch]		[hostsubnets]																[][]
				[get list watch]			[nodes]																	[][]
				[create get]				[clusternetworks]															[][]
system:sdn-reader		Verbs					Resources																Resource Names	Non-Resource URLs				Extension
				[get list watch]			[hostsubnets]																[][]
				[get list watch]			[nodes]																	[][]
				[get]					[clusternetworks]															[][]
system:webhook			Verbs					Resources																Resource Names	Non-Resource URLs				Extension
				[create get]				[buildconfigs/webhooks]															[][]
view				Verbs					Resources																Resource Names	Non-Resource URLs				Extension
				[get list watch]			[projects resourcegroup:allkube resourcegroup:allkube-status resourcegroup:allopenshift-status resourcegroup:exposedopenshift]		[][]

$ oc describe clusterPolicyBindings :default

$ oc describe clusterPolicyBindings :default
Name:						:default
Created:					4 hours ago
Labels:						<none>
Last Modified:					2015-06-10 17:22:26 +0000 UTC
Policy:						<none>
RoleBinding[basic-users]:
						Role:	basic-user
						Users:	[]
						Groups:	[system:authenticated]
RoleBinding[cluster-admins]:
						Role:	cluster-admin
						Users:	[]
						Groups:	[system:cluster-admins]
RoleBinding[cluster-readers]:
						Role:	cluster-reader
						Users:	[]
						Groups:	[system:cluster-readers]
RoleBinding[cluster-status-binding]:
						Role:	cluster-status
						Users:	[]
						Groups:	[system:authenticated system:unauthenticated]
RoleBinding[self-provisioners]:
						Role:	self-provisioner
						Users:	[]
						Groups:	[system:authenticated]
RoleBinding[system:build-controller]:
						Role:	system:build-controller
						Users:	[system:serviceaccount:openshift-infra:build-controller]
						Groups:	[]
RoleBinding[system:deployment-controller]:
						Role:	system:deployment-controller
						Users:	[system:serviceaccount:openshift-infra:deployment-controller]
						Groups:	[]
RoleBinding[system:masters]:
						Role:	system:master
						Users:	[]
						Groups:	[system:masters]
RoleBinding[system:node-proxiers]:
						Role:	system:node-proxier
						Users:	[]
						Groups:	[system:nodes]
RoleBinding[system:nodes]:
						Role:	system:node
						Users:	[]
						Groups:	[system:nodes]
RoleBinding[system:oauth-token-deleters]:
						Role:	system:oauth-token-deleter
						Users:	[]
						Groups:	[system:authenticated system:unauthenticated]
RoleBinding[system:registrys]:
						Role:	system:registry
						Users:	[]
						Groups:	[system:registries]
RoleBinding[system:replication-controller]:
						Role:	system:replication-controller
						Users:	[system:serviceaccount:openshift-infra:replication-controller]
						Groups:	[]
RoleBinding[system:routers]:
						Role:	system:router
						Users:	[]
						Groups:	[system:routers]
RoleBinding[system:sdn-readers]:
						Role:	system:sdn-reader
						Users:	[]
						Groups:	[system:nodes]
RoleBinding[system:webhooks]:
						Role:	system:webhook
						Users:	[]
						Groups:	[system:authenticated system:unauthenticated]

$ oc describe policyBindings :default

$ oc describe policyBindings :default -n joe-project
Name:					:default
Created:				About a minute ago
Labels:					<none>
Last Modified:				2015-06-10 21:55:06 +0000 UTC
Policy:					<none>
RoleBinding[admins]:
					Role:	admin
					Users:	[joe]
					Groups:	[]
RoleBinding[system:deployers]:
					Role:	system:deployer
					Users:	[system:serviceaccount:joe-project:deployer]
					Groups:	[]
RoleBinding[system:image-builders]:
					Role:	system:image-builder
					Users:	[system:serviceaccount:joe-project:builder]
					Groups:	[]
RoleBinding[system:image-pullers]:
					Role:	system:image-puller
					Users:	[]
					Groups:	[system:serviceaccounts:joe-project]

Adding, or binding, a role to users or groups gives the user or group the relevant access granted by the role. You can add and remove roles to and from users and groups using oadm policy commands.

When managing a user or group’s associated roles for a local policy using the following operations, a project may be specified with the -n flag. If it is not specified, then the current project is used.

You can also manage role bindings for the cluster policy using the following operations. The -n flag is not used used for these operations because the cluster policy uses non-namespaced resources.

For example, you can add the admin role to the alice user in joe-project by running:

$ oadm policy add-role-to-user admin alice -n joe-project

You can then view the local bindings and verify the addition in the output:

$ oc describe policyBindings :default -n joe-project
Name:					:default
Created:				5 minutes ago
Labels:					<none>
Last Modified:				2015-06-10 22:00:44 +0000 UTC
Policy:					<none>
RoleBinding[admins]:
					Role:	admin
					Users:	[alice joe] 1
					Groups:	[]
RoleBinding[system:deployers]:
					Role:	system:deployer
					Users:	[system:serviceaccount:joe-project:deployer]
					Groups:	[]
RoleBinding[system:image-builders]:
					Role:	system:image-builder
					Users:	[system:serviceaccount:joe-project:builder]
					Groups:	[]
RoleBinding[system:image-pullers]:
					Role:	system:image-puller
					Users:	[]
					Groups:	[system:serviceaccounts:joe-project]

Security context constraints allow administrators to control permissions for pods. To learn more about this API type please refer to the security context constraints (SCCs) architecture documentation. You may manage SCCs in your instance as normal API objects using the CLI.

To get a current list of SCCs:

$ oc get scc
NAME         PRIV      CAPS      HOSTDIR   SELINUX     RUNASUSER
privileged   true      []        true      RunAsAny    RunAsAny
restricted   false     []        false     MustRunAs   MustRunAsRange

To examine a particular SCC, use oc get, oc describe, oc export, or oc edit.

$ oc edit scc restricted
allowHostDirVolumePlugin: false
allowHostNetwork: false
allowHostPorts: false
allowPrivilegedContainer: false
allowedCapabilities: null
apiVersion: v1
groups:
- system:authenticated
kind: SecurityContextConstraints
metadata:
  creationTimestamp: 2015-09-08T07:37:54Z
  name: restricted 1
  resourceVersion: "58"
  selfxref: /api/v1/securitycontextconstraints/restricted
  uid: 849d9228-55fc-11e5-976b-080027c5bfa9
runAsUser:
  type: MustRunAsRange
seLinuxContext:
  type: MustRunAs

To create a new SCC, first define the SCC in a JSON or YAML file:

kind: SecurityContextConstraints
apiVersion: v1
metadata:
  name: scc-admin
allowPrivilegedContainer: true
runAsUser:
  type: RunAsAny
seLinuxContext:
  type: RunAsAny
users:
- my-admin-user
groups:
- my-admin-group

Then, run oc create passing the file to create it:

$ oc create -f scc_admin.yaml
securitycontextconstraints/scc-admin

$ oc get scc
NAME         PRIV      CAPS      HOSTDIR   SELINUX     RUNASUSER
privileged   true      []        true      RunAsAny    RunAsAny
restricted   false     []        false     MustRunAs   MustRunAsRange
scc-admin    true      []        false     RunAsAny    RunAsAny

To delete an SCC:

$ oc delete scc <scc_name>

To update an existing SCC:

$ oc edit scc <scc_name>

If you would like to reset your security context constraints to the default settings for any reason you may delete the existing security context constraints and restart your master. The default security context constraints will only be recreated if no security context constraints exist in the system.

The following describe common scenarios and procedures using SCCs.

$ oc edit scc privileged

allowHostDirVolumePlugin: true
allowPrivilegedContainer: true
apiVersion: v1
groups:
- system:cluster-admins
- system:nodes
kind: SecurityContextConstraints
metadata:
  creationTimestamp: 2015-06-15T20:44:53Z
  name: privileged
  resourceVersion: "58"
  selfxref: /api/v1/securitycontextconstraints/privileged
  uid: 602a0838-139f-11e5-8aa4-080027c5bfa9
runAsUser:
  type: RunAsAny
seLinuxContext:
  type: RunAsAny
users:
- system:serviceaccount:openshift-infra:build-controller
- e2e-user 1

$ cat <<EOF | oc create -n My_Project -f -
kind: ServiceAccount
apiVersion: v1
metadata:
  name: My_SVCACCT 1
EOF

$ oc edit scc privileged

   - system:serviceaccount:My_Project:My_SVCACCT

setcap cap_net_raw,cap_net_admin+p /usr/bin/ping

The Kubernetes pod scheduler is responsible for determining placement of new pods onto nodes within the cluster. It reads data from the pod and tries to find a node that is a good fit based on configured policies. It is completely independent and exists as a standalone/pluggable solution. It does not modify the pod and just creates a binding for the pod that ties the pod to the particular node.

The existing generic scheduler is the default platform-provided scheduler "engine" that selects a node to host the pod in a 3-step operation:

There are several predicates provided out of the box in Kubernetes. Some of these predicates can be customized by providing certain parameters. Multiple predicates can be combined to provide additional filtering of nodes.

{"name" : "PodFitsPorts"}

{"name" : "PodFitsResources"}

{"name" : "NoDiskConflict"}

{"name" : "MatchNodeSelector"}

{"name" : "HostName"}

{"name" : "Zone", "argument" : {"serviceAffinity" : {"labels" : ["zone"]}}}

{"name" : "RequireRegion", "argument" : {"labelsPresence" : {"labels" : ["region"], "presence" : true}}}

A custom set of priority functions can be specified to configure the scheduler. There are several priority functions provided out-of-the-box in Kubernetes. Some of these priority functions can be customized by providing certain parameters. Multiple priority functions can be combined and different weights can be given to each in order to impact the prioritization. A weight is required to be specified and cannot be 0 or negative.

{"name" : "LeastRequestedPriority", "weight" : 1}

{"name" : "BalancedResourceAllocation", "weight" : 1}

{"name" : "ServiceSpreadingPriority", "weight" : 1}

{"name" : "EqualPriority", "weight" : 1}

{"name" : "RackSpread", "weight" : 1, "argument" : {"serviceAntiAffinity" : {"label" : "rack"}}}

{"name" : "RackPreferred", "weight" : 1, "argument" : {"labelPreference" : {"label" : "rack"}}}

The selection of the predicate and priority functions defines the policy for the scheduler. Administrators can provide a JSON file that specifies the predicates and priority functions to configure the scheduler. The path to the scheduler policy file can be specified in the master configuration file. In the absence of the scheduler policy file, the default configuration gets applied.

It is important to note that the predicates and priority functions defined in the scheduler configuration file will completely override the default scheduler policy. If any of the default predicates and priority functions are required, they have to be explicitly specified in the scheduler configuration file.

One of the important use cases for scheduling within OpenShift is to support flexible affinity and anti-affinity policies.

The configuration below specifies the default scheduler configuration, if it were to be specified via the scheduler policy file.

{
	"kind" : "Policy",
	"version" : "v1",
	"predicates" : [
		{"name" : "PodFitsPorts"},
		{"name" : "PodFitsResources"},
		{"name" : "NoDiskConflict"},
		{"name" : "MatchNodeSelector"},
		{"name" : "HostName"}
	],
	"priorities" : [
		{"name" : "LeastRequestedPriority", "weight" : 1},
		{"name" : "BalancedResourceAllocation", "weight" : 1},
		{"name" : "ServiceSpreadingPriority", "weight" : 1}
	]
}

Three topological levels defined as region (affinity) -→ zone (affinity) -→ rack (anti-affinity)

{
	"kind" : "Policy",
	"version" : "v1",
	"predicates" : [
		...
		{"name" : "RegionZoneAffinity", "argument" : {"serviceAffinity" : {"labels" : ["region", "zone"]}}}
	],
	"priorities" : [
		...
		{"name" : "RackSpread", "weight" : 1, "argument" : {"serviceAntiAffinity" : {"label" : "rack"}}}
	]
}

Three topological levels defined as city (affinity) → building (anti-affinity) → room (anti-affinity):

{
	"kind" : "Policy",
	"version" : "v1",
	"predicates" : [
		...
		{"name" : "CityAffinity", "argument" : {"serviceAffinity" : {"labels" : ["city"]}}}
	],
	"priorities" : [
		...
		{"name" : "BuildingSpread", "weight" : 1, "argument" : {"serviceAntiAffinity" : {"label" : "building"}}},
		{"name" : "RoomSpread", "weight" : 1, "argument" : {"serviceAntiAffinity" : {"label" : "room"}}}
	]
}

Only use nodes with the 'region' label defined and prefer nodes with the 'zone' label defined:

{
	"kind" : "Policy",
	"version" : "v1",
	"predicates" : [
		...
		{"name" : "RequireRegion", "argument" : {"labelsPresence" : {"labels" : ["region"], "presence" : true}}}

	],
	"priorities" : [
		...
		{"name" : "ZonePreferred", "weight" : 1, "argument" : {"labelPreference" : {"label" : "zone", "presence" : true}}}
	]
}

Configuration example combining static and configurable predicates and priority functions:

{
	"kind" : "Policy",
	"version" : "v1",
	"predicates" : [
		...
		{"name" : "RegionAffinity", "argument" : {"serviceAffinity" : {"labels" : ["region"]}}},
		{"name" : "RequireRegion", "argument" : {"labelsPresence" : {"labels" : ["region"], "presence" : true}}},
		{"name" : "BuildingNodesAvoid", "argument" : {"labelsPresence" : {"labels" : ["building"], "presence" : false}}},
		{"name" : "PodFitsPorts"},
		{"name" : "MatchNodeSelector"}
	],
	"priorities" : [
		...
		{"name" : "ZoneSpread", "weight" : 2, "argument" : {"serviceAntiAffinity" : {"label" : "zone"}}},
		{"name" : "ZonePreferred", "weight" : 1, "argument" : {"labelPreference" : {"label" : "zone", "presence" : true}}},
		{"name" : "ServiceSpreadingPriority", "weight" : 1}
	]
}

As is the case with almost everything else in Kubernetes/OpenShift, the scheduler is built using a plug-in model and the current implementation itself is a plug-in. There are two ways to extend the scheduler functionality:

Over time, API objects created in OpenShift can accumulate in the etcd data store through normal user operations, such as when building and deploying applications.

As an administrator, you can periodically prune older versions of objects from your OpenShift instance that are no longer needed. For example, by pruning images you can delete older images and layers that are no longer in use, but are still taking up disk space.

The CLI groups prune operations under a common parent command.

$ oadm prune <object_type> <options>

This specifies:

In order to prune deployments that are no longer required by the system due to age and status, administrators may run the following command:

$ oadm prune deployments [<options>]

To see what a pruning operation would delete:

$ oadm prune deployments --orphans --keep-complete=5 --keep-failed=1 \
    --keep-younger-than=60m

To actually perform the prune operation:

$ oadm prune deployments --orphans --keep-complete=5 --keep-failed=1 \
    --keep-younger-than=60m --confirm

In order to prune builds that are no longer required by the system due to age and status, administrators may run the following command:

$ oadm prune builds [<options>]

To see what a pruning operation would delete:

$ oadm prune builds --orphans --keep-complete=5 --keep-failed=1 \
    --keep-younger-than=60m

To actually perform the prune operation:

$ oadm prune builds --orphans --keep-complete=5 --keep-failed=1 \
    --keep-younger-than=60m --confirm

In order to prune images that are no longer required by the system due to age and status, administrators may run the following command:

$ oadm prune images [<options>]

OpenShift uses the following logic to determine which images and layers to prune:

To see what a pruning operation would delete:

$ oadm prune images --keep-tag-revisions=3 --keep-younger-than=60m

To actually perform the prune operation:

$ oadm prune images --keep-tag-revisions=3 --keep-younger-than=60m --confirm

Depending on the underlying implementation, you can monitor a running router in multiple ways. This topic discusses the HAProxy template router and the components to check to ensure its health.

The HAProxy router exposes a web listener for the HAProxy statistics. Enter the router’s public IP address and the correctly configured port (1936 by default) to view the statistics page, and enter the administrator password when prompted. This password and port are configured during the router installation, but they can be found by viewing the haproxy.conf file on the container.

To view a router log, run the oc log command on the pod. Since the router is running as a plug-in process that manages the underlying implementation, the log is for the plug-in, not the actual HAProxy log.

routes.json

Routes are processed by the HAProxy router, and are stored both in memory, on disk, and in the HAProxy configuration file. The internal route representation, which is passed to the template to generate the HAProxy configuration file, is found in the /var/lib/containers/router/routes.json file. When troubleshooting a routing issue, view this file to see the data being used to drive configuration.

HAProxy configuration

You can find the HAProxy configuration and the backends that have been created for specific routes in the /var/lib/haproxy/conf/haproxy.conf file. The mapping files are found in the same directory. The helper frontend and backends use mapping files when mapping incoming requests to a backend.

Certificates

Certificates are stored in two places:

The files are keyed by the namespace and name of the route. The key, certificate, and CA certificate are concatenated into a single file. You can use OpenSSL to view the contents of these files.

This topic describes how to set up highly-available services on your OpenShift cluster.

The Kubernetes replication controller ensures that the deployment requirements, in particular the number of replicas, are satisfied when the appropriate resources are available. When run with two or more replicas, the router can be resilient to failures, providing a highly-available service. Depending on how the router instances are discovered (via a service, DNS entry, or IP addresses), this could impose operational requirements to handle failure cases when one or more router instances are "unreachable".

For some IP-based traffic services, virtual IP addresses (VIPs) should always be serviced for as long as a single instance is available. This simplifies the operational overhead and handles failure cases gracefully.

Use cases for high-availability include:

You can configure a highly-available router or network setup by running multiple instances of the pod and fronting them with a balancing tier. This can be something as simple as DNS round robin, or as complex as multiple load-balancing layers.

Using IP failover involves switching IP addresses to a redundant or stand-by set of nodes on failure conditions.

The oadm ipfailover command helps set up the VIP failover configuration. As an administrator, you can configure IP failover on an entire cluster, or on a subset of nodes, as defined by the labeled selector. If you are running in production, match the labeled selector with at least two nodes to ensure you have failover protection and provide a --replicas=<n> value that matches the number of nodes for the given labeled selector:

$ oadm ipfailover [<Ip_failover_config_name>] <options> --replicas=<n>

The oadm ipfailover command ensures that a failover pod runs on each of the nodes matching the constraints or label used. This pod uses VRRP (Virtual Router Redundancy Protocol) with Keepalived to ensure that the service on the watched port is available, and, if needed, Keepalived will automatically float the VIPs if the service is not available.

You can allow developers to create their own projects. An accessible endpoint exists that will provision a project according to a template. This endpoint is made accessible when a developer creates a new project.

The API server automatically provisions projects based on the template that is defined in the projectRequestTemplate parameter of the master-config.yaml file. If the parameter is not defined, the API server creates a default template that creates a project with the requested name, and assigns the requesting user to the "admin" role for that project.

To create your own custom project template:

When a project request is submitted, the API substitutes the following parameters into the template:

Access to the API is granted to developers with the self-provisioner role and the self-provisioners cluster role binding. This role is available to all authenticated developers by default.

Deleting the self-provisionerscluster role binding will deny permissions for self-provisioning any new projects. When disabling self-provisioning, set the projectRequestMessage parameter in the master-config.yaml file instructing developers on how to request a new project. This parameter is a string that will be presented to the developer in the web console and command line when they attempt to self-provision a project. For example:

Contact your system administrator at projectname@example.com to request a project.

or:

To request a new project, fill out the project request form located at
https://internal.example.com/openshift-project-request.

You can provision your OpenShift cluster with persistent storage using NFS. Some familiarity with Kubernetes and NFS is assumed.

The Kubernetes persistent volume framework allows administrators to provision a cluster with persistent storage and gives users a way to request those resources without having any knowledge of the underlying infrastructure.

For a detailed example, see the guide for WordPress and MySQL using NFS.

Storage must exist in the underlying infrastructure before it can be mounted as a volume in OpenShift. All that is required for NFS is a distinct list of servers and paths and the PersistentVolume API.

{
  "apiVersion": "v1",
  "kind": "PersistentVolume",
  "metadata": {
    "name": "pv0001"
  },
  "spec": {
    "capacity": {
        "storage": "5Gi"
    },
    "accessModes": [ "ReadWriteOnce" ],
    "nfs": {
        "path": "/tmp",
        "server": "172.17.0.2"
    },
    "persistentVolumeReclaimPolicy": "Recycle"
  }
}

NFS implements the Kubernetes Recyclable plug-in interface. Automatic processes handle reclamation tasks based on policies set on each persistent volume.

By default, persistent volumes are set to Retain. NFS volumes which are set to Recycle are scrubbed (i.e., rm -rf is run on the volume) after being released from their claim (i.e, after the user’s PersistentVolumeClaim bound to the volume is deleted). Once recycled, the NFS volume can be bound to a new claim.

As discussed, clusters can be provisioned with persistent storage using NFS in the following way:

They are many ways that you can use scripts to automate the above tasks. You can use an example Ansible playbook to help you get started.

By default, SELinux does not allow writing from a pod to a remote NFS server. The NFS volume mounts correctly, but is read-only.

To enable writing in SELinux on each node:

# setsebool -P virt_use_nfs 1

The -P option makes the bool persistent between reboots.

Additionally, in order to enable arbitrary container users to read and write the volume, each exported volume on the NFS server itself should conform to the following:

/<example_fs> *(rw,all_squash)

chown -R nfsnobody:nfsnobody /<example_fs>

chmod 777 /<example_fs>

This topic describes how administrators should work with iptables. openshift-sdn takes care of adding the necessary iptables rules to make it work. Kubernetes and Docker also manage iptables for port forwarding and services.

Docker doesn’t monitor the iptables rules that it adds for exposing ports from containers and hence if iptables service is restarted, then these rules are lost. So, to safely restart iptables, it is recommended that the rules are saved and restored.

$ iptables-save > /path/to/iptables.bkp
$ systemctl restart iptables
$ iptables-restore < /path/to/iptables.bkp

This topic describes how to set up container networking using existing switches and routers and the kernel networking stack in Linux. The setup requires that the network administrator or a script modifies the router or routers when new nodes are added to the cluster.

The following diagram shows the container networking setup described in this topic. It uses one Linux node with two network interface cards serving as a router, two switches, and three nodes connected to these switches.

The following describes a general network setup:

The following procedure assumes a Linux box with multiple NICs is used as a router. Modify the steps as required to use the syntax for a particular router:

Builds in OpenShift are run in privileged containers that have access to the Docker daemon socket. As a security measure, it is recommended to limit who can run builds and the strategy that is used for those builds. Custom builds are inherently less safe than Source builds, given that they can execute any code in the build with potentially full access to the node’s Docker socket. Docker build permission should also be granted with caution as a vulnerability in the Docker build logic could result in a privileges being granted on the host node.

By default, project administrators (the admin role) and project editors (the edit role) are granted permission to use all build strategies (Docker, Source-to-Image, and Custom).

You can control who can build with what build strategy using an authorization policy. Each build strategy has a corresponding build subresource. Granting permission to create on the build subresource allows the user to create builds of that type.

To prevent access to a particular build strategy globally, log in as a user with cluster-admin privileges and edit each of the default roles:

$ oc edit clusterrole admin
$ oc edit clusterrole edit

For each role, remove the line that corresponds to the resource of the strategy to disable:

kind: ClusterRole
metadata:
  name: admin
...

rules:
- attributeRestrictions: null
  resources:
  - builds/custom
  - builds/docker 1
  - builds/source
  - pods/exec
  - pods/portforward

...

To allow only a set of specific users to create builds with a particular strategy:

Similar to granting the build strategy role to a user globally, to allow only a set of specific users within a project to create builds with a particular strategy:

OpenShift uses image change triggers in a build configuration to detect when an image stream tag has been updated. You can use the oadm build-chain command to build a dependency tree that identifies which images would be affected by updating an image in a specified image stream.

The build-chain tool can determine which builds to trigger; it analyzes the output of those builds to determine if they will in turn update another image stream tag. If they do, the tool continues to follow the dependency tree. Lastly, it outputs a graph specifying the image stream tags that would be impacted by an update to the top-level tag. The default output syntax for this tool is set to a human-readable format; the DOT format is also supported.

The following table describes common build-chain usage and general syntax:

$ oadm build-chain <image-stream>

$ oadm build-chain <image-stream>:v2 \
    -o dot \
    | dot -T svg -o deps.svg

$ oadm build-chain <image-stream>:v1 \
    -n test --all

Administrators can customize the web console using extensions, which let you run scripts and load custom stylesheets when the web console loads. You can change the look and feel of nearly any aspect of the user interface in this way.

To add scripts and stylesheets, edit the master configuration file. The scripts and stylesheet files must exist on the Asset Server and are added with the following options:

assetConfig:
  ...
  extensionScripts:
    - /path/to/script1.js
    - /path/to/script2.js
    - ...
  extensionStylesheets:
    - /path/to/stylesheet1.css
    - /path/to/stylesheet2.css
    - ...

Relative paths are resolved relative to the master configuration file. To pick up configuration changes, restart the server.

Custom scripts and stylesheets are read once at server start time. To make developing extensions easier, you can reload scripts and stylesheets on every request by enabling development mode with the following setting:

assetConfig:
  ...
  extensionDevelopment: true

When set, the web console reloads any changes to existing extension script or stylesheet files when you refresh the page in your browser. You still must restart the server when adding new extension stylesheets or scripts, however. This setting is only recommended for testing changes and not for production.

The following examples show common ways you can customize the web console.

Customizing the Logo

The following style changes the logo in the web console header:

#header-logo {
  background-image: url("https://www.example.com/images/logo.png");
  width: 160px;
  height: 10px;
}

Replace the example.com URL with a URL to an actual image, and adjust the width and height. The ideal height is 10px.

Save the style to a file, for example logo.css, and add it to the master configuration file:

assetConfig:
  ...
  extensionStylesheets:
    - /path/to/logo.css

Changing the Header Color

The following style changes the header color to dark blue:

.navbar-header {
  background-color: #2B3856;
}

Save the style to a file, for example theme.css, and add it to the master configuration file:

assetConfig:
  ...
  extensionStylesheets:
    - /path/to/theme.css

Adding a Link to the Header

The following script adds a link into the web console header:

$(".navbar-utility").prepend('<li><a href="http://example.com/status/">System Status</a></li>');

Save this script to a file, for example nav-link.js, and add it to the master configuration file:

assetConfig:
  ...
  extensionScripts:
    - /path/to/nav-link.js

You can serve other files from the Asset Server as well. For example, you might want to make the CLI executable available for download from the web console or add images to use in a custom stylesheet.

Add the directory with the files you want using the following configuration option:

assetConfig:
  ...
  extensions:
    - name: images
      sourceDirectory: /path/to/my_images

The files under the /path/to/my_images directory will be available under the URL /<context>/extensions/images in the web console.

To reference these files from a stylesheet, you should generally use a relative path. For example:

#header-logo {
  background-image: url("../extensions/images/my-logo.png");
}

assetConfig:
  ...
  extensions:
    - name: my_extension
      sourceDirectory: /path/to/myExtension
     .adoc5Mode: true

You can also change the login page for the web console. Run the following command to create a template you can modify:

$ oadm create-login-template > login-template.adoc

Edit the file to change the styles or add content, but be careful not to remove any required parameters inside curly braces.

To use your custom login page, set the following option in the master configuration file:

oauthConfig:
  ...
  templates:
    login: /path/to/login-template.adoc

Relative paths are resolved relative to the master configuration file. You must restart the server after changing this configuration.

You can change the location a console user is sent to when logging out of the console by modifying the logoutURL parameter in the /etc/openshift/master/master-config.yaml file:

...
assetConfig:
  logoutURL: "http://www.example.com"
...

This can be useful when authenticating with Request Header and OAuth or OpenID identity providers, which require visiting an external URL to destroy single sign-on sessions.

Production environments can deny direct access to the Internet and instead have an HTTP or HTTPS proxy available. Configuring OpenShift to use these proxies can be as simple as setting standard environment variables in configuration or JSON files.

OpenShift node hosts need to perform push and pull operations to Docker registries. If you have a registry that does not need a proxy for nodes to access, include the NO_PROXY parameter with the registry’s host name, the registry service’s IP address, and service name. This blacklists that registry, leaving the external HTTP proxy as the only option.

S2I builds fetch dependencies from various locations. You can use a .sti/environment file to specify simple shell variables and OpenShift will react accordingly when seeing build images.

The following are the supported proxy environment variables with example values:

HTTP_PROXY=http://USERNAME:PASSWORD@10.0.1.1:8080/
HTTPS_PROXY=https://USERNAME:PASSWORD@10.0.0.1:8080/
NO_PROXY=master.hostname.example.com

The example templates available in OpenShift by default do not include settings for HTTP proxies. For existing applications based on these templates, modify the source section of the application’s build configuration and add proxy settings:

...
source:
  type: Git
  git:
    uri: https://github.com/openshift/ruby-hello-world
    httpProxy: http://proxy.example.com
    httpsProxy: https://proxy.example.com
...

This is similar to the process for using proxies for Git cloning.

You can set the NO_PROXY, HTTP_PROXY, and HTTPS_PROXY environment variables in the templates.spec.containers stanza in a deployment configuration to pass proxy connection information. The same can be done for configuring a Pod’s proxy at runtime:

...
containers:
- env:
  - name: "HTTP_PROXY"
    value: "http://USER:PASSWORD@IPADDR:PORT"
...

You can also use the oc env command to update an existing deployment configuration with a new environment variable:

$ oc env dc/frontend HTTP_PROXY=http://USER:PASSWORD@IPADDR:PORT

If you have a ConfigChange trigger set up in your OpenShift instance, the changes happen automatically. Otherwise, manually redeploy your application for the changes to take effect.

If your Git repository can only be accessed using a proxy, you can define the proxy to use in the source section of the BuildConfig. You can configure both a HTTP and HTTPS proxy to use. Both fields are optional.

...
source:
  type: Git
  git:
    uri: "https://github.com/openshift/ruby-hello-world"
    httpProxy: http://proxy.example.com
    httpsProxy: https://proxy.example.com
...

OpenShift Enterprise 3.0 release.