8.4.5. A/B deployments
The A/B deployment strategy lets you try a new version of the application in a limited way in the production environment. You can specify that the production version gets most of the user requests while a limited fraction of requests go to the new version.
Because you control the portion of requests to each version, as testing progresses you can increase the fraction of requests to the new version and ultimately stop using the previous version. As you adjust the request load on each version, the number of pods in each service might have to be scaled as well to provide the expected performance.
In addition to upgrading software, you can use this feature to experiment with versions of the user interface. Since some users get the old version and some the new, you can evaluate the user’s reaction to the different versions to inform design decisions.
For this to be effective, both the old and new versions must be similar enough that both can run at the same time. This is common with bug fix releases and when new features do not interfere with the old. The versions require N-1 compatibility to properly work together.
OpenShift Container Platform supports N-1 compatibility through the web console as well as the CLI.
8.4.5.1. Load balancing for A/B testing 링크 복사링크가 클립보드에 복사되었습니다!
The user sets up a route with multiple services. Each service handles a version of the application.
Each service is assigned a weight and the portion of requests to each service is the service_weight divided by the sum_of_weights. The weight for each service is distributed to the service’s endpoints so that the sum of the endpoint weights is the service weight.
The route can have up to four services. The weight for the service can be between 0 and 256. When the weight is 0, the service does not participate in load balancing but continues to serve existing persistent connections. When the service weight is not 0, each endpoint has a minimum weight of 1. Because of this, a service with a lot of endpoints can end up with higher weight than intended. In this case, reduce the number of pods to get the expected load balance weight.
Procedure
To set up the A/B environment:
Create the two applications and give them different names. Each creates a
Deploymentobject. The applications are versions of the same program; one is usually the current production version and the other the proposed new version.Create the first application. The following example creates an application called
ab-example-a:$ oc new-app openshift/deployment-example --name=ab-example-aCreate the second application:
$ oc new-app openshift/deployment-example:v2 --name=ab-example-bBoth applications are deployed and services are created.
Make the application available externally via a route. At this point, you can expose either. It can be convenient to expose the current production version first and later modify the route to add the new version.
$ oc expose svc/ab-example-aBrowse to the application at
ab-example-a.<project>.<router_domain>to verify that you see the expected version.When you deploy the route, the router balances the traffic according to the
weightsspecified for the services. At this point, there is a single service with defaultweight=1so all requests go to it. Adding the other service as analternateBackendsand adjusting theweightsbrings the A/B setup to life. This can be done by theoc set route-backendscommand or by editing the route.참고When using
alternateBackends, also use theroundrobinload balancing strategy to ensure requests are distributed as expected to the services based on weight.roundrobincan be set for a route by using a route annotation. See the Additional resources section for more information about route annotations.Setting the
oc set route-backendto0means the service does not participate in load balancing, but continues to serve existing persistent connections.참고Changes to the route just change the portion of traffic to the various services. You might have to scale the deployment to adjust the number of pods to handle the anticipated loads.
To edit the route, run:
$ oc edit route <route_name>Example output
apiVersion: route.openshift.io/v1 kind: Route metadata: name: route-alternate-service annotations: haproxy.router.openshift.io/balance: roundrobin # ... spec: host: ab-example.my-project.my-domain to: kind: Service name: ab-example-a weight: 10 alternateBackends: - kind: Service name: ab-example-b weight: 15 # ...
8.4.5.1.1. Managing weights of an existing route using the web console 링크 복사링크가 클립보드에 복사되었습니다!
Procedure
-
Navigate to the Networking
Routes page. -
Click the Actions menu
next to the route you want to edit and select Edit Route.
-
Edit the YAML file. Update the
weightto be an integer between0and256that specifies the relative weight of the target against other target reference objects. The value0suppresses requests to this back end. The default is100. Runoc explain routes.spec.alternateBackendsfor more information about the options. - Click Save.
8.4.5.1.2. Managing weights of an new route using the web console 링크 복사링크가 클립보드에 복사되었습니다!
-
Navigate to the Networking
Routes page. - Click Create Route.
- Enter the route Name.
- Select the Service.
- Click Add Alternate Service.
-
Enter a value for Weight and Alternate Service Weight. Enter a number between
0and255that depicts relative weight compared with other targets. The default is100. - Select the Target Port.
- Click Create.
8.4.5.1.3. Managing weights using the CLI 링크 복사링크가 클립보드에 복사되었습니다!
Procedure
To manage the services and corresponding weights load balanced by the route, use the
oc set route-backendscommand:$ oc set route-backends ROUTENAME \ [--zero|--equal] [--adjust] SERVICE=WEIGHT[%] [...] [options]For example, the following sets
ab-example-aas the primary service withweight=198andab-example-bas the first alternate service with aweight=2:$ oc set route-backends ab-example ab-example-a=198 ab-example-b=2This means 99% of traffic is sent to service
ab-example-aand 1% to serviceab-example-b.This command does not scale the deployment. You might be required to do so to have enough pods to handle the request load.
Run the command with no flags to verify the current configuration:
$ oc set route-backends ab-exampleExample output
NAME KIND TO WEIGHT routes/ab-example Service ab-example-a 198 (99%) routes/ab-example Service ab-example-b 2 (1%)To override the default values for the load balancing algorithm, adjust the annotation on the route by setting the algorithm to
roundrobin. For a route on OpenShift Container Platform, the default load balancing algorithm is set torandomorsourcevalues.To set the algorithm to
roundrobin, run the command:$ oc annotate routes/<route-name> haproxy.router.openshift.io/balance=roundrobinFor Transport Layer Security (TLS) passthrough routes, the default value is
source. For all other routes, the default israndom.To alter the weight of an individual service relative to itself or to the primary service, use the
--adjustflag. Specifying a percentage adjusts the service relative to either the primary or the first alternate (if you specify the primary). If there are other backends, their weights are kept proportional to the changed.The following example alters the weight of
ab-example-aandab-example-bservices:$ oc set route-backends ab-example --adjust ab-example-a=200 ab-example-b=10Alternatively, alter the weight of a service by specifying a percentage:
$ oc set route-backends ab-example --adjust ab-example-b=5%By specifying
+before the percentage declaration, you can adjust a weighting relative to the current setting. For example:$ oc set route-backends ab-example --adjust ab-example-b=+15%The
--equalflag sets theweightof all services to100:$ oc set route-backends ab-example --equalThe
--zeroflag sets theweightof all services to0. All requests then return with a 503 error.참고Not all routers may support multiple or weighted backends.
8.4.5.1.4. One service, multiple Deployment objects 링크 복사링크가 클립보드에 복사되었습니다!
Procedure
Create a new application, adding a label
ab-example=truethat will be common to all shards:$ oc new-app openshift/deployment-example --name=ab-example-a --as-deployment-config=true --labels=ab-example=true --env=SUBTITLE\=shardA$ oc delete svc/ab-example-aThe application is deployed and a service is created. This is the first shard.
Make the application available via a route, or use the service IP directly:
$ oc expose deployment ab-example-a --name=ab-example --selector=ab-example\=true$ oc expose service ab-example-
Browse to the application at
ab-example-<project_name>.<router_domain>to verify you see thev1image. Create a second shard based on the same source image and label as the first shard, but with a different tagged version and unique environment variables:
$ oc new-app openshift/deployment-example:v2 \ --name=ab-example-b --labels=ab-example=true \ SUBTITLE="shard B" COLOR="red" --as-deployment-config=true$ oc delete svc/ab-example-bAt this point, both sets of pods are being served under the route. However, because both browsers (by leaving a connection open) and the router (by default, through a cookie) attempt to preserve your connection to a back-end server, you might not see both shards being returned to you.
To force your browser to one or the other shard:
Use the
oc scalecommand to reduce replicas ofab-example-ato0.$ oc scale dc/ab-example-a --replicas=0Refresh your browser to show
v2andshard B(in red).Scale
ab-example-ato1replica andab-example-bto0:$ oc scale dc/ab-example-a --replicas=1; oc scale dc/ab-example-b --replicas=0Refresh your browser to show
v1andshard A(in blue).
If you trigger a deployment on either shard, only the pods in that shard are affected. You can trigger a deployment by changing the
SUBTITLEenvironment variable in eitherDeploymentobject:$ oc edit dc/ab-example-aor
$ oc edit dc/ab-example-b