Skip to main content

Redis Rolling Upgrade on Pivotal Cloud Foundry (PCF)

· 7 min read

Introduction​

In this article, I will show you how to update Redis Enterprise on PCF and see how Redis Enterprise cluster will guarantee a service continuity using out of the box failover.

If you need a Cloud Foundry application that calls Redis automatically you can use this project simple-redis-spring-demo-pcf.

For this article, I will upgrade Redis Enterprise for PCF from the version v5.4.2400147 to the latest version, currently v5.4.40700169.

Prerequisites

  • Pivotal Cloud Foundry up & running
    • Administrator access to Ops Manager and Apps Manager
  • One of more Redis databases running on PCF
    • My environment has2 databases in version v5.4.2400147
    • One wit replication (db:4) another one without replication (db:5)

Initial Environment​

Let's take a look to the environment before the update; for this you can access the Redis Enterprise Cluster Management Console:

Do not use this to create/delete a database, you must use Cloud Foundry to do it. (cf command or UI)

In the Web console, go to "Cluster" then "Configuration", you can see the version of Redis Labs Enterprise Cluster (5.4.0-24), and Redis (5.0.2) versions.

You can also use the rladmin command line to achieve this.

Checking Redis cluster using the command line

SSH to your Ops Manager and, bosh ssh to one of the Redis cluster VMs.

When I run the bosh vms command on my environment I can see the following VMs related to my Redis deployment:

The deployment is made of 5 VMs:

  • the 3 first VMs are the Redis Nodes
  • the 2 others are related to the PCF integration (Registrar and Service Broker)

We can look in more details into the role of each VMS in the cluster, for this I will bosh ssh into one of the nodes:

$ bosh -d redis-enterprise-[your-deployment-id] ssh redis-enterprise-node/[your-vm-id]

Once connected use the sudo rladmin status to look at the Redis cluster deployed on PCF.

In this cluster you see:

  • in the Cluster Nodes section that we have 3 nodes in version 5.4.0-24
  • in the Databases section that we have 2 database instances, the name is generated by Cloud Foundry. In this environment, the db:4 is replicated with shards on node:1 (master) and node:2 (slave/replica), while db:5 is not replicated.

Let's now see the Redis version of the databases using:

  • sudo rladmin status databases extra redis_version

As expected the version if 5.0.2, the same value that you have seen in the Web console.

Installing the latest version of Redis Enterprise for PCF​

Once the latest release of Redis Enterprise on PCF is imported, the upgrade is easy to do:

  1. Click on "Redis Enterprise on PCF" in the left menu.

  2. Click on the "+" link.

    • The tiles is updated to the new version, you can review the configuration, not needed in this tutorial.

  3. Click on "Review Pending Changes" button.

  4. Unselect all product except Redis

  5. Click Apply Changes

Once you have clicked the update process will start, and you can follow the progress using the log information.

Nevertheless, it is interesting to see what is happening behind the scene using the command line on the VMS.

The update process using PCF will do the following:

  • Update and restart each node one by one (the 5 nodes of the Redis Enterprise deployment)
  • during these steps, Redis Cluster will fail over moving the master and endpoint to another node to provide service continuity to the applications.

Let's look at the following screenshots to see how the rolling upgrade was done by PCF.

Starting Point​

The cluster is up and running with 3 nodes with the version 5.4.0-24, and the node:1 is the master of the cluster

Cluster Nodes:

Endpoints:

The node:1 is also the endpoint for the db:4

Shards:

You can see 3 shards in this deployment:

  • db:4 is replicated and has 2 shards the master on node:1 and a replica on node:2, the failover will automatically happen with no data loss.
  • db:5 is not replicated and has a single shard, so the database will be recreated fresh on a new node during the update.

So if you want to have a full service continuity with no data loss it is mandatory to use replication.

PCF Updating Node 1​

PCF has now started the process and stopped the node:1.

Cluster Nodes:

All the nodes are still on the "old version", but the cluster master has been moved now to node:2; so applications will continue to work.

The errors are here to indicate that the node:1 is not accessible, and the node:3 also raised an error since the replication link is not available.

Endpoints:

Here we see that the db:4 endpoint, now on node:2, Redis Enterprises cluster manager has moved the endpoint to this node automatically.

Shards:

  • db:4 is up and the master shard has been moved from node:1 to node:2
  • db:5 is not present anymore, a new master will be created automatically on node:3, but empty.

The fail over is done transparently with no impact for the application.

PCF is restarting the updated Node 1​

Once the node:1 VM is restarted with the updated version of Redis Enterprise you can see the new version number and status.

Cluster Nodes:

The 3 nodes of the cluster are up and running, and you can see that the node:1 has been updated to the new version 5.4.4-7.

The master is still the node:2

For a short time the cluster will have heterogeneous nodes, this is not an issue.

Shards:

You can see that the db:4 shards have the status OK, OLD VERSION that indicates that:

  • the database is up and running
  • but the database itself has not yet been updated to the latest Redis version

The update of the database is done automatically, so after a while, if you run the command sudo rladmin status databases extra redis_version you will see something like:

Updating all the nodes​

The PCF update will continue and upgrade:

  • node:2, Redis Cluster will move the masters (cluster, shard, endpoint) to another node, in our case node:1 for the replicated database (db:4)
  • once the node:2 is done the same work will be done on node 3.

Cluster Nodes:

All the nodes of the clusters are now updated to the latest version of Redis Enterprise (5.4.4-7) supported on PCF.

Shards:

The update of the database is done automatically, so after a while if your run the command sudo rladmin status databases extra redis_version you will see something like:

In this example I am doing a “minor upgrade”, from Redis Cluster 5.4.0/Redis 5.0.2 to Redis Cluster 5.4.4/Redis 5.0.4, and everything is done automatically.

If you are doing a major upgrade for example from 4.x to 5.x, the cluster will automatically be updated to the proper release, but you will have to manually update the existing databases as documented here.

Updating Redis on PCF Services​

During the update, you will see other VMs stopped and started in the process. These VMs are used for:

  • Redis Registrar
  • ResisLabs Service Broker

These services and nodes are not part of the "Redis Enterprise" per se, but are part of the integration with PCF.

Conclusion​

The update of the Redis Cluster is now complete:

  • All the nodes are on 5.4.4-7 (from 5.4.0-024)
  • All the databases have been updated to the new Redis 5.0.4 (from 5.0.2)

The upgrade has been done automatically without any interruptions of service:

  • PCF scripts have been responsible for upgrading, stoping and starting each part of the installation in the correct order
  • while Redis Enterprise Cluster has been responsible for keeping the databases available for the applications, during the process.