One of the most common use cases for Redis is to use it the database as a caching layer for your data, but Redis can do a lot more (I will publish new articles later)!
In this article, you will learn using a straightforward service, how to cache the result on some REST API calls to accelerate the data access, and also reduce the number of calls to external services.
For this example, I am using the "Redis Movie Database" application, a microservice-based application that I created to showcase and explain various features of Redis and Redis Enterprise.
In this article, I will explain how to secure your Redis databases using SSL (Secure Sockets Layer). In production, it is a good practice to use SSL to protect the data that are moving between various computers (client applications and Redis servers). Transport Level Security (TLS) guarantees that only allowed applications/computers are connected to the database, and also that data is not viewed or altered by a middle man process.
You can secure the connections between your client applications and Redis cluster using:
One-Way SSL: the client (your application) get the certificate from the server (Redis cluster), validate it, and then all communications are encrypted
Two-Way SSL: (aka mutual SSL) here both the client and the server authenticate each other and validate that both ends are trusted.
In this article, I will focus on the Two-Way SSL, and using Redis Enterprise.
In this article, I will show you how to update Redis Enterprise on PCF and see how Redis Enterprise cluster will guarantee a service continuity using out of the box failover.
As part of my on-boarding/training at RedisLabs I continue to play with the product, and I have decided today to install a local 3 nodes cluster of Redis Enterprise Software (RS); and show how easy is to move from a single node/shard database to a multi nodes highly available one.
Once your cluster is up & running, you will kill some containers to see how the system automatically fail-over to guarantee service continuity.
This is a perfect environment for learning, developing and testing your applications, but it is not supported in production; for production, you can use:
As you may have seen, I have joined Redis Labs a month ago; one of the first task as a new hire is to learn more about Redis. So I learned, and I am still learning.
This is when I discovered Redis Streams. I am a big fan of streaming-based applications so it is natural that I start with a small blog post explaining how to use Redis Streams and Java.
What is Redis Streams?
Redis Streams is a Redis Data Type, that represents a log so you can add new information/message in an append-only mode (this is not 100% accurate since you can remove messages from the log). Using Redis Streams you can build "Kafka Like" applications, what I mean by that you can:
create applications that publish and consume messages (nothing extraordinary here, you could already do that with Redis Pub/Sub)
consume messages that are published even when your client application (consumer) is not running. This is a big difference with Redis Pub/Sub
consume messages starting a specific offset, for example, read the whole history, or only new messages
In addition to this, Redis Streams has the concept of Consumer Groups. Redis Streams Consumer Groups, like Apache Kafka ones, allows the client applications to consume messages in a distributed fashion (multiple clients), providing an easy way to scale and create highly available systems.
In this project you will learn how to use the MapR-DB JSON REST API to:
Create and Delete tables
Create, Read, Update and Delete documents (CRUD)
MapR Extension Package 5.0 (MEP) introduced the MapR-DB JSON REST API that allow application to use REST to interact with MapR-DB JSON.
Most of the applications have to deal with "master/detail" type of data:
breweries and beer
department and employees
invoices and items
...
This is necessary for example to create application view like the following:
With Couchbase, and many of the document oriented databases you have different ways to deal with this, you can:
Create a single document for each master and embed all the children in it
Create a master and child documents and link them using an attribute.
In the first case when all the information are stored in a single document it is quite easy to use the entire set of data and for example create a screen that shows all the information, but what about the second case?
In this post I am explaining how it is possible to use Couchbase views to deal with that an make it easy to create master/detail views.
As an ex-Oracle employee, I am using the infamous SCOTT schema with the DEPT and EMP tables, as the first example. Then at the end I will extend this to the beer sample data provided with Couchbase.
Couchbase is a schema-less database, and you can store “anything you want” into it, but for this you need to use JSON documents and create 2 types of document : “department” and “employee”.
The way we usually do that is using a technical attribute to type the document. So the employee and department document will look as follow :
This shows just the document, in Couchbase you have to associate a document to a key. For this example I am using a simple pattern : type__id, for these documents the keys will look like the following:
dept__10
emp__20
You can use any pattern to create a key, for example for the employee you could chose to put an email.
Note the “dept_id” attribute in the employee document. This is the key of the department; you can see that as the “foreign key”. But remember, the relationship between the department and employee documents are managed entirely by the application, Couchbase Server does not enforce it.
I have created a Zip file that contains all the data, you can download it from here; and import the data into Couchbase using the cbdocloader utility. To import the data run the following command from a terminal window:
Queries inside Couchbase are based on views; and views build indexes, so we have to create a view, a "collated view" to be exact.
The idea behing a collated view is to produce an index where the keys are ordered so that a parent id appears first followed by its children. So we are generating an index that will look like:
This is in fact quite easy to do with Couchbase views. The only trick here is to control the order and be sure the master is always the first one, just before its children.
So to control this we can create an compound key that contains the department id, a "sorting" element and the name (beer or brewery)
So the map function of the view looks like the following:
The key is composed of:
the department id extracted from the department document itself or from the employee document depending of the type of document
an arbitrary number that is used to control the ordering. I put 0 for the department, 1 for the employee
the name of the department or the employee, this also allows to sort the result by name
In addition to the key, this view is used to emit some information about the salary of the employees. The salary is simply the sum of the salary plus thecommission when exists. The result of the view looks like:
With this view you can now use the result of the view to build report for your application. It is also possible to use parameters in your query to see only a part of the data, for example by departement, using for example startkey=["dept__20",0]&endkey=["dept__20",2] to view only the data -Department and Employees- of the deparment 20-Research.
You can create an equivalent view for the beer sample application where you print all the breweries and beers in the same report. The view is called "all_with_beers" in the design document "brewery". The view looks like:
Once you have publish it in production you can use it in the Beer Sample application, for this example I have modified the Java sample application.
Create a servlet to handle user request and on the /all URI.
The "BreweryAndBeerServlet" that calls the view using the following code :
The result of the query is set into the HttpRequest and the all.jsp page is executed. The JSP uses JSTL to print the information using the following code:
The JSP gets the items from the HTTP Request and loops on each items, then based on the type of the item the information is printed. The final result looks like :
In this example I am using Java to inject Tweets into Couchbase, you can obviously use another langage if you want to.
The sources of this project are available on my Github repository Twitter Injector for Couchbase you can also download the Binary version here, and execute the application from the command line, see Run The Application paragraph. Do not forget to create your Twitter oAuth keys (see next paragraph)
The first thing to do to be able to use the Twitter API is to create a set of keys. If you want to learn more about all these keys/tokens take a look to the oAuth protocol : http://oauth.net/
The following code is the main code of the application:
Some basic explanation:
The setUp() method simply reads the twitter4j.properties file from the classpath to build the Couchbase connection string.
The injectTweets opens the Couchbase connection -line 76- and calls the TwitterStream API.
A Listener is created and will receive all the onStatus(Status status) from Twitter. The most important method is onStatus() that receive the message and save it into Couchbase.
One interesting thing : since Couchbase is a JSON Document database it allows your to just take the JSON String and save it directly. cbClient.add(idStr,0 ,twitterMessage);
