Skip to main content

8 posts tagged with "nosql"

View All Tags

路 7 min read

One of the most common use cases for Redis is to use it the database as a caching layer for your data, but Redis can do a lot more (I will publish new articles later)!

In this article, you will learn using a straightforward service, how to cache the result on some REST API calls to accelerate the data access, and also reduce the number of calls to external services.

For this example, I am using the "Redis Movie Database" application, a microservice-based application that I created to showcase and explain various features of Redis and Redis Enterprise.

路 6 min read

In this article, I will explain how to secure your Redis databases using SSL (Secure Sockets Layer). In production, it is a good practice to use SSL to protect the data that are moving between various computers (client applications and Redis servers). Transport Level Security (TLS) guarantees that only allowed applications/computers are connected to the database, and also that data is not viewed or altered by a middle man process.

You can secure the connections between your client applications and Redis cluster using:

  • One-Way SSL: the client (your application) get the certificate from the server (Redis cluster), validate it, and then all communications are encrypted
  • Two-Way SSL: (aka mutual SSL) here both the client and the server authenticate each other and validate that both ends are trusted.

In this article, I will focus on the Two-Way SSL, and using Redis Enterprise.

路 7 min read

Introduction#

In this article, I will show you how to update Redis Enterprise on PCF and see how Redis Enterprise cluster will guarantee a service continuity using out of the box failover.

If you need a Cloud Foundry application that calls Redis automatically you can use this project simple-redis-spring-demo-pcf.

For this article, I will upgrade Redis Enterprise for PCF from the version v5.4.2400147 to the latest version, currently v5.4.40700169.

路 8 min read

As part of my on-boarding/training at RedisLabs I continue to play with the product, and I have decided today to install a local 3 nodes cluster of Redis Enterprise Software (RS); and show how easy is to move from a single node/shard database to a multi nodes highly available one.

Once your cluster is up & running, you will kill some containers to see how the system automatically fail-over to guarantee service continuity.

The deployment will look more or less like the schema below, (coming from RedisLabs documentation)

This is a perfect environment for learning, developing and testing your applications, but it is not supported in production; for production, you can use:

路 8 min read

As you may have seen, I have joined Redis Labs a month ago; one of the first task as a new hire is to learn more about Redis. So I learned, and I am still learning.

This is when I discovered Redis Streams. I am a big fan of streaming-based applications so it is natural that I start with a small blog post explaining how to use Redis Streams and Java.

What is Redis Streams?

Redis Streams is a Redis Data Type, that represents a log so you can add new information/message in an append-only mode (this is not 100% accurate since you can remove messages from the log). Using Redis Streams you can build "Kafka Like" applications, what I mean by that you can:

  • create applications that publish and consume messages (nothing extraordinary here, you could already do that with Redis Pub/Sub)
  • consume messages that are published even when your client application (consumer) is not running. This is a big difference with Redis Pub/Sub
  • consume messages starting a specific offset, for example, read the whole history, or only new messages

In addition to this, Redis Streams has the concept of Consumer Groups. Redis Streams Consumer Groups, like Apache Kafka ones, allows the client applications to consume messages in a distributed fashion (multiple clients), providing an easy way to scale and create highly available systems.

Enroll in the Redis University: Redis Streams to learn more and get certified.

Sample Application

The redis-streams-101-java GitHub Repository contains sample code that shows how to

  • post messages to a streams
  • consume messages using a consumer group

路 3 min read

Introduction#

In this project you will learn how to use the MapR-DB JSON REST API to:

Create and Delete tables Create, Read, Update and Delete documents (CRUD) MapR Extension Package 5.0 (MEP) introduced the MapR-DB JSON REST API that allow application to use REST to interact with MapR-DB JSON.

You can find information about the MapR-DB JSON REST API in the documentation: Using the MapR-DB JSON REST API

路 5 min read

Most of the applications have to deal with "master/detail" type of data:

  • breweries and beer
  • department and employees
  • invoices and items
  • ...

This is necessary for example to create application view like the following:

With Couchbase, and many of the document oriented databases you have different ways to deal with this, you can:

  • Create a single document for each master and embed all the children in it
  • Create a master and child documents and link them using an attribute.

In the first case when all the information are stored in a single document it is quite easy to use the entire set of data and for example create a screen that shows all the information, but what about the second case?

In this post I am explaining how it is possible to use Couchbase views to deal with that an make it easy to create master/detail views.

As an ex-Oracle employee, I am using the infamous SCOTT schema with the DEPT and EMP tables, as the first example. Then at the end I will extend this to the beer sample data provided with Couchbase.

The Data#

Couchbase is a schema-less database, and you can store 鈥渁nything you want鈥 into it, but for this you need to use JSON documents and create 2 types of document : 鈥渄epartment鈥 and 鈥渆mployee鈥.

The way we usually do that is using a technical attribute to type the document. So the employee and department document will look as follow :

Department

{  "type": "dept",  "id": 10,  "name": "Accounting",  "city": "New York"}

Employee

{  "type": "emp",  "id": 7782,  "name": "Blake",  "job": "Clark",  "manager": 7839,  "salary": 2450,  "dept_id": "dept__10"}

This shows just the document, in Couchbase you have to associate a document to a key. For this example I am using a simple pattern : type__id, for these documents the keys will look like the following:

  • dept__10
  • emp__20

You can use any pattern to create a key, for example for the employee you could chose to put an email.

Note the 鈥渄ept_id鈥 attribute in the employee document. This is the key of the department; you can see that as the 鈥渇oreign key鈥. But remember, the relationship between the department and employee documents are managed entirely by the application, Couchbase Server does not enforce it.

I have created a Zip file that contains all the data, you can download it from here; and import the data into Couchbase using the cbdocloader utility. To import the data run the following command from a terminal window:

./cbdocloader -n 127.0.0.1:8091 -u Administrator -p password -b default ~/Downloads/emp-dept.zip

You can learn more about the cbdocloader tool in the documentation.

The View#

Queries inside Couchbase are based on views; and views build indexes, so we have to create a view, a "collated view" to be exact.

The idea behing a collated view is to produce an index where the keys are ordered so that a parent id appears first followed by its children. So we are generating an index that will look like:

DEPT_10, AccountingDEPT_10, BlakeDEPT_10, MillerDEPT_20, ResearchDEPT_20, AdamsDEPT_20, Ford...

This is in fact quite easy to do with Couchbase views. The only trick here is to control the order and be sure the master is always the first one, just before its children.

So to control this we can create an compound key that contains the department id, a "sorting" element and the name (beer or brewery)

So the map function of the view looks like the following:

The key is composed of:

  • the department id extracted from the department document itself or from the employee document depending of the type of document
  • an arbitrary number that is used to control the ordering. I put 0 for the department, 1 for the employee
  • the name of the department or the employee, this also allows to sort the result by name

In addition to the key, this view is used to emit some information about the salary of the employees. The salary is simply the sum of the salary plus thecommission when exists. The result of the view looks like:

With this view you can now use the result of the view to build report for your application. It is also possible to use parameters in your query to see only a part of the data, for example by departement, using for example startkey=["dept__20",0]&endkey=["dept__20",2] to view only the data -Department and Employees- of the deparment 20-Research.

The Beer Sample Application#

You can create an equivalent view for the beer sample application where you print all the breweries and beers in the same report. The view is called "all_with_beers" in the design document "brewery". The view looks like:

Once you have publish it in production you can use it in the Beer Sample application, for this example I have modified the Java sample application.

Create a servlet to handle user request and on the /all URI.

The "BreweryAndBeerServlet" that calls the view using the following code :

The result of the query is set into the HttpRequest and the all.jsp page is executed. The JSP uses JSTL to print the information using the following code:

The JSP gets the items from the HTTP Request and loops on each items, then based on the type of the item the information is printed. The final result looks like :

This extension to the Beer Sample application is available here :https://github.com/tgrall/beersample-java/tree/BreweriesAndBeers

路 4 min read

An easy way to create large dataset when playing/demonstrating Couchbase -or any other NoSQL engine- is to inject Twitter feed into your database.

For this small application I am using:

In this example I am using Java to inject Tweets into Couchbase, you can obviously use another langage if you want to.

The sources of this project are available on my Github repository Twitter Injector for Couchbase you can also download the Binary version here, and execute the application from the command line, see Run The Application paragraph. Do not forget to create your Twitter oAuth keys (see next paragraph)

Create oAuth Keys#

The first thing to do to be able to use the Twitter API is to create a set of keys. If you want to learn more about all these keys/tokens take a look to the oAuth protocol : http://oauth.net/

1- Log in into the Twitter Development Portal : https://dev.twitter.com/

2- Create a new Application

Click on the "Create an App" link or go into the "User Menu > My Applications > Create a new application"

3- Enter the Application Details information

4- Click "Create Your Twitter Application" button

Your application's OAuth settings are now available :

5- Go down on the Application Settings page and click on the "Create My Access Token" button

You have now all the necessary information to create your application:

  • Consumer key
  • Consumer secret
  • Access token
  • Access token secret

These keys will be uses in the twitter4j.properties file when running the Java application from the command line

Create the Java Application#

The following code is the main code of the application:

Some basic explanation:

  • The setUp() method simply reads the twitter4j.properties file from the classpath to build the Couchbase connection string.
  • The injectTweets opens the Couchbase connection -line 76- and calls the TwitterStream API.
  • A Listener is created and will receive all the onStatus(Status status) from Twitter. The most important method is onStatus() that receive the message and save it into Couchbase.
  • One interesting thing : since Couchbase is a JSON Document database it allows your to just take the JSON String and save it directly. cbClient.add(idStr,0 ,twitterMessage);

Packaging#

To be able to execute the application directly from the Jar file, I am using the assembly plugin with the following informations from the pom.xml:

  ...<archive><manifest><mainclass>com.couchbase.demo.TwitterInjector</mainclass></manifest><manifestentries><class-path>.</class-path></manifestentries></archive>...

Some information:

  • The mainClass entry allows you to set which class to execute when running java -jar command.
  • The Class-Path entry allows you to set the current directory as part of the classpath where the program will search for the twitter4j.properties file.
  • The assembly file is also configure to include all the dependencies (Twitter4J, Couchbase client SDK, ...)

If you do want to build it from the sources, simply run :

mvn clean package

This will create the following Jar file ./target/CouchbaseTwitterInjector.jar

Run the Java Application#

Before running the application you must create a twitter4j.properties file with the following information :

twitter4j.jsonStoreEnabled=true
oauth.consumerKey=[YOUR CONSUMER KEY]oauth.consumerSecret=[YOUR CONSUMER SECRET KEY]oauth.accessToken=[YOUR ACCESS TOKEN]oauth.accessTokenSecret=[YOUR ACCESS TOKEN SECRET]
couchbase.uri.list=http://127.0.0.1:8091/poolscouchbase.bucket=defaultcouchbase.password=

Save the properties file and from the same location run:

jar -jar [path-to-jar]/CouchbaseTwitterInjector.jar

This will inject Tweets into your Couchbase Server. Enjoy !