Skip to main content

2 posts tagged with "couchbase"

View All Tags

· 5 min read

Most of the applications have to deal with "master/detail" type of data:

  • breweries and beer
  • department and employees
  • invoices and items
  • ...

This is necessary for example to create application view like the following:

With Couchbase, and many of the document oriented databases you have different ways to deal with this, you can:

  • Create a single document for each master and embed all the children in it
  • Create a master and child documents and link them using an attribute.

In the first case when all the information are stored in a single document it is quite easy to use the entire set of data and for example create a screen that shows all the information, but what about the second case?

In this post I am explaining how it is possible to use Couchbase views to deal with that an make it easy to create master/detail views.

As an ex-Oracle employee, I am using the infamous SCOTT schema with the DEPT and EMP tables, as the first example. Then at the end I will extend this to the beer sample data provided with Couchbase.

The Data

Couchbase is a schema-less database, and you can store “anything you want” into it, but for this you need to use JSON documents and create 2 types of document : “department” and “employee”.

The way we usually do that is using a technical attribute to type the document. So the employee and department document will look as follow :

Department

{
"type": "dept",
"id": 10,
"name": "Accounting",
"city": "New York"
}

Employee

{
"type": "emp",
"id": 7782,
"name": "Blake",
"job": "Clark",
"manager": 7839,
"salary": 2450,
"dept_id": "dept__10"
}

This shows just the document, in Couchbase you have to associate a document to a key. For this example I am using a simple pattern : type__id, for these documents the keys will look like the following:

  • dept__10
  • emp__20

You can use any pattern to create a key, for example for the employee you could chose to put an email.

Note the “dept_id” attribute in the employee document. This is the key of the department; you can see that as the “foreign key”. But remember, the relationship between the department and employee documents are managed entirely by the application, Couchbase Server does not enforce it.

I have created a Zip file that contains all the data, you can download it from here; and import the data into Couchbase using the cbdocloader utility. To import the data run the following command from a terminal window:

./cbdocloader -n 127.0.0.1:8091 -u Administrator -p password -b default ~/Downloads/emp-dept.zip

You can learn more about the cbdocloader tool in the documentation.

The View

Queries inside Couchbase are based on views; and views build indexes, so we have to create a view, a "collated view" to be exact.

The idea behing a collated view is to produce an index where the keys are ordered so that a parent id appears first followed by its children. So we are generating an index that will look like:

DEPT_10, Accounting
DEPT_10, Blake
DEPT_10, Miller
DEPT_20, Research
DEPT_20, Adams
DEPT_20, Ford
...

This is in fact quite easy to do with Couchbase views. The only trick here is to control the order and be sure the master is always the first one, just before its children.

So to control this we can create an compound key that contains the department id, a "sorting" element and the name (beer or brewery)

So the map function of the view looks like the following:

The key is composed of:

  • the department id extracted from the department document itself or from the employee document depending of the type of document
  • an arbitrary number that is used to control the ordering. I put 0 for the department, 1 for the employee
  • the name of the department or the employee, this also allows to sort the result by name

In addition to the key, this view is used to emit some information about the salary of the employees. The salary is simply the sum of the salary plus thecommission when exists. The result of the view looks like:

With this view you can now use the result of the view to build report for your application. It is also possible to use parameters in your query to see only a part of the data, for example by departement, using for example startkey=["dept__20",0]&endkey=["dept__20",2] to view only the data -Department and Employees- of the deparment 20-Research.

The Beer Sample Application

You can create an equivalent view for the beer sample application where you print all the breweries and beers in the same report. The view is called "all_with_beers" in the design document "brewery". The view looks like:

Once you have publish it in production you can use it in the Beer Sample application, for this example I have modified the Java sample application.

Create a servlet to handle user request and on the /all URI.

The "BreweryAndBeerServlet" that calls the view using the following code :

The result of the query is set into the HttpRequest and the all.jsp page is executed. The JSP uses JSTL to print the information using the following code:

The JSP gets the items from the HTTP Request and loops on each items, then based on the type of the item the information is printed. The final result looks like :

This extension to the Beer Sample application is available here :https://github.com/tgrall/beersample-java/tree/BreweriesAndBeers

· 4 min read

An easy way to create large dataset when playing/demonstrating Couchbase -or any other NoSQL engine- is to inject Twitter feed into your database.

For this small application I am using:

In this example I am using Java to inject Tweets into Couchbase, you can obviously use another langage if you want to.

The sources of this project are available on my Github repository Twitter Injector for Couchbase you can also download the Binary version here, and execute the application from the command line, see Run The Application paragraph. Do not forget to create your Twitter oAuth keys (see next paragraph)

Create oAuth Keys

The first thing to do to be able to use the Twitter API is to create a set of keys. If you want to learn more about all these keys/tokens take a look to the oAuth protocol : http://oauth.net/

1- Log in into the Twitter Development Portal : https://dev.twitter.com/

2- Create a new Application

Click on the "Create an App" link or go into the "User Menu > My Applications > Create a new application"

3- Enter the Application Details information

4- Click "Create Your Twitter Application" button

Your application's OAuth settings are now available :

5- Go down on the Application Settings page and click on the "Create My Access Token" button

You have now all the necessary information to create your application:

  • Consumer key
  • Consumer secret
  • Access token
  • Access token secret

These keys will be uses in the twitter4j.properties file when running the Java application from the command line

Create the Java Application

The following code is the main code of the application:

Some basic explanation:

  • The setUp() method simply reads the twitter4j.properties file from the classpath to build the Couchbase connection string.
  • The injectTweets opens the Couchbase connection -line 76- and calls the TwitterStream API.
  • A Listener is created and will receive all the onStatus(Status status) from Twitter. The most important method is onStatus() that receive the message and save it into Couchbase.
  • One interesting thing : since Couchbase is a JSON Document database it allows your to just take the JSON String and save it directly. cbClient.add(idStr,0 ,twitterMessage);

Packaging

To be able to execute the application directly from the Jar file, I am using the assembly plugin with the following informations from the pom.xml:

  ...
<archive>
<manifest>
<mainclass>com.couchbase.demo.TwitterInjector</mainclass>
</manifest>
<manifestentries>
<class-path>.</class-path>
</manifestentries>
</archive>
...

Some information:

  • The mainClass entry allows you to set which class to execute when running java -jar command.
  • The Class-Path entry allows you to set the current directory as part of the classpath where the program will search for the twitter4j.properties file.
  • The assembly file is also configure to include all the dependencies (Twitter4J, Couchbase client SDK, ...)

If you do want to build it from the sources, simply run :

mvn clean package

This will create the following Jar file ./target/CouchbaseTwitterInjector.jar

Run the Java Application

Before running the application you must create a twitter4j.properties file with the following information :

twitter4j.jsonStoreEnabled=true

oauth.consumerKey=[YOUR CONSUMER KEY]
oauth.consumerSecret=[YOUR CONSUMER SECRET KEY]
oauth.accessToken=[YOUR ACCESS TOKEN]
oauth.accessTokenSecret=[YOUR ACCESS TOKEN SECRET]

couchbase.uri.list=http://127.0.0.1:8091/pools
couchbase.bucket=default
couchbase.password=

Save the properties file and from the same location run:

jar -jar [path-to-jar]/CouchbaseTwitterInjector.jar

This will inject Tweets into your Couchbase Server. Enjoy !