Querying Couchbase with N1QL

1. Overview

In this article, we’ll be looking at querying a Couchbase Server with N1QL. In a simplified way, this is SQL for NoSQL databases – with the goal of making the transition from SQL/Relational databases to a NoSQL database system easier.

There are a couple of ways of interacting with the Couchbase Server; here, we’ll be using the Java SDK to interact with the database – as it is typical for Java applications.

2. Maven Dependencies

We assume that a local Couchbase Server has been set up already; if that’s not the case, this guide can help you get started.

Let’s now add the dependency for Couchbase Java SDK to pom.xml:

<dependency>
    <groupId>com.couchbase.client</groupId>
    <artifactId>java-client</artifactId>
    <version>2.5.0</version>
</dependency>

The latest version of Couchbase Java SDK can be found on Maven Central.

We’ll also be using Jackson library to map results returned from queries; let’s add its dependency to pom.xml as well:

<dependency>
    <groupId>com.fasterxml.jackson.core</groupId>
    <artifactId>jackson-databind</artifactId>
    <version>2.9.1</version>
</dependency>

The latest version of Jackson library can be found on Maven Central.

3. Connecting to a Couchbase Server

Now, that the project is set up with the right dependencies, let’s connect to Couchbase Server from a Java application.

First, we need to start the Couchbase Server – if it’s not running already.

A guide to starting and stopping a Couchbase Server can be found here.

Let’s connect to a Couchbase Bucket:

Cluster cluster = CouchbaseCluster.create("localhost");
Bucket bucket = cluster.openBucket("test");

What we did was to connect to the Couchbase Cluster and then obtain the Bucket object.

The name of the bucket in the Couchbase cluster is test and can be created using the Couchbase Web Console. When we’re done with all the database operations, we can close the particular bucket we’ve opened.

On the other hand, we can disconnect from the cluster – which will eventually close all the buckets:

bucket.close();
cluster.disconnect();

4. Inserting Documents

Couchbase is a document-oriented database system. Let’s add a new document to the test bucket:

JsonObject personObj = JsonObject.create()
  .put("name", "John")
  .put("email", "john@doe.com")
  .put("interests", JsonArray.from("Java", "Nigerian Jollof"));

String id = UUID.randomUUID().toString();
JsonDocument doc = JsonDocument.create(id, personObj);
bucket.insert(doc);

First, we created a JSON personObj and provided some initial data. Keys can be seen as columns in a relational database system.

From the person object, we created a JSON document using JsonDocument.create(), which we’ll insert into the bucket. Note that we generate a random id using java.util.UUID class.

The inserted document can be seen in the Couchbase Web Console at http://localhost:8091 or by calling the bucket.get() with its id:

System.out.println(bucket.get(id));

5. Basic N1QL SELECT Query

N1QL is a superset of SQL, and its syntax, naturally, looks similar.

For instance, the N1QL for selecting all documents in the test bucket is:

SELECT * FROM test

Let’s execute this query in the application:

bucket.bucketManager().createN1qlPrimaryIndex(true, false);

N1qlQueryResult result
  = bucket.query(N1qlQuery.simple("SELECT * FROM test"));

First, we create a primary index using the createN1qlPrimaryIndex(), it’ll be ignored if it has been created before; creating it is compulsory before any query can be executed.

Then we use the bucket.query() to execute the N1QL query.

N1qlQueryResult is an Iterable<N1qlQueryRow> object, and thus we can print out every row using forEach():

result.forEach(System.out::println);

From the returned result, we can get N1qlMetrics object by calling result.info(). From the metrics object, we can get insights about the returned result – for example, the result and the error count:

System.out.println("result count: " + result.info().resultCount());
System.out.println("error count: " + result.info().errorCount());

On the returned result, we can use the result.parseSuccess() to check if the query is syntactically correct and parsed successfully. We can use the result.finalSuccess() to determine if the execution of the query was successful.

6. N1QL Query Statements

Let’s take a look at the different N1QL Query statements and different ways of executing them via the Java SDK.

6.1. SELECT Statement

The SELECT statement in NIQL is just like a standard SQL SELECT. It consists of three parts:

• SELECT – defines the projection of the documents to be returned
• FROM – describes the keyspace to fetch the documents from; keyspace is synonymous with table name in SQL database systems
• WHERE – specifies the additional filtering criteria

The Couchbase Server comes with some sample buckets (databases). If they were not loaded during initial setup, the Settings section of the Web Console has a dedicated tab for setting them up.

We’ll be using the travel-sample bucket. The travel-sample bucket contains data for airlines, landmark, airports, hotels, and routes. The data model can be found here.

Let’s select 100 airline records from the travel-sample data:

String query = "SELECT name FROM `travel-sample` " +
  "WHERE type = 'airport' LIMIT 100";
N1qlQueryResult result1 = bucket.query(N1qlQuery.simple(query));

The N1QL query, as can be seen above, looks very similar to SQL. Note that the keyspace name has to be put in backtick (`) because it contains a hyphen.

N1qlQueryResult is just a wrapper around the raw JSON data returned from the database. It extends Iterable<N1qlQueryRow> and can be looped over.

Invoking result1.allRows() will return all the rows in a List<N1qlQueryRow> object. This is useful for processing results with the Stream API and/or accessing each result via index:

N1qlQueryRow row = result1.allRows().get(0);
JsonObject rowJson = row.value();
System.out.println("Name in First Row " + rowJson.get("name"));

We got the first row of the returned results, and we use row.value() to get a JsonObject – which maps the row to a key-value pair, and the key corresponds to the column name.

So we got the value of column, name, for the first row using the get(). It’s as easy as that.

So far we have been using simple N1QL query. Let’s look at the parameterized statement in N1QL.

In this query, we’re going to use the wildcard (*) symbol for selecting all the fields in the travel-sample records where type is an airport.

The type will be passed to the statement – as a parameter. Then we process the returned result:

JsonObject pVal = JsonObject.create().put("type", "airport");
String query = "SELECT * FROM `travel-sample` " +
  "WHERE type = $type LIMIT 100";
N1qlQueryResult r2 = bucket.query(N1qlQuery.parameterized(query, pVal));

We created a JsonObject to hold the parameters as a key-value pair. The value of the key ‘type’, in the pVal object, will be used to replace the $type placeholder in the query string.

N1qlQuery.parameterized() accepts a query string that contains one or more placeholders and a JsonObject as demonstrated above.

In the previous sample query above, we only select a column – name. This makes it easy to map the returned result into a JsonObject.

But now that we use the wildcard (*) in the select statement, it is not that simple. The returned result is a raw JSON string:

[  
  {  
    "travel-sample":{  
      "airportname":"Calais Dunkerque",
      "city":"Calais",
      "country":"France",
      "faa":"CQF",
      "geo":{  
        "alt":12,
        "lat":50.962097,
        "lon":1.954764
      },
      "icao":"LFAC",
      "id":1254,
      "type":"airport",
      "tz":"Europe/Paris"
    }
  },

So what we need is a way to map each row to a structure that allows us to access the data by specifying the column name.

Therefore, let’s create a method that will accept N1qlQueryResult and then map every row in the result to a JsonNode object.

We choose JsonNode because it can handle a broad range of JSON data structures and we can easily navigate it:

public static List<JsonNode> extractJsonResult(N1qlQueryResult result) {
  return result.allRows().stream()
    .map(row -> {
        try {
            return objectMapper.readTree(row.value().toString());
        } catch (IOException e) {
            logger.log(Level.WARNING, e.getLocalizedMessage());
            return null;
        }
    })
    .filter(Objects::nonNull)
    .collect(Collectors.toList());
}

We processed each row in the result using the Stream API. We mapped each row to a JsonNode object and then return the result as a List of JsonNodes. 

Now we can use the method to process the returned result from the last query:

List<JsonNode> list = extractJsonResult(r2);
System.out.println(
  list.get(0).get("travel-sample").get("airportname").asText());

From the example JSON output shown previously, every row has a key the correlates to the keyspace name specified in the SELECT query – which is travel-sample in this case.

So we got the first row in the result, which is a JsonNode. Then we traverse the node to get to the airportname key, that is then printed as a text.

The example raw JSON output shared earlier provides more clarity as per the structure of the returned result.

6.2. SELECT Statement Using N1QL DSL

Other than using raw string literals for building queries we can also use N1QL DSL which comes with the Java SDK we are using.

For example, the above string query can be formulated with the DSL thus:

Statement statement = select("*")
  .from(i("travel-sample"))
  .where(x("type").eq(s("airport")))
  .limit(100);
N1qlQueryResult r3 = bucket.query(N1qlQuery.simple(statement));

The DSL is fluent and can be interpreted easily. The data selection classes and methods are in com.couchbase.client.java.query.Select class.

Expression methods like i(), eq(), x(), s() are in com.couchbase.client.java.query.dsl.Expression class. Read more about the DSL here.

N1QL select statements can also have OFFSET, GROUP BY and ORDER BY clauses. The syntax is pretty much like that of standard SQL, and its reference can be found here.

The WHERE clause of N1QL can take Logical Operators AND, OR, and NOT in its definitions. In addition to this, N1QL has provision for comparison operators like >, ==, !=, IS NULL and others.

There are also other operators that make accessing stored documents easy – the string operators can be used to concatenate fields to form a single string, and the nested operators can be used to slice arrays and cherry pick fields or element.

Let’s see these in action.

This query selects the city column, concatenate the airportname and faa columns as portname_faa from the travel-sample bucket where the country column ends with ‘States’‘, and the latitude of the airport is greater than or equal to 70:

String query2 = "SELECT t.city, " +
  "t.airportname || \" (\" || t.faa || \")\" AS portname_faa " +
  "FROM `travel-sample` t " +
  "WHERE t.type=\"airport\"" +
  "AND t.country LIKE '%States'" +
  "AND t.geo.lat >= 70 " +
  "LIMIT 2";
N1qlQueryResult r4 = bucket.query(N1qlQuery.simple(query2));
List<JsonNode> list3 = extractJsonResult(r4);
System.out.println("First Doc : " + list3.get(0));

We can do the same thing using N1QL DSL:

Statement st2 = select(
  x("t.city, t.airportname")
  .concat(s(" (")).concat(x("t.faa")).concat(s(")")).as("portname_faa"))
  .from(i("travel-sample").as("t"))
  .where( x("t.type").eq(s("airport"))
  .and(x("t.country").like(s("%States")))
  .and(x("t.geo.lat").gte(70)))
  .limit(2);
N1qlQueryResult r5 = bucket.query(N1qlQuery.simple(st2));
//...

Let’s look at other statements in N1QL. We’ll be building on the knowledge we’ve acquired in this section.

6.3. INSERT Statement

The syntax for the insert statement in N1QL is:

INSERT INTO `travel-sample` ( KEY, VALUE )
VALUES("unique_key", { "id": "01", "type": "airline"})
RETURNING META().id as docid, *;

Where travel-sample is the keyspace name, unique_key is the required non-duplicate key for the value object that follows it.

The last segment is the RETURNING statement that specifies what gets returned.

In this case, the id of the inserted document is returned as docid. The wildcard (*) signifies that other attributes of the added document should be returned as well – separately from docid. See the sample result below.

Executing the following statement in the Query tab of Couchbase Web Console will insert a new record into the travel-sample bucket:

INSERT INTO `travel-sample` (KEY, VALUE)
VALUES('cust1293', {"id":"1293","name":"Sample Airline", "type":"airline"})
RETURNING META().id as docid, *

Let’s do the same thing from a Java app. First, we can use a raw query like this:

String query = "INSERT INTO `travel-sample` (KEY, VALUE) " +
  " VALUES(" +
  "\"cust1293\", " +
  "{\"id\":\"1293\",\"name\":\"Sample Airline\", \"type\":\"airline\"})" +
  " RETURNING META().id as docid, *";
N1qlQueryResult r1 = bucket.query(N1qlQuery.simple(query));
r1.forEach(System.out::println);

This will return the id  of the inserted document as docid separately and the complete document body separately:

{  
  "docid":"cust1293",
  "travel-sample":{  
    "id":"1293",
    "name":"Sample Airline",
    "type":"airline"
  }
}

However, since we’re using the Java SDK, we can do it the object way by creating a JsonDocument that is then inserted into the bucket via the Bucket API:

JsonObject ob = JsonObject.create()
  .put("id", "1293")
  .put("name", "Sample Airline")
  .put("type", "airline");
bucket.insert(JsonDocument.create("cust1295", ob));

Instead of using the insert() we can use upsert() which will update the document if there is an existing document with the same unique identifier cust1295.

As it is now, using insert() will throw an exception if that same unique id already exists.

The insert(), however, if successful, will return a JsonDocument that contains the unique id and entries of the inserted data.

The syntax for bulk insert using N1QL is:

INSERT INTO `travel-sample` ( KEY, VALUE )
VALUES("unique_key", { "id": "01", "type": "airline"}),
VALUES("unique_key", { "id": "01", "type": "airline"}),
VALUES("unique_n", { "id": "01", "type": "airline"})
RETURNING META().id as docid, *;

We can perform bulk operations with the Java SDK using Reactive Java that underlines the SDK. Let’s add ten documents into a bucket using batch process:

List<JsonDocument> documents = IntStream.rangeClosed(0,10)
  .mapToObj( i -> {
      JsonObject content = JsonObject.create()
        .put("id", i)
        .put("type", "airline")
        .put("name", "Sample Airline "  + i);
      return JsonDocument.create("cust_" + i, content);
  }).collect(Collectors.toList());

List<JsonDocument> r5 = Observable
  .from(documents)
  .flatMap(doc -> bucket.async().insert(doc))
  .toList()
  .last()
  .toBlocking()
  .single();

r5.forEach(System.out::println);

First, we generate ten documents and put them into a List; then we used RxJava to perform the bulk operation.

Finally, we print out the result of each insert – which has been accumulated to form a List.

The reference for performing bulk operations in the Java SDK can be found here. Also, the reference for insert statement can be found here.

6.4. UPDATE Statement

N1QL also has UPDATE statement. It can update documents identified by their unique keys. We can use the update statement to either SET (update) values of an attribute or UNSET (remove) an attribute altogether.

Let’s update one of the documents we recently inserted into the travel-sample bucket:

String query2 = "UPDATE `travel-sample` USE KEYS \"cust_1\" " +
  "SET name=\"Sample Airline Updated\" RETURNING name";
N1qlQueryResult result = bucket.query(N1qlQuery.simple(query2));
result.forEach(System.out::println);

In the above query, we updated the name attribute of a cust_1 entry in the bucket to Sample Airline Updated, and we instruct the query to return the updated name.

As stated earlier, we can also achieve the same thing by constructing a JsonDocument with the same id and use the upsert() of Bucket API to update the document:

JsonObject o2 = JsonObject.create()
  .put("name", "Sample Airline Updated");
bucket.upsert(JsonDocument.create("cust_1", o2));

In this next query, let’s use the UNSET command to remove the name attribute and return the affected document:

String query3 = "UPDATE `travel-sample` USE KEYS \"cust_2\" " +
  "UNSET name RETURNING *";
N1qlQueryResult result1 = bucket.query(N1qlQuery.simple(query3));
result1.forEach(System.out::println);

The returned JSON string is:

{  
  "travel-sample":{  
    "id":2,
    "type":"airline"
  }
}

Take note of the missing name attribute – it has been removed from the document object. N1QL update syntax reference can be found here.

So we have a look at inserting new documents and updating documents. Now let’s look at the final piece of the CRUD acronym – DELETE.

6.5. DELETE Statement

Let’s use the DELETE query to delete some of the documents we have created earlier. We’ll use the unique id to identify the document with the USE KEYS keyword:

String query4 = "DELETE FROM `travel-sample` USE KEYS \"cust_50\"";
N1qlQueryResult result4 = bucket.query(N1qlQuery.simple(query4));

N1QL DELETE statement also takes a WHERE clause. So we can use conditions to select the records to be deleted:

String query5 = "DELETE FROM `travel-sample` WHERE id = 0 RETURNING *";
N1qlQueryResult result5 = bucket.query(N1qlQuery.simple(query5));

We can also use the remove() from the bucket API directly:

bucket.remove("cust_2");

Much simpler right? Yes, but now we also know how to do it using N1QL. The reference doc for DELETE syntax can be found here.

7. N1QL Functions and Sub-queries

N1QL did not just resemble SQL regarding syntax alone; it goes all the way to some functionalities. In SQL, we’ve some functions like COUNT() that can be used within the query string.

N1QL, in the same fashion, has its functions that can be used in the query string.

For example, this query will return the total number of landmark records that are in the travel-sample bucket:

SELECT COUNT(*) as landmark_count FROM `travel-sample` WHERE type = 'landmark'

In previous examples above, we’ve used the META function in UPDATE statement to return the id of updated document.

There are string method that can trim trailing white spaces, make lower and upper case letters and even check if a string contains a token. Let’s use some of these functions in a query:

Let’s use some of these functions in a query:

INSERT INTO `travel-sample` (KEY, VALUE) 
VALUES(LOWER(UUID()), 
  {"id":LOWER(UUID()), "name":"Sample Airport Rand", "created_at": NOW_MILLIS()})
RETURNING META().id as docid, *

The query above inserts a new entry into the travel-sample bucket. It uses the UUID() function to generate a unique random id which was converted to lower case using the LOWER() function. 

The NOW_MILLIS() method was used to set the current time, in milliseconds, as the value of the created_at attribute. The complete reference of N1QL functions can be found here.

Sub-queries come in handy at times, and N1QL has provision for them. Still using the travel-sample bucket, let’s select the destination airport of all routes for a particular airline – and get the country they are located in:

SELECT DISTINCT country FROM `travel-sample` WHERE type = "airport" AND faa WITHIN 
  (SELECT destinationairport 
  FROM `travel-sample` t WHERE t.type = "route" and t.airlineid = "airline_10")

The sub-query in the above query is enclosed within parentheses and returns the destinationairport attribute, of all routes associated with airline_10, as a collection.

The destinationairport attributes correlate to the faa attribute on airport documents in the travel-sample bucket. The WITHIN keyword is part of collection operators in N1QL.

Now, that we’ve got the country of destination airport of all routes for airline_10. Let’s do something interesting by looking for hotels within that country:

SELECT name, price, address, country FROM `travel-sample` h 
WHERE h.type = "hotel" AND h.country WITHIN
  (SELECT DISTINCT country FROM `travel-sample` 
  WHERE type = "airport" AND faa WITHIN 
  (SELECT destinationairport FROM `travel-sample` t 
  WHERE t.type = "route" and t.airlineid = "airline_10" )
  ) LIMIT 100

The previous query was used as a sub-query in the WHERE constraint of the outermost query. Take note of the DISTINCT keyword – it does the same thing as in SQL – returns non-duplicate data.

All the query examples here can be executed using the SDK as demonstrated earlier in this article.

8. Conclusion

N1QL takes the process of querying the document-based database like Couchbase to another whole level. It doesn’t only simplify this process, it also makes switching from a relational database system a lot easier as well.

We’ve looked at the N1QL query in this article; the main documentation can be found here. And you can learn about Spring Data Couchbase here.

As always, the complete source code is available over on Github.