1. We let MongoDB generate the _id primary key. What are the pros and cons of generating the primary key ourselves instead?
2. are there any other indexes that may be useful? hint: What if you needed a search input field in the application?
answers are available at the end of the chapter.
MongoDB Node.js Driver
This is the Node.js driver that lets you connect and interact with the MongoDB server. It is, as you probably guessed, an npm module. It provides methods very similar to what you saw in the mongo shell, but not exactly the same.
Another option is to use the mongoose client, which is an object-document-mapper as discussed in Chapter 1. But since I want you to explore the nuts and bolts of how the native driver works, which works similar to the shell commands, let’s use the native driver itself.
To start, let’s install the driver:
$ npm install mongodb --save
To connect to the database from a Node.js program, you call the connect method on the MongoClient object provided by the module. The mongodb module exports many functions and objects, of which MongoClient gives you the ability to act as a client, mainly, the connect method. The parameter to the connect function is a URL-like string starting with mongodb:// followed by the server name, and then the database name separated by a /. You can optionally include a port after the server name, separated by a :, but if you’re using the default port, this can be skipped.
Once you acquire a connection, to get a handle to any collection, you need to call its collection() method. To this method, you supply the name of the collection as the parameter to indicate which collection. Then, the CRUD operation methods can be called on the handle to the collection. For example, to connect to a database called playground and fetch some documents from a collection called employees, you do the following:
...
const MongoClient = require('mongodb').MongoClient;
MongoClient.connect('mongodb://localhost/playground', function(err, db) { db.collection('employees').find().toArray(function(err, docs) {
console.log('Result of find:', docs);
db.close();
});
});
...
Chapter 6 ■ Using MongoDB
In the above, a find() returns a cursor which you could iterate over. Calling toArray() on the cursor runs through all the documents and makes an array out of them. It calls the callback when the array is ready to be processed, passing the array as a parameter to the callback.
To insert a document, you need to use the insertOne() method on the collection, and pass it one parameter: the object to be inserted. The result of an insert contains various things, one of which is the new _id that has been generated, in a property called insertedId.
Note that all calls to the driver are asynchronous calls, which means that you don’t get the result of the call as a return value to the function call. In the above example, you supplied a callback to each of the MongoDB driver methods. When the result is ready, these callbacks will be called. The MongoDB driver documentation gives you three different paradigms for dealing with this asynchronous nature: the callbacks paradigm, one using promises, and another using the co module and generator functions. Let’s explore these three options, and one more option using the async module, which is not mentioned in the driver documentation.
Let’s do all this in a JavaScript program called trymongo.js so that you have a ready test if and when required. Let’s initialize this file with a test wrapper that we will use to exercise each of the paradigms. We’ll use a command line argument to the program to call a different function for each of the paradigms. Command line arguments to Node.js programs are available in the array process.argv. The first two elements in the array are node and the name of the program. All user arguments follow after these two. The initial contents of the file are shown in Listing 6-2.
Listing 6-2. trymongo.js: Initial Contents, Command Line Arguments Handling 'use strict';
const MongoClient = require('mongodb');
function usage() { console.log('Usage:');
console.log('node', __filename, '<option>');
console.log('Where option is one of:');
console.log(' callbacks Use the callbacks paradigm');
console.log(' promises Use the Promises paradigm');
console.log(' generator Use the Generator paradigm');
console.log(' async Use the async module');
}
if (process.argv.length < 3) {
console.log("Incorrect number of arguments");
usage();
} else {
if (process.argv[2] === 'callbacks') { testWithCallbacks();
} else if (process.argv[2] === 'promises') { testWithPromises();
Chapter 6 ■ Using MongoDB
103 } else if (process.argv[2] === 'generator') {
testWithGenerator();
} else if (process.argv[2] === 'async') { testWithAsync();
} else {
console.log("Invalid option:", process.argv[2]);
usage();
} }
We gave a name and associated a function with each of the paradigms: Callbacks, Promises, Generator, and Async. We will fill in the functions in the following subsections.
Callbacks
The conventional and oldest way to deal with asynchronous calls is to provide a callback to handle the result of the operation. As seen in the above example, it is minimal JavaScript. You pass a callback to the method, with the first parameter of the callback expecting any errors, and the second (or more) parameters expecting the result of the operation.
Callbacks are easy to understand and reason about. They work in conventional ES5 JavaScript as well. They have very little constructs that you need to learn. The callback paradigm is not particular to the MongoDB driver. There are many APIs, including the core Node.js library, that follow this paradigm.
Listing 6-3 shows how to get a database connection, use the connection to insert into a collection, and then use the result of the insert operation to retrieve the object. As you can see, one problem with this paradigm is that it can get deeply nested and complicated, depending on the depth of the chain: the result of one operation being passed to the next.
Listing 6-3. trymongo.js, testWithCallbacks: Callbacks Paradigm function testWithCallbacks() {
MongoClient.connect('mongodb://localhost/playground', function(err, db) { db.collection('employees').insertOne({id: 1, name: 'A. Callback'},
function(err, result) {
console.log("Result of insert:", result.insertedId);
db.collection('employees').find({id: 1}).toArray(function(err, docs) { console.log('Result of find:', docs);
db.close();
});
});
});
}
Chapter 6 ■ Using MongoDB
It can get even more deeply nested when you also have to handle errors (or other) conditions: you’ll soon find yourself writing the same set of statements multiple times (imagine db.close() in every error condition). This is often referred to as callback hell.
The only remedy if you want to stick to the callback paradigm is to split each small piece of code into its own function and pass that function as a parameter to a call, chaining the callback along.
Promises
Using ES2015 promises, the nesting can be avoided, and the chaining can become seemingly sequential. The above can be written as shown in Listing 6-4, using the promises paradigm.
Listing 6-4. trymongo.js, testWithPromises: Promises Paradigm function testWithPromises() {
let db;
MongoClient.connect('mongodb://localhost/playground').then(connection => { db = connection;
return db.collection('employees')....insertOne({id: 1, name: 'B. Promises'});
}).then(result => {
console.log("Result of insert:", result.insertedId);
return db.collection('employees').find({id: 1}).toArray();
}).then(docs => {
console.log('Result of find:', docs);
db.close();
}).catch(err => {
console.log('ERROR', err);
});
}
The result of every call is a promise, on to which you attach a then, which returns another promise, and so on. Finally, there is a catch block that consumes all errors.
Assuming all calls throw errors, you’ll find that error handling isn’t needed in each individual block: just one final catch() for errors at any stage is enough.
Generator and co Module
ES2015 introduces generator functions, which can be exited temporarily and called again.
The temporary exits are done using the yield statement. Between multiple calls, the function retains the execution state. These functions are declared using an asterisk after the function keyword, like function*().
Chapter 6 ■ Using MongoDB
105 A module called co takes advantage of ES2015 generators and promises to make asynchronous calls look sequential. It achieves this by asking you to sequence the asynchronous calls within one function. Then, the co module makes multiple calls to this function, where each asynchronous step temporarily exits the function.
To try this paradigm, let’s first install the co module:
$ npm install co
We did not use --save because we will not be using this module other than for trying out this paradigm. The previous example can be rewritten using the co module as shown in Listing 6-5.
Listing 6-5. trymongo.js, testWithGenerator: Generator Paradigm with co Module function testWithGenerator() {
const co = require('co');
co(function*() {
const db = yield MongoClient.connect('mongodb://localhost/playground');
const result = yield db.collection('employees')...insertOne({id: 1, name: 'C. Generator'});
console.log('Result of insert:', result.insertedId);
const docs = yield db.collection('employees').find({id: 1}).toArray();
console.log('Result of find:', docs);
db.close();
}).catch(err => {
console.log('ERROR', err);
});
}
As you can see, every asynchronous call was preceded by the keyword yield. This causes a temporary return from the function, after which the function can be resumed where it left off, if called again. The co module does the repeated calling, which is why we needed to wrap the function around co().
The async Module
Yet another way to manage callbacks is by using the async module, though this method is not mentioned in the MongoDB driver documentation. To start, let’s install the async module (again without --save because we will not be using this paradigm in the application, only trying it out in this section.)
$ npm install async
Chapter 6 ■ Using MongoDB
Apart from many other useful utilities for managing asynchronous calls, this module provides a method called waterfall, which lets you pipe the result of one asynchronous call to another. This method takes an array of functions to run. Each function is passed the results of the previous function, and a callback (which takes an error and results as its parameters). Each function in the array must call this callback when it is done. The results are passed through as a waterfall from one function to the next, that is, the outputs of one are passed to the next.
Since all the driver methods follow the same callback convention of error and results, it’s easy to pass the callbacks through the waterfall. Listing 6-6 demonstrates this.
Listing 6-6. trymongo.js, testWithAsync: Async Paradigm function testWithAsync() {
const async = require('async');
let db;
async.waterfall([
next => {
MongoClient.connect('mongodb://localhost/playground', next);
},
(connection, next) => { db = connection;
db.collection('employees').insertOne({id: 1, name: 'D. Async'}, next);
},
(insertResult, next) => {
console.log('Insert result:', insertResult.insertedId);
db.collection('employees').find({id: 1}).toArray(next);
},
(docs, next) => {
console.log('Result of find:', docs);
db.close();
next(null, 'All done');
}
], (err, result) => { if (err)
console.log('ERROR', err);
else
console.log(result);
});
}
Only in the last function did we explicitly call the callback with a null error and a string as the final result. In all of the other function calls, we just passed the callback through to the MongoDB driver method. The driver function calls the callback when the results are available. We also named the callback next, just to be clear that it will be the next function in the array that will be called when done.
Chapter 6 ■ Using MongoDB
107 Choosing any one of the paradigms is a matter of taste and familiarity. All of them are valid choices; it is really up to you as to which one you pick. For the purpose of the Issue Tracker application, I am going to pick the promises paradigm, mainly because it does not depend on any other module. Also, it lets you perform parallel tasks if you choose to do so for certain operations.
Reading from MongoDB
Let’s first modify the List API to read from the database instead of the in-memory list of issues on your server. Since we’ve initialized the database with a set of initial issues, you should be able to test this easily.
To start, we have to include the MongoDB driver that we have already installed, in server.js. Next, we need to connect to the MongoDB server and keep the connection open for future calls to the database. We do this rather than acquire a connection in every request because creating a connection takes a bit of time. Also, the MongoDB driver keeps a pool of connections automatically and reuses them if you reuse the connection handle. So, we’ll just save the connection in a global variable called db for later use.
We’ll also start the Express server only once we get the connection. Let’s call the database issuetracker. Listing 6-7 shows the connection acquisition and modified server startup section in server.js.
Listing 6-7. server.js: Modified Initialization Sequence with MongoDB connection ...
const MongoClient = require('mongodb').MongoClient;
...
let db;
MongoClient.connect('mongodb://localhost/issuetracker').then(connection => { db = connection;
app.listen(3000, () => {
console.log('App started on port 3000');
});
}).catch(error => {
console.log('ERROR:', error);
});
...
Now, we can modify the endpoint handler for /api/issues to read from the database. All we need to do is call a find() on the issues collection, convert it to an array, and return the documents returned by this call. Listing 6-8 shows the modified endpoint handler.
Chapter 6 ■ Using MongoDB
Listing 6-8. server.js: List API Modified to Use MongoDB app.get('/api/issues', (req, res) => {
db.collection('issues').find().toArray().then(issues => { const metadata = { total_count: issues.length };
res.json({ _metadata: metadata, records: issues }) }).catch(error => {
console.log(error);
res.status(500).json({ message: `Internal Server Error: ${error}` });
});
});
■Caution never skip the catch block when using promises. if you do so, any runtime error within any of the blocks will not be caught and will be silently ignored. if you don’t have a catch block, you may fail to “catch” errors in your code, even those such as a typo in one of the variable names.
Since we have changed the field name id to _id, the front-end code referring to id also needs to change. There are two places in App.jsx where this has to be done. Listing 6-9 shows the changed lines, with the change highlighted in bold.
Listing 6-9. App.jsx: Front-End Changes, Replacing id with _id ...
<td>{props.issue._id}</td>
...
const issueRows = props.issues.map(issue =><IssueRow key={issue._id} issue={issue} />)
...
Finally, now that even the List API can return a non-successful HTTP Status code, let’s make sure that we handle that in the front end, just like we did for the Create API, by looking at response.ok. The modified method loadData() is show in Listing 6-10.
Listing 6-10. App.jsx, IssueList loadData(): Error Handling loadData() {
fetch('/api/issues').then(response => { if (response.ok) {
response.json().then(data => {
console.log("Total count of records:", data._metadata.total_count);
data.records.forEach(issue => {
issue.created = new Date(issue.created);
if (issue.completionDate)
issue.completionDate = new Date(issue.completionDate);
});
Chapter 6 ■ Using MongoDB
109 this.setState({ issues: data.records });
});
} else {
response.json().then(error => {
alert("Failed to fetch issues:" + error.message) });
}
}).catch(err => {
alert("Error in fetching data from server:", err);
});
}
Now, the changes can be tested. Refresh the browser, and the two issues that we initialized using the mongo shell script should be displayed. The only change is that the ID is now a long string instead of what used to be 1 and 2. Of course, adding a new issue won’t work, which we’ll deal with in the next section.