This module allows data to be saved, retrieved, searched for, and deleted in a MongoDB instance. MongoDB is a great match for persisting vert.x data since it natively handles JSON (BSON) documents.
####To use this module you must have a MongoDB instance running on your network.
This is a multi-threaded worker module.
This module requires a MongoDB server to be available on the network.
The module name is mongo-persistor.
The mongo-persistor module takes the following configuration:
{
"address": <address>,
"host": <host>,
"port": <port>,
"db_name": <db_name>,
"pool_size": <pool_size>,
"fake": <fake>,
"use_ssl": <bool>,
"read_preference": <e.g. "nearest" or "primary" etecetera>
}
For example:
{
"address": "test.my_persistor",
"host": "192.168.1.100",
"port": 27000,
"pool_size": 20,
"db_name": "my_db",
"read_preference": "nearest"
}
Let's take a look at each field in turn:
addressThe main address for the module. Every module has a main address. Defaults tovertx.mongopersistor.hostHost name or ip address of the MongoDB instance. Defaults tolocalhost.portPort at which the MongoDB instance is listening. Defaults to27017.db_nameName of the database in the MongoDB instance to use. Defaults todefault_db.pool_sizeThe number of socket connections the module instance should maintain to the MongoDB server. Default is 10.fakeIf true then a fake in memory Mongo DB server is used instead (using Fongo). Useful for testing!use_sslenable SSL based connections. See http://docs.mongodb.org/manual/tutorial/configure-ssl/ for more details. Defaults tofalse.read_preferenceis the read preferences, see http://docs.mongodb.org/manual/core/read-preference/. Default is "primary".
If you want to use sharding or a replica set then you need to provide a list of seed addresses, these take priority over the host/port combination. For example:
{
"address": "test.my_persistor",
"seeds": [
{ host: "192.168.1.100", port: 27000 },
{ host: "192.168.1.101", port: 27001 }
],
"pool_size": 20,
"db_name": "my_db"
}
The seeds variable takes a list of objects which specify the host and port of each member of your seed list.
The module supports the following operations
Saves a document in the database.
To save a document send a JSON message to the module main address:
{
"action": "save",
"collection": <collection>,
"document": <document>
}
Where:
collectionis the name of the MongoDB collection that you wish to save the document in. This field is mandatory.documentis the JSON document that you wish to save.
An example would be:
{
"action": "save",
"collection": "users",
"document": {
"name": "tim",
"age": 1000,
"shoesize": 3.14159,
"username": "tim",
"password": "wibble"
}
}
When the save complete successfully, a reply message is sent back to the sender with the following data:
{
"status": "ok"
}
The reply will also contain a field _id if the document that was saved didn't specify an id, this will be an automatically generated UUID, for example:
{
"status": "ok"
"_id": "ffeef2a7-5658-4905-a37c-cfb19f70471d"
}
If you save a document which already possesses an _id field, and a document with the same id already exists in the database, then the document will be updated.
If an error occurs in saving the document a reply is returned:
{
"status": "error",
"message": <message>
}
Where
messageis an error message.
Updates a document in the database. Uses the Mongodb update function: http://www.mongodb.org/display/DOCS/Updating
To update a document send a JSON message to the module main address:
{
"action": "update",
"collection": <collection>,
"criteria": {
<criteria>
},
"objNew" : {
<objNew>
},
upsert : <upsert>
multi: <multi>
}
Where:
collectionis the name of the MongoDB collection that you wish to save the document in. This field is mandatory.
An example would be:
{
"action": "update",
"collection": "users",
"criteria": {
"_id": "tim",
},
objNew : {
$inc: {
age: 30
}
},
upsert : true,
multi : false
}
Finds matching documents in the database.
To find documents send a JSON message to the module main address:
{
"action": "find",
"collection": <collection>,
"matcher": <matcher>,
"sort": <sort_query>,
"keys": <keys>,
"skip": <offset>,
"limit": <limit>,
"timeout": <cursor timeout>,
"hint": <index hint>,
"batch_size": <batch_size>
}
Where:
collectionis the name of the MongoDB collection that you wish to search in in. This field is mandatory.matcheris a JSON object that you want to match against to find matching documents. This obeys the normal MongoDB matching rules.sort_queryprovides an order for sorting the responses that you are returned.keysis an optional JSON object that contains the fields that should be returned for matched documents. See MongoDB manual for more information. Example: { "name": 1 } will only return objects with _id and the name fieldskipis a number which determines the number of documents to skip. This is optional. By default no documents are skipped.limitis a number which determines the maximum total number of documents to return. This is optional. By default all documents are returned.timeoutis a positive number which determines how many milliseconds a cursor containing more data will be held onto. This is optional. By default, a cursor is held onto for 10 seconds.hintis either a string, specifying the name of the index to use, or a JSON object, representing the structure of the index to use. This is optional.batch_sizeis a number which determines how many documents to return in each reply JSON message. It's optional and the default value is100. Batching is discussed in more detail below.
An example would be:
{
"action": "find",
"collection": "orders",
"matcher": {
"item": "cheese"
}
}
This would return all orders where the item field has the value cheese.
When the find complete successfully, a reply message is sent back to the sender with the following data:
{
"status": "ok",
"results": <results>
}
Where
*results is a JSON array containing the results of the find operation. For example:
{
"status": "ok",
"results": [
{
"user": "tim",
"item": "cheese",
"total": 123.45
},
{
"user": "bob",
"item": "cheese",
"total": 12.23
},
{
"user": "jane",
"item": "cheese",
"total": 50.05
}
]
}
If an error occurs in finding the documents a reply is returned:
{
"status": "error",
"message": <message>
}
Where
*message is an error message.
If you would like to paginate your result :
{
"action": "find",
"collection": "orders",
"skip" : 10,
"limit" : 10,
"matcher": {}
}
You decide to display 10 documents per page. This message will retrieve second page.
Equivalence in mongoDB:
db.order.find().skip(10).limit(10)
If a find returns many documents we do not want to load them all up into memory at once and send them in a single JSON message since this could result in the server running out of RAM.
Instead, if there are more than batch_size documents to be found, the module will send a maxium of batch_size documents in each reply, and send multiple replies.
When you receive a reply to the find message containing batch_size documents the status field of the reply will be set to more-exist if there are more documents available.
To get the next batch of documents you just reply to the reply with an empty JSON message, and specify a reply handler in which to receive the next batch.
For instance, in JavaScript you might do something like:
function processResults(results) {
// Process the data
}
function createReplyHandler() {
return function(reply, replier) {
// Got some results - process them
processResults(reply.results);
if (reply.status === 'more-exist') {
// Get next batch
replier({}, createReplyHandler());
}
}
}
// Send the find request
eb.send('foo.myPersistor', {
action: 'find',
collection: 'items',
matcher: {}
}, createReplyHandler());
If there is more data to be requested and you do not reply to get the next batch within a timeout (see timeout parameter), then the underlying MongoDB cursor will be closed, and any further attempts to request more will fail.
Finds a single matching document in the database.
To find a document send a JSON message to the module main address:
{
"action": "findone",
"collection": <collection>,
"matcher": <matcher>,
"keys": <keys>
}
Where:
collectionis the name of the MongoDB collection that you wish to search in in. This field is mandatory.matcheris a JSON object that you want to match against to find a matching document. This obeys the normal MongoDB matching rules.keysis an optional JSON object that contains the fields that should be returned for matched documents. See MongoDB manual for more information. Example: { "name": 1 } will only return objects with _id and the name field
If more than one document matches, just the first one will be returned.
An example would be:
{
"action": "findone",
"collection": "items",
"matcher": {
"_id": "ffeef2a7-5658-4905-a37c-cfb19f70471d"
}
}
This would return the item with the specified id.
When the find complete successfully, a reply message is sent back to the sender with the following data:
{
"status": "ok",
"result": <result>
}
If an error occurs in finding the documents a reply is returned:
{
"status": "error",
"message": <message>
}
Where
*message is an error message.
The findAndModify command atomically modifies and returns a single document. By default, the returned document does not include the modifications made on the update. To return the document with the modifications made on the update, use the new option. See http://docs.mongodb.org/manual/reference/command/findAndModify/ for details:
{
"action": "find_and_modify",
"collection": <collection>,
"matcher": <document>,
"sort": <document>,
"remove": <boolean>,
"update": <document>,
"new": <boolean>,
"fields": <document>,
"upsert": <boolean>
}
When the operation is successful a reply message is sent back to the sender with the relevant document:
{
"status": "ok",
"result": <document>
}
Otherwise, it sends an error:
{
"status": "error",
"message": <string>
}
An example would be:
{
"action": "find_and_modify",
"collection": "counters",
"matcher": { "_id": "people" },
"update": { "$inc": { "seq": 1 } },
"new": true
}
This would find a document in the counters collection with an _id of "people" and increment its seq field and reply successfully with the new document, as new was set to true.
Counts the number of documents within a collection:
{ "action": "count", "collection": , "matcher": }
Where:
collectionis the name of the MongoDB collection that you wish to delete from. This field is mandatory.matcheris a JSON object that you want to match against to count matching documents. This obeys the normal MongoDB matching rules.
All documents within the collection will be counted.
An example would be:
{
"action": "count",
"collection": "items",
"matcher": {
"active": true
}
}
This should return the count of all documents that have the attribute active set to true.
When the count completes successfully, a reply message is sent back to the sender with the following data:
{
"status": "ok",
"count": <count>
}
Where
*count is the number of documents in the collection that matched the matcher.
If an error occurs in finding the documents a reply is returned:
{
"status": "error",
"message": <message>
}
Where
*message is an error message.
Deletes a matching documents in the database.
To delete documents send a JSON message to the module main address:
{
"action": "delete",
"collection": <collection>,
"matcher": <matcher>
}
Where:
collectionis the name of the MongoDB collection that you wish to delete from. This field is mandatory.matcheris a JSON object that you want to match against to delete matching documents. This obeys the normal MongoDB matching rules.
All documents that match will be deleted.
An example would be:
{
"action": "delete",
"collection": "items",
"matcher": {
"_id": "ffeef2a7-5658-4905-a37c-cfb19f70471d"
}
}
This would delete the item with the specified id.
When the find complete successfully, a reply message is sent back to the sender with the following data:
{
"status": "ok",
"number": <number>
}
Where
*number is the number of documents deleted.
If an error occurs in finding the documents a reply is returned:
{
"status": "error",
"message": <message>
}
Where
*message is an error message.
Returns the list of collection names in the db:
{ "action": "get_collections" }
All collections within the current db will be returned if they exist.
An example would be:
{
"action": "get_collections"
}
This should return the list of all collections within the db
When get_collections completes successfully, a reply message is sent back to the sender with the following data:
{
"status": "ok",
"collections": [
<listOfCollections>
]
}
Where
- is a list containing each collection name in the db.
If an error occurs in finding the documents a reply is returned:
{
"status": "error",
"message": <message>
}
Where
messageis an error message.
Returns statistics about the db:
{ "action": "collection_stats", "collection": }
Where:
collectionis the name of the MongoDB collection that you wish to get statistics on in the db. This field is mandatory.
An example would be:
{
"action": "collection_stats",
"collection": "items"
}
This will return the statistics for the items collection within the db.
When collection_stats completes successfully, a reply message is sent back to the sender with the following data:
{
"status": "ok",
"stats": {
"serverUsed":"localhost/127.0.0.1:27017",
"ns"": "test_coll.items",
"count": 1,
"size": 136,
"avgObjSize": 136.0,
"storageSize": 8192,
"numExtents": 1,
"nindexes": 1,
"lastExtentSize": 8192,
"paddingFactor": 1.0,
"systemFlags": 1,
"userFlags": 0,
"totalIndexSize": 8176,
"indexSizes": {
"_id_":8176
},
"ok":1.0
}
}
Instead of putting placeholders in the DOC here are almost all the real values from the test_client.js run which tests collectionStats I did change the "ns" value to match the collection name in the sample above.
If an error occurs in finding the documents a reply is returned:
{
"status": "error",
"message": <message>
}
Where
messageis an error message.
Drops a collection from the db:
{ "action": "drop_collection", "collection": }
Where:
collectionis the name of the MongoDB collection that you wish to drop from the db. This field is mandatory.
The collection will be removed from the db. This means that all the documents within the collection are gone. Use with CARE
An example would be:
{
"action": "drop_collection",
"collection": "items"
}
This should return an "ok" response, but nothing else. Check out the test_client.js to see our test to make sure that dropCollection works. We just then go retrieve the collection list to make sure the collection we dropped is not in the collection list returned but "getCollections" action
When the drop completes successfully, a reply message is sent back to the sender with the following data:
{
"status": "ok",
}
If an error occurs in finding the documents a reply is returned:
{
"status": "error",
"message": <message>
}
Where
messageis an error message.
Runs an arbitrary MongoDB command.
Command can be used to run more advanced MongoDB features, such as using Mapreduce. There is a complete list of commands at http://docs.mongodb.org/manual/reference/command/.
An example that just pings to make sure the mongo instance is up would be:
{
"action": "command",
"command": "{ ping: 1 }"
}
You would expect a result something like:
{
"result": {
"serverUsed":"localhost/127.0.0.1:27017",
"ok":1.0
},
"status":"ok"
}
The operations save, update and delete have an optional field called "writeConcern". Setting this property in your request changes the "consistency" of that operation.
This allows each call case to overwrite the db's default WriteConcern setting.
Since certain use cases might need more or less consistency than the databases default setting.
By default MongoDB sets the database to the least restrictive value which can lead to data loss on failure of the system.
By being able to set it to a higher setting you can get to your required consistency.
The property is "write_concern" and can be set to any of the constant names in the Java MongoDB Driver WriteConcern class as a String.
WriteConcern has a valueOf method that takes that String and converts it to a fully configured WriteConcern class.
Valid values are "NONE", "NORMAL", "SAFE", "MAJORITY", "FSYNC_SAFE", "JOURNAL_SAFE" and "REPLICAS_SAFE"
An example for delete would be
{
"action": "delete",
"collection": <collection>,
"matcher": <matcher>,
"writeConcern": "SAFE"
}