Cygnus could deal with changes in expiration and collection size (peding: collection size)

[fgalan]

In https://github.com/telefonicaid/fiware-cygnus/blob/master/cygnus-common/src/main/java/com/telefonica/iot/cygnus/backends/mongo/MongoBackendImpl.java we have to functions for createCollection:

public void createCollection(String dbName, String collectionName, long dataExpiration) throws MongoException

public void createCollection(String dbName, String collectionName, long collectionsSize, long maxDocuments) throws MongoException

(the second one assuming that dataExpiration has been removed, as described in #2159)

They work perfectly for new collections but for existing collections Cygnus is not able to change the dateExpirations or collection size settings.

With regards with expiration index...

MongoDB does't allow to use createIndex() if an expiration index already exists and the expiration parameter is different, i.e:

> db.x.getIndexes()
[
...
        {
                "v" : 2,
                "key" : {
                        "createdAt" : 1
                },
                "name" : "createdAt_1",
                "expireAfterSeconds" : 10
        },
...
]
> db.x.ensureIndex( { "createdAt": 1 }, { expireAfterSeconds: 30 } )
{
        "ok" : 0,
        "errmsg" : "Index with name: createdAt_1 already exists with different options",
        "code" : 85,
        "codeName" : "IndexOptionsConflict"
}

so Cygnus would need to remove the old index and create the new one.

With regards with collection size...

It seems MongoDB allows to use the the convertToCapped command to do the resize (see https://stackoverflow.com/a/22814104/1485926) although it is not clear if that command work with already capped collections (see my comment in that answer about it)

[fgalan]

Side note: it would be great if Cygnus can cover these cases but, if it is too complex, a possible solution is through documentation, I mean, to define the manual procedures to do that and include them in Cygnus documentation.

[Madhu1029]

Hi @fgalan ,

With regards to expiration index...
so Cygnus would need to remove the old index and create the new one.

I am working on this.

[Madhu1029]

Hi @fgalan ,

With regards to expiration index...
I have updated the code for this.

With regards to collection size...
The resizing of capped collection is available in 6.0 version of MongoDB as mentioned in https://jira.mongodb.org/browse/SERVER-1864.
db.runCommand({collMod: "collectionName", cappedSize: maxSize, cappedMax: maxDocs})

If resizing need to be supported on Cygnus with lower version of MongoDB then we need to follow below approach:

1. Run below command, but it will not persist the existing indexes.
   db.runCommand({"convertToCapped": "collectionName", size: 1000000000});
2. Create the previous index again.

We need to perform above steps for each existing collection.
So, as per my understanding, it will not be good idea to support resizing with lower version of MongoDB. Please let me know your opinion.

[fgalan]

Please focus on the "With regards to expiration index" case.

The "With regards with collection size" is not very important, as we don't typically run Cygnus in this mode.

[fgalan]

PR #2211 (changes in date expiration).

Collection size changes are pending although, as commented above that case is less important and probably doesn't work the effort to be covered by the implementation.