What is and does optimizing step of a project during indexing?

[ChristopheBordieu]

Hi team,

While indexing (project based), I see into the logs that I have:

• 2057 occurences of "Done traversal ..."
• 2057 occurences of "Done indexing ..."
• 1587 occurences of "Done optimizing ..."

What is doimg this optimizing step?
What triggers it that could explain it is not run for each project?

[vladak]

The "optimization" for index databases is run at the end of index database update. It basically reduces number of segments for the index database to 1:

opengrok/opengrok-indexer/src/main/java/org/opengrok/indexer/index/IndexDatabase.java

Lines 869 to 870 in 1a2ed5c

	wrt = new IndexWriter(indexDirectory, conf);
	wrt.forceMerge(1); // this is deprecated and not needed anymore

As specified in the Lucene documentation:

Forces merge policy to merge segments until there are <= maxNumSegments. The actual merges to be executed are determined by the MergePolicy.

This is a horribly costly operation, especially when you pass a small maxNumSegments; usually you should only call this if the index is static (will no longer be changed).

and as noted on SO, in the above source code comment and various remarks in some OpenGrok issues, it can probably be removed.

The optimization step is done only for index databases that changed.

The optimization step can be controlled using the --optimize indexer option and associated optimizeDatabase indexer tunable.

[ChristopheBordieu]

Hi @vladak

Thanks for your reply.
This night I disabled "optimize" step while indexing. Very interesting!

Indexing duration: from ~4h30 to ~3h (for ~ 2k projects and ~ 35k repos processed)
Webapp config update duration: from ~31 min (!) to ~11 min (conf file size is 24M)

Also, another interesting point is: while indexing, without optimizing indexes, the temporary ~400G of OG data disk that were used (then relased when indexing is terminated) before are no more consumed!

[vladak]

Thanks for the data.

The change in the configuration update is curious - I wonder what is the breakdown of this operation (might need to add more stats to the code). As for the disk space consumed, I guess this makes sense as the segment reduction likely needs some temporary space for segment merging.

[ChristopheBordieu]

From the comments of your updated code, I understand that you don't really know if optimizing the indexes could have an impact on the performance of the OG webapp, am I wrong?

In term of indexing runtime, for sure, it is better.

Would you advice to optimize indexes regularly (eg: during week-end)?

[vladak]

Firstly, based on my experiments so far, I am not sure simply avoiding the "optimization" step is possible given the way Lucene is used. It looks like this breaks terms traversal, at least for the /{project}/files API endpoint. The same way of traversing the index is used in the indexer to detect files to be updated/removed, however there are no test failures. PR #3983 definitely needs more work.

[vladak]

Because of this, I'd advise against turning the optimization step off until this is cleared.

[vladak]

Looks like this is caused by incorrect programming of terms traversal (using iterator in a bad way) so should be possible to resolve. Still, I'd be careful with disabling the "optimization" step.

[ChristopheBordieu]

Ok. Will wait for your clarification.

[vladak]

Based on my analysis in #3983 (comment) , the indexer still works fine (functionality wise) with the optimization being off. As for the performance aspect, this needs to be clarified. It looks like Lucene does not let the segment count to grow forever based on https://blog.mikemccandless.com/2011/02/visualizing-lucenes-segment-merges.html - it seems the number of segments is actually reduced during indexing.