Why one of my project is not visible suddenly

[jkjha]

I am running docker version of opengrok and right now it has a few projects. While browsing one of the project (of source code size 5GB), pod crashed and I got 502 error but again it recovered and I could browse the project but when I go back to home page, I don't see that project again. What could have gone wrong? I see indexer is now running again for the project but nothing is visible on webapp. I have somehow lost the logs during that time, but will try to reproduce but meanwhile if you have any info, please let me know.

[jkjha]

Is it related to

        - name: CHECK_INDEX
          value: "true"

where its recerating whole index data? However, I am able to access the direct link of the project including all features history/ annotations https://opengrok.company.com/project_name

but it doesn't show in project dropdown and home dashboard list.

[vladak]

The InvocationTargetException exception is harmless (see #2211).

[vladak]

If there is a dirty file in the index directory and the indexed field of given project is set to false, this means that either the indexer is still running or it crashed. See the indexer logs.

[jkjha]

I don't find any other error yet in indexer log...last few lines of indexer log:

2022-12-06 05:34:09.244+0000 FINE t45 PendingFileCompleter.completeInner: deleted 12,920 file(s) (took 0:13:30)
2022-12-06 05:34:09.244+0000 FINE t45 PendingFileCompleter.completeInner: renamed 0 file(s) (took 0 ms)
2022-12-06 05:34:09.245+0000 FINE t45 PendingFileCompleter.completeInner: affirmed links for 0 path(s) (took 0 ms)
2022-12-06 05:34:09.245+0000 FINE t45 IndexDatabase.finishWriting: completed 12,920 object(s)
2022-12-06 05:34:22.355+0000 INFO t1 Statistics.logIt: Done indexing data of all repositories (took 2:58:55)
2022-12-06 05:34:22.372+0000 INFO t1 Statistics.logIt: Indexer finished (took 2:58:59)

but I see the opengrok container has restarted several times with below error:

INFO: Suggesters for [censored-project-name] were successfully rebuilt (took 9 minutes 50 seconds)
Received signal 15
Terminating Tomcat <Popen: returncode: None args: ['/usr/local/tomcat/bin/catalina.sh', 'run']>
Received signal 15
Terminating Tomcat <Popen: returncode: None args: ['/usr/local/tomcat/bin/catalina.sh', 'run']>
Dec 06, 2022 8:35:36 AM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["http-nio-8080"]
Dec 06, 2022 8:35:36 AM org.apache.catalina.core.StandardService stopInternal
INFO: Stopping service [Catalina]
Sync done
Dec 06, 2022 8:35:36 AM org.opengrok.indexer.configuration.WatchDogService stop
INFO: Watchdog stopped
Dec 06, 2022 8:35:36 AM org.apache.catalina.loader.WebappClassLoaderBase checkThreadLocalMapForLeaks
SEVERE: The web application [ROOT] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@59338941]) and a value of type [net.openhft.chronicle.hash.impl.ContextHolder] (value [net.openhft.chronicle.hash.impl.ContextHolder@6d22b17e]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Dec 06, 2022 8:35:36 AM org.apache.catalina.loader.WebappClassLoaderBase checkThreadLocalMapForLeaks
SEVERE: The web application [ROOT] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@59338941]) and a value of type [net.openhft.chronicle.hash.impl.ContextHolder] (value [net.openhft.chronicle.hash.impl.ContextHolder@591950cc]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Dec 06, 2022 8:35:36 AM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["http-nio-8080"]
Dec 06, 2022 8:35:36 AM org.apache.coyote.AbstractProtocol destroy
INFO: Destroying ProtocolHandler ["http-nio-8080"]
could not get configuration from web application: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
Waiting for reindex to be triggered

[vladak]

You might be hitting #3650. If the container is terminated while the indexer is running, this might result in the symptoms you are seeing.

[jkjha]

yes, thats seems to be exactly what I am facing. Almost during the end of indexing, I see below log and the container gets restarted...

Dec 06, 2022 11:28:13 AM org.opengrok.suggest.Suggester shutdownAndAwaitTermination
INFO: Suggesters for [censored-project-name] were successfully rebuilt (took 8 minutes 8 seconds)
Received signal 15
Terminating Tomcat <Popen: returncode: None args: ['/usr/local/tomcat/bin/catalina.sh', 'run']>
Received signal 15
Terminating Tomcat <Popen: returncode: None args: ['/usr/local/tomcat/bin/catalina.sh', 'run']>
Dec 06, 2022 11:35:35 AM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["http-nio-8080"]
Sync done
Dec 06, 2022 11:35:35 AM org.apache.catalina.core.StandardService stopInternal
INFO: Stopping service [Catalina]
Dec 06, 2022 11:35:36 AM org.opengrok.indexer.configuration.WatchDogService stop
INFO: Watchdog stopped
Dec 06, 2022 11:35:36 AM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["http-nio-8080"]
Dec 06, 2022 11:35:36 AM org.apache.coyote.AbstractProtocol destroy
INFO: Destroying ProtocolHandler ["http-nio-8080"]
could not get configuration from web application: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
Waiting for reindex to be triggered

Unfortunately, I do not see any suggested workaround for this task except downgrading to 1.6.9 which I really want to avoid. Please let me know if there are any possible workaround.