OpenGrok webapp fails to start after adding a project

[abhi31oct]

Describe the bug

I have organised my open Gronk in multiple projects, and projects are organised in project groups like-

ProjectGroup1-
-- Project1
-- Project2
ProjectGroup2-
-- Project3
-- Project4
ProjectGroup3-
-- Project5
-- Project6

A total of 17 projects were created, and till project 16, everything was working fine. However when I added Project 17, and indexed it,
opengrok app stopped working. I am getting this error - HTTP Status 404 – Not Found

bash# /opt/tomcat/bin/catalina.sh version
Using CATALINA_BASE: /opt/tomcat
Using CATALINA_HOME: /opt/tomcat
Using CATALINA_TMPDIR: /opt/tomcat/temp
Using JRE_HOME: /usr/local
Using CLASSPATH: /opt/tomcat/bin/bootstrap.jar:/opt/tomcat/bin/tomcat-juli.jar
Using CATALINA_OPTS:
Server version: Apache Tomcat/10.1.23
Server built: Apr 16 2024 12:17:18 UTC
Server number: 10.1.23.0
OS Name: Linux
OS Version: 4.18.0-513.11.1.el8_9.x86_64
Architecture: amd64
JVM Version: 21.0.2+13-LTS
JVM Vendor: Red Hat, Inc.

bash# ls -lh /opt/tomcat/webapps/
total 51M
drwxr-xr-x 16 root root 4.0K May 10 01:19 docs
drwxr-xr-x 7 root root 99 May 10 01:19 examples
drwxr-xr-x 6 root root 79 May 10 01:19 host-manager
drwxr-xr-x 6 root root 114 May 10 01:19 manager
drwxr-xr-x 3 root root 4.0K May 10 01:19 ROOT
drwxr-xr-x 6 root root 4.0K May 22 10:04 source
-rwxr-xr-x 1 root root 51M May 22 10:04 source.war
[abhisheksing@dc5lp-fae-app01 src]$

To Reproduce
Steps to reproduce the behavior: It happens suddenly after adding the 17th project and indexing that

Expected behavior
Opengrok app should work

[vladak]

Given that the webapp is no longer present hints that there is something fundamentally wrong with it or it was undeployed for some reason. You need to check the webapp/server logs to see what is going on.

[abhi31oct]

@vladak I tried restarting Tomcat and deploy the opengrok again, still facing the same issue.

[abhi31oct]

@vladak Can you please guide me to solve this issue?

[vladak]

what is in the webapp/server logs ? how did you add the project exactly ?

[abhi31oct]

@vladak when I am trying to deploy again, I can see following logs-

 bash# tail -f catalina.out
31-May-2024 09:10:00.357 INFO [Catalina-utility-2] org.apache.catalina.startup.HostConfig.undeploy Undeploying context [/source]
31-May-2024 09:10:00.415 INFO [Catalina-utility-2] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive [/opt/tomcat/webapps/source.war]
31-May-2024 09:10:00.421 WARNING [Catalina-utility-2] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Context] failed to set property [antiJARLocking] to [true]


31-May-2024 09:10:04.631 INFO [Catalina-utility-2] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
31-May-2024 09:10:04.746 INFO [Catalina-utility-2] org.opengrok.indexer.Metrics.<clinit> configuring PrometheusRegistry
31-May-2024 09:10:04.819 INFO [Catalina-utility-2] org.opengrok.web.WebappListener.contextInitialized Starting webapp with version 1.13.7 (596074cd506d4664560dedb8b97be5b40372b373)
31-May-2024 09:10:04.821 INFO [Catalina-utility-2] org.opengrok.indexer.configuration.Configuration.read Reading configuration from /opengrok/etc/configuration.xml
31-May-2024 09:10:04.868 INFO [Catalina-utility-2] org.opengrok.web.WebappListener.contextInitialized running inside Apache Tomcat/10.1.23
31-May-2024 09:10:04.873 WARNING [Catalina-utility-2] org.opengrok.indexer.framework.PluginFramework.reload Plugin directory not found or not readable: null. All requests allowed.
31-May-2024 09:10:04.878 INFO [Catalina-utility-2] org.opengrok.indexer.configuration.Configuration.read Reading configuration from /opengrok/etc/configuration.xml
31-May-2024 09:10:04.887 SEVERE [Catalina-utility-2] org.apache.catalina.core.StandardContext.startInternal One or more listeners failed to start. Full details will be found in the appropriate container log file
31-May-2024 09:10:04.888 SEVERE [Catalina-utility-2] org.apache.catalina.core.StandardContext.startInternal Context [/source] startup failed due to previous errors
31-May-2024 09:10:04.899 INFO [Catalina-utility-2] org.opengrok.indexer.configuration.WatchDogService.stop Watchdog stopped
Exception in thread "Thread-4" java.lang.NullPointerException
        at java.base/java.util.Objects.requireNonNull(Objects.java:233)
        at java.base/sun.nio.fs.UnixFileSystem.getPath(UnixFileSystem.java:296)
        at java.base/java.nio.file.Path.of(Path.java:148)
        at java.base/java.nio.file.Paths.get(Paths.java:69)
        at org.opengrok.web.api.v1.suggester.provider.service.impl.SuggesterServiceImpl.getAllProjectIndexDirs(SuggesterServiceImpl.java:345)
        at org.opengrok.web.api.v1.suggester.provider.service.impl.SuggesterServiceImpl.lambda$initSuggester$3(SuggesterServiceImpl.java:333)
        at java.base/java.lang.Thread.run(Thread.java:1583)
31-May-2024 09:10:04.915 INFO [Catalina-utility-2] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive [/opt/tomcat/webapps/source.war] has finished in [4,500] ms

[abhi31oct]

@vladak I am using below commands to add a project and index the project-

Command to add a project-
opengrok-projadm -b /opengrok -a SDK11 -U http://localhost:8080/source/

Command to index a project

curl -s -X GET http://localhost:8080/source/api/v1/configuration -o fresh_config.xml
opengrok-indexer -a /opengrok/dist/lib/opengrok.jar --     -c /usr/local/bin/ctags     -U 'http://localhost:8080/source'     -R fresh_config.xml     -s /opengrok/src -d /opengrok/data -H -P -S -G -H SDK11 SDK11

[abhi31oct]

@vladak I redeployed everything, and it started working. I was able to use opengrok for two days, but suddenly it stopped working without any trigger-

1. I have started the tomcat again
2. Re-deployed opengrok

I can see failures while deploying, Can you please help to figure out what is the issue?

Error observed in Catalina.out-

06-Jun-2024 02:31:59.879 INFO [Catalina-utility-2] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive [/opt/tomcat/webapps/source.war]
06-Jun-2024 02:31:59.881 WARNING [Catalina-utility-2] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Context] failed to set property [antiJARLocking] to [true]
06-Jun-2024 02:32:02.172 INFO [Catalina-utility-2] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
06-Jun-2024 02:32:02.233 INFO [Catalina-utility-2] org.opengrok.indexer.Metrics.<clinit> configuring PrometheusRegistry
06-Jun-2024 02:32:02.272 INFO [Catalina-utility-2] org.opengrok.web.WebappListener.contextInitialized Starting webapp with version 1.13.7 (596074cd506d4664560dedb8b97be5b40372b373)
06-Jun-2024 02:32:02.274 INFO [Catalina-utility-2] org.opengrok.indexer.configuration.Configuration.read Reading configuration from /opengrok/etc/configuration.xml
06-Jun-2024 02:32:02.304 INFO [Catalina-utility-2] org.opengrok.web.WebappListener.contextInitialized running inside Apache Tomcat/10.1.23
06-Jun-2024 02:32:02.308 WARNING [Catalina-utility-2] org.opengrok.indexer.framework.PluginFramework.reload Plugin directory not found or not readable: null. All requests allowed.
06-Jun-2024 02:32:02.312 INFO [Catalina-utility-2] org.opengrok.indexer.configuration.Configuration.read Reading configuration from /opengrok/etc/configuration.xml
06-Jun-2024 02:32:02.319 SEVERE [Catalina-utility-2] org.apache.catalina.core.StandardContext.startInternal One or more listeners failed to start. Full details will be found in the appropriate container log file
06-Jun-2024 02:32:02.319 SEVERE [Catalina-utility-2] org.apache.catalina.core.StandardContext.startInternal Context [/source] startup failed due to previous errors
06-Jun-2024 02:32:02.326 INFO [Catalina-utility-2] org.opengrok.indexer.configuration.WatchDogService.stop Watchdog stopped
06-Jun-2024 02:32:02.340 WARNING [Catalina-utility-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [source] appears to have started a thread named [Thread-5] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 org.opengrok.web.api.v1.suggester.provider.service.impl.SuggesterServiceImpl.getAllProjectIndexDirs(SuggesterServiceImpl.java:345)
 org.opengrok.web.api.v1.suggester.provider.service.impl.SuggesterServiceImpl.lambda$initSuggester$3(SuggesterServiceImpl.java:333)
 java.base/java.lang.Thread.run(Thread.java:1583)
06-Jun-2024 02:32:02.343 INFO [Catalina-utility-2] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive [/opt/tomcat/webapps/source.war] has finished in [2,465] ms
06-Jun-2024 02:32:02.346 INFO [Thread-5] org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading Illegal access: this web application instance has been stopped already. Could not load [java.nio.file.Paths]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
        java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load [java.nio.file.Paths]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
                at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading(WebappClassLoaderBase.java:1373)
                at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1361)
                at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1198)
                at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1165)
                at org.opengrok.web.api.v1.suggester.provider.service.impl.SuggesterServiceImpl.getAllProjectIndexDirs(SuggesterServiceImpl.java:345)
                at org.opengrok.web.api.v1.suggester.provider.service.impl.SuggesterServiceImpl.lambda$initSuggester$3(SuggesterServiceImpl.java:333)
                at java.base/java.lang.Thread.run(Thread.java:1583)
Exception in thread "Thread-5" java.lang.NoClassDefFoundError: java/nio/file/Paths
        at org.opengrok.web.api.v1.suggester.provider.service.impl.SuggesterServiceImpl.getAllProjectIndexDirs(SuggesterServiceImpl.java:345)
        at org.opengrok.web.api.v1.suggester.provider.service.impl.SuggesterServiceImpl.lambda$initSuggester$3(SuggesterServiceImpl.java:333)
        at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.ClassNotFoundException: Illegal access: this web application instance has been stopped already. Could not load [java.nio.file.Paths]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
        at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1363)
        at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1198)
        at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1165)
        ... 3 more
Caused by: java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load [java.nio.file.Paths]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
        at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading(WebappClassLoaderBase.java:1373)
        at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1361)
        ... 5 more

[abhi31oct]

@vladak Can you please help me out on possible issues or how to debug this issue?

[abhi31oct]

@vladak After removing the project group, I did not face this issue.
Looks like project group logic has some issues. After diving the projects in groups, there is no issue, but at midnight, when the suggester task runs, it reproduce the issue, looks like suggester does not understand project group related XML