SEVERE: cannot reach OpenGrok web application on http://localhost:8080/: Found

[sysyi]

Hosting Opengrok in Docker container.
Tomcat Version: Apache Tomcat/10.1.8
Opengrok Version: 1.12.6

To mention: I'm able to access the web application at https://server.test.com

Describe the configuration
I have enabled SSL for tomcat in Docker, with below conf.

server.xml ->

<Connector port="8080" protocol="HTTP/1.1"
               connectionTimeout="20000"
               proxyName="server.test.com"
            redirectPort="443"/>
    <Connector port="443" protocol="org.apache.coyote.http11.Http11Nio2Protocol" SSLEnabled="true"
               scheme="https" secure="true"
             enableLookups="false" disableUploadTimeout="true"
            clientAuth="false" sslProtocol="TLS"
            defaultSSLHostConfigName="test">
    <UpgradeProtocol className="org.apache.coyote.http2.Http2Protocol" />
    <SSLHostConfig hostName="test" protocols="TLSv1.2">
        <Certificate certificateKeystoreFile="/etc/ssl/server.test.com.keystore"
                     certificateKeystorePassword="opengrok1"
                     type="RSA" />
</SSLHostConfig>
</Connector>

web.xml ->

<security-constraint>
        <web-resource-collection>
            <web-resource-name>HTTPSOnly</web-resource-name>
            <url-pattern>/*</url-pattern>
        </web-resource-collection>
        <user-data-constraint>
            <transport-guarantee>CONFIDENTIAL</transport-guarantee>
        </user-data-constraint>
    </security-constraint

After this configuration when restart my container i get below errors:

INFO [main] org.apache.catalina.startup.Catalina.start Server startup in [14692] milliseconds
Tomcat is ready
Adding project rct
could not add project 'rct' in web application: <Response [200]>
Merging commands with environment
Sync starting
command 'opengrok-reindex-project --printoutput --api_timeout 8 --jar /opengrok/lib/opengrok.jar -U http://localhost:8080/ -P w1m -- --connectTimeout 8 -r dirbased -G -m 256 --leadingWildCards on -c /usr/local/bin/ctags -U http://localhost:8080/ -H w1m' for project w1m failed with code 1, breaking

Additional context
Also when i checked i'm unable to get any info from http://localhost:8080/api/v1 or http://localhost:8080/source/api/v1 or http://localhost:8080/source/api/v1/configuration

/usr/local/tomcat# opengrok-reindex-project --printoutput --api_timeout 30 --jar /opengrok/lib/opengrok.jar -U http://localhost:8080/ -P rmc -- --connectTimeout 30 -r dirbased -G -m 256 --leadingWildCards on -c /usr/local/bin/ctags -U http://localhost:8080/ -H rmc
Aug 09, 2024 10:28:37 AM org.opengrok.indexer.configuration.Configuration read
INFO: Reading configuration from /tmp/tmpotj0uxe5
Aug 09, 2024 10:28:37 AM org.opengrok.indexer.index.Indexer parseOptions
INFO: Indexer options: [-R, /tmp/tmpotj0uxe5, --connectTimeout, 30, -r, dirbased, -G, -m, 256, --leadingWildCards, on, -c, /usr/local/bin/ctags, -U, http://localhost:8080/, -H, rmc]
Aug 09, 2024 10:28:37 AM org.opengrok.indexer.util.HostUtil isWebAppReachable
SEVERE: cannot reach OpenGrok web application on http://localhost:8080/: Found
http://localhost:8080/ is not reachable and the -U option was specified, exiting.
Indexer command for project rmc failed (return code 1)

can someone please provide some guidance on this?
Hi @vladak , Can you please help with this?

[vladak]

The indexer has to be able to reach the webapp in order to send/retrieve information about projects/repositories etc. That's why the -U indexer option is used. This URL is stored in the sync.yml configuration file and is actually used by the main Docker program itself (in fact it is hard-coded on

opengrok/docker/start.py

Line 159 in 9fd1f26

uri = "http://localhost:8080/" + url_root

). The indexer should be able to reach the http://localhost:8080/ which the configuration prohibits due to the redirect. So, either you will need to "poke a hole" in the SSL configuration to allow the URL to be reachable from the localhost or you will need to modify the start.py and change the configuration to use different URL (https://localhost:8080 might work).

I'd say this is a bit beyond of what the Docker image was designed for, because it requires modification of the configuration files delivered in the image and also the code. If you go down the sync.yml path, you will be on your own in terms of keeping the configuration up to date if and when the original configuration changes.

[vladak]

The question is whether the other issues you are seeing are related to the changes done to web.xml and elsewhere. I'd recommend checking with vanilla OpenGrok configuration first.

The NPE (NullPointerException) was likely accompanied by a log message in the web server logs. In the Docker image these are emitted to the console.

As for whether adding stuff manually to configuration.xml: this is not a good idea as the changes will be wiped out by next resync. This is used with the per-project workflow used by the Docker image. If there is a project/repository missing in the configuration, it is prudent to find out why it was not discovered. The project/repository scan is done by the webapp, triggered by an API call via start.py.

Debugging all of this in the Docker image is likely sub-optimal.

[sysyi]

I'm not aware what vanilla OpenGrok configuration is, will look for it.

The NPE (NullPointerException) was likely accompanied by a log message in the web server logs. In the Docker image these are emitted to the console. - Okay got it.

As for whether adding stuff manually to configuration.xml: this is not a good idea as the changes will be wiped out by next resync. - Understood, thanks for clarification.

I'm trying to clone submodule separately now, as i still didn't find out why it's not getting scanned properly, it will take some time i guess.
So far i have used below commands:

opengrok-projadm -b /opengrok -a vqd -U http://localhost:8080/ --jar /opengrok/lib/opengrok.jar

opengrok-indexer -a /opengrok/lib/opengrok.jar -- --depth 8 --progress -c /usr/local/bin/ctags -U http://localhost:8080/ -R /opengrok/etc/configuration.xml -H -P vqd vqd

After these commands, i was able to see submodule vqd in webapp.

Questions:

1. Is there a possibility in docker opengrok to index/re-index specific folder inside a project.
   Example: for my rmc repo if i want to re-index specific submodule folder, like using opengrok index command by giving path - rmc/<submodule_folder>

2. Is it possible to reindex single file in opengrok, manually ?

Asking as before your answer i tried to update configuration.xml from my side and saw this type of warning while running opengrok-reindex command:
command used was: opengrok-reindex-project --printoutput --api_timeout 8 --jar /opengrok/lib/opengrok.jar -U http://localhost:8080/ -P rmc -- --connectTimeout 8 -r dirbased -G -m 256 --leadingWildCards on -c /usr/local/bin/ctags -U http://localhost:8080/ --depth 8 -H rmc

warning:

2024 7:17:27 AM org.opengrok.indexer.index.IndexDatabase lambda$indexParallel$4
WARNING: ERROR addFile(): '/opengrok/src/rmc/tp/wrt'
java.io.FileNotFoundException: /opengrok/src/rmc/tp/wrt (Is a directory)

[vladak]

By vanilla configuration I meant stock (unmodified) OpenGrok docker image.

Is there a possibility in docker opengrok to index/re-index specific folder inside a project.

No.

Is it possible to reindex single file in opengrok, manually ?

Generally no, however there are some ways. If the file is part of Git/Mercurial repository, you'd have to disable history based reindex and touch it - not sure about possible negative side effects in that case. Otherwise the file can be touched and the indexer should pick it up.

[vladak]

As for the FileNotFoundException for the directory, you might be hitting #4528 (fixed).

[sysyi]

Okay got it, we haven't modified the image from our end.
But still see that some of the subfolders aren't getting re-indexed - it's for some projects have tried reindexing multiple times but it doesn't work. I don't see those folders under history as well.
Is my understanding correct - once those subfolders are available under history folder then it will reindex those properly ?