OpenGrok 1.4.1 in Docker, indexing fails, problems with CTags

[innerground]

Guys,
I cannot make OpenGrok indexing work.
I either get c-tags timeout, tagLine == null,interrupted--ctags process died :

WARNING: File '/XXX/aa/bb/cc/dd/AType.java' interrupted--ctags process died
Aug 28, 2020 11:33:05 AM org.opengrok.indexer.analysis.Ctags doCtags
WARNING: Ctags process exited with exit value 137
/opengrok/start.sh: line 17:     8 Killed                  java -d64 -Xms2g -Xmx8g -jar /opengrok/dist/lib/opengrok.jar -c /usr/bin/ctags -v --progress -P -T 6 -O on --leadingWildCards on -s /opengrok/src -d /opengrok/data -W /opengrok/etc/configuration.xml -R /opengrok/etc/configuration-ro.xml

I tried CTags from Debian Beaver directly, also built the latest version from git but It always fail (Along with some other tags found on some forums).
Is there a particular change hash recommanded for CTags to be used with OpenGrok 1.4.1
Thanks,
Best regards,

[idodeclare]

I tested universal-ctags/ctags@6ca23b5 from 27 Aug 2020 with no problem.

If you run the OpenGrok indexer with --help ctags, it will give you the (long) ctags command line. You could run it manually (leaving off --filter=yes) on your source file to see if that reveals any more about the problem.

[vladak]

Exit value 137 means the ctags process was terminated with SIGKILL. OOM Linux killer in action perhaps ?

[vladak]

Also, you can try to see if this is data sensitive (probably is) and isolate the repositories/files that cause the problem.

[innerground]

Here is the CTags version:
root@a9e4be3c0f1c:/build/ctags# ctags --help
Universal Ctags 0.0.0(36a6dd19), Copyright (C) 2015 Universal Ctags Team
Universal Ctags is derived from Exuberant Ctags.
Exuberant Ctags 5.8, Copyright (C) 1996-2009 Darren Hiebert
Compiled: Aug 30 2020, 05:22:20
URL: https://ctags.io/
Optional compiled features: +wildcards, +regex, +iconv, +option-directory, +xpath, +json, +interactive, +sandbox, +yaml, +packcc

Depending on the version I compile, the process fails, but at a different percentage:
WARNING: Terminating ctags process for file '/src/XXX/src/com/sun/enterprise/web/connector/grizzly/AsyncReadTask.java' due to timeout 240 seconds
Aug 30, 2020 6:23:33 AM org.opengrok.indexer.index.IndexDatabase addFile
WARNING: File '/XXX/src/com/sun/enterprise/web/connector/grizzly/AsyncReadTask.java' interrupted--ctags timeout
Aug 30, 2020 6:23:48 AM org.opengrok.indexer.analysis.Ctags readTags
WARNING: ctags: Unexpected end of file!
Aug 30, 2020 6:24:09 AM org.opengrok.indexer.analysis.Ctags readTags
WARNING: ctags exited with code: 137
Aug 30, 2020 6:24:09 AM org.opengrok.indexer.index.IndexDatabase lambda$null$2
WARNING: No retry: /src/XXX/src/com/sun/enterprise/web/connector/grizzly/AsyncReadTask.java
avAug 30, 2020 6:36:27 AM org.opengrok.indexer.analysis.Ctags doCtags
WARNING: Terminating ctags process for file '/src/XXX/src/aa/bb/_TypeInterface.java' due to timeout 240 seconds
Aug 30, 2020 6:36:32 AM org.opengrok.indexer.analysis.Ctags readTags
WARNING: ctags: Unexpected end of file!
Aug 30, 2020 6:37:25 AM org.opengrok.indexer.index.IndexDatabase addFile
WARNING: File '/XXX/src/aa/bb/_TypeInterface.java' interrupted--ctags timeout
Aug 30, 2020 6:37:38 AM org.opengrok.indexer.analysis.Ctags readTags
WARNING: ctags exited with code: 137
Aug 30, 2020 6:37:43 AM org.opengrok.indexer.analysis.Ctags doCtags
WARNING: Ctags process exited with exit value 137
Aug 30, 2020 6:37:49 AM org.opengrok.indexer.index.IndexDatabase addFile
WARNING: File '/XXX/src/com/amazonaws/services/apigateway/model/transform/CreateApiKeyResultJsonUnmarshaller.java' interrupted--ctags process died
Aug 30, 2020 6:42:30 AM org.opengrok.indexer.analysis.Ctags doCtags
WARNING: Terminating ctags process for file '/src/XXX/src/aa/bb/_TypeInterface.java' due to timeout 240 seconds
Aug 30, 2020 6:42:34 AM org.opengrok.indexer.analysis.Ctags readTags
WARNING: ctags: Unexpected end of file!
Aug 30, 2020 6:43:16 AM org.opengrok.indexer.index.IndexDatabase addFile
WARNING: File '/XXX/src/aa/bb/_TypeInterface.java' interrupted--ctags timeout
Aug 30, 2020 6:43:35 AM org.opengrok.indexer.analysis.Ctags readTags
WARNING: ctags exited with code: 137
Aug 30, 2020 6:43:35 AM org.opengrok.indexer.index.IndexDatabase lambda$null$2
WARNING: No retry: /src/XXX/src/aa/bb/_TypeInterface.java
Aug 30, 2020 6:44:24 AM org.opengrok.indexer.analysis.Ctags doCtags
WARNING: Ctags process exited with exit value 137
Aug 30, 2020 6:44:28 AM org.opengrok.indexer.index.IndexDatabase addFile
WARNING: File '/XXX/src/org/codehaus/jackson/jaxrs/JacksonJaxbJsonProvider.java' interrupted--ctags process died
Aug 30, 2020 6:49:42 AM org.opengrok.indexer.analysis.Ctags doCtags
WARNING: Terminating ctags process for file '/src/XXX/src/com/sun/enterprise/web/connector/grizzly/AsyncTask.java' due to timeout 240 seconds
Aug 30, 2020 6:49:46 AM org.opengrok.indexer.analysis.Ctags readTags
WARNING: ctags: Unexpected end of file!
Aug 30, 2020 6:50:08 AM org.opengrok.indexer.index.IndexDatabase addFile
WARNING: File '/XXX/src/com/sun/enterprise/web/connector/grizzly/AsyncTask.java' interrupted--ctags timeout
Aug 30, 2020 6:51:01 AM org.opengrok.indexer.analysis.Ctags doCtags
WARNING: Ctags process exited with exit value 137
Aug 30, 2020 6:51:03 AM org.opengrok.indexer.analysis.Ctags readTags
WARNING: ctags exited with code: 137
Aug 30, 2020 6:51:04 AM org.opengrok.indexer.index.IndexDatabase addFile
WARNING: File '/XXX/src/com/yyy/server/Builder.java' interrupted--ctags process died
/opengrok/start.sh: line 17: 9 Killed java -d64 -Xms2g -Xmx8g -jar /opengrok/dist/lib/opengrok.jar -c /usr/bin/ctags -v --progress -P -T 6 -O on --leadingWildCards on -s /opengrok/src -d /opengrok/data -W /opengrok/etc/configuration.xml -R /opengrok/etc/configuration-ro.xml

I am a bit stuck, the java source code is nothing complicated...Is there a way to get extra output?

[vladak]

This means the ctags process did not finish processing of given file in 4 minutes. Did you try running the ctags process standalone as suggested earlier ? Does this happen for all Java files or just for some ? Might be worth seeing what the long running ctags processes are up to e.g. using the usuall tools such as strace, ltrace (to see calls into libc), top etc.

[vladak]

Also, is the /XXX/src local or network file system ? Can the ctags process be stuck on I/O ?

[innerground]

Also, is the /XXX/src local or network file system ? Can the ctags process be stuck on I/O ?

It is a docker path, I tried to access to the file, no problem there.

[innerground]

This means the ctags process did not finish processing of given file in 4 minutes. Did you try running the ctags process standalone as suggested earlier ? Does this happen for all Java files or just for some ? Might be worth seeing what the long running ctags processes are up to e.g. using the usuall tools such as strace, ltrace (to see calls into libc), top etc.

I ran ctags on the files which fail, and It works fine... FYI, It is a docker container.

[innerground]

I am getting error, but the main problem is the ctags exit with code : 137... OOM, there are some options around that, gonna make some extra tests.

[vladak]

The error 137 means the process was forcibly terminated:

$ kill -l 137
KILL

and the indexer logs above confirm this was due to timeout.

You need to trace the ctags processes while they are running in the container to see what is going on.

[innerground]

I ran the docker with -m 16g --oom-kill-disable but I got the same result.
I am going to trace ctags to find out.
Also, there is something which I find weird, even if I limit the threads with -T flag, OpenGrok spawns one process per available thread systematically, Is that normal?

[innerground]

It is a resource problem, running the indexing with -T 1 worked.
I still do not understand with OpenGrok still spawn 1 process per thread even if I specified -T 1 (That is why It was failing with -T > 1).

[vladak]

Also, there is something which I find weird, even if I limit the threads with -T flag, OpenGrok spawns one process per available thread systematically, Is that normal?

I think that's expected. Each indexer thread needs one ctags process.

[vladak]

I am curious what kind of resource problem this is.

[innerground]

Also, there is something which I find weird, even if I limit the threads with -T flag, OpenGrok spawns one process per available thread systematically, Is that normal?

I think that's expected. Each indexer thread needs one ctags process.

@vladak Check the screenshot, why opengrok is spawning so many processes (If each process runs with Xmx, of course one of them is gonna run of memory)?

[vladak]

These are probably just userland threads of the same process. The green color indicates them - try pressing Shift+H to toggle the thread view.

See https://unix.stackexchange.com/questions/10362/why-does-htop-show-more-process-than-ps and https://peteris.rocks/blog/htop/

[vladak]

Similarly, try running pstree -p on the indexer process ID with and without -T. Actually, when you run without -T and with -t it will even display Java thread names.

[vladak]

Also, to get overall info on resources/process/threads in a Docker container, use docker container stats. For more detailed view with threads you can use docker container <INSERT_CONTAINER_NAME> top -eLf

[masatake]

Can I see the input file causing the error 137 ?