Indexing failing with error Caused by: java.io.IOException: Broken pipe #4247

jkjha · 2023-03-28T10:19:16Z

jkjha
Mar 28, 2023

I have deployed opengrok webapp as container and the projects are mounted using volume mount. After placing the source code here /opengrok/src it used to index the project correctly but suddenly now it started failing with error:

Caused by: java.io.IOException: Broken pipe

The full error log is below... any idea what could be wrong? I tried redeploying the container but still the same error and opengrok has stopped indexing any project:

command 'opengrok-reindex-project -J=-XX:-UseGCOverheadLimit -J=-Xmx36g -J=-server --jar /opengrok/lib/opengrok.jar -t /opengrok/etc/logging.properties.template -p %PROJ% -d /opengrok/log/<project_name> -P <project_name> -U http://localhost:8080/ -- --renamedHistory off -r dirbased -G -m 4096 -c /usr/local/bin/ctags -U http://localhost:8080/ -o /opengrok/etc/ctags.config -H <project_name>' for project <project_name> failed with code 1, breaking

...
...

Saving configuration to /opengrok/etc/configuration.xml
Mar 28, 2023 10:03:08 AM org.glassfish.jersey.server.ServerRuntime$Responder writeResponse
SEVERE: An I/O error has occurred while writing a response message entity to the container output stream.
org.glassfish.jersey.server.internal.process.MappableException: org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe
        at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:67)
        at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:139)
        at org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1116)
        at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:677)
        at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:385)
        at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:375)
        at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:264)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:240)
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:697)
        at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
        at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:357)
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:311)
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:223)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:158)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:185)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:158)
        at org.opengrok.web.CookieFilter.doFilter(CookieFilter.java:50)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:185)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:158)
        at org.opengrok.web.AuthorizationFilter.doFilter(AuthorizationFilter.java:68)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:185)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:158)
        at org.opengrok.web.StatisticsFilter.doFilter(StatisticsFilter.java:67)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:185)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:158)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:177)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:542)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:119)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:357)
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:400)
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:859)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1734)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
        at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
        at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe
        at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:349)
        at org.apache.catalina.connector.OutputBuffer.flushByteBuffer(OutputBuffer.java:780)
        at org.apache.catalina.connector.OutputBuffer.append(OutputBuffer.java:685)
        at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:384)
        at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:362)
        at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:96)
        at org.glassfish.jersey.servlet.internal.ResponseWriter$NonCloseableOutputStreamWrapper.write(ResponseWriter.java:302)
        at org.glassfish.jersey.message.internal.CommittingOutputStream.write(CommittingOutputStream.java:200)
        at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$UnCloseableOutputStream.write(WriterInterceptorExecutor.java:276)
        at java.base/sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:233)
        at java.base/sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:303)
        at java.base/sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:281)
        at java.base/sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)
        at java.base/sun.nio.cs.StreamEncoder.write(StreamEncoder.java:135)
        at java.base/java.io.OutputStreamWriter.write(OutputStreamWriter.java:226)
        at org.glassfish.jersey.message.internal.ReaderWriter.writeToAsString(ReaderWriter.java:169)
        at org.glassfish.jersey.message.internal.AbstractMessageReaderWriterProvider.writeToAsString(AbstractMessageReaderWriterProvider.java:107)
        at org.glassfish.jersey.message.internal.StringMessageProvider.writeTo(StringMessageProvider.java:76)
        at org.glassfish.jersey.message.internal.StringMessageProvider.writeTo(StringMessageProvider.java:36)
        at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.invokeWriteTo(WriterInterceptorExecutor.java:242)
        at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.aroundWriteTo(WriterInterceptorExecutor.java:227)
        at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:139)
        at org.glassfish.jersey.server.internal.JsonWithPaddingInterceptor.aroundWriteTo(JsonWithPaddingInterceptor.java:85)
        at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:139)
        at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:61)
        ... 49 more
Caused by: java.io.IOException: Broken pipe
        at java.base/sun.nio.ch.FileDispatcherImpl.write0(Native Method)
        at java.base/sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
        at java.base/sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:113)
        at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:79)
        at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:50)
        at java.base/sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:462)
        at org.apache.tomcat.util.net.NioChannel.write(NioChannel.java:118)
        at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.doWrite(NioEndpoint.java:1374)
        at org.apache.tomcat.util.net.SocketWrapperBase.doWrite(SocketWrapperBase.java:764)
        at org.apache.tomcat.util.net.SocketWrapperBase.writeBlocking(SocketWrapperBase.java:589)
        at org.apache.tomcat.util.net.SocketWrapperBase.write(SocketWrapperBase.java:533)
        at org.apache.coyote.http11.Http11OutputBuffer$SocketOutputBuffer.doWrite(Http11OutputBuffer.java:547)

jkjha · 2023-03-28T11:25:40Z

jkjha
Mar 28, 2023
Author

I tried to run the indexer command manually using exact same above command inside the container and it worked completely fine...

opengrok-reindex-project -J=-XX:-UseGCOverheadLimit -J=-Xmx36g -J=-server --jar /opengrok/lib/opengrok.jar -t /opengrok/etc/logging.properties.template -p %PROJ% -d /opengrok/log/<project_name> -P <project_name> -U http://localhost:8080/ -- --renamedHistory off -r dirbased -G -m 4096 -c /usr/local/bin/ctags -U http://localhost:8080/ -o /opengrok/etc/ctags.config -H <project_name>'

It confuses me more what could be the issue and why auto indexing not working. I have set "NOMIRROR" true in my container.

0 replies

jkjha · 2023-03-28T11:28:07Z

jkjha
Mar 28, 2023
Author

one important log might be below while container runs auto indexing:

WARNING: Terminating process of command [/usr/bin/p4 changes -t -m 1 $'...#have'] in directory /opengrok/src/<project_name> due to timeout 5 seconds

I see this command runs fine when I do manual indexing (manually running above command).

0 replies

jkjha · 2023-03-28T11:47:40Z

jkjha
Mar 28, 2023
Author

pasting some more logs just for reference... may be irrelevant (from the container)... if it helps:

WARNING: Failed to scan [file:/usr/local/tomcat/webapps/ROOT/WEB-INF/lib/javax.el-3.0.0.jar] from classloader hierarchy
java.nio.file.NoSuchFileException: /usr/local/tomcat/webapps/ROOT/WEB-INF/lib/javax.el-3.0.0.jar

1 reply

vladak Mar 28, 2023
Maintainer

Not related. See #3390.

vladak · 2023-03-28T13:00:09Z

vladak
Mar 28, 2023
Maintainer

What is the problem you are trying to solve ? Is there some broken functionality or is it just the exception reported in the logs ?

The ClientAbortException seems to happen when a client terminates the connection prematurely.

The question is what what HTTP client request this exception maps to. Keep in mind that the container logs mix both Tomcat and Indexer and repository synchronization logs so it might be confusing.

21 replies

jkjha Mar 29, 2023
Author

Hi @vladak , I took your latest code from master and built my docker image. I also passed API_TIMEOUT to 60. But I still see its erroring out in 3 seconds:

This is 1st few line of start log:

And still same error essage I see... please note the read timeout = 3

Can you please suggest.

vladak Mar 29, 2023
Maintainer

Ha, the timeout value needs to propagate even further - to opengrok-reindex-project.

vladak Mar 29, 2023
Maintainer

And thanks for the testing !

vladak Mar 29, 2023
Maintainer

PR #4253 attempts to fix this.

jkjha Mar 30, 2023
Author

I have retrieved the most recent version from the master branch following your recent correction, and it appears that everything is functioning properly now. If any problems arise, I will provide an update here.
Thank you very much for your prompt resolution.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Indexing failing with error Caused by: java.io.IOException: Broken pipe #4247

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 4 comments 22 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

Indexing failing with error Caused by: java.io.IOException: Broken pipe #4247

jkjha Mar 28, 2023

Replies: 4 comments · 22 replies

jkjha Mar 28, 2023 Author

jkjha Mar 28, 2023 Author

jkjha Mar 28, 2023 Author

vladak Mar 28, 2023 Maintainer

vladak Mar 28, 2023 Maintainer

jkjha Mar 29, 2023 Author

vladak Mar 29, 2023 Maintainer

vladak Mar 29, 2023 Maintainer

vladak Mar 29, 2023 Maintainer

jkjha Mar 30, 2023 Author

jkjha
Mar 28, 2023

Replies: 4 comments 22 replies

jkjha
Mar 28, 2023
Author

jkjha
Mar 28, 2023
Author

jkjha
Mar 28, 2023
Author

vladak Mar 28, 2023
Maintainer

vladak
Mar 28, 2023
Maintainer

jkjha Mar 29, 2023
Author

vladak Mar 29, 2023
Maintainer

vladak Mar 29, 2023
Maintainer

vladak Mar 29, 2023
Maintainer

jkjha Mar 30, 2023
Author