Occational ClassCircularityError based on SkyWalking Java Agent 8.16.0

[yizhouw11]

I developed a couple of plugins based on and to use with Java agent 8.16.0.

Problem: after every few hundred restarts of a container app, a few errors like java.lang.ClassCircularityError: java/util/concurrent/ThreadLocalRandom occur in ONE of the containers at request processing. Or at some backend thread where none of the classes in the stack trace were enhanced. When it happens, the QPS is always at a relatively high level. (ThreadLocalRandom for 98% of the case, but yes the exception does occur with other classes as well). According to the stack trace, ThreadLocalRandom was invoked while java.util.Map expands. However, we did not enhance ThreadLocalRandom or Map, etc.

I've tried to locate the direct cause by adding or removing plugins or codes I developed, but the result indicates that the error occurs upon a COMPOSITION of factors, so currently it is hard to narrow down.

Would there be any advice for addressing the problem? Thank you!

[wu-sheng]

AFAIK, ThreadLocalRandom is never used by the agent, and we don't instrument this class too. So, if your error is about this class, it is related to the upstream.

[wu-sheng]

ClassCircularityError is sometimes made when different threads trying to load through static blocks. In SkyWallking, we usually don't use static blocks.

[wu-sheng]

ClassCircularityError should report the relative classes, you should be able to find a clue from there.

[yizhouw11]

Thank you for the reply.

I've tried removing all static blocks in my customized plugins but still got this error. Also, I've tried rebasing my plugins onto the latest SkyWalking Java Agent 9.3.0 version but still got this error.

I've encountered several classes that follow this java.lang.ClassCircularityError, including java/util/concurrent/ThreadLocalRandom, org.apache.kafka.clients.MetadataCache$PartitionInfoAndEpoch and org.hibernate.validator.internal.constraintvalidators.bv.number.bound.MaxValidatorForBigDecimal. If we take the most common one ThreadLocalRandom as an example, the error stacks under TheadLocalRandom are various as well. Some error stacks are related to NioEndpoint class in Tomcat, some to sofamq, and some to StandardSession class in Catalina. So it's hard to find effective clues from error class or error stack.

[bert82503]

Hi, brother. @wu-sheng
Other application encounter this issue, which occurs when the app has just finished start run ready for K8s Liveness Probe (存活探针). And the app doesn't recover until it restarts.
Thanks!

The environment

• skywalking-agent-9.1.0
• Dubbo Java 3.2.10
• jdk1.8.0_333

Shortly after the application was launched, this exception occurred.
应用启动后不久，出现了这个异常。

the full exception stack:

java.lang.ClassCircularityError: java/util/concurrent/ThreadLocalRandom
  at java.util.concurrent.ConcurrentHashMap.addCount(ConcurrentHashMap.java:2263)
  at java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1070)
  at java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:1006)
  at org.apache.dubbo.remoting.exchange.support.DefaultFuture.<init>(DefaultFuture.java:99)
  at org.apache.dubbo.remoting.exchange.support.DefaultFuture.newFuture(DefaultFuture.java:128)
  at org.apache.dubbo.remoting.exchange.support.header.HeaderExchangeChannel.request(HeaderExchangeChannel.java:153)
  at org.apache.dubbo.remoting.exchange.support.header.HeaderExchangeClient.request(HeaderExchangeClient.java:109)
  at org.apache.dubbo.rpc.protocol.dubbo.ReferenceCountExchangeClient.request(ReferenceCountExchangeClient.java:95)
  at org.apache.dubbo.rpc.protocol.dubbo.DubboInvoker.doInvoke(DubboInvoker.java:134)
  at org.apache.dubbo.rpc.protocol.AbstractInvoker.doInvokeAndReturn(AbstractInvoker.java:248)
  at org.apache.dubbo.rpc.protocol.AbstractInvoker.invoke(AbstractInvoker.java:191)
  at org.apache.dubbo.rpc.listener.ListenerInvokerWrapper.invoke(ListenerInvokerWrapper.java:71)
  at org.apache.dubbo.rpc.filter.RpcExceptionFilter.invoke(RpcExceptionFilter.java:40)
  at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349)
  at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CallbackRegistrationInvoker.invoke(FilterChainBuilder.java:197)
  at org.apache.dubbo.rpc.protocol.ReferenceCountInvokerWrapper.invoke(ReferenceCountInvokerWrapper.java:106)
  at org.apache.dubbo.rpc.cluster.support.AbstractClusterInvoker.invokeWithContext(AbstractClusterInvoker.java:412)
  at org.apache.dubbo.rpc.cluster.support.FailfastClusterInvoker.doInvoke(FailfastClusterInvoker.java:48)
  at org.apache.dubbo.rpc.cluster.support.AbstractClusterInvoker.invoke(AbstractClusterInvoker.java:366)
  at org.apache.dubbo.rpc.cluster.router.RouterSnapshotFilter.invoke(RouterSnapshotFilter.java:46)
  at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349)
  at org.apache.dubbo.monitor.support.MonitorFilter.$sw$original$invoke$1vamrv0(MonitorFilter.java:108)
  at org.apache.dubbo.monitor.support.MonitorFilter.$sw$original$invoke$1vamrv0$accessor$$sw$p4vq621(MonitorFilter.java)
  at org.apache.dubbo.monitor.support.MonitorFilter$$sw$auxiliary$8n9i9c3.call(Unknown Source)
  at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstMethodsInter.intercept(InstMethodsInter.java:86)
  at org.apache.dubbo.monitor.support.MonitorFilter.invoke(MonitorFilter.java)
  at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349)
  at org.apache.dubbo.rpc.cluster.filter.support.MetricsClusterFilter.invoke(MetricsClusterFilter.java:57)
  at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349)
  at org.apache.dubbo.rpc.protocol.dubbo.filter.FutureFilter.invoke(FutureFilter.java:52)
  at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349)
  at org.apache.dubbo.rpc.cluster.filter.support.ObservationSenderFilter.invoke(ObservationSenderFilter.java:62)
  at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349)
  at org.apache.dubbo.spring.security.filter.ContextHolderParametersSelectedTransferFilter.invoke(ContextHolderParametersSelectedTransferFilter.java:40)
  at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349)
  at org.apache.dubbo.metrics.filter.MetricsFilter.invoke(MetricsFilter.java:86)
  at org.apache.dubbo.rpc.cluster.filter.support.MetricsConsumerFilter.invoke(MetricsConsumerFilter.java:38)
  at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:349)
  at org.apache.dubbo.rpc.cluster.filter.support.ConsumerClassLoaderFilter.invoke(ConsumerClassLoaderFilter.java:40)

关键根因日志

java.lang.ClassCircularityError: java/util/concurrent/ThreadLocalRandom
  at java.util.concurrent.ConcurrentHashMap.addCount(ConcurrentHashMap.java:2263)

ConcurrentHashMap.addCount(ConcurrentHashMap.java:2263)

ThreadLocalRandom.getProbe()

sun.misc.Unsafe#getInt(java.lang.Object, long)
native方法

The full detail please to see apache/dubbo#14578

#10207

[yizhouw11]

@wu-sheng @bert82503 This error is caused by ByteBuddy and a similar issue was proposed there. The usage of ConcurrentHashMap, whose addCount() method uses ThreadLocalRandom if there exists thread contention, was introduced since ByteBuddy 1.12.11 via this commit. By means of ByteBuddy, ClassFileTransformer.transform() will trigger class loading of ThreadLocalRandom which is probably being loaded, in which case ClassCircularityError will be thrown. An issue in the OpenJDK community provides detailed explanations of this exception and proposes to adjust the name of the thrown error message.

To fix this error in the SkyWalking level, preloading ThreadLocalRandom before calling ByteBuddy will work. Referring to another java agent, I tried to add codes below to PluginBootstrap which prevents this error according to my testing.

// Preload ThreadLocalRandom in case of intermittent ClassCircularityError since ByteBuddy 1.12.11
static {
    try {
        Class.forName("java.util.concurrent.ThreadLocalRandom");
    } catch (Exception e) {
        LOGGER.warn("Preload ThreadLocalRandom failure.");
    }
}

[wu-sheng]

Thanks for the update. This seems a good respond from openjdk community.

If you want to propose a pull request, I am good to review that PR.

[wu-sheng]

Added through apache/skywalking-java#718. And will be released in 9.4.0