[GEODE-10081] Freeze during launch of locator

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.14.2
• Component/s: gfsh
• Assignee: Unassigned
• Reporter: Julien Gilles
• Created: 24/Feb/22 14:58
• Updated: 10/May/22 17:21

Description

We are using gfsh with a script to automatically launch a locator and a server.

During the launch of the locator, the process gfsh just freezes randomly. The locator process is forked, but gfsh remains frozen.

A jstack on the process shows that the process is stuck on an internal NativeLibrary.load call. This one happens when gfsh try to attach to the process : internally the jvm loads the libattach.so library, but the call never return.

To be complete, Geode is here deployed inside a docker container, using ubi8 as base image, java is jdk11.0.11 - we have tested with 11.0.13, same issue.

 

The stack is the following (full jstack dump attached)

"main" #1 prio=5 os_prio=0 cpu=2402.78ms elapsed=1924.45s allocated=236M defined_classes=10346 tid=0x00007fc904024000 nid=0x16 runnable  [0x00007fc90a84a000]
   java.lang.Thread.State: RUNNABLE
    at java.lang.ClassLoader$NativeLibrary.load0(java.base@11.0.11/Native Method)
    at java.lang.ClassLoader$NativeLibrary.load(java.base@11.0.11/Unknown Source)
    at java.lang.ClassLoader$NativeLibrary.loadLibrary(java.base@11.0.11/Unknown Source)
    - locked <0x00000000823efe60> (a java.util.HashSet)
    at java.lang.ClassLoader.loadLibrary0(java.base@11.0.11/Unknown Source)
    at java.lang.ClassLoader.loadLibrary(java.base@11.0.11/Unknown Source)
    at java.lang.Runtime.loadLibrary0(java.base@11.0.11/Unknown Source)
    - locked <0x00000000828224b8> (a java.lang.Runtime)
    at java.lang.System.loadLibrary(java.base@11.0.11/Unknown Source)
    at sun.tools.attach.VirtualMachineImpl.<clinit>(jdk.attach@11.0.11/Unknown Source)
    at sun.tools.attach.AttachProviderImpl.attachVirtualMachine(jdk.attach@11.0.11/Unknown Source)
    at com.sun.tools.attach.VirtualMachine.attach(jdk.attach@11.0.11/Unknown Source)
    at org.apache.geode.internal.process.MBeanProcessController.getJMXServiceURL(MBeanProcessController.java:227)
    at org.apache.geode.internal.process.MBeanProcessController.connect(MBeanProcessController.java:192)
    at org.apache.geode.internal.process.MBeanProcessController.invokeOperationOnTargetMBean(MBeanProcessController.java:159)
    at org.apache.geode.internal.process.MBeanProcessController.status(MBeanProcessController.java:136)
    at org.apache.geode.internal.process.MBeanProcessController.status(MBeanProcessController.java:81)
    at org.apache.geode.internal.process.MBeanOrFileProcessController.status(MBeanOrFileProcessController.java:37)
    at org.apache.geode.distributed.LocatorLauncher.statusWithWorkingDirectory(LocatorLauncher.java:1012)
    at org.apache.geode.distributed.LocatorLauncher.status(LocatorLauncher.java:940)
    at org.apache.geode.distributed.LocatorLauncher$LocatorState.fromDirectory(LocatorLauncher.java:2077)
    at org.apache.geode.management.internal.cli.commands.StartLocatorCommand.doStartLocator(StartLocatorCommand.java:267)
    at org.apache.geode.management.internal.cli.commands.StartLocatorCommand.startLocator(StartLocatorCommand.java:133)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@11.0.11/Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@11.0.11/Unknown Source)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.11/Unknown Source)
    at java.lang.reflect.Method.invoke(java.base@11.0.11/Unknown Source)
    at org.springframework.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:282)
    at org.apache.geode.management.internal.cli.remote.CommandExecutor.callInvokeMethod(CommandExecutor.java:139)
    at org.apache.geode.management.internal.cli.remote.CommandExecutor.invokeCommand(CommandExecutor.java:149)