[CARBONDATA-1878] JVM crash after off-heap-sort disabled

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.3.0
• Fix Version/s: 1.3.0
• Component/s: core
• Assignee: Chuanyin Xu
• Reporter: Chuanyin Xu
• Created: 08/Dec/17 03:44
• Updated: 14/Dec/17 06:27
• Resolved: 14/Dec/17 06:27

Description

1. SCENARIO

Recently I have fixed some issues in Carbondata. To perform a full test to cover all the code that has been modified by me, I performed some iteration of the whole test case. Each iteration is started with different key configurations that will affect the flow in the code.

After I set `enable.offheap.sort=false` (default value is true) in the configuration, running tests will always end up with JVM crash error. The error messages are shown as below:

```
#

1. A fatal error has been detected by the Java Runtime Environment:
   #
2. SIGSEGV (0xb) at pc=0x00007f346b207ff1, pid=144619, tid=0x00007f346c2fc700
   #
3. JRE version: Java(TM) SE Runtime Environment (8.0_111-b14) (build 1.8.0_111-b14)
4. Java VM: Java HotSpot(TM) 64-Bit Server VM (25.111-b14 mixed mode linux-amd64 compressed oops)
5. Problematic frame:
6. V [libjvm.so+0xa90ff1] Unsafe_SetNativeShort+0x51
   #
7. Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
   #
8. An error report file with more information is saved as:
9. /home/xu/ws/carbondata/hs_err_pid144619.log
   #
10. If you would like to submit a bug report, please visit:
11. http://bugreport.java.com/bugreport/crash.jsp
    #

Process finished with exit code 134 (interrupted by signal 6: SIGABRT)
```

1. STEPS TOREPRODUCE

The error can be easily reproduced in different ways. Here I will provide a simple way to reproduce it:

1. Find the test case `DateDataTypeDirectDictionaryTest`.

2. Add the following code in the method `beforeAll`.
```
CarbonProperties.getInstance()
.addProperty(CarbonCommonConstants.ENABLE_OFFHEAP_SORT, "false")
```

3. Run this test case.

4. You will find the test failed with the above error.

5. Replace the code in Step2 with the following code:
```
CarbonProperties.getInstance()
.addProperty(CarbonCommonConstants.ENABLE_OFFHEAP_SORT, "true")
```

6. Run this test case.

7. The test is success without error.

1. ANALYZE & RESOLVE

I have reproduced this error and analyzed the core dump file. The final stack message in core dump looks like below:

```
Thread 73303: (state = IN_VM)

• sun.misc.Unsafe.putShort(long, short) @bci=0 (Interpreted frame)
• org.apache.carbondata.core.indexstore.UnsafeMemoryDMStore.addToUnsafe(org.apache.carbondata.core.indexstore.schema.CarbonRowSchema, org.apache.carbondata.core.indexstore.row.DataMapRow, int) @bci=781, line=150 (Interpreted frame)
• org.apache.carbondata.core.indexstore.UnsafeMemoryDMStore.addIndexRowToUnsafe(org.apache.carbondata.core.indexstore.row.DataMapRow) @bci=59, line=99 (Interpreted frame)
  ...
  ```

After inspecting the code, I found there lies bug in `UnsafeMemoryDMStore line=150` while writing length to unsafe memory – It writes with wrong base object.

Issue Links

• links to: GitHub Pull Request #1633