Details
-
Bug
-
Status: Open
-
Minor
-
Resolution: Unresolved
-
1.4.1
-
None
-
None
-
Spark 2.2
Description
Steps :
sparS.sql("create external table if not exists thread_x stored by 'carbondata' location 'D:/KernelTest_New/KernetTest/KernelTest_New/KernetTest/thread/thread'").show();
sparS.sql("select count
from thread_x").show(); ----> 300000 record count
try{
System.out.println("count called after write");
Thread t1 = new Thread(new Trial(sparS));
Thread t2 = new Thread(new Trial(sparS));
//Each thread is holding 2 records each
t1.setName("T1");
t2.setName("T2");
t1.start();
Thread.sleep(10);
t2.start();
t1.join();
t2.join();
sparS.sql("select count from thread_x").show();
System.out.println("counrt called after load");
Actual Issue : sparS.sql("select count from thread_x").show(); -----> 300002
Which implies Two different threads are overwriting the same carbondatafile.
Expected : 300004 record count should be displayed.
(ie) A carbonfile has to be created for each thread. Concurrent load is failing.
From logs its is seen that two different threads are overwriting the same carbondatafile
Line 298: 18/08/22 16:58:10 INFO CarbonUtil: WriterForwardPool: thread_x Copying C:\Users\SWX593~1\AppData\Local\Temp\\carbon16096431985350_0\Fact\Part0\Segment_1534937289563\0\part-0-0_batchno0-0-1534937289563-1534937289561.carbondata --> D:/KernelTest_New/KernetTest/KernelTest_New/KernetTest/thread/thread
Line 302: 18/08/22 16:58:10 INFO CarbonUtil: WriterForwardPool: thread_x Copying C:\Users\SWX593~1\AppData\Local\Temp\\carbon16096431985350_0\Fact\Part0\Segment_1534937289563\0\0_batchno0-0-1534937289563-1534937289561.carbonindex --> D:/KernelTest_New/KernetTest/KernelTest_New/KernetTest/thread/thread
Line 374: 18/08/22 16:58:11 INFO CarbonUtil: WriterForwardPool: thread_x Copying C:\Users\SWX593~1\AppData\Local\Temp\\carbon16096980581610_0\Fact\Part0\Segment_1534937289563\0\part-0-0_batchno0-0-1534937289563-1534937289561.carbondata --> D:/KernelTest_New/KernetTest/KernelTest_New/KernetTest/thread/thread
Line 378: 18/08/22 16:58:11 INFO CarbonUtil: WriterForwardPool: thread_x Copying C:\Users\SWX593~1\AppData\Local\Temp\\carbon16096980581610_0\Fact\Part0\Segment_1534937289563\0\0_batchno0-0-1534937289563-1534937289561.carbonindex --> D:/KernelTest_New/KernetTest/KernelTest_New/KernetTest/thread/thread
