[TAJO-1830] Fix race condition in HdfsServiceTracker

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.10.0
• Fix Version/s: 0.11.0, 0.12.0
• Component/s: TajoMaster, Unit Test
• Assignee: Jinho Kim
• Reporter: Jinho Kim
• Created: 09/Sep/15 02:48
• Updated: 11/Sep/15 06:30
• Resolved: 11/Sep/15 05:40

Description

If HdfsServiceTracker reads active-master file before the file not closed, active-master file is empty.

{format}
testAutoFailOver(org.apache.tajo.ha.TestHAServiceHDFSImpl) Time elapsed: 4.668 sec <<< ERROR!
org.apache.tajo.exception.TajoRuntimeException: org.apache.tajo.client.v2.exception.ClientConnectionException: java.io.EOFException
at org.apache.tajo.client.SessionConnection.getTajoMasterConnection(SessionConnection.java:141)
at org.apache.tajo.client.SessionConnection.<init>(SessionConnection.java:113)
at org.apache.tajo.client.TajoClientImpl.<init>(TajoClientImpl.java:62)
at org.apache.tajo.client.TajoClientImpl.<init>(TajoClientImpl.java:86)
at org.apache.tajo.client.TajoClientImpl.<init>(TajoClientImpl.java:82)
at org.apache.tajo.ha.TestHAServiceHDFSImpl.verifyDataBaseAndTable(TestHAServiceHDFSImpl.java:152)
at org.apache.tajo.ha.TestHAServiceHDFSImpl.testAutoFailOver(TestHAServiceHDFSImpl.java:82)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
Caused by: org.apache.tajo.client.v2.exception.ClientConnectionException: java.io.EOFException
... 30 more
Caused by: org.apache.tajo.service.ServiceTrackerException: java.io.EOFException
at org.apache.tajo.ha.HdfsServiceTracker.getAddressElements(HdfsServiceTracker.java:508)
at org.apache.tajo.ha.HdfsServiceTracker.getClientServiceAddress(HdfsServiceTracker.java:409)
at org.apache.tajo.client.SessionConnection.getTajoMasterAddr(SessionConnection.java:361)
at org.apache.tajo.client.SessionConnection.getTajoMasterConnection(SessionConnection.java:130)
... 29 more
Caused by: java.io.EOFException
at java.io.DataInputStream.readUnsignedShort(DataInputStream.java:340)
at java.io.DataInputStream.readUTF(DataInputStream.java:589)
at java.io.DataInputStream.readUTF(DataInputStream.java:564)
at org.apache.tajo.ha.HdfsServiceTracker.getAddressElements(HdfsServiceTracker.java:498)
... 32 more
at org.apache.tajo.ha.HdfsServiceTracker.createMasterFile(HdfsServiceTracker.java:249)
at org.apache.tajo.ha.HdfsServiceTracker.register(HdfsServiceTracker.java:155)
at org.apache.tajo.ha.HdfsServiceTracker$PingChecker.run(HdfsServiceTracker.java:374)
at java.lang.Thread.run(Thread.java:745){format}