Hive
  1. Hive
  2. HIVE-3537

release locks at the end of move tasks

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.11.0
    • Component/s: Locking, Query Processor
    • Labels:
      None

      Description

      Look at HIVE-3106 for details.

      In order to make sure that concurrency is not an issue for multi-table
      inserts, the current option is to introduce a dependency task, which thereby
      delays the creation of all partitions. It would be desirable to release the
      locks for the outputs as soon as the move task is completed. That way, for
      multi-table inserts, the concurrency can be enabled without delaying any
      table.

      Currently, the movetask contains a input/output, but they do not seem to be
      populated correctly.

      1. hive.3537.9.patch
        14 kB
        Namit Jain
      2. hive.3537.8.patch
        14 kB
        Namit Jain
      3. hive.3537.7.patch
        14 kB
        Namit Jain
      4. hive.3537.6.patch
        14 kB
        Namit Jain
      5. hive.3537.5.patch
        14 kB
        Namit Jain
      6. hive.3537.4.patch
        14 kB
        Namit Jain
      7. hive.3537.3.patch
        14 kB
        Namit Jain
      8. hive.3537.2.patch
        14 kB
        Namit Jain
      9. hive.3537.10.patch
        19 kB
        Namit Jain
      10. hive.3537.1.patch
        11 kB
        Namit Jain

        Activity

        Namit Jain created issue -
        Namit Jain made changes -
        Field Original Value New Value
        Assignee Namit Jain [ namit ]
        Carl Steinbach made changes -
        Component/s Locking [ 12317303 ]
        Show
        Namit Jain added a comment - https://reviews.facebook.net/D5931
        Namit Jain made changes -
        Attachment hive.3537.1.patch [ 12547967 ]
        Namit Jain made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Kevin Wilfong made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Namit Jain made changes -
        Attachment hive.3537.2.patch [ 12555732 ]
        Hide
        Namit Jain added a comment -

        comments addressed

        Show
        Namit Jain added a comment - comments addressed
        Namit Jain made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Namit Jain made changes -
        Attachment hive.3537.3.patch [ 12556124 ]
        Hide
        Kevin Wilfong added a comment -

        +1

        Show
        Kevin Wilfong added a comment - +1
        Kevin Wilfong made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Hide
        Kevin Wilfong added a comment -

        Lots of NPE's when running tests.

        Show
        Kevin Wilfong added a comment - Lots of NPE's when running tests.
        Namit Jain made changes -
        Attachment hive.3537.4.patch [ 12561253 ]
        Namit Jain made changes -
        Attachment hive.3537.5.patch [ 12561429 ]
        Namit Jain made changes -
        Attachment hive.3537.6.patch [ 12561441 ]
        Namit Jain made changes -
        Attachment hive.3537.7.patch [ 12561634 ]
        Namit Jain made changes -
        Attachment hive.3537.8.patch [ 12563901 ]
        Namit Jain made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Namit Jain made changes -
        Attachment hive.3537.9.patch [ 12563996 ]
        Hide
        Namit Jain added a comment -

        refreshed and fixed tests

        Show
        Namit Jain added a comment - refreshed and fixed tests
        Hide
        Ashutosh Chauhan added a comment -

        Namit,
        Will this result in fix introduced in HIVE-3106 redundant, i.e., dependency task (and thus config variable) introduced will no longer be required. If not, than config variable should always be true, isn't it for concurrency to work correctly?

        Show
        Ashutosh Chauhan added a comment - Namit, Will this result in fix introduced in HIVE-3106 redundant, i.e., dependency task (and thus config variable) introduced will no longer be required. If not, than config variable should always be true, isn't it for concurrency to work correctly?
        Hide
        Ashutosh Chauhan added a comment -

        Also, seems like it won't be easy to write a test case for this. One possibility I can think of is to write a junit test-case (instead of .q) which tests for the lock data in ZK and asserts that lock data is wiped out of ZK, after query doing multi-insert finishes.

        Show
        Ashutosh Chauhan added a comment - Also, seems like it won't be easy to write a test case for this. One possibility I can think of is to write a junit test-case (instead of .q) which tests for the lock data in ZK and asserts that lock data is wiped out of ZK, after query doing multi-insert finishes.
        Hide
        Namit Jain added a comment -

        @Ashutosh, I agree - with this, HIVE-3106 is redundant.
        Do you want me to remove that code/config, or leave it around for now - I cant think of a good usecase why would be useful other than locking,
        but intuitively, it seems to be good to allow multi-table insert to finish atomically.

        Show
        Namit Jain added a comment - @Ashutosh, I agree - with this, HIVE-3106 is redundant. Do you want me to remove that code/config, or leave it around for now - I cant think of a good usecase why would be useful other than locking, but intuitively, it seems to be good to allow multi-table insert to finish atomically.
        Hide
        Ashutosh Chauhan added a comment -

        Do you want me to remove that code/config,

        Yes please. Dependency task approach looks very hacky to me. We should really get rid of it.

        Show
        Ashutosh Chauhan added a comment - Do you want me to remove that code/config, Yes please. Dependency task approach looks very hacky to me. We should really get rid of it.
        Hide
        Namit Jain added a comment -

        Ashutosh Chauhan, I was thinking about it more, and I think we should leave it for some users.
        What if I want atomicity for some reason - Without dependency, it is possible that:

        the query fails (for some reason), but some outputs get changed, and we dont even know which one.

        Show
        Namit Jain added a comment - Ashutosh Chauhan , I was thinking about it more, and I think we should leave it for some users. What if I want atomicity for some reason - Without dependency, it is possible that: the query fails (for some reason), but some outputs get changed, and we dont even know which one.
        Hide
        Namit Jain added a comment -

        ping Ashutosh Chauhan, what do you think ?

        Show
        Namit Jain added a comment - ping Ashutosh Chauhan , what do you think ?
        Hide
        Ashutosh Chauhan added a comment -

        Before HIVE-3106, for a multi-insert case, if query failed it was possible that some partitions gets inserted while others don't inspite of acquiring table level locks. HIVE-3106 fixed that by introducing a fake dependency task which was added at the end of the plan because of which partitions will be added only after all the MR jobs are finished and data is generated for all tables. This jira fixes the issue where a lock on table is released as soon as partition addition on that particular table is completed. This ensures consistency for that table as in no other updates could be made on that table until lock is released. Atomicity you are referring to in your previous comment is when users want either all of the tables are changed or none at all. Are all these statements correct? Wants to make sure I understood all the nuances here.

        Show
        Ashutosh Chauhan added a comment - Before HIVE-3106 , for a multi-insert case, if query failed it was possible that some partitions gets inserted while others don't inspite of acquiring table level locks. HIVE-3106 fixed that by introducing a fake dependency task which was added at the end of the plan because of which partitions will be added only after all the MR jobs are finished and data is generated for all tables. This jira fixes the issue where a lock on table is released as soon as partition addition on that particular table is completed. This ensures consistency for that table as in no other updates could be made on that table until lock is released. Atomicity you are referring to in your previous comment is when users want either all of the tables are changed or none at all. Are all these statements correct? Wants to make sure I understood all the nuances here.
        Hide
        Namit Jain added a comment -

        Atomicity you are referring to in your previous comment is when users want either all of the tables are changed or none at all.

        YES, this has nothing to do with locking.

        Show
        Namit Jain added a comment - Atomicity you are referring to in your previous comment is when users want either all of the tables are changed or none at all. YES, this has nothing to do with locking.
        Hide
        Ashutosh Chauhan added a comment -

        I agree, atomicity is an useful property to guarantee, I know of use-cases where folks have wanted this from Hive. I wish we had a cleaner approach instead of fake dependency task, but if thats the best we can come up with, I am fine with that.
        +1

        Show
        Ashutosh Chauhan added a comment - I agree, atomicity is an useful property to guarantee, I know of use-cases where folks have wanted this from Hive. I wish we had a cleaner approach instead of fake dependency task, but if thats the best we can come up with, I am fine with that. +1
        Hide
        Namit Jain added a comment -

        Ashutosh Chauhan, I am assuming you are taking care of testing and committing.
        If you want me to to do it, let me know.

        Show
        Namit Jain added a comment - Ashutosh Chauhan , I am assuming you are taking care of testing and committing. If you want me to to do it, let me know.
        Hide
        Ashutosh Chauhan added a comment -

        OK. I am running tests now. Will commit if tests pass. BTW, there is no tests in the patch, I assume you did some manual testing of the patch, since its hard to write a test-case for this. Curious how did you test, since it seems like it will be hard to simulate failure before the patch.

        Show
        Ashutosh Chauhan added a comment - OK. I am running tests now. Will commit if tests pass. BTW, there is no tests in the patch, I assume you did some manual testing of the patch, since its hard to write a test-case for this. Curious how did you test, since it seems like it will be hard to simulate failure before the patch.
        Hide
        Namit Jain added a comment -

        Dont need to simulate a failure. Can have a multi-table insert where one of the inserts take a much longer than time than the other ?
        Manually verified that the locks are getting released when one of the branches is completed.

        Show
        Namit Jain added a comment - Dont need to simulate a failure. Can have a multi-table insert where one of the inserts take a much longer than time than the other ? Manually verified that the locks are getting released when one of the branches is completed.
        Hide
        Ashutosh Chauhan added a comment -

        Seems like there is some problem. My tests have been running for more than 15 hours and they seem to have been stuck. e.g., following tests extraordinarily long to complete:

         [junit] Done query: bucketmapjoin5.q elapsedTime=4421s
         [junit] Done query: bucketmapjoin4.q elapsedTime=4419s
         [junit] Done query: bucketmapjoin2.q elapsedTime=6625s
         [junit] Done query: bucketmapjoin3.q elapsedTime=4414s
        

        I was able to capture jstack for one of it.

        jstack 14269
        2013-01-16 23:29:23
        Full thread dump Java HotSpot(TM) Server VM (10.0-b19 mixed mode):
        
        "Attach Listener" daemon prio=10 tid=0x08786400 nid=0x74a4 waiting on condition [0x00000000..0x00000000]
           java.lang.Thread.State: RUNNABLE
        
        "main-EventThread" daemon prio=10 tid=0xce4ae000 nid=0x63ac waiting on condition [0xcedfe000..0xcedff030]
           java.lang.Thread.State: WAITING (parking)
        	at sun.misc.Unsafe.park(Native Method)
        	- parking to wait for  <0xd684c6b8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
        	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
        	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:493)
        
        "main-SendThread(localhost:21818)" daemon prio=10 tid=0xcf142400 nid=0x63ab runnable [0xce9ad000..0xce9adfb0]
           java.lang.Thread.State: RUNNABLE
        	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
        	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
        	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
        	- locked <0xd6854078> (a sun.nio.ch.Util$1)
        	- locked <0xd6854068> (a java.util.Collections$UnmodifiableSet)
        	- locked <0xd6853bf8> (a sun.nio.ch.EPollSelectorImpl)
        	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
        	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:274)
        	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
        
        "main-EventThread" daemon prio=10 tid=0xce4aa800 nid=0x63aa waiting on condition [0xce95c000..0xce95d130]
           java.lang.Thread.State: WAITING (parking)
        	at sun.misc.Unsafe.park(Native Method)
        	- parking to wait for  <0xd6856b70> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
        	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
        	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:493)
        
        "main-SendThread(localhost:21818)" daemon prio=10 tid=0xce4aa400 nid=0x63a9 runnable [0xcf5d7000..0xcf5d80b0]
           java.lang.Thread.State: RUNNABLE
        	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
        	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
        	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
        	- locked <0xd6857720> (a sun.nio.ch.Util$1)
        	- locked <0xd6857710> (a java.util.Collections$UnmodifiableSet)
        	- locked <0xd68572a0> (a sun.nio.ch.EPollSelectorImpl)
        	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
        	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:274)
        	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
        
        "main-EventThread" daemon prio=10 tid=0xcf4f3c00 nid=0x37f1 waiting on condition [0xcf35c000..0xcf35d030]
           java.lang.Thread.State: WAITING (parking)
        	at sun.misc.Unsafe.park(Native Method)
        	- parking to wait for  <0xd5528060> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
        	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
        	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:493)
        
        "main-SendThread(localhost:21818)" daemon prio=10 tid=0xcf11f000 nid=0x37f0 runnable [0xce9fe000..0xce9fefb0]
           java.lang.Thread.State: RUNNABLE
        	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
        	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
        	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
        	- locked <0xd5528d48> (a sun.nio.ch.Util$1)
        	- locked <0xd5528d38> (a java.util.Collections$UnmodifiableSet)
        	- locked <0xd55288c8> (a sun.nio.ch.EPollSelectorImpl)
        	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
        	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:274)
        	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
        
        "derby.rawStoreDaemon" daemon prio=10 tid=0x08edd800 nid=0x37e6 in Object.wait() [0xcf3ad000..0xcf3ae030]
           java.lang.Thread.State: TIMED_WAITING (on object monitor)
        	at java.lang.Object.wait(Native Method)
        	- waiting on <0xd50539f0> (a org.apache.derby.impl.services.daemon.BasicDaemon)
        	at org.apache.derby.impl.services.daemon.BasicDaemon.rest(Unknown Source)
        	- locked <0xd50539f0> (a org.apache.derby.impl.services.daemon.BasicDaemon)
        	at org.apache.derby.impl.services.daemon.BasicDaemon.run(Unknown Source)
        	at java.lang.Thread.run(Thread.java:619)
        
        "Timer-0" daemon prio=10 tid=0x09070c00 nid=0x37e3 in Object.wait() [0xcf3fe000..0xcf3ff0b0]
           java.lang.Thread.State: WAITING (on object monitor)
        	at java.lang.Object.wait(Native Method)
        	- waiting on <0xd4e87170> (a java.util.TaskQueue)
        	at java.lang.Object.wait(Object.java:485)
        	at java.util.TimerThread.mainLoop(Timer.java:483)
        	- locked <0xd4e87170> (a java.util.TaskQueue)
        	at java.util.TimerThread.run(Timer.java:462)
        
        "derby.antiGC" daemon prio=10 tid=0x08da4800 nid=0x37e2 in Object.wait() [0xcf586000..0xcf586e30]
           java.lang.Thread.State: WAITING (on object monitor)
        	at java.lang.Object.wait(Native Method)
        	- waiting on <0xd4e8a188> (a org.apache.derby.impl.services.monitor.AntiGC)
        	at java.lang.Object.wait(Object.java:485)
        	at org.apache.derby.impl.services.monitor.AntiGC.run(Unknown Source)
        	- locked <0xd4e8a188> (a org.apache.derby.impl.services.monitor.AntiGC)
        	at java.lang.Thread.run(Thread.java:619)
        
        "ProcessThread(sid:0 cport:-1):" prio=10 tid=0xcfed1000 nid=0x37d7 waiting on condition [0xcf679000..0xcf679e30]
           java.lang.Thread.State: WAITING (parking)
        	at sun.misc.Unsafe.park(Native Method)
        	- parking to wait for  <0xd4df4e60> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
        	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
        	at org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:119)
        
        "SyncThread:0" prio=10 tid=0xcfed4800 nid=0x37d6 waiting on condition [0xcf6ca000..0xcf6cadb0]
           java.lang.Thread.State: WAITING (parking)
        	at sun.misc.Unsafe.park(Native Method)
        	- parking to wait for  <0xd4df74d0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
        	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
        	at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:97)
        
        "SessionTracker" prio=10 tid=0xcfd99800 nid=0x37d5 in Object.wait() [0xcf71b000..0xcf71bf30]
           java.lang.Thread.State: TIMED_WAITING (on object monitor)
        	at java.lang.Object.wait(Native Method)
        	- waiting on <0xd4df7e80> (a org.apache.zookeeper.server.SessionTrackerImpl)
        	at org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:146)
        	- locked <0xd4df7e80> (a org.apache.zookeeper.server.SessionTrackerImpl)
        
        "NIOServerCxn.Factory:0.0.0.0/0.0.0.0:21818" daemon prio=10 tid=0xcfcd9000 nid=0x37d4 runnable [0xcf76c000..0xcf76ceb0]
           java.lang.Thread.State: RUNNABLE
        	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
        	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
        	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
        	- locked <0xd4d4a100> (a sun.nio.ch.Util$1)
        	- locked <0xd4d4a0f0> (a java.util.Collections$UnmodifiableSet)
        	- locked <0xd4d49ec0> (a sun.nio.ch.EPollSelectorImpl)
        	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
        	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:194)
        	at java.lang.Thread.run(Thread.java:619)
        
        "Low Memory Detector" daemon prio=10 tid=0x0868e400 nid=0x37d0 runnable [0x00000000..0x00000000]
           java.lang.Thread.State: RUNNABLE
        
        "CompilerThread1" daemon prio=10 tid=0x0868cc00 nid=0x37cf waiting on condition [0x00000000..0xd00f05f8]
           java.lang.Thread.State: RUNNABLE
        
        "CompilerThread0" daemon prio=10 tid=0x0868a400 nid=0x37ce waiting on condition [0x00000000..0xd0171578]
           java.lang.Thread.State: RUNNABLE
        
        "Signal Dispatcher" daemon prio=10 tid=0x08689000 nid=0x37cd runnable [0x00000000..0xd01c2b10]
           java.lang.Thread.State: RUNNABLE
        
        "Finalizer" daemon prio=10 tid=0x0866e000 nid=0x37cc in Object.wait() [0xd0413000..0xd0413db0]
           java.lang.Thread.State: WAITING (on object monitor)
        	at java.lang.Object.wait(Native Method)
        	- waiting on <0xd4dcd590> (a java.lang.ref.ReferenceQueue$Lock)
        	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
        	- locked <0xd4dcd590> (a java.lang.ref.ReferenceQueue$Lock)
        	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
        	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
        
        "Reference Handler" daemon prio=10 tid=0x0866cc00 nid=0x37cb in Object.wait() [0xd0464000..0xd0464f30]
           java.lang.Thread.State: WAITING (on object monitor)
        	at java.lang.Object.wait(Native Method)
        	- waiting on <0xd4d54fa8> (a java.lang.ref.Reference$Lock)
        	at java.lang.Object.wait(Object.java:485)
        	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
        	- locked <0xd4d54fa8> (a java.lang.ref.Reference$Lock)
        
        "main" prio=10 tid=0x08583000 nid=0x37c1 waiting on condition [0xf7ee7000..0xf7ee8208]
           java.lang.Thread.State: TIMED_WAITING (sleeping)
        	at java.lang.Thread.sleep(Native Method)
        	at org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager.unlockWithRetry(ZooKeeperHiveLockManager.java:415)
        	at org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager.unlock(ZooKeeperHiveLockManager.java:404)
        	at org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager.releaseLocks(ZooKeeperHiveLockManager.java:251)
        	at org.apache.hadoop.hive.ql.Driver.releaseLocks(Driver.java:862)
        	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:948)
        	at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259)
        	at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
        	at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413)
        	at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:348)
        	at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:774)
        	at org.apache.hadoop.hive.cli.TestCliDriver.runTest(TestCliDriver.java:5609)
        	at org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin5(TestCliDriver.java:1657)
        	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        	at java.lang.reflect.Method.invoke(Method.java:597)
        	at junit.framework.TestCase.runTest(TestCase.java:168)
        	at junit.framework.TestCase.runBare(TestCase.java:134)
        	at junit.framework.TestResult$1.protect(TestResult.java:110)
        	at junit.framework.TestResult.runProtected(TestResult.java:128)
        	at junit.framework.TestResult.run(TestResult.java:113)
        	at junit.framework.TestCase.run(TestCase.java:124)
        	at junit.framework.TestSuite.runTest(TestSuite.java:232)
        	at junit.framework.TestSuite.run(TestSuite.java:227)
        	at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518)
        	at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052)
        	at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906)
        "VM Thread" prio=10 tid=0x08669c00 nid=0x37ca runnable 
        
        "GC task thread#0 (ParallelGC)" prio=10 tid=0x08589c00 nid=0x37c2 runnable 
        
        "GC task thread#1 (ParallelGC)" prio=10 tid=0x0858b000 nid=0x37c3 runnable 
        
        "GC task thread#2 (ParallelGC)" prio=10 tid=0x0858c000 nid=0x37c4 runnable 
        
        "GC task thread#3 (ParallelGC)" prio=10 tid=0x0858d000 nid=0x37c5 runnable 
        
        "GC task thread#4 (ParallelGC)" prio=10 tid=0x0858e000 nid=0x37c6 runnable 
        
        "GC task thread#5 (ParallelGC)" prio=10 tid=0x0858f400 nid=0x37c7 runnable 
        
        "GC task thread#6 (ParallelGC)" prio=10 tid=0x08590400 nid=0x37c8 runnable 
        
        "GC task thread#7 (ParallelGC)" prio=10 tid=0x08591400 nid=0x37c9 runnable 
        
        "VM Periodic Task Thread" prio=10 tid=0x0868fc00 nid=0x37d1 waiting on condition 
        
        JNI global references: 888
        
        

        Though none of the tests have failed and there is no deadlock in this jstack. But this stack and extra-ordinarily long run time of these tests suggest, something is up here. Are you observing similar behavior in your environment?

        Show
        Ashutosh Chauhan added a comment - Seems like there is some problem. My tests have been running for more than 15 hours and they seem to have been stuck. e.g., following tests extraordinarily long to complete: [junit] Done query: bucketmapjoin5.q elapsedTime=4421s [junit] Done query: bucketmapjoin4.q elapsedTime=4419s [junit] Done query: bucketmapjoin2.q elapsedTime=6625s [junit] Done query: bucketmapjoin3.q elapsedTime=4414s I was able to capture jstack for one of it. jstack 14269 2013-01-16 23:29:23 Full thread dump Java HotSpot(TM) Server VM (10.0-b19 mixed mode): "Attach Listener" daemon prio=10 tid=0x08786400 nid=0x74a4 waiting on condition [0x00000000..0x00000000] java.lang. Thread .State: RUNNABLE "main-EventThread" daemon prio=10 tid=0xce4ae000 nid=0x63ac waiting on condition [0xcedfe000..0xcedff030] java.lang. Thread .State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0xd684c6b8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:493) "main-SendThread(localhost:21818)" daemon prio=10 tid=0xcf142400 nid=0x63ab runnable [0xce9ad000..0xce9adfb0] java.lang. Thread .State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked <0xd6854078> (a sun.nio.ch.Util$1) - locked <0xd6854068> (a java.util.Collections$UnmodifiableSet) - locked <0xd6853bf8> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:274) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035) "main-EventThread" daemon prio=10 tid=0xce4aa800 nid=0x63aa waiting on condition [0xce95c000..0xce95d130] java.lang. Thread .State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0xd6856b70> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:493) "main-SendThread(localhost:21818)" daemon prio=10 tid=0xce4aa400 nid=0x63a9 runnable [0xcf5d7000..0xcf5d80b0] java.lang. Thread .State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked <0xd6857720> (a sun.nio.ch.Util$1) - locked <0xd6857710> (a java.util.Collections$UnmodifiableSet) - locked <0xd68572a0> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:274) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035) "main-EventThread" daemon prio=10 tid=0xcf4f3c00 nid=0x37f1 waiting on condition [0xcf35c000..0xcf35d030] java.lang. Thread .State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0xd5528060> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:493) "main-SendThread(localhost:21818)" daemon prio=10 tid=0xcf11f000 nid=0x37f0 runnable [0xce9fe000..0xce9fefb0] java.lang. Thread .State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked <0xd5528d48> (a sun.nio.ch.Util$1) - locked <0xd5528d38> (a java.util.Collections$UnmodifiableSet) - locked <0xd55288c8> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:274) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035) "derby.rawStoreDaemon" daemon prio=10 tid=0x08edd800 nid=0x37e6 in Object .wait() [0xcf3ad000..0xcf3ae030] java.lang. Thread .State: TIMED_WAITING (on object monitor) at java.lang. Object .wait(Native Method) - waiting on <0xd50539f0> (a org.apache.derby.impl.services.daemon.BasicDaemon) at org.apache.derby.impl.services.daemon.BasicDaemon. rest (Unknown Source) - locked <0xd50539f0> (a org.apache.derby.impl.services.daemon.BasicDaemon) at org.apache.derby.impl.services.daemon.BasicDaemon.run(Unknown Source) at java.lang. Thread .run( Thread .java:619) "Timer-0" daemon prio=10 tid=0x09070c00 nid=0x37e3 in Object .wait() [0xcf3fe000..0xcf3ff0b0] java.lang. Thread .State: WAITING (on object monitor) at java.lang. Object .wait(Native Method) - waiting on <0xd4e87170> (a java.util.TaskQueue) at java.lang. Object .wait( Object .java:485) at java.util.TimerThread.mainLoop(Timer.java:483) - locked <0xd4e87170> (a java.util.TaskQueue) at java.util.TimerThread.run(Timer.java:462) "derby.antiGC" daemon prio=10 tid=0x08da4800 nid=0x37e2 in Object .wait() [0xcf586000..0xcf586e30] java.lang. Thread .State: WAITING (on object monitor) at java.lang. Object .wait(Native Method) - waiting on <0xd4e8a188> (a org.apache.derby.impl.services.monitor.AntiGC) at java.lang. Object .wait( Object .java:485) at org.apache.derby.impl.services.monitor.AntiGC.run(Unknown Source) - locked <0xd4e8a188> (a org.apache.derby.impl.services.monitor.AntiGC) at java.lang. Thread .run( Thread .java:619) "ProcessThread(sid:0 cport:-1):" prio=10 tid=0xcfed1000 nid=0x37d7 waiting on condition [0xcf679000..0xcf679e30] java.lang. Thread .State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0xd4df4e60> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) at org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:119) "SyncThread:0" prio=10 tid=0xcfed4800 nid=0x37d6 waiting on condition [0xcf6ca000..0xcf6cadb0] java.lang. Thread .State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0xd4df74d0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:97) "SessionTracker" prio=10 tid=0xcfd99800 nid=0x37d5 in Object .wait() [0xcf71b000..0xcf71bf30] java.lang. Thread .State: TIMED_WAITING (on object monitor) at java.lang. Object .wait(Native Method) - waiting on <0xd4df7e80> (a org.apache.zookeeper.server.SessionTrackerImpl) at org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:146) - locked <0xd4df7e80> (a org.apache.zookeeper.server.SessionTrackerImpl) "NIOServerCxn.Factory:0.0.0.0/0.0.0.0:21818" daemon prio=10 tid=0xcfcd9000 nid=0x37d4 runnable [0xcf76c000..0xcf76ceb0] java.lang. Thread .State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked <0xd4d4a100> (a sun.nio.ch.Util$1) - locked <0xd4d4a0f0> (a java.util.Collections$UnmodifiableSet) - locked <0xd4d49ec0> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:194) at java.lang. Thread .run( Thread .java:619) "Low Memory Detector" daemon prio=10 tid=0x0868e400 nid=0x37d0 runnable [0x00000000..0x00000000] java.lang. Thread .State: RUNNABLE "CompilerThread1" daemon prio=10 tid=0x0868cc00 nid=0x37cf waiting on condition [0x00000000..0xd00f05f8] java.lang. Thread .State: RUNNABLE "CompilerThread0" daemon prio=10 tid=0x0868a400 nid=0x37ce waiting on condition [0x00000000..0xd0171578] java.lang. Thread .State: RUNNABLE "Signal Dispatcher" daemon prio=10 tid=0x08689000 nid=0x37cd runnable [0x00000000..0xd01c2b10] java.lang. Thread .State: RUNNABLE "Finalizer" daemon prio=10 tid=0x0866e000 nid=0x37cc in Object .wait() [0xd0413000..0xd0413db0] java.lang. Thread .State: WAITING (on object monitor) at java.lang. Object .wait(Native Method) - waiting on <0xd4dcd590> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116) - locked <0xd4dcd590> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159) "Reference Handler" daemon prio=10 tid=0x0866cc00 nid=0x37cb in Object .wait() [0xd0464000..0xd0464f30] java.lang. Thread .State: WAITING (on object monitor) at java.lang. Object .wait(Native Method) - waiting on <0xd4d54fa8> (a java.lang.ref.Reference$Lock) at java.lang. Object .wait( Object .java:485) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116) - locked <0xd4d54fa8> (a java.lang.ref.Reference$Lock) "main" prio=10 tid=0x08583000 nid=0x37c1 waiting on condition [0xf7ee7000..0xf7ee8208] java.lang. Thread .State: TIMED_WAITING (sleeping) at java.lang. Thread .sleep(Native Method) at org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager.unlockWithRetry(ZooKeeperHiveLockManager.java:415) at org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager.unlock(ZooKeeperHiveLockManager.java:404) at org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager.releaseLocks(ZooKeeperHiveLockManager.java:251) at org.apache.hadoop.hive.ql.Driver.releaseLocks(Driver.java:862) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:948) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:348) at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:774) at org.apache.hadoop.hive.cli.TestCliDriver.runTest(TestCliDriver.java:5609) at org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin5(TestCliDriver.java:1657) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at junit.framework.TestCase.runTest(TestCase.java:168) at junit.framework.TestCase.runBare(TestCase.java:134) at junit.framework.TestResult$1.protect(TestResult.java:110) at junit.framework.TestResult.runProtected(TestResult.java:128) at junit.framework.TestResult.run(TestResult.java:113) at junit.framework.TestCase.run(TestCase.java:124) at junit.framework.TestSuite.runTest(TestSuite.java:232) at junit.framework.TestSuite.run(TestSuite.java:227) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906) "VM Thread " prio=10 tid=0x08669c00 nid=0x37ca runnable "GC task thread#0 (ParallelGC)" prio=10 tid=0x08589c00 nid=0x37c2 runnable "GC task thread#1 (ParallelGC)" prio=10 tid=0x0858b000 nid=0x37c3 runnable "GC task thread#2 (ParallelGC)" prio=10 tid=0x0858c000 nid=0x37c4 runnable "GC task thread#3 (ParallelGC)" prio=10 tid=0x0858d000 nid=0x37c5 runnable "GC task thread#4 (ParallelGC)" prio=10 tid=0x0858e000 nid=0x37c6 runnable "GC task thread#5 (ParallelGC)" prio=10 tid=0x0858f400 nid=0x37c7 runnable "GC task thread#6 (ParallelGC)" prio=10 tid=0x08590400 nid=0x37c8 runnable "GC task thread#7 (ParallelGC)" prio=10 tid=0x08591400 nid=0x37c9 runnable "VM Periodic Task Thread " prio=10 tid=0x0868fc00 nid=0x37d1 waiting on condition JNI global references: 888 Though none of the tests have failed and there is no deadlock in this jstack. But this stack and extra-ordinarily long run time of these tests suggest, something is up here. Are you observing similar behavior in your environment?
        Hide
        Namit Jain added a comment -

        Ashutosh Chauhan, let me take a look ? Thanks

        Show
        Namit Jain added a comment - Ashutosh Chauhan , let me take a look ? Thanks
        Namit Jain made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Namit Jain made changes -
        Attachment hive.3537.10.patch [ 12565339 ]
        Hide
        Namit Jain added a comment -

        Found the issue - there was a issue in the equals method for lock, whereby the lock data was not getting fetched.
        Due to that, although the lock was getting released, but hive thought that the lock was still present, and it was again
        repeatedly trying to release it before giving up.

        Manually verified above tests finish normally - running the whole suite now.

        Show
        Namit Jain added a comment - Found the issue - there was a issue in the equals method for lock, whereby the lock data was not getting fetched. Due to that, although the lock was getting released, but hive thought that the lock was still present, and it was again repeatedly trying to release it before giving up. Manually verified above tests finish normally - running the whole suite now.
        Hide
        Namit Jain added a comment -

        Ashutosh Chauhan, the tests finished in appropriate time.
        Can you take a look ?

        I have updated the phabricator entry and uploaded the new patch.

        Show
        Namit Jain added a comment - Ashutosh Chauhan , the tests finished in appropriate time. Can you take a look ? I have updated the phabricator entry and uploaded the new patch.
        Hide
        Ashutosh Chauhan added a comment -

        Looks good. Running tests. Will commit if tests pass.

        Show
        Ashutosh Chauhan added a comment - Looks good. Running tests. Will commit if tests pass.
        Hide
        Ashutosh Chauhan added a comment -

        Committed to trunk. Thanks, Namit!

        Show
        Ashutosh Chauhan added a comment - Committed to trunk. Thanks, Namit!
        Ashutosh Chauhan made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Fix Version/s 0.11.0 [ 12323587 ]
        Hide
        Hudson added a comment -

        Integrated in hive-trunk-hadoop1 #27 (See https://builds.apache.org/job/hive-trunk-hadoop1/27/)
        HIVE-3537 : release locks at the end of move tasks (Namit via Ashutosh Chauhan) (Revision 1435492)

        Result = FAILURE
        hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1435492
        Files :

        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Context.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/EmbeddedLockManager.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveLockObject.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLock.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
        Show
        Hudson added a comment - Integrated in hive-trunk-hadoop1 #27 (See https://builds.apache.org/job/hive-trunk-hadoop1/27/ ) HIVE-3537 : release locks at the end of move tasks (Namit via Ashutosh Chauhan) (Revision 1435492) Result = FAILURE hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1435492 Files : /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Context.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/EmbeddedLockManager.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveLockObject.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLock.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
        Ashutosh Chauhan made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Hide
        Hudson added a comment -

        Integrated in Hive-trunk-hadoop2 #75 (See https://builds.apache.org/job/Hive-trunk-hadoop2/75/)
        HIVE-3537 : release locks at the end of move tasks (Namit via Ashutosh Chauhan) (Revision 1435492)

        Result = FAILURE
        hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1435492
        Files :

        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Context.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/EmbeddedLockManager.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveLockObject.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLock.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
        Show
        Hudson added a comment - Integrated in Hive-trunk-hadoop2 #75 (See https://builds.apache.org/job/Hive-trunk-hadoop2/75/ ) HIVE-3537 : release locks at the end of move tasks (Namit via Ashutosh Chauhan) (Revision 1435492) Result = FAILURE hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1435492 Files : /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Context.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/EmbeddedLockManager.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveLockObject.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLock.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
        Hide
        Hudson added a comment -

        Integrated in Hive-trunk-h0.21 #1924 (See https://builds.apache.org/job/Hive-trunk-h0.21/1924/)
        HIVE-3537 : release locks at the end of move tasks (Namit via Ashutosh Chauhan) (Revision 1435492)

        Result = FAILURE
        hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1435492
        Files :

        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Context.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/EmbeddedLockManager.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveLockObject.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLock.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
        Show
        Hudson added a comment - Integrated in Hive-trunk-h0.21 #1924 (See https://builds.apache.org/job/Hive-trunk-h0.21/1924/ ) HIVE-3537 : release locks at the end of move tasks (Namit via Ashutosh Chauhan) (Revision 1435492) Result = FAILURE hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1435492 Files : /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Context.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/EmbeddedLockManager.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveLockObject.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLock.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
        Owen O'Malley made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Patch Available Patch Available Open Open
        28d 18h 13m 3 Namit Jain 17/Jan/13 10:26
        Open Open Patch Available Patch Available
        77d 5h 52m 4 Ashutosh Chauhan 19/Jan/13 05:41
        Patch Available Patch Available Resolved Resolved
        12h 27m 1 Ashutosh Chauhan 19/Jan/13 18:09
        Resolved Resolved Closed Closed
        117d 3h 1m 1 Owen O'Malley 16/May/13 22:10

          People

          • Assignee:
            Namit Jain
            Reporter:
            Namit Jain
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development