Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.5.4
    • Component/s: None
    • Labels:
      None
    • Target Version/s:

      Description

      TestAMRecovery fails sometimes on testVertexPartiallyFinished_XXX.
      The scenario is that we'd like kill AM when vertex is partially finished ( with 2 tasks, task_0 is finished and task_1 is running). When in recovery, task_0 should not rerun and task_1 should rerun. ( We use the recovery log(TaskAttemptFinishedEvent) to judge whether task is rerun)
      Currently, using VertexManager.onSourceTaskCompleted to control when to kill AM, but it is not perfect. VertexManager.onSourceTaskCompleted is not invoked at the moment task attempt is finished ( TaskAttempt send event to Task to tell TaskAttempt is finsihed, and then Task send event to Vertex to trigger VM.onSourceTaskCompleted)
      The following case is possible: task_0 finished -> task_1 finished -> VM.onSourceTaskCompleted -> VM.onSourceTaskCompleted
      In this case, we will take it as partially completed in the first VM.onSourceTaskCompleted, but actually the vertex is fully completed.

      1. TEZ-1642.patch
        8 kB
        Jeff Zhang
      2. TEZ-1642-2.patch
        8 kB
        Jeff Zhang
      3. TEZ-1642-3.patch
        23 kB
        Jeff Zhang
      4. TEZ-1642-4.patch
        37 kB
        Jeff Zhang
      5. TEZ-1642-5.patch
        17 kB
        Jeff Zhang

        Activity

        Hide
        zjffdu Jeff Zhang added a comment -

        Attach the patch.

        Hitesh Shah, please help review it
        Reproduce the error, looks like resource allocation delay is one of the reason.
        2 main changes:

        • Change the AM RM heartbeat internal to 100
        • Change the MiniTezCluster's NodeManager number to 2

        Although using Thread.sleep to control whether vertex is partially completed or fully completed is not perfect, run the test 2 hours continually, no fail happen. Still keep it running, will update it tomorrow.

        BTW, another way is to kill the AM in state machine transition callback, will try that later.

        Show
        zjffdu Jeff Zhang added a comment - Attach the patch. Hitesh Shah , please help review it Reproduce the error, looks like resource allocation delay is one of the reason. 2 main changes: Change the AM RM heartbeat internal to 100 Change the MiniTezCluster's NodeManager number to 2 Although using Thread.sleep to control whether vertex is partially completed or fully completed is not perfect, run the test 2 hours continually, no fail happen. Still keep it running, will update it tomorrow. BTW, another way is to kill the AM in state machine transition callback, will try that later.
        Hide
        hitesh Hitesh Shah added a comment -

        Patch looks fine for the most part.

        How is printHistoryEvents() meant to be help someone trying to debug a test failure? It has no information on the first and last events and also there is no additional log saying parsing events from log1 or log2. Also, it might be helpful to log a statement at the start of each test case so that one can easily figure out which events were generated as part of which test case.

        Show
        hitesh Hitesh Shah added a comment - Patch looks fine for the most part. How is printHistoryEvents() meant to be help someone trying to debug a test failure? It has no information on the first and last events and also there is no additional log saying parsing events from log1 or log2. Also, it might be helpful to log a statement at the start of each test case so that one can easily figure out which events were generated as part of which test case.
        Hide
        hitesh Hitesh Shah added a comment -

        using Thread.sleep to control whether vertex is partially completed or fully completed

        This can be controlled by using a custom vertex manager which schedules its own tasks and use the state change notifier ( for task completions ) to decide when to kill the AM.

        Show
        hitesh Hitesh Shah added a comment - using Thread.sleep to control whether vertex is partially completed or fully completed This can be controlled by using a custom vertex manager which schedules its own tasks and use the state change notifier ( for task completions ) to decide when to kill the AM.
        Hide
        zjffdu Jeff Zhang added a comment -

        Run the test for the whole night. Unfortunately it fails this morning, but the fail reason seems weird, it complains the DAG failed, but after I check the app logs and recovery logs, both show dag is succeeded, not sure whether this is bug on client api or bug in AM.

        testVertexCompletelyFinished_ScatterGather(org.apache.tez.test.TestAMRecovery) 
        java.lang.AssertionError: expected:<SUCCEEDED> but was:<FAILED>
        	at org.junit.Assert.fail(Assert.java:88)
        	at org.junit.Assert.failNotEquals(Assert.java:743)
        	at org.junit.Assert.assertEquals(Assert.java:118)
        	at org.junit.Assert.assertEquals(Assert.java:144)
        	at org.apache.tez.test.TestAMRecovery.runDAGAndVerify(TestAMRecovery.java:427)
        	at org.apache.tez.test.TestAMRecovery.testVertexCompletelyFinished_ScatterGather(TestAMRecovery.java:381)
        
        Show
        zjffdu Jeff Zhang added a comment - Run the test for the whole night. Unfortunately it fails this morning, but the fail reason seems weird, it complains the DAG failed, but after I check the app logs and recovery logs, both show dag is succeeded, not sure whether this is bug on client api or bug in AM. testVertexCompletelyFinished_ScatterGather(org.apache.tez.test.TestAMRecovery) java.lang.AssertionError: expected:<SUCCEEDED> but was:<FAILED> at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:743) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:144) at org.apache.tez.test.TestAMRecovery.runDAGAndVerify(TestAMRecovery.java:427) at org.apache.tez.test.TestAMRecovery.testVertexCompletelyFinished_ScatterGather(TestAMRecovery.java:381)
        Hide
        zjffdu Jeff Zhang added a comment -

        This can be controlled by using a custom vertex manager which schedules its own tasks and use the state change notifier ( for task completions ) to decide when to kill the AM.

        Yes, Currently I use custom vertex manager, and kill AM in VertexManager.onSourceTaskCompleted

        Show
        zjffdu Jeff Zhang added a comment - This can be controlled by using a custom vertex manager which schedules its own tasks and use the state change notifier ( for task completions ) to decide when to kill the AM. Yes, Currently I use custom vertex manager, and kill AM in VertexManager.onSourceTaskCompleted
        Hide
        hitesh Hitesh Shah added a comment -

        What was the original failure being addressed?

        Show
        hitesh Hitesh Shah added a comment - What was the original failure being addressed?
        Hide
        zjffdu Jeff Zhang added a comment -

        The original failure is that AM is killed when vertex is completely finished in test that AM should been killed when vertex is partially finished.
        The original failure looks like been addressed, no such failure happen in the 7 hours' testing.

        Show
        zjffdu Jeff Zhang added a comment - The original failure is that AM is killed when vertex is completely finished in test that AM should been killed when vertex is partially finished. The original failure looks like been addressed, no such failure happen in the 7 hours' testing.
        Hide
        hitesh Hitesh Shah added a comment -

        +1 after the logging comment above is addresed.

        Show
        hitesh Hitesh Shah added a comment - +1 after the logging comment above is addresed.
        Hide
        zjffdu Jeff Zhang added a comment - - edited

        Additional log is added.

        Committed to both master & branch-0.5

        Show
        zjffdu Jeff Zhang added a comment - - edited Additional log is added. Committed to both master & branch-0.5
        Hide
        hitesh Hitesh Shah added a comment -

        Jeff Zhang Which patch was committed? There does not seem to be an updated patch to this jira.

        Show
        hitesh Hitesh Shah added a comment - Jeff Zhang Which patch was committed? There does not seem to be an updated patch to this jira.
        Hide
        zjffdu Jeff Zhang added a comment -

        Hitesh Shah, attach the new patch. But it fails again in my local box. Need to continue check it.

        Show
        zjffdu Jeff Zhang added a comment - Hitesh Shah , attach the new patch. But it fails again in my local box. Need to continue check it.
        Hide
        hitesh Hitesh Shah added a comment -

        Jeff Zhang Please revert the commit in that case.

        Show
        hitesh Hitesh Shah added a comment - Jeff Zhang Please revert the commit in that case.
        Hide
        zjffdu Jeff Zhang added a comment -

        commit reverted

        Show
        zjffdu Jeff Zhang added a comment - commit reverted
        Hide
        zjffdu Jeff Zhang added a comment -

        Hitesh Shah, I find another issue that it is not guaranteed that task_0 is scheduled before task_1.
        The most difficult case is the vertex partially completed, for vertex_1, try to kill AM when task_0 is completed, and task_1 is still running. By using vertexmanager's onSourceTaskCompleted is not perfect.
        The following case is possible: task_0 finished -> task_1 finished -> trigger VM.onSourceTaskCompleted -> trigger VM.onSourceTaskCompleted
        In this case, we will take it as partially completed in the first VM.onSourceTaskCompleted, but actually the vertex is fully completed.

        I am trying to add custom callback on TaskImpl's state machine transition, it should be able to avoid the case above, will post the update later.

        Show
        zjffdu Jeff Zhang added a comment - Hitesh Shah , I find another issue that it is not guaranteed that task_0 is scheduled before task_1. The most difficult case is the vertex partially completed, for vertex_1, try to kill AM when task_0 is completed, and task_1 is still running. By using vertexmanager's onSourceTaskCompleted is not perfect. The following case is possible: task_0 finished -> task_1 finished -> trigger VM.onSourceTaskCompleted -> trigger VM.onSourceTaskCompleted In this case, we will take it as partially completed in the first VM.onSourceTaskCompleted, but actually the vertex is fully completed. I am trying to add custom callback on TaskImpl's state machine transition, it should be able to avoid the case above, will post the update later.
        Hide
        zjffdu Jeff Zhang added a comment -

        Hitesh Shah Attach new patch.

        • Using the callback of TaskImpl's state machine to control when to kill AM rather than using custom VertexManager
        • This method is a little aggressive by adding other custom code in AM.
        • run it for the last 8 hours, no failure. Still keep it running.
        Show
        zjffdu Jeff Zhang added a comment - Hitesh Shah Attach new patch. Using the callback of TaskImpl's state machine to control when to kill AM rather than using custom VertexManager This method is a little aggressive by adding other custom code in AM. run it for the last 8 hours, no failure. Still keep it running.
        Hide
        hitesh Hitesh Shah added a comment -

        This is probably a bad idea. A better approach may to be use the notifier APIs introduced where a VM can subscribe to certain events or potentially support a onTaskCompleted() api on the VM plugin.

        Please update the jira description with the actual issue so that we can look at potential options.

        Show
        hitesh Hitesh Shah added a comment - This is probably a bad idea. A better approach may to be use the notifier APIs introduced where a VM can subscribe to certain events or potentially support a onTaskCompleted() api on the VM plugin. Please update the jira description with the actual issue so that we can look at potential options.
        Hide
        hitesh Hitesh Shah added a comment -

        Can testVertexPartiallyFinished_XXX be achieved by only scheduling 1 task, waiting a certain amount of time ( or launching a thread to poll the task state ) and then halting the jvm? On the second attempt, the VM should schedule both tasks unlike attempt 1 where only 1 task is scheduled?

        Show
        hitesh Hitesh Shah added a comment - Can testVertexPartiallyFinished_XXX be achieved by only scheduling 1 task, waiting a certain amount of time ( or launching a thread to poll the task state ) and then halting the jvm? On the second attempt, the VM should schedule both tasks unlike attempt 1 where only 1 task is scheduled?
        Hide
        zjffdu Jeff Zhang added a comment -

        Can testVertexPartiallyFinished_XXX be achieved by only scheduling 1 task, waiting a certain amount of time ( or launching a thread to poll the task state ) and then halting the jvm? On the second attempt, the VM should schedule both tasks unlike attempt 1 where only 1 task is scheduled?

        Sounds a good idea, will try that.

        Show
        zjffdu Jeff Zhang added a comment - Can testVertexPartiallyFinished_XXX be achieved by only scheduling 1 task, waiting a certain amount of time ( or launching a thread to poll the task state ) and then halting the jvm? On the second attempt, the VM should schedule both tasks unlike attempt 1 where only 1 task is scheduled? Sounds a good idea, will try that.
        Hide
        hitesh Hitesh Shah added a comment - - edited

        Also, it looks for a DAG with V0 -> V1, the VM of V1 is being used to stop the AM when task 0 of V0 completes. This is definitely a very flaky test. The better approach would be have the VM of V0 control the scheduling and kill the AM as needed.

        Show
        hitesh Hitesh Shah added a comment - - edited Also, it looks for a DAG with V0 -> V1, the VM of V1 is being used to stop the AM when task 0 of V0 completes. This is definitely a very flaky test. The better approach would be have the VM of V0 control the scheduling and kill the AM as needed.
        Hide
        zjffdu Jeff Zhang added a comment -

        Can testVertexPartiallyFinished_XXX be achieved by only scheduling 1 task, waiting a certain amount of time ( or launching a thread to poll the task state ) and then halting the jvm? On the second attempt, the VM should schedule both tasks unlike attempt 1 where only 1 task is scheduled?

        Looks like this won't work. The key thing is to keep task_0 finished and task_1 is still running. Scheduling 1 task only can make task_0 finished but can not make task_1 running.

        Also, it looks for a DAG with V0 -> V1, the VM of V1 is being used to stop the AM when task 0 of V0 completes. This is definitely a very flaky test. The better approacj would be have to VM of V0 control the scheduling and kill the AM as needed.

        But currently VM don't have much context about the its tasks' state, unless it can get notification of its tasks' state change

        Show
        zjffdu Jeff Zhang added a comment - Can testVertexPartiallyFinished_XXX be achieved by only scheduling 1 task, waiting a certain amount of time ( or launching a thread to poll the task state ) and then halting the jvm? On the second attempt, the VM should schedule both tasks unlike attempt 1 where only 1 task is scheduled? Looks like this won't work. The key thing is to keep task_0 finished and task_1 is still running. Scheduling 1 task only can make task_0 finished but can not make task_1 running. Also, it looks for a DAG with V0 -> V1, the VM of V1 is being used to stop the AM when task 0 of V0 completes. This is definitely a very flaky test. The better approacj would be have to VM of V0 control the scheduling and kill the AM as needed. But currently VM don't have much context about the its tasks' state, unless it can get notification of its tasks' state change
        Hide
        zjffdu Jeff Zhang added a comment -

        It would be better to allow VM get the notification of Task & TaskAttempt ( by callback of state machine rather than send event), so that we can have more accurate control on when to kill AM.
        And I think this would be useful for system test of recovery, we can design more complicated scenario when we have this feature. Although unit test can cover some logic of recovery, system test is more important to verify the recovery function.
        My initial thought is that we can implement this by enhance org.apache.tez.dag.app.dag.StateChangeNotifier

        Show
        zjffdu Jeff Zhang added a comment - It would be better to allow VM get the notification of Task & TaskAttempt ( by callback of state machine rather than send event), so that we can have more accurate control on when to kill AM. And I think this would be useful for system test of recovery, we can design more complicated scenario when we have this feature. Although unit test can cover some logic of recovery, system test is more important to verify the recovery function. My initial thought is that we can implement this by enhance org.apache.tez.dag.app.dag.StateChangeNotifier
        Hide
        zjffdu Jeff Zhang added a comment - - edited

        Attach a new patch, the main change in the patch is add task/taskattempt state notification in VertexManager Hitesh Shah Siddharth Seth Bikas Saha Please help review the new patch.

        • Currently, only put task/taskattempt state notification in VertexManagerPluginContextImpl, didn't expose it in user-facing API, this should has the least impact on the existing code. If in future it is necessary, we can move it to user-facing API. The notification happens just after the state change by using the callback of StateMachine transition rather than using send event asynchronously, this allow us has more accurate control on the running status.
        • Currently, only listen the task/taskattempt succeeded notification, In future it can extend to other state change notification.
        • Changes on the TestAMRecovery
          • Remove the verification of TOTAL_LAUNCHED_TASKS, because VM of v1 can only gurattne it is partially completed or fully completed when killed, but can not make sure the status of v2 ( using sleep is not perfect solution)
          • Only verify one task is completed in the testcase of XXXPartiallyCompeted rather than assume it is task_0, because it is not guaranteed that task_0 is scheduled before task_1
        Show
        zjffdu Jeff Zhang added a comment - - edited Attach a new patch, the main change in the patch is add task/taskattempt state notification in VertexManager Hitesh Shah Siddharth Seth Bikas Saha Please help review the new patch. Currently, only put task/taskattempt state notification in VertexManagerPluginContextImpl, didn't expose it in user-facing API, this should has the least impact on the existing code. If in future it is necessary, we can move it to user-facing API. The notification happens just after the state change by using the callback of StateMachine transition rather than using send event asynchronously, this allow us has more accurate control on the running status. Currently, only listen the task/taskattempt succeeded notification, In future it can extend to other state change notification. Changes on the TestAMRecovery Remove the verification of TOTAL_LAUNCHED_TASKS, because VM of v1 can only gurattne it is partially completed or fully completed when killed, but can not make sure the status of v2 ( using sleep is not perfect solution) Only verify one task is completed in the testcase of XXXPartiallyCompeted rather than assume it is task_0, because it is not guaranteed that task_0 is scheduled before task_1
        Hide
        bikassaha Bikas Saha added a comment -

        The state change notifier change seems a bit heavy handed to make this test pass. Also, the test is casting out the impl class of vertexmanager to access its non-API methods. This is not good as people tend to look at test code to see how API's are being used.
        Is the intent to fail the vertex after 1 task is finished but not the second? This can be achieved by only controlling when these tasks get scheduled. We can stop deriving from the input ready vertex manager and control the scheduling in a custom manner. It can schedule task 1 but not task 2. Then wait for task 1 to finish. Then die. This way the vertex is partially finished. Then recovered AM schedules both tasks.

        Does that work?

        Show
        bikassaha Bikas Saha added a comment - The state change notifier change seems a bit heavy handed to make this test pass. Also, the test is casting out the impl class of vertexmanager to access its non-API methods. This is not good as people tend to look at test code to see how API's are being used. Is the intent to fail the vertex after 1 task is finished but not the second? This can be achieved by only controlling when these tasks get scheduled. We can stop deriving from the input ready vertex manager and control the scheduling in a custom manner. It can schedule task 1 but not task 2. Then wait for task 1 to finish. Then die. This way the vertex is partially finished. Then recovered AM schedules both tasks. Does that work?
        Hide
        zjffdu Jeff Zhang added a comment -

        Is the intent to fail the vertex after 1 task is finished but not the second? This can be achieved by only controlling when these tasks get scheduled. We can stop deriving from the input ready vertex manager and control the scheduling in a custom manner. It can schedule task 1 but not task 2. Then wait for task 1 to finish. Then die. This way the vertex is partially finished. Then recovered AM schedules both tasks.

        It can only guarantee that taks_1 is done, but can not guarantee that task_2 is running, The purpose of this test case is to make task_1 done while task_2 is running, and verify that task_1 is not re-run but task_2 is rerun when recover.
        The purpose I make VM get notification from the task/taskattempt completed is that only in the main AsyncDispatcher the status of DAG/Vertex/Task/TaskAttempt is determined (state change will block on the main AsyncDispatcher).
        Or I can loose the test case scenario requirement, only guarantee that task_1 is done but not guarantee the state of task_2

        Show
        zjffdu Jeff Zhang added a comment - Is the intent to fail the vertex after 1 task is finished but not the second? This can be achieved by only controlling when these tasks get scheduled. We can stop deriving from the input ready vertex manager and control the scheduling in a custom manner. It can schedule task 1 but not task 2. Then wait for task 1 to finish. Then die. This way the vertex is partially finished. Then recovered AM schedules both tasks. It can only guarantee that taks_1 is done, but can not guarantee that task_2 is running, The purpose of this test case is to make task_1 done while task_2 is running, and verify that task_1 is not re-run but task_2 is rerun when recover. The purpose I make VM get notification from the task/taskattempt completed is that only in the main AsyncDispatcher the status of DAG/Vertex/Task/TaskAttempt is determined (state change will block on the main AsyncDispatcher). Or I can loose the test case scenario requirement, only guarantee that task_1 is done but not guarantee the state of task_2
        Hide
        bikassaha Bikas Saha added a comment -

        How crucial is it for task 2 to be running vs being incomplete ? The test still needs to check that task 1 is not re-executed but task 2 is executed. So IMO, task 2 not scheduled should be fine for the purpose of this test. Having task 2 as running would definitely matter when we start supporting re-attaching to running tasks after AM restart but we are not there now. IMO, we should do the simpler thing for now.

        Show
        bikassaha Bikas Saha added a comment - How crucial is it for task 2 to be running vs being incomplete ? The test still needs to check that task 1 is not re-executed but task 2 is executed. So IMO, task 2 not scheduled should be fine for the purpose of this test. Having task 2 as running would definitely matter when we start supporting re-attaching to running tasks after AM restart but we are not there now. IMO, we should do the simpler thing for now.
        Hide
        zjffdu Jeff Zhang added a comment -

        Attach a new patch, Bikas Saha Hitesh Shah Please help review.

        • Lose the constraint of the test scenario, Use Bikas Saha's suggestion to only keep task_1 finished while task_2 is not started.
        • Use a custom VM of V1 to control whether schedule only one task when it is partially finished test case, and kill AM in the VM of V2 when it get the notification of TaskAttempt of V1 is completed. Remove the verification of TOTAL_LAUNCHED_TASKS, and NUM_KILLED_TASKS, because it is not guaranteed that whether tasks of V2 is running when AM is killed.
        Show
        zjffdu Jeff Zhang added a comment - Attach a new patch, Bikas Saha Hitesh Shah Please help review. Lose the constraint of the test scenario, Use Bikas Saha 's suggestion to only keep task_1 finished while task_2 is not started. Use a custom VM of V1 to control whether schedule only one task when it is partially finished test case, and kill AM in the VM of V2 when it get the notification of TaskAttempt of V1 is completed. Remove the verification of TOTAL_LAUNCHED_TASKS, and NUM_KILLED_TASKS, because it is not guaranteed that whether tasks of V2 is running when AM is killed.
        Hide
        bikassaha Bikas Saha added a comment -

        Approach in the VM looks fine to me. Though I am not sure about the other changes in the existing test code that were caused due to this change. Maybe Hitesh Shah should sign off for this.

        Show
        bikassaha Bikas Saha added a comment - Approach in the VM looks fine to me. Though I am not sure about the other changes in the existing test code that were caused due to this change. Maybe Hitesh Shah should sign off for this.
        Hide
        hitesh Hitesh Shah added a comment -

        +1

        Show
        hitesh Hitesh Shah added a comment - +1
        Hide
        zjffdu Jeff Zhang added a comment -

        Thanks for review. Hitesh Shah, Bikas Saha Committed to master

        Show
        zjffdu Jeff Zhang added a comment - Thanks for review. Hitesh Shah , Bikas Saha Committed to master
        Hide
        zjffdu Jeff Zhang added a comment -

        backport it to branch-0.5
        commit c033965e7ae582c91884c86e0bc5edadb75c68ea
        Author: Jeff Zhang <zjffdu@apache.org>
        Date: Fri Dec 19 10:43:04 2014 +0800

        TEZ-1642. TestAMRecovery sometimes fail (zjffdu)

        (cherry picked from commit 1dd725f7976de91a93f7cf25fb922278c4993af8)

        Conflicts:
        CHANGES.txt

        Show
        zjffdu Jeff Zhang added a comment - backport it to branch-0.5 commit c033965e7ae582c91884c86e0bc5edadb75c68ea Author: Jeff Zhang <zjffdu@apache.org> Date: Fri Dec 19 10:43:04 2014 +0800 TEZ-1642 . TestAMRecovery sometimes fail (zjffdu) (cherry picked from commit 1dd725f7976de91a93f7cf25fb922278c4993af8) Conflicts: CHANGES.txt
        Hide
        zjffdu Jeff Zhang added a comment - - edited

        Update master's CHANGES.txt : Move TEZ-1642, TEZ-1934 to 0.5.4

        commit 758d5a6b9441fcd92e37fae5885e793ece8f0457 (HEAD, master)
        Author: Jeff Zhang <zjffdu@apache.org>
        Date: Thu Jan 22 10:09:29 2015 +0800

        Update CHNAGES.txt: Move TEZ-1642, TEZ-1934 to 0.5.4 (zjffdu)

        Show
        zjffdu Jeff Zhang added a comment - - edited Update master's CHANGES.txt : Move TEZ-1642 , TEZ-1934 to 0.5.4 commit 758d5a6b9441fcd92e37fae5885e793ece8f0457 (HEAD, master) Author: Jeff Zhang <zjffdu@apache.org> Date: Thu Jan 22 10:09:29 2015 +0800 Update CHNAGES.txt: Move TEZ-1642 , TEZ-1934 to 0.5.4 (zjffdu)
        Hide
        hitesh Hitesh Shah added a comment -

        Can you also please update CHANGES.txt in branch-0.6?

        Show
        hitesh Hitesh Shah added a comment - Can you also please update CHANGES.txt in branch-0.6?
        Hide
        zjffdu Jeff Zhang added a comment -

        Hitesh Shah Update CHANGES.txt in branch-0.6.

        commit c206223647de57abc1cf3a69c5b0e38a6a634b19 (HEAD, origin/branch-0.6, branch-0.6)
        Author: Jeff Zhang <zjffdu@apache.org>
        Date: Thu Jan 22 11:32:27 2015 +0800

        Update CHANGES.txt: Move TEZ-1642,TEZ-1934 to 0.5.4 (zjffdu)

        Show
        zjffdu Jeff Zhang added a comment - Hitesh Shah Update CHANGES.txt in branch-0.6. commit c206223647de57abc1cf3a69c5b0e38a6a634b19 (HEAD, origin/branch-0.6, branch-0.6) Author: Jeff Zhang <zjffdu@apache.org> Date: Thu Jan 22 11:32:27 2015 +0800 Update CHANGES.txt: Move TEZ-1642 , TEZ-1934 to 0.5.4 (zjffdu)
        Hide
        hitesh Hitesh Shah added a comment -

        Closing issue as 0.5.4, 0.6.1 and 0.7.0 have been released.

        Show
        hitesh Hitesh Shah added a comment - Closing issue as 0.5.4, 0.6.1 and 0.7.0 have been released.

          People

          • Assignee:
            zjffdu Jeff Zhang
            Reporter:
            zjffdu Jeff Zhang
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development