Hadoop Common
  1. Hadoop Common
  2. HADOOP-649

Jobs without any map and reduce operations seems to be lost after their execution

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.6.1
    • Fix Version/s: 0.12.0
    • Component/s: None
    • Labels:
      None

      Description

      When a job does not provide any file splits (because there is no data present), the job successes but then it is lost by the job tracker. Is this normal ?

      1. HADOOP-649-0.11.2.patch
        2 kB
        Thomas Friol
      2. HADOOP-649.patch
        0.8 kB
        Thomas Friol
      3. HADOOP-649.patch
        0.8 kB
        Thomas Friol

        Activity

        Hide
        Thomas Friol added a comment -

        After a long investigation, i finally found why these kind of jobs where retired by the jobtracker. This is because in this case, job finished time is never setted.
        So here is a patch to correct this problem.
        It could be great if my patch is correct to integrate it in the next 0.9.0 release if possible.
        Thanks.

        Show
        Thomas Friol added a comment - After a long investigation, i finally found why these kind of jobs where retired by the jobtracker. This is because in this case, job finished time is never setted. So here is a patch to correct this problem. It could be great if my patch is correct to integrate it in the next 0.9.0 release if possible. Thanks.
        Hide
        Thomas Friol added a comment -

        If my patch is correct and commited (I have tested it), I think that HADOOP-648 issue can also be resolved in the same time as this one because in fact, it is the same issue.

        Show
        Thomas Friol added a comment - If my patch is correct and commited (I have tested it), I think that HADOOP-648 issue can also be resolved in the same time as this one because in fact, it is the same issue.
        Hide
        Thomas Friol added a comment -

        The previous patch is not really good. In fact, the finished time must be setted before the JobStatus in order not to be retire by the job tracker.
        Here is new and valid patch.

        Show
        Thomas Friol added a comment - The previous patch is not really good. In fact, the finished time must be setted before the JobStatus in order not to be retire by the job tracker. Here is new and valid patch.
        Hide
        Thomas Friol added a comment -

        Has somenone looked to this patch ?
        I think it is quiet important to fix this issue since some NPEs are regularly thrown by the JobInProgress class.
        This patch has been tested on the 0.9.0 version and works perfectly.
        Thanks for attention.

        Show
        Thomas Friol added a comment - Has somenone looked to this patch ? I think it is quiet important to fix this issue since some NPEs are regularly thrown by the JobInProgress class. This patch has been tested on the 0.9.0 version and works perfectly. Thanks for attention.
        Hide
        Sylvain Wallez added a comment -

        Can someone have a look at this one-liner? Thanks

        Show
        Sylvain Wallez added a comment - Can someone have a look at this one-liner? Thanks
        Hide
        Owen O'Malley added a comment -

        When a patch is ready to be looked at you need to push the "submit patch" button to make it's state "patch available." We use the the patch available state to find the patches to look at. This looks like a good change assuming it still applies to head.

        +1

        Show
        Owen O'Malley added a comment - When a patch is ready to be looked at you need to push the "submit patch" button to make it's state "patch available." We use the the patch available state to find the patches to look at. This looks like a good change assuming it still applies to head. +1
        Hide
        Thomas Friol added a comment -

        Sorry. Didn't know about this practice. Will take care of that next time.
        Thanks for your attention.

        Show
        Thomas Friol added a comment - Sorry. Didn't know about this practice. Will take care of that next time. Thanks for your attention.
        Hide
        Thomas Friol added a comment -

        Patch available

        Show
        Thomas Friol added a comment - Patch available
        Hide
        Hadoop QA added a comment -

        +1, because http://issues.apache.org/jira/secure/attachment/12346228/HADOOP-649.patch applied and successfully tested against trunk revision r506778.

        Show
        Hadoop QA added a comment - +1, because http://issues.apache.org/jira/secure/attachment/12346228/HADOOP-649.patch applied and successfully tested against trunk revision r506778.
        Hide
        Doug Cutting added a comment -

        This looks good to me. It would be best to have a unit test, to make sure that it really fixes the issue and that we don't break it again in the future. Thomas, are you willing to write one?

        Also, please don't include the bug number in the patch itself. We don't want the code to get cluttered with comments documenting its history: we leave history to subversion.

        Thanks!

        Show
        Doug Cutting added a comment - This looks good to me. It would be best to have a unit test, to make sure that it really fixes the issue and that we don't break it again in the future. Thomas, are you willing to write one? Also, please don't include the bug number in the patch itself. We don't want the code to get cluttered with comments documenting its history: we leave history to subversion. Thanks!
        Hide
        Doug Cutting added a comment -

        > It would be best to have a unit test

        Probably this can be done by simply adding a new test or slightly modifying an existing unit test, rather than constructing an entirely new test for this bug.

        Show
        Doug Cutting added a comment - > It would be best to have a unit test Probably this can be done by simply adding a new test or slightly modifying an existing unit test, rather than constructing an entirely new test for this bug.
        Hide
        Thomas Friol added a comment -

        I will add a unit test and remove my comment from the previous patch.

        Show
        Thomas Friol added a comment - I will add a unit test and remove my comment from the previous patch.
        Hide
        Thomas Friol added a comment -

        Here is the new patch including the unit test.

        Show
        Thomas Friol added a comment - Here is the new patch including the unit test.
        Hide
        Hadoop QA added a comment -

        +1, because http://issues.apache.org/jira/secure/attachment/12351118/HADOOP-649-0.11.2.patch applied and successfully tested against trunk revision r507276.

        Show
        Hadoop QA added a comment - +1, because http://issues.apache.org/jira/secure/attachment/12351118/HADOOP-649-0.11.2.patch applied and successfully tested against trunk revision r507276.
        Hide
        Doug Cutting added a comment -

        I just committed this. Thanks, Thomas!

        Show
        Doug Cutting added a comment - I just committed this. Thanks, Thomas!

          People

          • Assignee:
            Owen O'Malley
            Reporter:
            Thomas Friol
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development