Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1982

[Rumen] TraceBuilder's output shows jobname as NULL for jobhistory files with valid jobnames

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.22.0
    • Component/s: tools/rumen
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      TraceBuilder fails to extract configuration properties (like job-name) from the job-conf if the job-conf has the properties stored using the deprecated keys.

      1. 1982.v1.patch
        10 kB
        Ravi Gummadi
      2. 1982.v1.1.patch
        11 kB
        Ravi Gummadi

        Activity

        Hide
        Ranjit Mathew added a comment - - edited

        An example fragment is:
        --------------------------------------- 8< ---------------------------------------
        {
        "priority" : "NORMAL",
        "jobID" : "job_201007280625_0001",
        "user" : "gridperf",
        "jobName" : null,
        "mapTasks" : [ {
        "startTime" : 1280298532016,
        "taskID" : "task_201007280625_0001_m_000323",
        "taskType" : "MAP",
        "attempts" : [ {
        "location" :

        { "layers" : [ "98\\.137\\.102\\.64", "node1\\.yahoo\\.com" ] }

        ,
        "hostName" : "/98\\.137\\.102\\.64/node1\\.yahoo
        .com",
        "startTime" : 1280298532028,
        "finishTime" : 1280298597501,
        "result" : "SUCCESS",
        --------------------------------------- 8< ---------------------------------------

        This is because Rumen directly uses Properties instead of Configuration - the
        new key-name is "mapreduce.job.name", while the old key-name was
        "mapred.job.name". Using Configuration insulates against such changes.

        Show
        Ranjit Mathew added a comment - - edited An example fragment is: --------------------------------------- 8< --------------------------------------- { "priority" : "NORMAL", "jobID" : "job_201007280625_0001", "user" : "gridperf", "jobName" : null, "mapTasks" : [ { "startTime" : 1280298532016, "taskID" : "task_201007280625_0001_m_000323", "taskType" : "MAP", "attempts" : [ { "location" : { "layers" : [ "98\\.137\\.102\\.64", "node1\\.yahoo\\.com" ] } , "hostName" : "/98\\.137\\.102\\.64/node1\\.yahoo .com", "startTime" : 1280298532028, "finishTime" : 1280298597501, "result" : "SUCCESS", --------------------------------------- 8< --------------------------------------- This is because Rumen directly uses Properties instead of Configuration - the new key-name is "mapreduce.job.name", while the old key-name was "mapred.job.name". Using Configuration insulates against such changes.
        Hide
        Amar Kamat added a comment -

        Also TraceBuilder extracts various job configuration properties like

        1. job-name
        2. queue-names
        3. mapred-child task jvm settings etc

        from job's conf. So this bug is seen only when there is a jobconf present in the jobhistory folder.

        Show
        Amar Kamat added a comment - Also TraceBuilder extracts various job configuration properties like job-name queue-names mapred-child task jvm settings etc from job's conf. So this bug is seen only when there is a jobconf present in the jobhistory folder.
        Hide
        Ravi Gummadi added a comment -

        Attaching patch fixing the issue. Added testcase that fails without the fix and passes with the fix.

        Show
        Ravi Gummadi added a comment - Attaching patch fixing the issue. Added testcase that fails without the fix and passes with the fix.
        Hide
        Ravi Gummadi added a comment -

        Attaching new patch with minor changes to the comments and assertion failure messages in testcases.

        Show
        Ravi Gummadi added a comment - Attaching new patch with minor changes to the comments and assertion failure messages in testcases.
        Hide
        Ravi Gummadi added a comment -

        All unit tests passed on my local machine.

        ant test-patch gave:

        [exec] +1 overall.
        [exec]
        [exec] +1 @author. The patch does not contain any @author tags.
        [exec]
        [exec] +1 tests included. The patch appears to include 8 new or modified tests.
        [exec]
        [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
        [exec]
        [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
        [exec]
        [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
        [exec]
        [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.

        Show
        Ravi Gummadi added a comment - All unit tests passed on my local machine. ant test-patch gave: [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 8 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
        Hide
        Amar Kamat added a comment -

        The latest patch looks good.

        Show
        Amar Kamat added a comment - The latest patch looks good.
        Hide
        Amareshwari Sriramadasu added a comment -

        I just committed this. Thanks Ravi !

        In Long term, I think Rumen should be changed to use Configuration directly instead of parsing conf from Properies.

        Show
        Amareshwari Sriramadasu added a comment - I just committed this. Thanks Ravi ! In Long term, I think Rumen should be changed to use Configuration directly instead of parsing conf from Properies.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #523 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/523/)

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #523 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/523/ )

          People

          • Assignee:
            Ravi Gummadi
            Reporter:
            Amar Kamat
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development