Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1982

[Rumen] TraceBuilder's output shows jobname as NULL for jobhistory files with valid jobnames

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.22.0
    • tools/rumen
    • None
    • Reviewed

    Description

      TraceBuilder fails to extract configuration properties (like job-name) from the job-conf if the job-conf has the properties stored using the deprecated keys.

      Attachments

        1. 1982.v1.1.patch
          11 kB
          Ravi Gummadi
        2. 1982.v1.patch
          10 kB
          Ravi Gummadi

        Activity

          ranjit Ranjit Mathew added a comment - - edited

          An example fragment is:
          --------------------------------------- 8< ---------------------------------------
          {
          "priority" : "NORMAL",
          "jobID" : "job_201007280625_0001",
          "user" : "gridperf",
          "jobName" : null,
          "mapTasks" : [ {
          "startTime" : 1280298532016,
          "taskID" : "task_201007280625_0001_m_000323",
          "taskType" : "MAP",
          "attempts" : [ {
          "location" :

          { "layers" : [ "98\\.137\\.102\\.64", "node1\\.yahoo\\.com" ] }

          ,
          "hostName" : "/98\\.137\\.102\\.64/node1\\.yahoo
          .com",
          "startTime" : 1280298532028,
          "finishTime" : 1280298597501,
          "result" : "SUCCESS",
          --------------------------------------- 8< ---------------------------------------

          This is because Rumen directly uses Properties instead of Configuration - the
          new key-name is "mapreduce.job.name", while the old key-name was
          "mapred.job.name". Using Configuration insulates against such changes.

          ranjit Ranjit Mathew added a comment - - edited An example fragment is: --------------------------------------- 8< --------------------------------------- { "priority" : "NORMAL", "jobID" : "job_201007280625_0001", "user" : "gridperf", "jobName" : null, "mapTasks" : [ { "startTime" : 1280298532016, "taskID" : "task_201007280625_0001_m_000323", "taskType" : "MAP", "attempts" : [ { "location" : { "layers" : [ "98\\.137\\.102\\.64", "node1\\.yahoo\\.com" ] } , "hostName" : "/98\\.137\\.102\\.64/node1\\.yahoo .com", "startTime" : 1280298532028, "finishTime" : 1280298597501, "result" : "SUCCESS", --------------------------------------- 8< --------------------------------------- This is because Rumen directly uses Properties instead of Configuration - the new key-name is "mapreduce.job.name", while the old key-name was "mapred.job.name". Using Configuration insulates against such changes.
          amar_kamat Amar Kamat added a comment -

          Also TraceBuilder extracts various job configuration properties like

          1. job-name
          2. queue-names
          3. mapred-child task jvm settings etc

          from job's conf. So this bug is seen only when there is a jobconf present in the jobhistory folder.

          amar_kamat Amar Kamat added a comment - Also TraceBuilder extracts various job configuration properties like job-name queue-names mapred-child task jvm settings etc from job's conf. So this bug is seen only when there is a jobconf present in the jobhistory folder.
          ravidotg Ravi Gummadi added a comment -

          Attaching patch fixing the issue. Added testcase that fails without the fix and passes with the fix.

          ravidotg Ravi Gummadi added a comment - Attaching patch fixing the issue. Added testcase that fails without the fix and passes with the fix.
          ravidotg Ravi Gummadi added a comment -

          Attaching new patch with minor changes to the comments and assertion failure messages in testcases.

          ravidotg Ravi Gummadi added a comment - Attaching new patch with minor changes to the comments and assertion failure messages in testcases.
          ravidotg Ravi Gummadi added a comment -

          All unit tests passed on my local machine.

          ant test-patch gave:

          [exec] +1 overall.
          [exec]
          [exec] +1 @author. The patch does not contain any @author tags.
          [exec]
          [exec] +1 tests included. The patch appears to include 8 new or modified tests.
          [exec]
          [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
          [exec]
          [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
          [exec]
          [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
          [exec]
          [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.

          ravidotg Ravi Gummadi added a comment - All unit tests passed on my local machine. ant test-patch gave: [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 8 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
          amar_kamat Amar Kamat added a comment -

          The latest patch looks good.

          amar_kamat Amar Kamat added a comment - The latest patch looks good.

          I just committed this. Thanks Ravi !

          In Long term, I think Rumen should be changed to use Configuration directly instead of parsing conf from Properies.

          amareshwari Amareshwari Sriramadasu added a comment - I just committed this. Thanks Ravi ! In Long term, I think Rumen should be changed to use Configuration directly instead of parsing conf from Properies.
          hudson Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #523 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/523/)

          hudson Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #523 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/523/ )

          People

            ravidotg Ravi Gummadi
            amar_kamat Amar Kamat
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: