Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5778

JobSummary does not escape newlines in the job name

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.10, 2.3.0
    • Fix Version/s: 0.23.11, 2.4.0
    • Component/s: jobhistoryserver
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      JobSummary is not escaping newlines in the job name. This can result in a job summary log entry that spans multiple lines when users are expecting one-job-per-line output.

      1. MAPREDUCE-5778.patch
        1.0 kB
        Akira AJISAKA
      2. MAPREDUCE-5778.2.patch
        1 kB
        Akira AJISAKA
      3. MAPREDUCE-5778.3.patch
        4 kB
        Akira AJISAKA
      4. MAPREDUCE-5778.4.patch
        4 kB
        Akira AJISAKA

        Issue Links

          Activity

          Jason Lowe created issue -
          Akira AJISAKA made changes -
          Field Original Value New Value
          Assignee Akira AJISAKA [ ajisakaa ]
          Hide
          Akira AJISAKA added a comment -

          Attaching a patch.

          Show
          Akira AJISAKA added a comment - Attaching a patch.
          Akira AJISAKA made changes -
          Attachment MAPREDUCE-5778.patch [ 12632688 ]
          Akira AJISAKA made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Target Version/s 2.4.0 [ 12326141 ]
          Akira AJISAKA made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Akira AJISAKA made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Tsuyoshi Ozawa added a comment -

          Thank you for taking this JIRA, Akira. Can you attach job summaries before and after your modification?

          Show
          Tsuyoshi Ozawa added a comment - Thank you for taking this JIRA, Akira. Can you attach job summaries before and after your modification?
          Hide
          Akira AJISAKA added a comment -

          Thank you for your comment, Tsuyoshi.
          In fact the modification couldn't escape newlines. I'll attach a new patch and job summaries before and after.

          Show
          Akira AJISAKA added a comment - Thank you for your comment, Tsuyoshi. In fact the modification couldn't escape newlines. I'll attach a new patch and job summaries before and after.
          Tsuyoshi Ozawa made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Akira AJISAKA added a comment -

          Looked around the logs, I found ApplicationSummary does not escape newlines also.
          I filed YARN-1789 to track this.

          Show
          Akira AJISAKA added a comment - Looked around the logs, I found ApplicationSummary does not escape newlines also. I filed YARN-1789 to track this.
          Akira AJISAKA made changes -
          Link This issue is related to YARN-1789 [ YARN-1789 ]
          Akira AJISAKA made changes -
          Attachment MAPREDUCE-5778.2.patch [ 12632991 ]
          Hide
          Akira AJISAKA added a comment -

          Attaching a patch, it escapes newlines as follows:
          before

          2014-03-06 07:25:31,476 INFO org.apache.hadoop.mapreduce.jobhistory.JobSummary: jobId=job_1394057583500_0002,submitTime=1394058253665,launchTime=1394058261607,firstMapTaskLaunchTime=1394058264006,firstReduceTaskLaunchTime=1394058271899,finishTime=1394058277328,resourcesPerMap=512,resourcesPerReduce=512,numMaps=1,numReduces=1,user=root,queue=default,status=SUCCEEDED,mapSlotSeconds=5,reduceSlotSeconds=5,jobName=word
          count
          

          after

          2014-03-06 10:09:15,338 INFO org.apache.hadoop.mapreduce.jobhistory.JobSummary: jobId=job_1394067952388_0001,submitTime=1394067985813,launchTime=1394068001404,firstMapTaskLaunchTime=1394068003984,firstReduceTaskLaunchTime=1394068014719,finishTime=1394068023681,resourcesPerMap=512,resourcesPerReduce=512,numMaps=1,numReduces=1,user=root,queue=default,status=SUCCEEDED,mapSlotSeconds=8,reduceSlotSeconds=8,jobName=word[LF]count
          
          Show
          Akira AJISAKA added a comment - Attaching a patch, it escapes newlines as follows: before 2014-03-06 07:25:31,476 INFO org.apache.hadoop.mapreduce.jobhistory.JobSummary: jobId=job_1394057583500_0002,submitTime=1394058253665,launchTime=1394058261607,firstMapTaskLaunchTime=1394058264006,firstReduceTaskLaunchTime=1394058271899,finishTime=1394058277328,resourcesPerMap=512,resourcesPerReduce=512,numMaps=1,numReduces=1,user=root,queue= default ,status=SUCCEEDED,mapSlotSeconds=5,reduceSlotSeconds=5,jobName=word count after 2014-03-06 10:09:15,338 INFO org.apache.hadoop.mapreduce.jobhistory.JobSummary: jobId=job_1394067952388_0001,submitTime=1394067985813,launchTime=1394068001404,firstMapTaskLaunchTime=1394068003984,firstReduceTaskLaunchTime=1394068014719,finishTime=1394068023681,resourcesPerMap=512,resourcesPerReduce=512,numMaps=1,numReduces=1,user=root,queue= default ,status=SUCCEEDED,mapSlotSeconds=8,reduceSlotSeconds=8,jobName=word[LF]count
          Akira AJISAKA made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Tsuyoshi Ozawa added a comment -

          Looks good. Can you add tests?

          Show
          Tsuyoshi Ozawa added a comment - Looks good. Can you add tests?
          Hide
          Jason Lowe added a comment -

          Personally I'd prefer the original escaping, CR=\r and LF=\n as that's straightforward to translate back to the original job string. [CR] and [LF] are non-standard.

          Show
          Jason Lowe added a comment - Personally I'd prefer the original escaping, CR=\r and LF=\n as that's straightforward to translate back to the original job string. [CR] and [LF] are non-standard.
          Akira AJISAKA made changes -
          Attachment MAPREDUCE-5778.3.patch [ 12633207 ]
          Hide
          Akira AJISAKA added a comment -

          Thanks Tsuyoshi and Jason for the reviews!
          I attached a patch that reflects your comments.

          Show
          Akira AJISAKA added a comment - Thanks Tsuyoshi and Jason for the reviews! I attached a patch that reflects your comments.
          Tsuyoshi Ozawa made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Tsuyoshi Ozawa made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Tsuyoshi Ozawa added a comment -

          LGTM. Waiting for Jenkins report.

          Show
          Tsuyoshi Ozawa added a comment - LGTM. Waiting for Jenkins report.
          Hide
          Tsuyoshi Ozawa added a comment -

          Akira AJISAKA, I noticed one minor point while waiting for Jenkins:

           private static JobSummary summary = new JobSummary();
          

          We should avoid creating static object if we can. Can you fix it? Thanks

          Show
          Tsuyoshi Ozawa added a comment - Akira AJISAKA , I noticed one minor point while waiting for Jenkins: private static JobSummary summary = new JobSummary(); We should avoid creating static object if we can. Can you fix it? Thanks
          Hide
          Akira AJISAKA added a comment -

          Thanks, I fixed it.

          Show
          Akira AJISAKA added a comment - Thanks, I fixed it.
          Akira AJISAKA made changes -
          Attachment MAPREDUCE-5778.4.patch [ 12633825 ]
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12633825/MAPREDUCE-5778.4.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4410//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4410//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12633825/MAPREDUCE-5778.4.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4410//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4410//console This message is automatically generated.
          Hide
          Jason Lowe added a comment -

          +1 lgtm. Committing this.

          Show
          Jason Lowe added a comment - +1 lgtm. Committing this.
          Hide
          Jason Lowe added a comment -

          Thanks, Akira! I committed this to trunk, branch-2, branch-2.4, and branch-0.23.

          Show
          Jason Lowe added a comment - Thanks, Akira! I committed this to trunk, branch-2, branch-2.4, and branch-0.23.
          Jason Lowe made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Fix Version/s 0.23.11 [ 12324663 ]
          Fix Version/s 2.4.0 [ 12326141 ]
          Resolution Fixed [ 1 ]
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #5312 (See https://builds.apache.org/job/Hadoop-trunk-Commit/5312/)
          MAPREDUCE-5778. JobSummary does not escape newlines in the job name. Contributed by Akira AJISAKA (jlowe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1576940)

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobSummary.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestJobSummary.java
          Show
          Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #5312 (See https://builds.apache.org/job/Hadoop-trunk-Commit/5312/ ) MAPREDUCE-5778 . JobSummary does not escape newlines in the job name. Contributed by Akira AJISAKA (jlowe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1576940 ) /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobSummary.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestJobSummary.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #508 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/508/)
          MAPREDUCE-5778. JobSummary does not escape newlines in the job name. Contributed by Akira AJISAKA (jlowe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1576940)

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobSummary.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestJobSummary.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #508 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/508/ ) MAPREDUCE-5778 . JobSummary does not escape newlines in the job name. Contributed by Akira AJISAKA (jlowe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1576940 ) /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobSummary.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestJobSummary.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #1700 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1700/)
          MAPREDUCE-5778. JobSummary does not escape newlines in the job name. Contributed by Akira AJISAKA (jlowe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1576940)

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobSummary.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestJobSummary.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #1700 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1700/ ) MAPREDUCE-5778 . JobSummary does not escape newlines in the job name. Contributed by Akira AJISAKA (jlowe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1576940 ) /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobSummary.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestJobSummary.java
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1725 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1725/)
          MAPREDUCE-5778. JobSummary does not escape newlines in the job name. Contributed by Akira AJISAKA (jlowe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1576940)

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobSummary.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestJobSummary.java
          Show
          Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1725 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1725/ ) MAPREDUCE-5778 . JobSummary does not escape newlines in the job name. Contributed by Akira AJISAKA (jlowe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1576940 ) /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobSummary.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestJobSummary.java
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Patch Available Patch Available Open Open
          5d 20h 9m 3 Tsuyoshi Ozawa 10/Mar/14 20:43
          Open Open Patch Available Patch Available
          4h 5m 4 Tsuyoshi Ozawa 10/Mar/14 20:43
          Patch Available Patch Available Resolved Resolved
          2d 1h 18m 1 Jason Lowe 12/Mar/14 22:01
          Resolved Resolved Closed Closed
          28d 15h 9m 1 Arun C Murthy 10/Apr/14 14:11

            People

            • Assignee:
              Akira AJISAKA
              Reporter:
              Jason Lowe
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development