Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6616

Fail to create jobhistory file if there are some multibyte characters in the job name

    Details

    • Hadoop Flags:
      Reviewed

      Description

      When creating jobhistory file, job name is trimmed within 50 characters by default, and the name is URL-encoded after the job name is trimmed. Therefore, if there are some multibyte characters in the job name, the encoded job name can be longer than 50 characters. Eventually it can break the limit of the file name (Usually 255 characters).

      1. MAPREDUCE-6616-test.patch
        2 kB
        Akira Ajisaka
      2. MAPREDUCE-6616.0.patch
        21 kB
        Kousuke Saruta
      3. MAPREDUCE-6616.1.patch
        23 kB
        Kousuke Saruta

        Issue Links

          Activity

          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #9207 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9207/)
          MAPREDUCE-6616. Fail to create jobhistory file if there are some (aajisaka: rev df99ea8a92d600e669606d41d3887bd004e7a3cc)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/FileNameIndexUtils.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/jobhistory/TestFileNameIndexUtils.java
          • hadoop-mapreduce-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9207 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9207/ ) MAPREDUCE-6616 . Fail to create jobhistory file if there are some (aajisaka: rev df99ea8a92d600e669606d41d3887bd004e7a3cc) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/FileNameIndexUtils.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/jobhistory/TestFileNameIndexUtils.java hadoop-mapreduce-project/CHANGES.txt
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Committed this to trunk, branch-2, and branch-2.8. Thanks Kousuke Saruta for the contribution!

          Show
          ajisakaa Akira Ajisaka added a comment - Committed this to trunk, branch-2, and branch-2.8. Thanks Kousuke Saruta for the contribution!
          Hide
          ajisakaa Akira Ajisaka added a comment -

          +1, committing this.

          Show
          ajisakaa Akira Ajisaka added a comment - +1, committing this.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 10m 53s trunk passed
          +1 compile 0m 36s trunk passed with JDK v1.8.0_66
          +1 compile 0m 31s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 21s trunk passed
          +1 mvnsite 0m 37s trunk passed
          +1 mvneclipse 0m 20s trunk passed
          +1 findbugs 1m 19s trunk passed
          +1 javadoc 0m 41s trunk passed with JDK v1.8.0_66
          +1 javadoc 0m 41s trunk passed with JDK v1.7.0_91
          +1 mvninstall 0m 32s the patch passed
          +1 compile 0m 33s the patch passed with JDK v1.8.0_66
          +1 javac 0m 33s the patch passed
          +1 compile 0m 28s the patch passed with JDK v1.7.0_91
          +1 javac 0m 28s the patch passed
          +1 checkstyle 0m 17s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common: patch generated 0 new + 3 unchanged - 12 fixed = 3 total (was 15)
          +1 mvnsite 0m 34s the patch passed
          +1 mvneclipse 0m 15s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 1m 36s the patch passed
          +1 javadoc 0m 37s the patch passed with JDK v1.8.0_66
          +1 javadoc 0m 37s the patch passed with JDK v1.7.0_91
          +1 unit 1m 11s hadoop-mapreduce-client-common in the patch passed with JDK v1.8.0_66.
          +1 unit 1m 5s hadoop-mapreduce-client-common in the patch passed with JDK v1.7.0_91.
          +1 asflicense 0m 27s Patch does not generate ASF License warnings.
          25m 55s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12784934/MAPREDUCE-6616.1.patch
          JIRA Issue MAPREDUCE-6616
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 8b0cda89aa40 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 86560a4
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6293/testReport/
          modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common
          Max memory used 76MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6293/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 10m 53s trunk passed +1 compile 0m 36s trunk passed with JDK v1.8.0_66 +1 compile 0m 31s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 21s trunk passed +1 mvnsite 0m 37s trunk passed +1 mvneclipse 0m 20s trunk passed +1 findbugs 1m 19s trunk passed +1 javadoc 0m 41s trunk passed with JDK v1.8.0_66 +1 javadoc 0m 41s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 32s the patch passed +1 compile 0m 33s the patch passed with JDK v1.8.0_66 +1 javac 0m 33s the patch passed +1 compile 0m 28s the patch passed with JDK v1.7.0_91 +1 javac 0m 28s the patch passed +1 checkstyle 0m 17s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common: patch generated 0 new + 3 unchanged - 12 fixed = 3 total (was 15) +1 mvnsite 0m 34s the patch passed +1 mvneclipse 0m 15s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 36s the patch passed +1 javadoc 0m 37s the patch passed with JDK v1.8.0_66 +1 javadoc 0m 37s the patch passed with JDK v1.7.0_91 +1 unit 1m 11s hadoop-mapreduce-client-common in the patch passed with JDK v1.8.0_66. +1 unit 1m 5s hadoop-mapreduce-client-common in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 27s Patch does not generate ASF License warnings. 25m 55s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12784934/MAPREDUCE-6616.1.patch JIRA Issue MAPREDUCE-6616 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 8b0cda89aa40 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 86560a4 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6293/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6293/console This message was automatically generated.
          Hide
          sarutak Kousuke Saruta added a comment -

          Fixed style issues and brushed up.

          Show
          sarutak Kousuke Saruta added a comment - Fixed style issues and brushed up.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Thanks Kousuke Saruta for creating the patch. Mostly looks good to me. Minor nits:
          1.

          // amount size of first 5 characters < 23
          // 23 < amount size of first 6 characters
          

          I'm thinking we should comment why the size is equal to 22, so the above comment can be rewritten as follows:

          // total size of the first 5 characters = 22
          // 23 < total size of the first 6 characters
          

          2.

          import  static java.nio.charset.StandardCharsets.UTF_8;
          

          Would you remove unnecessary whitespace between "import" and "static"?
          3.

               //NumReduces
          -    sb.append(indexInfo.getNumReduces());
          +    sb.append(encodeJobHistoryFileName(
          +        String.valueOf(indexInfo.getNumReduces())));
               sb.append(DELIMITER);
          -    
          +
               //JobStatus
          -    sb.append(indexInfo.getJobStatus());
          +    sb.append(encodeJobHistoryFileName(indexInfo.getJobStatus()));
               sb.append(DELIMITER);
          -    
          +
               //QueueName
          -    sb.append(escapeDelimiters(getQueueName(indexInfo)));
          +    sb.append(escapeDelimiters(encodeJobHistoryFileName(
          +            getQueueName(indexInfo))));
          

          The patch includes mixed continuation indents (4 spaces and 8 spaces). Would you make the indents consistent?

          Show
          ajisakaa Akira Ajisaka added a comment - Thanks Kousuke Saruta for creating the patch. Mostly looks good to me. Minor nits: 1. // amount size of first 5 characters < 23 // 23 < amount size of first 6 characters I'm thinking we should comment why the size is equal to 22, so the above comment can be rewritten as follows: // total size of the first 5 characters = 22 // 23 < total size of the first 6 characters 2. import static java.nio.charset.StandardCharsets.UTF_8; Would you remove unnecessary whitespace between "import" and "static"? 3. //NumReduces - sb.append(indexInfo.getNumReduces()); + sb.append(encodeJobHistoryFileName( + String .valueOf(indexInfo.getNumReduces()))); sb.append(DELIMITER); - + //JobStatus - sb.append(indexInfo.getJobStatus()); + sb.append(encodeJobHistoryFileName(indexInfo.getJobStatus())); sb.append(DELIMITER); - + //QueueName - sb.append(escapeDelimiters(getQueueName(indexInfo))); + sb.append(escapeDelimiters(encodeJobHistoryFileName( + getQueueName(indexInfo)))); The patch includes mixed continuation indents (4 spaces and 8 spaces). Would you make the indents consistent?
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 34s trunk passed
          +1 compile 0m 17s trunk passed with JDK v1.8.0_66
          +1 compile 0m 20s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 14s trunk passed
          +1 mvnsite 0m 26s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 0m 51s trunk passed
          +1 javadoc 0m 20s trunk passed with JDK v1.8.0_66
          +1 javadoc 0m 26s trunk passed with JDK v1.7.0_91
          +1 mvninstall 0m 21s the patch passed
          +1 compile 0m 14s the patch passed with JDK v1.8.0_66
          +1 javac 0m 14s the patch passed
          +1 compile 0m 17s the patch passed with JDK v1.7.0_91
          +1 javac 0m 17s the patch passed
          +1 checkstyle 0m 12s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common: patch generated 0 new + 4 unchanged - 11 fixed = 4 total (was 15)
          +1 mvnsite 0m 23s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 1m 2s the patch passed
          +1 javadoc 0m 17s the patch passed with JDK v1.8.0_66
          +1 javadoc 0m 24s the patch passed with JDK v1.7.0_91
          +1 unit 0m 37s hadoop-mapreduce-client-common in the patch passed with JDK v1.8.0_66.
          +1 unit 0m 44s hadoop-mapreduce-client-common in the patch passed with JDK v1.7.0_91.
          +1 asflicense 0m 18s Patch does not generate ASF License warnings.
          16m 47s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12784406/MAPREDUCE-6616.0.patch
          JIRA Issue MAPREDUCE-6616
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux a484468e967f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / cf8af7b
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6286/testReport/
          modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common
          Max memory used 77MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6286/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 34s trunk passed +1 compile 0m 17s trunk passed with JDK v1.8.0_66 +1 compile 0m 20s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 14s trunk passed +1 mvnsite 0m 26s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 0m 51s trunk passed +1 javadoc 0m 20s trunk passed with JDK v1.8.0_66 +1 javadoc 0m 26s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 21s the patch passed +1 compile 0m 14s the patch passed with JDK v1.8.0_66 +1 javac 0m 14s the patch passed +1 compile 0m 17s the patch passed with JDK v1.7.0_91 +1 javac 0m 17s the patch passed +1 checkstyle 0m 12s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common: patch generated 0 new + 4 unchanged - 11 fixed = 4 total (was 15) +1 mvnsite 0m 23s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 2s the patch passed +1 javadoc 0m 17s the patch passed with JDK v1.8.0_66 +1 javadoc 0m 24s the patch passed with JDK v1.7.0_91 +1 unit 0m 37s hadoop-mapreduce-client-common in the patch passed with JDK v1.8.0_66. +1 unit 0m 44s hadoop-mapreduce-client-common in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 18s Patch does not generate ASF License warnings. 16m 47s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12784406/MAPREDUCE-6616.0.patch JIRA Issue MAPREDUCE-6616 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux a484468e967f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / cf8af7b Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6286/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common Max memory used 77MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6286/console This message was automatically generated.
          Hide
          sarutak Kousuke Saruta added a comment -

          I've added the initial patch.

          Show
          sarutak Kousuke Saruta added a comment - I've added the initial patch.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Attaching a patch to reproduce this issue.

          Show
          ajisakaa Akira Ajisaka added a comment - Attaching a patch to reproduce this issue.

            People

            • Assignee:
              sarutak Kousuke Saruta
              Reporter:
              ajisakaa Akira Ajisaka
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development