Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-15850

CopyCommitter#concatFileChunks should check that the blocks per chunk is not 0

Details

    Description

      I was investigating test failure of TestIncrementalBackupWithBulkLoad from hbase against hadoop 3.1.1

      hbase MapReduceBackupCopyJob$BackupDistCp would create listing file:

              LOG.debug("creating input listing " + listing + " , totalRecords=" + totalRecords);
              cfg.set(DistCpConstants.CONF_LABEL_LISTING_FILE_PATH, listing);
              cfg.setLong(DistCpConstants.CONF_LABEL_TOTAL_NUMBER_OF_RECORDS, totalRecords);
      

      For the test case, two bulk loaded hfiles are in the listing:

      2018-10-13 14:09:24,123 DEBUG [Time-limited test] mapreduce.MapReduceBackupCopyJob$BackupDistCp(195): BackupDistCp : hdfs://localhost:42796/user/hbase/test-data/160aeab5-6bca-9f87-465e-2517a0c43119/data/default/test-1539439707496/96b5a3613d52f4df1ba87a1cef20684c/f/394e6d39a9b94b148b9089c4fb967aad_SeqId_205_
      2018-10-13 14:09:24,125 DEBUG [Time-limited test] mapreduce.MapReduceBackupCopyJob$BackupDistCp(195): BackupDistCp : hdfs://localhost:42796/user/hbase/test-data/160aeab5-6bca-9f87-465e-2517a0c43119/data/default/test-1539439707496/96b5a3613d52f4df1ba87a1cef20684c/f/a7599081e835440eb7bf0dd3ef4fd7a5_SeqId_205_
      2018-10-13 14:09:24,125 DEBUG [Time-limited test] mapreduce.MapReduceBackupCopyJob$BackupDistCp(197): BackupDistCp execute for 2 files of 10242
      

      Later on, CopyCommitter#concatFileChunks would throw the following exception:

      2018-10-13 14:09:25,351 WARN  [Thread-936] mapred.LocalJobRunner$Job(590): job_local1795473782_0004
      java.io.IOException: Inconsistent sequence file: current chunk file org.apache.hadoop.tools.CopyListingFileStatus@bb8826ee{hdfs://localhost:42796/user/hbase/test-data/       160aeab5-6bca-9f87-465e-2517a0c43119/data/default/test-1539439707496/96b5a3613d52f4df1ba87a1cef20684c/f/a7599081e835440eb7bf0dd3ef4fd7a5_SeqId_205_ length = 5100 aclEntries  = null, xAttrs = null} doesnt match prior entry org.apache.hadoop.tools.CopyListingFileStatus@243d544d{hdfs://localhost:42796/user/hbase/test-data/160aeab5-6bca-9f87-465e-   2517a0c43119/data/default/test-1539439707496/96b5a3613d52f4df1ba87a1cef20684c/f/394e6d39a9b94b148b9089c4fb967aad_SeqId_205_ length = 5142 aclEntries = null, xAttrs = null}
        at org.apache.hadoop.tools.mapred.CopyCommitter.concatFileChunks(CopyCommitter.java:276)
        at org.apache.hadoop.tools.mapred.CopyCommitter.commitJob(CopyCommitter.java:100)
        at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:567)
      

      The above warning shouldn't happen - the two bulk loaded hfiles are independent.

      From the contents of the two CopyListingFileStatus instances, we can see that their isSplit() return false. Otherwise the following from toString should be logged:

          if (isSplit()) {
            sb.append(", chunkOffset = ").append(this.getChunkOffset());
            sb.append(", chunkLength = ").append(this.getChunkLength());
          }
      

      From hbase side, we can specify one bulk loaded hfile per job but that defeats the purpose of using DistCp.

      Attachments

        1. testIncrementalBackupWithBulkLoad-output.txt
          1.09 MB
          Ted Yu
        2. HADOOP-15850.v6.patch
          2 kB
          Ted Yu
        3. HADOOP-15850.v5.patch
          2 kB
          Ted Yu
        4. HADOOP-15850.v4.patch
          1 kB
          Ted Yu
        5. HADOOP-15850.v3.patch
          1 kB
          Ted Yu
        6. HADOOP-15850.v2.patch
          1 kB
          Ted Yu
        7. HADOOP-15850.branch-3.0.patch
          2 kB
          Wei-Chiu Chuang

        Issue Links

          Activity

            yuzhihong@gmail.com Ted Yu added a comment - This is hbase code: https://github.com/apache/hbase/blob/master/hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/mapreduce/MapReduceBackupCopyJob.java#L153
            yuzhihong@gmail.com Ted Yu added a comment -

            The quoted test output was from testIncrementalBackupWithBulkLoad-output.txt

            yuzhihong@gmail.com Ted Yu added a comment - The quoted test output was from testIncrementalBackupWithBulkLoad-output.txt
            yuzhihong@gmail.com Ted Yu added a comment - - edited

            yzhangal:
            When you have chance, can you take a look ?
            Maybe I missed some existing DistCp functionality.

            Thanks

            yuzhihong@gmail.com Ted Yu added a comment - - edited yzhangal : When you have chance, can you take a look ? Maybe I missed some existing DistCp functionality. Thanks
            yuzhihong@gmail.com Ted Yu added a comment -

            CopyCommitter#concatFileChunks is private.
            It is not straight forward to override the method from DistCp user POV.

            yuzhihong@gmail.com Ted Yu added a comment - CopyCommitter#concatFileChunks is private. It is not straight forward to override the method from DistCp user POV.
            yuzhihong@gmail.com Ted Yu added a comment - - edited

            I wonder if the check for mismatching FileStatus should be refined this way:

            diff --git a/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyCommitter.java b/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/ma
            index 07eacb0..6177454 100644
            --- a/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyCommitter.java
            +++ b/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyCommitter.java
            @@ -266,9 +266,9 @@ private void concatFileChunks(Configuration conf) throws IOException {
                         // Two neighboring chunks have to be consecutive ones for the same
                         // file, for them to be merged
                         if (!srcFileStatus.getPath().equals(lastFileStatus.getPath()) ||
            -                srcFileStatus.getChunkOffset() !=
            +                lastFileStatus.isSplit() && (srcFileStatus.getChunkOffset() !=
                             (lastFileStatus.getChunkOffset() +
            -                lastFileStatus.getChunkLength())) {
            +                lastFileStatus.getChunkLength()))) {
                           String emsg = "Inconsistent sequence file: current " +
                               "chunk file " + srcFileStatus + " doesnt match prior " +
                               "entry " + lastFileStatus;
            

            The additional clause checks that lastFileStatus represents a split.

            stevel@apache.org yzhangal weichiu
            What do you think ?

            yuzhihong@gmail.com Ted Yu added a comment - - edited I wonder if the check for mismatching FileStatus should be refined this way: diff --git a/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyCommitter.java b/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/ma index 07eacb0..6177454 100644 --- a/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyCommitter.java +++ b/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyCommitter.java @@ -266,9 +266,9 @@ private void concatFileChunks(Configuration conf) throws IOException { // Two neighboring chunks have to be consecutive ones for the same // file, for them to be merged if (!srcFileStatus.getPath().equals(lastFileStatus.getPath()) || - srcFileStatus.getChunkOffset() != + lastFileStatus.isSplit() && (srcFileStatus.getChunkOffset() != (lastFileStatus.getChunkOffset() + - lastFileStatus.getChunkLength())) { + lastFileStatus.getChunkLength()))) { String emsg = "Inconsistent sequence file: current " + "chunk file " + srcFileStatus + " doesnt match prior " + "entry " + lastFileStatus; The additional clause checks that lastFileStatus represents a split. stevel@apache.org yzhangal weichiu What do you think ?
            weichiu Wei-Chiu Chuang added a comment - - edited

            Ted, if I understand it correctly (I have limited experience with distcp and hbase backup)
            You're hitting a bug hidden in HADOOP-11794. You don't seem to be using -blocksperchunk feature, but the implementation thinks you do.

            For the context, HADOOP-11794 allows distcp to copy a source file to multiple chunked intermediate files, and then "stitch" them together to the final file. This is a useful feature for storage systems that have long latency but high throughput.

            Is it reproducible for HBase on Hadoop 3.1.1?

            weichiu Wei-Chiu Chuang added a comment - - edited Ted, if I understand it correctly (I have limited experience with distcp and hbase backup) You're hitting a bug hidden in HADOOP-11794 . You don't seem to be using -blocksperchunk feature, but the implementation thinks you do. For the context, HADOOP-11794 allows distcp to copy a source file to multiple chunked intermediate files, and then "stitch" them together to the final file. This is a useful feature for storage systems that have long latency but high throughput. Is it reproducible for HBase on Hadoop 3.1.1?
            yuzhihong@gmail.com Ted Yu added a comment -

            Running the backup test against hadoop 3.0.x / 3.1.y , this is easily reproducible.

            I was aware of HADOOP-11794 and wondering why the per chunk feature kicks in.

            yuzhihong@gmail.com Ted Yu added a comment - Running the backup test against hadoop 3.0.x / 3.1.y , this is easily reproducible. I was aware of HADOOP-11794 and wondering why the per chunk feature kicks in.
            yuzhihong@gmail.com Ted Yu added a comment - - edited

            weichiu:
            See the link to MapReduceBackupCopyJob.java in my first comment .
            We invoke DistCp programmatically.

            yuzhihong@gmail.com Ted Yu added a comment - - edited weichiu : See the link to MapReduceBackupCopyJob.java in my first comment . We invoke DistCp programmatically.
            yuzhihong@gmail.com Ted Yu added a comment -

            The DistCpOptions instance for the DistCp session is not passed to CopyCommitter.
            If we have per chunk information, concatFileChunks call should depend on its value.

            yuzhihong@gmail.com Ted Yu added a comment - The DistCpOptions instance for the DistCp session is not passed to CopyCommitter. If we have per chunk information, concatFileChunks call should depend on its value.
            yuzhihong@gmail.com Ted Yu added a comment -

            I tried to add '-blocksperchunk 0' option when invoking DistCp:

            2018-10-17 02:33:53,708 DEBUG [Time-limited test] mapreduce.MapReduceBackupCopyJob(416): New DistCp options: [-async, -blocksperchunk, 0, hdfs://localhost:34344/user/hbase/test-data/78931012-3303-fc71-e289-5a9726f1bfcc/data/default/test-1539743586635/2e17accd93f78be97c0f585e68f283d6/f/46480cbed054406c9ef52ff123729938_SeqId_205_, hdfs://localhost:34344/user/hbase/test-data/78931012-3303-fc71-e289-5a9726f1bfcc/data/default/test-1539743586635/2e17accd93f78be97c0f585e68f283d6/f/7e3cc96eb3f7447cb4f925df947d1fa3_SeqId_205_, hdfs://localhost:34344/backupUT/backup_1539743624592]
            

            Still encountered 'Inconsistent sequence file' error.

            yuzhihong@gmail.com Ted Yu added a comment - I tried to add '-blocksperchunk 0' option when invoking DistCp: 2018-10-17 02:33:53,708 DEBUG [Time-limited test] mapreduce.MapReduceBackupCopyJob(416): New DistCp options: [-async, -blocksperchunk, 0, hdfs: //localhost:34344/user/hbase/test-data/78931012-3303-fc71-e289-5a9726f1bfcc/data/ default /test-1539743586635/2e17accd93f78be97c0f585e68f283d6/f/46480cbed054406c9ef52ff123729938_SeqId_205_, hdfs://localhost:34344/user/hbase/test-data/78931012-3303-fc71-e289-5a9726f1bfcc/data/ default /test-1539743586635/2e17accd93f78be97c0f585e68f283d6/f/7e3cc96eb3f7447cb4f925df947d1fa3_SeqId_205_, hdfs://localhost:34344/backupUT/backup_1539743624592] Still encountered 'Inconsistent sequence file' error.
            stevel@apache.org Steve Loughran added a comment -

            Distcp code is pretty complex these days, I wouldn't be surprised if problems like this exist

            stevel@apache.org Steve Loughran added a comment - Distcp code is pretty complex these days, I wouldn't be surprised if problems like this exist
            yuzhihong@gmail.com Ted Yu added a comment - - edited

            In order to retrieve the per chunk information in CopyCommitter ctor,
            how about using the config key DistCpOptionSwitch.BLOCKS_PER_CHUNK.getConfigLabel() ?
            CopyCommitter can selectively skip concatenation when the value for the config is 0.

            yuzhihong@gmail.com Ted Yu added a comment - - edited In order to retrieve the per chunk information in CopyCommitter ctor, how about using the config key DistCpOptionSwitch.BLOCKS_PER_CHUNK.getConfigLabel() ? CopyCommitter can selectively skip concatenation when the value for the config is 0.
            yuzhihong@gmail.com Ted Yu added a comment -

            I tested the patch v2 in two ways:

            • when there is no "-blocksperchunk" option specified, TestIncrementalBackupWithBulkLoad passes
            • when positive value for "-blocksperchunk" option is specified, concatFileChunks is called - resulting in previously reported error.
            yuzhihong@gmail.com Ted Yu added a comment - I tested the patch v2 in two ways: when there is no "-blocksperchunk" option specified, TestIncrementalBackupWithBulkLoad passes when positive value for "-blocksperchunk" option is specified, concatFileChunks is called - resulting in previously reported error.
            hadoopqa Hadoop QA added a comment -
            -1 overall



            Vote Subsystem Runtime Comment
            0 reexec 0m 21s Docker mode activated.
                  Prechecks
            +1 @author 0m 0s The patch does not contain any @author tags.
            -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
                  trunk Compile Tests
            +1 mvninstall 19m 8s trunk passed
            +1 compile 0m 25s trunk passed
            +1 checkstyle 0m 18s trunk passed
            +1 mvnsite 0m 29s trunk passed
            +1 shadedclient 11m 59s branch has no errors when building and testing our client artifacts.
            +1 findbugs 0m 33s trunk passed
            +1 javadoc 0m 20s trunk passed
                  Patch Compile Tests
            +1 mvninstall 0m 27s the patch passed
            +1 compile 0m 22s the patch passed
            +1 javac 0m 22s the patch passed
            -0 checkstyle 0m 14s hadoop-tools/hadoop-distcp: The patch generated 2 new + 29 unchanged - 0 fixed = 31 total (was 29)
            +1 mvnsite 0m 24s the patch passed
            +1 whitespace 0m 0s The patch has no whitespace issues.
            +1 shadedclient 12m 45s patch has no errors when building and testing our client artifacts.
            +1 findbugs 0m 38s the patch passed
            +1 javadoc 0m 18s the patch passed
                  Other Tests
            +1 unit 11m 41s hadoop-distcp in the patch passed.
            +1 asflicense 0m 26s The patch does not generate ASF License warnings.
            61m 8s



            Subsystem Report/Notes
            Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1
            JIRA Issue HADOOP-15850
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12944356/HADOOP-15850.v2.patch
            Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
            uname Linux 1837f39890d6 3.13.0-144-generic #193-Ubuntu SMP Thu Mar 15 17:03:53 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality /testptch/patchprocess/precommit/personality/provided.sh
            git revision trunk / d54f559
            maven version: Apache Maven 3.3.9
            Default Java 1.8.0_181
            findbugs v3.1.0-RC1
            checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/15383/artifact/out/diff-checkstyle-hadoop-tools_hadoop-distcp.txt
            Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/15383/testReport/
            Max. process+thread count 339 (vs. ulimit of 10000)
            modules C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp
            Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/15383/console
            Powered by Apache Yetus 0.8.0 http://yetus.apache.org

            This message was automatically generated.

            hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 21s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.       trunk Compile Tests +1 mvninstall 19m 8s trunk passed +1 compile 0m 25s trunk passed +1 checkstyle 0m 18s trunk passed +1 mvnsite 0m 29s trunk passed +1 shadedclient 11m 59s branch has no errors when building and testing our client artifacts. +1 findbugs 0m 33s trunk passed +1 javadoc 0m 20s trunk passed       Patch Compile Tests +1 mvninstall 0m 27s the patch passed +1 compile 0m 22s the patch passed +1 javac 0m 22s the patch passed -0 checkstyle 0m 14s hadoop-tools/hadoop-distcp: The patch generated 2 new + 29 unchanged - 0 fixed = 31 total (was 29) +1 mvnsite 0m 24s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 shadedclient 12m 45s patch has no errors when building and testing our client artifacts. +1 findbugs 0m 38s the patch passed +1 javadoc 0m 18s the patch passed       Other Tests +1 unit 11m 41s hadoop-distcp in the patch passed. +1 asflicense 0m 26s The patch does not generate ASF License warnings. 61m 8s Subsystem Report/Notes Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 JIRA Issue HADOOP-15850 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12944356/HADOOP-15850.v2.patch Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle uname Linux 1837f39890d6 3.13.0-144-generic #193-Ubuntu SMP Thu Mar 15 17:03:53 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/patchprocess/precommit/personality/provided.sh git revision trunk / d54f559 maven version: Apache Maven 3.3.9 Default Java 1.8.0_181 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/15383/artifact/out/diff-checkstyle-hadoop-tools_hadoop-distcp.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/15383/testReport/ Max. process+thread count 339 (vs. ulimit of 10000) modules C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/15383/console Powered by Apache Yetus 0.8.0 http://yetus.apache.org This message was automatically generated.
            yuzhihong@gmail.com Ted Yu added a comment - - edited

            stevel@apache.org yzhangal weichiu
            Mind taking a look at patch v3 ?

            Thanks

            yuzhihong@gmail.com Ted Yu added a comment - - edited stevel@apache.org yzhangal weichiu Mind taking a look at patch v3 ? Thanks
            hadoopqa Hadoop QA added a comment -
            -1 overall



            Vote Subsystem Runtime Comment
            0 reexec 0m 20s Docker mode activated.
                  Prechecks
            +1 @author 0m 0s The patch does not contain any @author tags.
            -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
                  trunk Compile Tests
            +1 mvninstall 19m 11s trunk passed
            +1 compile 0m 27s trunk passed
            +1 checkstyle 0m 19s trunk passed
            +1 mvnsite 0m 29s trunk passed
            +1 shadedclient 13m 20s branch has no errors when building and testing our client artifacts.
            +1 findbugs 0m 45s trunk passed
            +1 javadoc 0m 24s trunk passed
                  Patch Compile Tests
            +1 mvninstall 0m 27s the patch passed
            +1 compile 0m 22s the patch passed
            +1 javac 0m 22s the patch passed
            -0 checkstyle 0m 14s hadoop-tools/hadoop-distcp: The patch generated 1 new + 29 unchanged - 0 fixed = 30 total (was 29)
            +1 mvnsite 0m 24s the patch passed
            +1 whitespace 0m 0s The patch has no whitespace issues.
            +1 shadedclient 12m 41s patch has no errors when building and testing our client artifacts.
            +1 findbugs 0m 37s the patch passed
            +1 javadoc 0m 19s the patch passed
                  Other Tests
            +1 unit 13m 3s hadoop-distcp in the patch passed.
            +1 asflicense 0m 24s The patch does not generate ASF License warnings.
            64m 12s



            Subsystem Report/Notes
            Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1
            JIRA Issue HADOOP-15850
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12944366/HADOOP-15850.v3.patch
            Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
            uname Linux 00891bf59a6b 3.13.0-144-generic #193-Ubuntu SMP Thu Mar 15 17:03:53 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality /testptch/patchprocess/precommit/personality/provided.sh
            git revision trunk / d54f559
            maven version: Apache Maven 3.3.9
            Default Java 1.8.0_181
            findbugs v3.1.0-RC1
            checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/15384/artifact/out/diff-checkstyle-hadoop-tools_hadoop-distcp.txt
            Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/15384/testReport/
            Max. process+thread count 317 (vs. ulimit of 10000)
            modules C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp
            Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/15384/console
            Powered by Apache Yetus 0.8.0 http://yetus.apache.org

            This message was automatically generated.

            hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 20s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.       trunk Compile Tests +1 mvninstall 19m 11s trunk passed +1 compile 0m 27s trunk passed +1 checkstyle 0m 19s trunk passed +1 mvnsite 0m 29s trunk passed +1 shadedclient 13m 20s branch has no errors when building and testing our client artifacts. +1 findbugs 0m 45s trunk passed +1 javadoc 0m 24s trunk passed       Patch Compile Tests +1 mvninstall 0m 27s the patch passed +1 compile 0m 22s the patch passed +1 javac 0m 22s the patch passed -0 checkstyle 0m 14s hadoop-tools/hadoop-distcp: The patch generated 1 new + 29 unchanged - 0 fixed = 30 total (was 29) +1 mvnsite 0m 24s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 shadedclient 12m 41s patch has no errors when building and testing our client artifacts. +1 findbugs 0m 37s the patch passed +1 javadoc 0m 19s the patch passed       Other Tests +1 unit 13m 3s hadoop-distcp in the patch passed. +1 asflicense 0m 24s The patch does not generate ASF License warnings. 64m 12s Subsystem Report/Notes Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 JIRA Issue HADOOP-15850 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12944366/HADOOP-15850.v3.patch Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle uname Linux 00891bf59a6b 3.13.0-144-generic #193-Ubuntu SMP Thu Mar 15 17:03:53 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/patchprocess/precommit/personality/provided.sh git revision trunk / d54f559 maven version: Apache Maven 3.3.9 Default Java 1.8.0_181 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/15384/artifact/out/diff-checkstyle-hadoop-tools_hadoop-distcp.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/15384/testReport/ Max. process+thread count 317 (vs. ulimit of 10000) modules C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/15384/console Powered by Apache Yetus 0.8.0 http://yetus.apache.org This message was automatically generated.
            stevel@apache.org Steve Loughran added a comment -

            seems reasonable to me, but others who know more of DistCP need to look at it too.

            one nit: CopyCommitter now uses SLF4J, so the log statement should be

             LOG.debug("blocks per chunk {}", blocksPerChunk)
            
            stevel@apache.org Steve Loughran added a comment - seems reasonable to me, but others who know more of DistCP need to look at it too. one nit: CopyCommitter now uses SLF4J, so the log statement should be LOG.debug( "blocks per chunk {}" , blocksPerChunk)
            hadoopqa Hadoop QA added a comment -
            -1 overall



            Vote Subsystem Runtime Comment
            0 reexec 0m 21s Docker mode activated.
                  Prechecks
            +1 @author 0m 0s The patch does not contain any @author tags.
            -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
                  trunk Compile Tests
            +1 mvninstall 20m 59s trunk passed
            +1 compile 0m 26s trunk passed
            +1 checkstyle 0m 18s trunk passed
            +1 mvnsite 0m 28s trunk passed
            +1 shadedclient 12m 11s branch has no errors when building and testing our client artifacts.
            +1 findbugs 0m 33s trunk passed
            +1 javadoc 0m 19s trunk passed
                  Patch Compile Tests
            +1 mvninstall 0m 27s the patch passed
            +1 compile 0m 22s the patch passed
            +1 javac 0m 22s the patch passed
            -0 checkstyle 0m 13s hadoop-tools/hadoop-distcp: The patch generated 1 new + 29 unchanged - 0 fixed = 30 total (was 29)
            +1 mvnsite 0m 23s the patch passed
            +1 whitespace 0m 0s The patch has no whitespace issues.
            +1 shadedclient 12m 54s patch has no errors when building and testing our client artifacts.
            +1 findbugs 0m 38s the patch passed
            +1 javadoc 0m 17s the patch passed
                  Other Tests
            +1 unit 12m 3s hadoop-distcp in the patch passed.
            +1 asflicense 0m 25s The patch does not generate ASF License warnings.
            63m 36s



            Subsystem Report/Notes
            Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1
            JIRA Issue HADOOP-15850
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12944559/HADOOP-15850.v4.patch
            Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
            uname Linux cdcdbb6b9968 4.4.0-134-generic #160~14.04.1-Ubuntu SMP Fri Aug 17 11:07:07 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality /testptch/patchprocess/precommit/personality/provided.sh
            git revision trunk / 2202e00
            maven version: Apache Maven 3.3.9
            Default Java 1.8.0_181
            findbugs v3.1.0-RC1
            checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/15387/artifact/out/diff-checkstyle-hadoop-tools_hadoop-distcp.txt
            Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/15387/testReport/
            Max. process+thread count 340 (vs. ulimit of 10000)
            modules C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp
            Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/15387/console
            Powered by Apache Yetus 0.8.0 http://yetus.apache.org

            This message was automatically generated.

            hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 21s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.       trunk Compile Tests +1 mvninstall 20m 59s trunk passed +1 compile 0m 26s trunk passed +1 checkstyle 0m 18s trunk passed +1 mvnsite 0m 28s trunk passed +1 shadedclient 12m 11s branch has no errors when building and testing our client artifacts. +1 findbugs 0m 33s trunk passed +1 javadoc 0m 19s trunk passed       Patch Compile Tests +1 mvninstall 0m 27s the patch passed +1 compile 0m 22s the patch passed +1 javac 0m 22s the patch passed -0 checkstyle 0m 13s hadoop-tools/hadoop-distcp: The patch generated 1 new + 29 unchanged - 0 fixed = 30 total (was 29) +1 mvnsite 0m 23s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 shadedclient 12m 54s patch has no errors when building and testing our client artifacts. +1 findbugs 0m 38s the patch passed +1 javadoc 0m 17s the patch passed       Other Tests +1 unit 12m 3s hadoop-distcp in the patch passed. +1 asflicense 0m 25s The patch does not generate ASF License warnings. 63m 36s Subsystem Report/Notes Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 JIRA Issue HADOOP-15850 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12944559/HADOOP-15850.v4.patch Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle uname Linux cdcdbb6b9968 4.4.0-134-generic #160~14.04.1-Ubuntu SMP Fri Aug 17 11:07:07 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/patchprocess/precommit/personality/provided.sh git revision trunk / 2202e00 maven version: Apache Maven 3.3.9 Default Java 1.8.0_181 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/15387/artifact/out/diff-checkstyle-hadoop-tools_hadoop-distcp.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/15387/testReport/ Max. process+thread count 340 (vs. ulimit of 10000) modules C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/15387/console Powered by Apache Yetus 0.8.0 http://yetus.apache.org This message was automatically generated.

            I am hoping to review the patch later today, because I'm really interested in it. stevel@apache.org please wait for me.

            weichiu Wei-Chiu Chuang added a comment - I am hoping to review the patch later today, because I'm really interested in it. stevel@apache.org please wait for me.

            I think the fix makes sense. In addition, you should remove the following line in TestCopyCommitter in order to test the fix. (Comment out the line, the tests fail without the fix)

            TestCopyCommitter.java
                // Unset listing file path since the config is shared by
                // multiple tests, and some test doesn't set it, such as
                // testNoCommitAction, but the distcp code will check it.
                config.set(DistCpConstants.CONF_LABEL_LISTING_FILE_PATH, "");
            
            weichiu Wei-Chiu Chuang added a comment - I think the fix makes sense. In addition, you should remove the following line in TestCopyCommitter in order to test the fix. (Comment out the line, the tests fail without the fix) TestCopyCommitter.java // Unset listing file path since the config is shared by // multiple tests, and some test doesn't set it, such as // testNoCommitAction, but the distcp code will check it. config.set(DistCpConstants.CONF_LABEL_LISTING_FILE_PATH, "");
            yuzhihong@gmail.com Ted Yu added a comment -

            Thanks for the review, looks like this bug could have been discovered sooner.

            yuzhihong@gmail.com Ted Yu added a comment - Thanks for the review, looks like this bug could have been discovered sooner.
            hadoopqa Hadoop QA added a comment -
            +1 overall



            Vote Subsystem Runtime Comment
            0 reexec 15m 43s Docker mode activated.
                  Prechecks
            +1 @author 0m 0s The patch does not contain any @author tags.
            +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
                  trunk Compile Tests
            +1 mvninstall 19m 0s trunk passed
            +1 compile 0m 28s trunk passed
            +1 checkstyle 0m 20s trunk passed
            +1 mvnsite 0m 30s trunk passed
            +1 shadedclient 11m 33s branch has no errors when building and testing our client artifacts.
            +1 findbugs 0m 31s trunk passed
            +1 javadoc 0m 17s trunk passed
                  Patch Compile Tests
            +1 mvninstall 0m 25s the patch passed
            +1 compile 0m 20s the patch passed
            +1 javac 0m 20s the patch passed
            -0 checkstyle 0m 14s hadoop-tools/hadoop-distcp: The patch generated 1 new + 43 unchanged - 0 fixed = 44 total (was 43)
            +1 mvnsite 0m 22s the patch passed
            +1 whitespace 0m 0s The patch has no whitespace issues.
            +1 shadedclient 11m 40s patch has no errors when building and testing our client artifacts.
            +1 findbugs 0m 40s the patch passed
            +1 javadoc 0m 19s the patch passed
                  Other Tests
            +1 unit 12m 53s hadoop-distcp in the patch passed.
            +1 asflicense 0m 27s The patch does not generate ASF License warnings.
            75m 56s



            Subsystem Report/Notes
            Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1
            JIRA Issue HADOOP-15850
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12944651/HADOOP-15850.v5.patch
            Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
            uname Linux 7ef2b22cdca6 4.4.0-133-generic #159-Ubuntu SMP Fri Aug 10 07:31:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality /testptch/patchprocess/precommit/personality/provided.sh
            git revision trunk / 13cc0f5
            maven version: Apache Maven 3.3.9
            Default Java 1.8.0_181
            findbugs v3.1.0-RC1
            checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/15392/artifact/out/diff-checkstyle-hadoop-tools_hadoop-distcp.txt
            Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/15392/testReport/
            Max. process+thread count 443 (vs. ulimit of 10000)
            modules C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp
            Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/15392/console
            Powered by Apache Yetus 0.8.0 http://yetus.apache.org

            This message was automatically generated.

            hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 15m 43s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests +1 mvninstall 19m 0s trunk passed +1 compile 0m 28s trunk passed +1 checkstyle 0m 20s trunk passed +1 mvnsite 0m 30s trunk passed +1 shadedclient 11m 33s branch has no errors when building and testing our client artifacts. +1 findbugs 0m 31s trunk passed +1 javadoc 0m 17s trunk passed       Patch Compile Tests +1 mvninstall 0m 25s the patch passed +1 compile 0m 20s the patch passed +1 javac 0m 20s the patch passed -0 checkstyle 0m 14s hadoop-tools/hadoop-distcp: The patch generated 1 new + 43 unchanged - 0 fixed = 44 total (was 43) +1 mvnsite 0m 22s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 shadedclient 11m 40s patch has no errors when building and testing our client artifacts. +1 findbugs 0m 40s the patch passed +1 javadoc 0m 19s the patch passed       Other Tests +1 unit 12m 53s hadoop-distcp in the patch passed. +1 asflicense 0m 27s The patch does not generate ASF License warnings. 75m 56s Subsystem Report/Notes Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 JIRA Issue HADOOP-15850 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12944651/HADOOP-15850.v5.patch Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle uname Linux 7ef2b22cdca6 4.4.0-133-generic #159-Ubuntu SMP Fri Aug 10 07:31:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/patchprocess/precommit/personality/provided.sh git revision trunk / 13cc0f5 maven version: Apache Maven 3.3.9 Default Java 1.8.0_181 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/15392/artifact/out/diff-checkstyle-hadoop-tools_hadoop-distcp.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/15392/testReport/ Max. process+thread count 443 (vs. ulimit of 10000) modules C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/15392/console Powered by Apache Yetus 0.8.0 http://yetus.apache.org This message was automatically generated.
            stevel@apache.org Steve Loughran added a comment -

            Looking good; one little nit from checkstyle

            ./hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyCommitter.java:105:        concatFileChunks(conf);: 'if' child has incorrect indentation level 8, expected level should be 6. [Indentation]
            
            stevel@apache.org Steve Loughran added a comment - Looking good; one little nit from checkstyle ./hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyCommitter.java:105: concatFileChunks(conf);: ' if ' child has incorrect indentation level 8, expected level should be 6. [Indentation]
            hadoopqa Hadoop QA added a comment -
            +1 overall



            Vote Subsystem Runtime Comment
            0 reexec 14m 40s Docker mode activated.
                  Prechecks
            +1 @author 0m 0s The patch does not contain any @author tags.
            +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
                  trunk Compile Tests
            +1 mvninstall 22m 44s trunk passed
            +1 compile 0m 32s trunk passed
            +1 checkstyle 0m 23s trunk passed
            +1 mvnsite 0m 37s trunk passed
            +1 shadedclient 13m 41s branch has no errors when building and testing our client artifacts.
            +1 findbugs 0m 46s trunk passed
            +1 javadoc 0m 27s trunk passed
                  Patch Compile Tests
            +1 mvninstall 0m 30s the patch passed
            +1 compile 0m 25s the patch passed
            +1 javac 0m 25s the patch passed
            +1 checkstyle 0m 16s the patch passed
            +1 mvnsite 0m 29s the patch passed
            +1 whitespace 0m 0s The patch has no whitespace issues.
            +1 shadedclient 14m 35s patch has no errors when building and testing our client artifacts.
            +1 findbugs 0m 46s the patch passed
            +1 javadoc 0m 20s the patch passed
                  Other Tests
            +1 unit 15m 2s hadoop-distcp in the patch passed.
            +1 asflicense 1m 12s The patch does not generate ASF License warnings.
            88m 2s



            Subsystem Report/Notes
            Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1
            JIRA Issue HADOOP-15850
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12944725/HADOOP-15850.v6.patch
            Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
            uname Linux 88e430ae2252 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27 10:45:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality /testptch/patchprocess/precommit/personality/provided.sh
            git revision trunk / 9bd1832
            maven version: Apache Maven 3.3.9
            Default Java 1.8.0_181
            findbugs v3.1.0-RC1
            Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/15398/testReport/
            Max. process+thread count 335 (vs. ulimit of 10000)
            modules C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp
            Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/15398/console
            Powered by Apache Yetus 0.8.0 http://yetus.apache.org

            This message was automatically generated.

            hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 14m 40s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests +1 mvninstall 22m 44s trunk passed +1 compile 0m 32s trunk passed +1 checkstyle 0m 23s trunk passed +1 mvnsite 0m 37s trunk passed +1 shadedclient 13m 41s branch has no errors when building and testing our client artifacts. +1 findbugs 0m 46s trunk passed +1 javadoc 0m 27s trunk passed       Patch Compile Tests +1 mvninstall 0m 30s the patch passed +1 compile 0m 25s the patch passed +1 javac 0m 25s the patch passed +1 checkstyle 0m 16s the patch passed +1 mvnsite 0m 29s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 shadedclient 14m 35s patch has no errors when building and testing our client artifacts. +1 findbugs 0m 46s the patch passed +1 javadoc 0m 20s the patch passed       Other Tests +1 unit 15m 2s hadoop-distcp in the patch passed. +1 asflicense 1m 12s The patch does not generate ASF License warnings. 88m 2s Subsystem Report/Notes Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 JIRA Issue HADOOP-15850 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12944725/HADOOP-15850.v6.patch Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle uname Linux 88e430ae2252 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27 10:45:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/patchprocess/precommit/personality/provided.sh git revision trunk / 9bd1832 maven version: Apache Maven 3.3.9 Default Java 1.8.0_181 findbugs v3.1.0-RC1 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/15398/testReport/ Max. process+thread count 335 (vs. ulimit of 10000) modules C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/15398/console Powered by Apache Yetus 0.8.0 http://yetus.apache.org This message was automatically generated.

            +1

            weichiu Wei-Chiu Chuang added a comment - +1

            Pushed v6 patch to trunk, branch-3.2, branch-3.1, branch-3.0. Thanks yuzhihong@gmail.com for the patch and stevel@apache.org for the review

            weichiu Wei-Chiu Chuang added a comment - Pushed v6 patch to trunk, branch-3.2, branch-3.1, branch-3.0. Thanks yuzhihong@gmail.com for the patch and stevel@apache.org for the review

            The branch-3.0 backport doesn't compile. Reverted for now.

            [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project hadoop-distcp: Compilation failure
            [ERROR] /Users/weichiu/sandbox/upstream/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyCommitter.java:[85,38] incompatible types: int cannot be converted to java.lang.Throwable

            weichiu Wei-Chiu Chuang added a comment - The branch-3.0 backport doesn't compile. Reverted for now. [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project hadoop-distcp: Compilation failure [ERROR] /Users/weichiu/sandbox/upstream/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyCommitter.java: [85,38] incompatible types: int cannot be converted to java.lang.Throwable

            Made a trivial change to the cherry-pick for branch-3.0. Pushed the commit and posted the patch for posterity.

            weichiu Wei-Chiu Chuang added a comment - Made a trivial change to the cherry-pick for branch-3.0. Pushed the commit and posted the patch for posterity.
            busbey Sean Busbey added a comment -

            This seems more severe than "Major". Am I correct that this impacts downstream users of DistCp beyond HBase?

            busbey Sean Busbey added a comment - This seems more severe than "Major". Am I correct that this impacts downstream users of DistCp beyond HBase?
            hudson Hudson added a comment -

            SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #15275 (See https://builds.apache.org/job/Hadoop-trunk-Commit/15275/)
            HADOOP-15850. CopyCommitter#concatFileChunks should check that the (weichiu: rev e2cecb681e2aab8b7c5465719cac53dce407a64c)

            • (edit) hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/mapred/TestCopyCommitter.java
            • (edit) hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyCommitter.java
            hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #15275 (See https://builds.apache.org/job/Hadoop-trunk-Commit/15275/ ) HADOOP-15850 . CopyCommitter#concatFileChunks should check that the (weichiu: rev e2cecb681e2aab8b7c5465719cac53dce407a64c) (edit) hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/mapred/TestCopyCommitter.java (edit) hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyCommitter.java

            Am I correct that this impacts downstream users of DistCp beyond HBase?

            Most likely yes.

            weichiu Wei-Chiu Chuang added a comment - Am I correct that this impacts downstream users of DistCp beyond HBase? Most likely yes.

            Pushed to branch-2 and branch-2.9.

            weichiu Wei-Chiu Chuang added a comment - Pushed to branch-2 and branch-2.9.

            People

              yuzhihong@gmail.com Ted Yu
              yuzhihong@gmail.com Ted Yu
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: