Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6680

JHS UserLogDir scan algorithm sometime could skip directory with update in CloudFS (Azure FileSystem, S3, etc.)

    Details

    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      In our cluster based on a Cloud FileSystem, we notice JHS sometimes could skip directory with .jhist file in scanning.
      The behavior is like:
      First round scan, doesn't found .jhist file:

      16/04/13 11:14:34 DEBUG azure.NativeAzureFileSystem: Found path as a directory with 6 files in it.
      16/04/13 11:14:34 DEBUG hs.HistoryFileManager: Found 0 files
      ...
      

      Then, we see "Scan not needed of ..." for the same directory every 3 minutes until application failed as timeout.

      From our analysis, we found the root cause is: most of Cloud File System (Azure FS, S3, etc.) is truncating file/directory modification time to seconds instead of milliseconds - which could due to limit of http protocol (from discussion at: https://forums.aws.amazon.com/thread.jspa?messageID=476615).

      So if the time sequence is happen to be: latest non .jhist file modification on directory happens at T1, directory scanning happens at T2, .jhist file added to directory at T3. If we have T1< T2 < T3 and T1 is equal to T3 after truncating to seconds, this issue could appear.

      1. MAPREDUCE-6680.patch
        2 kB
        Junping Du
      2. MAPREDUCE-6680-v2.patch
        2 kB
        Junping Du
      3. MAPREDUCE-6680-v3.patch
        2 kB
        Junping Du

        Activity

        Hide
        vinodkv Vinod Kumar Vavilapalli added a comment -

        Closing the JIRA as part of 2.7.3 release.

        Show
        vinodkv Vinod Kumar Vavilapalli added a comment - Closing the JIRA as part of 2.7.3 release.
        Hide
        djp Junping Du added a comment -

        Thanks Jian He for review and commit!

        Show
        djp Junping Du added a comment - Thanks Jian He for review and commit!
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #9640 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9640/)
        MAPREDUCE-6680. JHS UserLogDir scan algorithm sometime could skip (jianhe: rev 1e48eefe5800975ea0c4295c9911ae3f572ed37d)

        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryFileManager.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9640 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9640/ ) MAPREDUCE-6680 . JHS UserLogDir scan algorithm sometime could skip (jianhe: rev 1e48eefe5800975ea0c4295c9911ae3f572ed37d) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryFileManager.java
        Hide
        jianhe Jian He added a comment -

        Committed to trunk, branch-2, branch-2.8, branch-2.7, thanks Junping !

        Show
        jianhe Jian He added a comment - Committed to trunk, branch-2, branch-2.8, branch-2.7, thanks Junping !
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 12s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 6m 58s trunk passed
        +1 compile 0m 15s trunk passed with JDK v1.8.0_77
        +1 compile 0m 18s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 14s trunk passed
        +1 mvnsite 0m 23s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 0m 34s trunk passed
        +1 javadoc 0m 12s trunk passed with JDK v1.8.0_77
        +1 javadoc 0m 16s trunk passed with JDK v1.7.0_95
        +1 mvninstall 0m 18s the patch passed
        +1 compile 0m 12s the patch passed with JDK v1.8.0_77
        +1 javac 0m 12s the patch passed
        +1 compile 0m 15s the patch passed with JDK v1.7.0_95
        +1 javac 0m 15s the patch passed
        +1 checkstyle 0m 11s the patch passed
        +1 mvnsite 0m 21s the patch passed
        +1 mvneclipse 0m 10s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 0m 42s the patch passed
        +1 javadoc 0m 10s the patch passed with JDK v1.8.0_77
        +1 javadoc 0m 13s the patch passed with JDK v1.7.0_95
        +1 unit 5m 37s hadoop-mapreduce-client-hs in the patch passed with JDK v1.8.0_77.
        +1 unit 5m 59s hadoop-mapreduce-client-hs in the patch passed with JDK v1.7.0_95.
        +1 asflicense 0m 19s Patch does not generate ASF License warnings.
        24m 58s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:fbe3e86
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12799378/MAPREDUCE-6680-v3.patch
        JIRA Issue MAPREDUCE-6680
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 2114dac365e7 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / cb3ca46
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6447/testReport/
        modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6447/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 6m 58s trunk passed +1 compile 0m 15s trunk passed with JDK v1.8.0_77 +1 compile 0m 18s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 14s trunk passed +1 mvnsite 0m 23s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 0m 34s trunk passed +1 javadoc 0m 12s trunk passed with JDK v1.8.0_77 +1 javadoc 0m 16s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 18s the patch passed +1 compile 0m 12s the patch passed with JDK v1.8.0_77 +1 javac 0m 12s the patch passed +1 compile 0m 15s the patch passed with JDK v1.7.0_95 +1 javac 0m 15s the patch passed +1 checkstyle 0m 11s the patch passed +1 mvnsite 0m 21s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 42s the patch passed +1 javadoc 0m 10s the patch passed with JDK v1.8.0_77 +1 javadoc 0m 13s the patch passed with JDK v1.7.0_95 +1 unit 5m 37s hadoop-mapreduce-client-hs in the patch passed with JDK v1.8.0_77. +1 unit 5m 59s hadoop-mapreduce-client-hs in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 19s Patch does not generate ASF License warnings. 24m 58s Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12799378/MAPREDUCE-6680-v3.patch JIRA Issue MAPREDUCE-6680 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 2114dac365e7 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / cb3ca46 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6447/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6447/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        djp Junping Du added a comment -

        (T1 != T3) or (T1 == T3 but T1.toSeconds >= T2.toSeconds)

        I mean (T1 != T3) or (T1.toSeconds == T2.toSeconds) or (T1.toSeconds == (T2.toSeconds +1))

        Show
        djp Junping Du added a comment - (T1 != T3) or (T1 == T3 but T1.toSeconds >= T2.toSeconds) I mean (T1 != T3) or (T1.toSeconds == T2.toSeconds) or (T1.toSeconds == (T2.toSeconds +1))
        Hide
        djp Junping Du added a comment -

        we can record latest scanning time on userDir as T2, and we scan the list of directory in case: (T1 != T3) or (T1 = T3 but T1.toSeconds = T2.toSeconds), that can get rid of skip problem and not involve many unnecessary scan - given the chance file update happen at the same second with scan time is very low (scan interval is default to be 3 minutes).

        Another possibility is T1 (X second Y millisecond) and T3 (X second Z millisecond) is cast to second (X+1) in FS. To address this, we need to check (T1 != T3) or (T1 == T3 but T1.toSeconds >= T2.toSeconds). Updated this in v3 patch.

        Show
        djp Junping Du added a comment - we can record latest scanning time on userDir as T2, and we scan the list of directory in case: (T1 != T3) or (T1 = T3 but T1.toSeconds = T2.toSeconds), that can get rid of skip problem and not involve many unnecessary scan - given the chance file update happen at the same second with scan time is very low (scan interval is default to be 3 minutes). Another possibility is T1 (X second Y millisecond) and T3 (X second Z millisecond) is cast to second (X+1) in FS. To address this, we need to check (T1 != T3) or (T1 == T3 but T1.toSeconds >= T2.toSeconds). Updated this in v3 patch.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 9s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 6m 52s trunk passed
        +1 compile 0m 16s trunk passed with JDK v1.8.0_77
        +1 compile 0m 17s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 16s trunk passed
        +1 mvnsite 0m 24s trunk passed
        +1 mvneclipse 0m 12s trunk passed
        +1 findbugs 0m 33s trunk passed
        +1 javadoc 0m 13s trunk passed with JDK v1.8.0_77
        +1 javadoc 0m 15s trunk passed with JDK v1.7.0_95
        +1 mvninstall 0m 18s the patch passed
        +1 compile 0m 12s the patch passed with JDK v1.8.0_77
        +1 javac 0m 12s the patch passed
        +1 compile 0m 15s the patch passed with JDK v1.7.0_95
        +1 javac 0m 15s the patch passed
        +1 checkstyle 0m 11s the patch passed
        +1 mvnsite 0m 20s the patch passed
        +1 mvneclipse 0m 11s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 0m 41s the patch passed
        +1 javadoc 0m 10s the patch passed with JDK v1.8.0_77
        +1 javadoc 0m 13s the patch passed with JDK v1.7.0_95
        +1 unit 5m 30s hadoop-mapreduce-client-hs in the patch passed with JDK v1.8.0_77.
        +1 unit 5m 52s hadoop-mapreduce-client-hs in the patch passed with JDK v1.7.0_95.
        +1 asflicense 0m 17s Patch does not generate ASF License warnings.
        24m 32s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:fbe3e86
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12799287/MAPREDUCE-6680-v2.patch
        JIRA Issue MAPREDUCE-6680
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 9fa1feedf7fd 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / d8b729e
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6444/testReport/
        modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6444/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 9s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 6m 52s trunk passed +1 compile 0m 16s trunk passed with JDK v1.8.0_77 +1 compile 0m 17s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 16s trunk passed +1 mvnsite 0m 24s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 0m 33s trunk passed +1 javadoc 0m 13s trunk passed with JDK v1.8.0_77 +1 javadoc 0m 15s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 18s the patch passed +1 compile 0m 12s the patch passed with JDK v1.8.0_77 +1 javac 0m 12s the patch passed +1 compile 0m 15s the patch passed with JDK v1.7.0_95 +1 javac 0m 15s the patch passed +1 checkstyle 0m 11s the patch passed +1 mvnsite 0m 20s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 41s the patch passed +1 javadoc 0m 10s the patch passed with JDK v1.8.0_77 +1 javadoc 0m 13s the patch passed with JDK v1.7.0_95 +1 unit 5m 30s hadoop-mapreduce-client-hs in the patch passed with JDK v1.8.0_77. +1 unit 5m 52s hadoop-mapreduce-client-hs in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 17s Patch does not generate ASF License warnings. 24m 32s Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12799287/MAPREDUCE-6680-v2.patch JIRA Issue MAPREDUCE-6680 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 9fa1feedf7fd 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d8b729e Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6444/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6444/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 11s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 6m 52s trunk passed
        +1 compile 0m 16s trunk passed with JDK v1.8.0_77
        +1 compile 0m 18s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 13s trunk passed
        +1 mvnsite 0m 24s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 0m 33s trunk passed
        +1 javadoc 0m 12s trunk passed with JDK v1.8.0_77
        +1 javadoc 0m 16s trunk passed with JDK v1.7.0_95
        +1 mvninstall 0m 18s the patch passed
        +1 compile 0m 13s the patch passed with JDK v1.8.0_77
        +1 javac 0m 13s the patch passed
        +1 compile 0m 15s the patch passed with JDK v1.7.0_95
        +1 javac 0m 15s the patch passed
        +1 checkstyle 0m 12s the patch passed
        +1 mvnsite 0m 20s the patch passed
        +1 mvneclipse 0m 11s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 0m 41s the patch passed
        +1 javadoc 0m 10s the patch passed with JDK v1.8.0_77
        +1 javadoc 0m 13s the patch passed with JDK v1.7.0_95
        +1 unit 5m 33s hadoop-mapreduce-client-hs in the patch passed with JDK v1.8.0_77.
        +1 unit 5m 54s hadoop-mapreduce-client-hs in the patch passed with JDK v1.7.0_95.
        +1 asflicense 0m 18s Patch does not generate ASF License warnings.
        24m 44s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:fbe3e86
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12799287/MAPREDUCE-6680-v2.patch
        JIRA Issue MAPREDUCE-6680
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 0541f6ebbf45 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / d8b729e
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6443/testReport/
        modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6443/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 6m 52s trunk passed +1 compile 0m 16s trunk passed with JDK v1.8.0_77 +1 compile 0m 18s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 13s trunk passed +1 mvnsite 0m 24s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 0m 33s trunk passed +1 javadoc 0m 12s trunk passed with JDK v1.8.0_77 +1 javadoc 0m 16s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 18s the patch passed +1 compile 0m 13s the patch passed with JDK v1.8.0_77 +1 javac 0m 13s the patch passed +1 compile 0m 15s the patch passed with JDK v1.7.0_95 +1 javac 0m 15s the patch passed +1 checkstyle 0m 12s the patch passed +1 mvnsite 0m 20s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 41s the patch passed +1 javadoc 0m 10s the patch passed with JDK v1.8.0_77 +1 javadoc 0m 13s the patch passed with JDK v1.7.0_95 +1 unit 5m 33s hadoop-mapreduce-client-hs in the patch passed with JDK v1.8.0_77. +1 unit 5m 54s hadoop-mapreduce-client-hs in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 18s Patch does not generate ASF License warnings. 24m 44s Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12799287/MAPREDUCE-6680-v2.patch JIRA Issue MAPREDUCE-6680 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 0541f6ebbf45 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d8b729e Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6443/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6443/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        djp Junping Du added a comment -

        v2 patch fix checkstyle issue.

        Show
        djp Junping Du added a comment - v2 patch fix checkstyle issue.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 11m 38s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 6m 55s trunk passed
        +1 compile 0m 15s trunk passed with JDK v1.8.0_77
        +1 compile 0m 18s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 14s trunk passed
        +1 mvnsite 0m 24s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 0m 34s trunk passed
        +1 javadoc 0m 12s trunk passed with JDK v1.8.0_77
        +1 javadoc 0m 16s trunk passed with JDK v1.7.0_95
        +1 mvninstall 0m 18s the patch passed
        +1 compile 0m 13s the patch passed with JDK v1.8.0_77
        +1 javac 0m 13s the patch passed
        +1 compile 0m 16s the patch passed with JDK v1.7.0_95
        +1 javac 0m 16s the patch passed
        -1 checkstyle 0m 12s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs: patch generated 1 new + 16 unchanged - 0 fixed = 17 total (was 16)
        +1 mvnsite 0m 20s the patch passed
        +1 mvneclipse 0m 11s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 0m 41s the patch passed
        +1 javadoc 0m 11s the patch passed with JDK v1.8.0_77
        +1 javadoc 0m 13s the patch passed with JDK v1.7.0_95
        +1 unit 5m 34s hadoop-mapreduce-client-hs in the patch passed with JDK v1.8.0_77.
        +1 unit 5m 57s hadoop-mapreduce-client-hs in the patch passed with JDK v1.7.0_95.
        +1 asflicense 0m 20s Patch does not generate ASF License warnings.
        36m 18s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:fbe3e86
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12799262/MAPREDUCE-6680.patch
        JIRA Issue MAPREDUCE-6680
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux fb28783d1eb4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / d8b729e
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6442/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-hs.txt
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6442/testReport/
        modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6442/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 11m 38s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 6m 55s trunk passed +1 compile 0m 15s trunk passed with JDK v1.8.0_77 +1 compile 0m 18s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 14s trunk passed +1 mvnsite 0m 24s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 0m 34s trunk passed +1 javadoc 0m 12s trunk passed with JDK v1.8.0_77 +1 javadoc 0m 16s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 18s the patch passed +1 compile 0m 13s the patch passed with JDK v1.8.0_77 +1 javac 0m 13s the patch passed +1 compile 0m 16s the patch passed with JDK v1.7.0_95 +1 javac 0m 16s the patch passed -1 checkstyle 0m 12s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs: patch generated 1 new + 16 unchanged - 0 fixed = 17 total (was 16) +1 mvnsite 0m 20s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 41s the patch passed +1 javadoc 0m 11s the patch passed with JDK v1.8.0_77 +1 javadoc 0m 13s the patch passed with JDK v1.7.0_95 +1 unit 5m 34s hadoop-mapreduce-client-hs in the patch passed with JDK v1.8.0_77. +1 unit 5m 57s hadoop-mapreduce-client-hs in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 20s Patch does not generate ASF License warnings. 36m 18s Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12799262/MAPREDUCE-6680.patch JIRA Issue MAPREDUCE-6680 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux fb28783d1eb4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d8b729e Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6442/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-hs.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6442/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6442/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        djp Junping Du added a comment -

        Put a patch with proper fix. The unit test could be very hard given the issue appears to be platform related and intermittently. I think the fix is straight-forward enough.

        Show
        djp Junping Du added a comment - Put a patch with proper fix. The unit test could be very hard given the issue appears to be platform related and intermittently. I think the fix is straight-forward enough.
        Hide
        djp Junping Du added a comment -

        Given analysis in description, the solution is: we can record latest scanning time on userDir as T2, and we scan the list of directory in case: (T1 != T3) or (T1 = T3 but T1.toSeconds = T2.toSeconds), that can get rid of skip problem and not involve many unnecessary scan - given the chance file update happen at the same second with scan time is very low (scan interval is default to be 3 minutes).

        Show
        djp Junping Du added a comment - Given analysis in description, the solution is: we can record latest scanning time on userDir as T2, and we scan the list of directory in case: (T1 != T3) or (T1 = T3 but T1.toSeconds = T2.toSeconds), that can get rid of skip problem and not involve many unnecessary scan - given the chance file update happen at the same second with scan time is very low (scan interval is default to be 3 minutes).

          People

          • Assignee:
            djp Junping Du
            Reporter:
            djp Junping Du
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development