Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.6.0
    • Fix Version/s: 2.8.0, 3.0.0-alpha2
    • Component/s: fs, test
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      HADOOP-11584 has shown that the contract tests are not validating that FileStatus.getBlocksize() must be >0 for any analytics jobs to partition workload correctly.

      Clarify in text and add test to do this. Test MUST be designed to work against eventually consistent filesystems where getFileStatus() may not be immediately visible, by retrying operation if FS declares it is an object store.

      1. HADOOP-11601-001.patch
        11 kB
        Steve Loughran
      2. HADOOP-11601-002.patch
        8 kB
        Steve Loughran
      3. HADOOP-11601-003.patch
        9 kB
        Steve Loughran
      4. HADOOP-11601-004.patch
        9 kB
        Steve Loughran
      5. HADOOP-11601-005.patch
        9 kB
        Steve Loughran
      6. HADOOP-11601-006.patch
        9 kB
        Mingliang Liu
      7. HADOOP-11601-007.patch
        9 kB
        Mingliang Liu

        Issue Links

          Activity

          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10855 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10855/)
          HADOOP-11601. Enhance FS spec & tests to mandate (liuml07: rev ae8849fe378e11b9db642ef7784c8e6a08199b96)

          • (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/AbstractContractCreateTest.java
          • (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/ContractTestUtils.java
          • (edit) hadoop-common-project/hadoop-common/src/site/markdown/filesystem/filesystem.md
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10855 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10855/ ) HADOOP-11601 . Enhance FS spec & tests to mandate (liuml07: rev ae8849fe378e11b9db642ef7784c8e6a08199b96) (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/AbstractContractCreateTest.java (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/ContractTestUtils.java (edit) hadoop-common-project/hadoop-common/src/site/markdown/filesystem/filesystem.md
          Hide
          liuml07 Mingliang Liu added a comment -

          Committed to trunk through branch-2.8 branches. Thanks Steve Loughran for your contribution; thanks Daniel Templeton and Masatake Iwasaki for review.

          Show
          liuml07 Mingliang Liu added a comment - Committed to trunk through branch-2.8 branches. Thanks Steve Loughran for your contribution; thanks Daniel Templeton and Masatake Iwasaki for review.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/hadoop/pull/50

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/hadoop/pull/50
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user liuml07 commented on the issue:

          https://github.com/apache/hadoop/pull/50

          +1

          Will commit this week.

          Show
          githubbot ASF GitHub Bot added a comment - Github user liuml07 commented on the issue: https://github.com/apache/hadoop/pull/50 +1 Will commit this week.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 6m 49s trunk passed
          +1 compile 9m 33s trunk passed
          +1 checkstyle 0m 29s trunk passed
          +1 mvnsite 1m 1s trunk passed
          +1 mvneclipse 0m 18s trunk passed
          +1 findbugs 1m 26s trunk passed
          +1 javadoc 0m 47s trunk passed
          +1 mvninstall 0m 36s the patch passed
          +1 compile 9m 9s the patch passed
          +1 javac 9m 9s the patch passed
          +1 checkstyle 0m 29s hadoop-common-project/hadoop-common: The patch generated 0 new + 18 unchanged - 2 fixed = 18 total (was 20)
          +1 mvnsite 1m 1s the patch passed
          +1 mvneclipse 0m 19s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 33s the patch passed
          +1 javadoc 0m 48s the patch passed
          +1 unit 8m 22s hadoop-common in the patch passed.
          +1 asflicense 0m 32s The patch does not generate ASF License warnings.
          45m 16s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-11601
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12839273/HADOOP-11601-007.patch
          Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle
          uname Linux 1cc372d48797 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 04a024b
          Default Java 1.8.0_111
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11082/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11082/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 6m 49s trunk passed +1 compile 9m 33s trunk passed +1 checkstyle 0m 29s trunk passed +1 mvnsite 1m 1s trunk passed +1 mvneclipse 0m 18s trunk passed +1 findbugs 1m 26s trunk passed +1 javadoc 0m 47s trunk passed +1 mvninstall 0m 36s the patch passed +1 compile 9m 9s the patch passed +1 javac 9m 9s the patch passed +1 checkstyle 0m 29s hadoop-common-project/hadoop-common: The patch generated 0 new + 18 unchanged - 2 fixed = 18 total (was 20) +1 mvnsite 1m 1s the patch passed +1 mvneclipse 0m 19s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 33s the patch passed +1 javadoc 0m 48s the patch passed +1 unit 8m 22s hadoop-common in the patch passed. +1 asflicense 0m 32s The patch does not generate ASF License warnings. 45m 16s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-11601 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12839273/HADOOP-11601-007.patch Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle uname Linux 1cc372d48797 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 04a024b Default Java 1.8.0_111 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11082/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11082/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 12s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 7m 45s trunk passed
          +1 compile 10m 21s trunk passed
          +1 checkstyle 0m 30s trunk passed
          +1 mvnsite 1m 6s trunk passed
          +1 mvneclipse 0m 17s trunk passed
          +1 findbugs 1m 34s trunk passed
          +1 javadoc 0m 48s trunk passed
          +1 mvninstall 0m 40s the patch passed
          +1 compile 9m 58s the patch passed
          +1 javac 9m 58s the patch passed
          -0 checkstyle 0m 32s hadoop-common-project/hadoop-common: The patch generated 4 new + 18 unchanged - 2 fixed = 22 total (was 20)
          +1 mvnsite 1m 4s the patch passed
          +1 mvneclipse 0m 19s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 34s the patch passed
          +1 javadoc 0m 49s the patch passed
          +1 unit 8m 9s hadoop-common in the patch passed.
          +1 asflicense 0m 33s The patch does not generate ASF License warnings.
          48m 0s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-11601
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12839262/HADOOP-11601-006.patch
          Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle
          uname Linux d972e88fba03 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 04a024b
          Default Java 1.8.0_111
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11081/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11081/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11081/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 7m 45s trunk passed +1 compile 10m 21s trunk passed +1 checkstyle 0m 30s trunk passed +1 mvnsite 1m 6s trunk passed +1 mvneclipse 0m 17s trunk passed +1 findbugs 1m 34s trunk passed +1 javadoc 0m 48s trunk passed +1 mvninstall 0m 40s the patch passed +1 compile 9m 58s the patch passed +1 javac 9m 58s the patch passed -0 checkstyle 0m 32s hadoop-common-project/hadoop-common: The patch generated 4 new + 18 unchanged - 2 fixed = 22 total (was 20) +1 mvnsite 1m 4s the patch passed +1 mvneclipse 0m 19s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 34s the patch passed +1 javadoc 0m 49s the patch passed +1 unit 8m 9s hadoop-common in the patch passed. +1 asflicense 0m 33s The patch does not generate ASF License warnings. 48m 0s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-11601 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12839262/HADOOP-11601-006.patch Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle uname Linux d972e88fba03 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 04a024b Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11081/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11081/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11081/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          liuml07 Mingliang Liu added a comment -

          +1

          Rebased from trunk branch and fixed minor conflicts.

          Will commit this week if no more comments.

          Show
          liuml07 Mingliang Liu added a comment - +1 Rebased from trunk branch and fixed minor conflicts. Will commit this week if no more comments.
          Hide
          iwasakims Masatake Iwasaki added a comment -

          I ran TestHDFSContractCreate, TestS3NContractCreate and TestS3AContractCreate with the patch and succeeded.

          Show
          iwasakims Masatake Iwasaki added a comment - I ran TestHDFSContractCreate , TestS3NContractCreate and TestS3AContractCreate with the patch and succeeded.
          Hide
          templedf Daniel Templeton added a comment -

          Finally! Looks good. My last comment is on this code:

                if (!fs.exists(path)) {
          
                  if (isSupported(IS_BLOBSTORE)) {
                    // object store: downgrade to a skip so that the failure is visible
                    // in test results
                    skip( "Filesystem is an object store and newly created files are not immediately visible");
                  }
                  assertPathExists("expected path to be visible before file closed",
                      path);
                }
          

          Since the assertPathExists() call will check !fs.exists(path), this seems redundant. You can get the same behavior with fewer checks with:

                if (isSupported(IS_BLOBSTORE) && !fs.exists(path)) {
                  // object store: downgrade to a skip so that the failure is visible
                  // in test results
                  skip( "Filesystem is an object store and newly created files are not immediately visible");
                }
          
                assertPathExists("expected path to be visible before file closed",
                    path);
          
          Show
          templedf Daniel Templeton added a comment - Finally! Looks good. My last comment is on this code: if (!fs.exists(path)) { if (isSupported(IS_BLOBSTORE)) { // object store: downgrade to a skip so that the failure is visible // in test results skip( "Filesystem is an object store and newly created files are not immediately visible" ); } assertPathExists( "expected path to be visible before file closed" , path); } Since the assertPathExists() call will check !fs.exists(path) , this seems redundant. You can get the same behavior with fewer checks with: if (isSupported(IS_BLOBSTORE) && !fs.exists(path)) { // object store: downgrade to a skip so that the failure is visible // in test results skip( "Filesystem is an object store and newly created files are not immediately visible" ); } assertPathExists( "expected path to be visible before file closed" , path);
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 21s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 8m 10s trunk passed
          +1 compile 7m 25s trunk passed
          +1 checkstyle 0m 23s trunk passed
          +1 mvnsite 0m 55s trunk passed
          +1 mvneclipse 0m 12s trunk passed
          +1 findbugs 1m 24s trunk passed
          +1 javadoc 0m 46s trunk passed
          +1 mvninstall 0m 40s the patch passed
          +1 compile 7m 14s the patch passed
          +1 javac 7m 14s the patch passed
          -0 checkstyle 0m 24s hadoop-common-project/hadoop-common: The patch generated 5 new + 44 unchanged - 3 fixed = 49 total (was 47)
          +1 mvnsite 0m 54s the patch passed
          +1 mvneclipse 0m 12s the patch passed
          -1 whitespace 0m 0s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 findbugs 1m 32s the patch passed
          +1 javadoc 0m 46s the patch passed
          -1 unit 7m 47s hadoop-common in the patch failed.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          40m 56s



          Reason Tests
          Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12817118/HADOOP-11601-005.patch
          JIRA Issue HADOOP-11601
          Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle
          uname Linux b7aa6448bf62 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 0fd3980
          Default Java 1.8.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/9955/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/9955/artifact/patchprocess/whitespace-eol.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9955/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9955/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9955/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 21s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 8m 10s trunk passed +1 compile 7m 25s trunk passed +1 checkstyle 0m 23s trunk passed +1 mvnsite 0m 55s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 24s trunk passed +1 javadoc 0m 46s trunk passed +1 mvninstall 0m 40s the patch passed +1 compile 7m 14s the patch passed +1 javac 7m 14s the patch passed -0 checkstyle 0m 24s hadoop-common-project/hadoop-common: The patch generated 5 new + 44 unchanged - 3 fixed = 49 total (was 47) +1 mvnsite 0m 54s the patch passed +1 mvneclipse 0m 12s the patch passed -1 whitespace 0m 0s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 findbugs 1m 32s the patch passed +1 javadoc 0m 46s the patch passed -1 unit 7m 47s hadoop-common in the patch failed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 40m 56s Reason Tests Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12817118/HADOOP-11601-005.patch JIRA Issue HADOOP-11601 Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle uname Linux b7aa6448bf62 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0fd3980 Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/9955/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/9955/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9955/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9955/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9955/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Patch 005; rebased against trunk, trying to cut more PR references from the comments. If this still isn't taking I'm going to delete some of the Yetus-generated comments and retry

          Show
          stevel@apache.org Steve Loughran added a comment - Patch 005; rebased against trunk, trying to cut more PR references from the comments. If this still isn't taking I'm going to delete some of the Yetus-generated comments and retry
          Hide
          stevel@apache.org Steve Loughran added a comment -

          I see —and I see the fix. Edited the relevant comment

          Show
          stevel@apache.org Steve Loughran added a comment - I see —and I see the fix. Edited the relevant comment
          Hide
          aw Allen Wittenauer added a comment - - edited

          Yetus is working exactly as it is supposed to. You can even test this yourself:

          $ dev-support/bin/smart-apply-patch HADOOP-11601
          Processing: HADOOP-11601
          HADOOP-11601 appears to be a Github PR. Switching Modes.
          GITHUB PR #50 is being downloaded at Thu Jun 30 07:09:27 PDT 2016 from
          https://github.com/apache/hadoop/pull/50
          Patch from GITHUB PR #50 is being downloaded at Thu Jun 30 07:09:28 PDT 2016 from
          https://github.com/apache/hadoop/pull/50.patch
          ERROR: Aborting! HADOOP-11601 cannot be verified.
          
          $ wget https://github.com/apache/hadoop/pull/50.patch
          --2016-06-30 07:09:36--  https://github.com/apache/hadoop/pull/50.patch
          Resolving github.com... 192.30.253.113
          Connecting to github.com|192.30.253.113|:443... connected.
          HTTP request sent, awaiting response... 302 Found
          Location: https://patch-diff.githubusercontent.com/raw/apache/hadoop/pull/50.patch [following]
          --2016-06-30 07:09:36--  https://patch-diff.githubusercontent.com/raw/apache/hadoop/pull/50.patch
          Resolving patch-diff.githubusercontent.com... 192.30.253.113
          Connecting to patch-diff.githubusercontent.com|192.30.253.113|:443... connected.
          HTTP request sent, awaiting response... 200 OK
          Cookie coming from patch-diff.githubusercontent.com attempted to set domain to github.com
          Length: unspecified [text/plain]
          Saving to: '50.patch'
          
          50.patch                                           [  <=>                                                                                                ]  11.11K  38.5KB/s    in 0.3s    
          
          2016-06-30 07:09:37 (38.5 KB/s) - '50.patch' saved [11377]
          
          $ git apply 50.patch
          50.patch:149: trailing whitespace.
            
          50.patch:175: trailing whitespace.
            
          50.patch:181: trailing whitespace.
              try( 
          50.patch:213: trailing whitespace.
              
          50.patch:219: trailing whitespace.
            
          error: patch failed: hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/AbstractContractCreateTest.java:163
          error: hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/AbstractContractCreateTest.java: patch does not apply
          error: patch failed: hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/ContractTestUtils.java:23
          error: hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/ContractTestUtils.java: patch does not apply
          

          Again: you've switched from github to patch attach and Yetus is programmed such that once you go github, you need to do a lot of surgery on the JIRA issue to switch back. See also YETUS-362.

          Show
          aw Allen Wittenauer added a comment - - edited Yetus is working exactly as it is supposed to. You can even test this yourself: $ dev-support/bin/smart-apply-patch HADOOP-11601 Processing: HADOOP-11601 HADOOP-11601 appears to be a Github PR. Switching Modes. GITHUB PR #50 is being downloaded at Thu Jun 30 07:09:27 PDT 2016 from https: //github.com/apache/hadoop/pull/50 Patch from GITHUB PR #50 is being downloaded at Thu Jun 30 07:09:28 PDT 2016 from https: //github.com/apache/hadoop/pull/50.patch ERROR: Aborting! HADOOP-11601 cannot be verified. $ wget https: //github.com/apache/hadoop/pull/50.patch --2016-06-30 07:09:36-- https: //github.com/apache/hadoop/pull/50.patch Resolving github.com... 192.30.253.113 Connecting to github.com|192.30.253.113|:443... connected. HTTP request sent, awaiting response... 302 Found Location: https: //patch-diff.githubusercontent.com/raw/apache/hadoop/pull/50.patch [following] --2016-06-30 07:09:36-- https: //patch-diff.githubusercontent.com/raw/apache/hadoop/pull/50.patch Resolving patch-diff.githubusercontent.com... 192.30.253.113 Connecting to patch-diff.githubusercontent.com|192.30.253.113|:443... connected. HTTP request sent, awaiting response... 200 OK Cookie coming from patch-diff.githubusercontent.com attempted to set domain to github.com Length: unspecified [text/plain] Saving to: '50.patch' 50.patch [ <=> ] 11.11K 38.5KB/s in 0.3s 2016-06-30 07:09:37 (38.5 KB/s) - '50.patch' saved [11377] $ git apply 50.patch 50.patch:149: trailing whitespace. 50.patch:175: trailing whitespace. 50.patch:181: trailing whitespace. try ( 50.patch:213: trailing whitespace. 50.patch:219: trailing whitespace. error: patch failed: hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/AbstractContractCreateTest.java:163 error: hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/AbstractContractCreateTest.java: patch does not apply error: patch failed: hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/ContractTestUtils.java:23 error: hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/ContractTestUtils.java: patch does not apply Again: you've switched from github to patch attach and Yetus is programmed such that once you go github, you need to do a lot of surgery on the JIRA issue to switch back. See also YETUS-362 .
          Hide
          stevel@apache.org Steve Loughran added a comment -

          it does apply to trunk, I just tested locally. Allen Wittenauer: is yetus having problems today?

          Show
          stevel@apache.org Steve Loughran added a comment - it does apply to trunk, I just tested locally. Allen Wittenauer : is yetus having problems today?
          Hide
          hadoopqa Hadoop QA added a comment - - edited
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 patch 0m 6s HADOOP-11601 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



          Subsystem Report/Notes
          JIRA Issue HADOOP-11601
          GITHUB PR https://github.com/apache/hadoop/pull/ <space> 50
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9906/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - - edited -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 6s HADOOP-11601 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Issue HADOOP-11601 GITHUB PR https://github.com/apache/hadoop/pull/ <space> 50 Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9906/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Patch 004; this is patch 003 rebased to trunk again

          Show
          stevel@apache.org Steve Loughran added a comment - Patch 004; this is patch 003 rebased to trunk again
          Hide
          hadoopqa Hadoop QA added a comment - - edited
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 patch 0m 6s HADOOP-11601 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



          Subsystem Report/Notes
          JIRA Issue HADOOP-11601
          GITHUB PR https://github.com/apache/hadoop/pull/ <space> 50
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9904/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - - edited -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 6s HADOOP-11601 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Issue HADOOP-11601 GITHUB PR https://github.com/apache/hadoop/pull/ <space> 50 Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9904/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          path 003; address dan's comments, resync with trunk

          Show
          stevel@apache.org Steve Loughran added a comment - path 003; address dan's comments, resync with trunk
          Hide
          templedf Daniel Templeton added a comment -

          Thanks, for the patch Steve Loughran.

          My comments:

          -that is too small will result in either too many jobs being submitted
          -for efficient work, or the `JobSubmissionClient` running out of memory.
          -
          +that is too small will result in badly partitioned workload.
          

          Your change here correctly states the root effect, whereas what you're removing states the visible side effects. Seems to me that they're both useful. Maybe add your statement with the existing statement as a clarifying subordinate phrase?

                  assertPathExists("expected path to be visible before anything written",
                      path);
          

          I don't understand the message here. Didn't you just write 'a' and then flush it?

              assertTrue("fs.getDefaultBlockSize(path) size is invalid " + defaultBlockSize,
                  defaultBlockSize > 0);
          

          Would it be helpful here to replace path with the actual path?

             * @param parent Test file parent dir path
             *    thrown if an I/O error occurs while writing or reading the test file
             *    <i>other than file not found</i>
             * @return the status
             * @throws IOException
          

          The throws message does not follow the @throws tag.

          In the getFileStatusEventually() method, would it be better to include a short (~10ms) sleep to reduce the impact during parallel testing?

          Show
          templedf Daniel Templeton added a comment - Thanks, for the patch Steve Loughran . My comments: -that is too small will result in either too many jobs being submitted - for efficient work, or the `JobSubmissionClient` running out of memory. - +that is too small will result in badly partitioned workload. Your change here correctly states the root effect, whereas what you're removing states the visible side effects. Seems to me that they're both useful. Maybe add your statement with the existing statement as a clarifying subordinate phrase? assertPathExists( "expected path to be visible before anything written" , path); I don't understand the message here. Didn't you just write 'a' and then flush it? assertTrue( "fs.getDefaultBlockSize(path) size is invalid " + defaultBlockSize, defaultBlockSize > 0); Would it be helpful here to replace path with the actual path? * @param parent Test file parent dir path * thrown if an I/O error occurs while writing or reading the test file * <i>other than file not found</i> * @ return the status * @ throws IOException The throws message does not follow the @throws tag. In the getFileStatusEventually() method, would it be better to include a short (~10ms) sleep to reduce the impact during parallel testing?
          Hide
          stevel@apache.org Steve Loughran added a comment -

          synced with trunk

          Show
          stevel@apache.org Steve Loughran added a comment - synced with trunk
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 8m 47s trunk passed
          +1 compile 9m 55s trunk passed with JDK v1.8.0_66
          +1 compile 10m 0s trunk passed with JDK v1.7.0_85
          +1 checkstyle 0m 19s trunk passed
          +1 mvnsite 1m 13s trunk passed
          +1 mvneclipse 0m 16s trunk passed
          +1 findbugs 2m 8s trunk passed
          +1 javadoc 1m 6s trunk passed with JDK v1.8.0_66
          +1 javadoc 1m 14s trunk passed with JDK v1.7.0_85
          +1 mvninstall 1m 44s the patch passed
          +1 compile 9m 41s the patch passed with JDK v1.8.0_66
          +1 javac 9m 41s the patch passed
          +1 compile 10m 3s the patch passed with JDK v1.7.0_85
          +1 javac 10m 3s the patch passed
          +1 checkstyle 0m 18s the patch passed
          +1 mvnsite 1m 9s the patch passed
          +1 mvneclipse 0m 15s the patch passed
          -1 whitespace 0m 0s The patch has 7 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 findbugs 2m 12s the patch passed
          +1 javadoc 1m 4s the patch passed with JDK v1.8.0_66
          +1 javadoc 1m 11s the patch passed with JDK v1.7.0_85
          -1 unit 8m 18s hadoop-common in the patch failed with JDK v1.8.0_66.
          +1 unit 8m 45s hadoop-common in the patch passed with JDK v1.7.0_85.
          +1 asflicense 0m 26s Patch does not generate ASF License warnings.
          81m 16s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.ipc.TestDecayRpcScheduler



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Issue HADOOP-11601
          GITHUB PR https://github.com/apache/hadoop/pull/50
          Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle
          uname Linux 599761bf506a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / a6f20d8
          findbugs v3.0.0
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/8120/artifact/patchprocess/whitespace-eol.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8120/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt
          unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8120/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt
          JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8120/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Max memory used 75MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8120/console
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8120/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 8m 47s trunk passed +1 compile 9m 55s trunk passed with JDK v1.8.0_66 +1 compile 10m 0s trunk passed with JDK v1.7.0_85 +1 checkstyle 0m 19s trunk passed +1 mvnsite 1m 13s trunk passed +1 mvneclipse 0m 16s trunk passed +1 findbugs 2m 8s trunk passed +1 javadoc 1m 6s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 14s trunk passed with JDK v1.7.0_85 +1 mvninstall 1m 44s the patch passed +1 compile 9m 41s the patch passed with JDK v1.8.0_66 +1 javac 9m 41s the patch passed +1 compile 10m 3s the patch passed with JDK v1.7.0_85 +1 javac 10m 3s the patch passed +1 checkstyle 0m 18s the patch passed +1 mvnsite 1m 9s the patch passed +1 mvneclipse 0m 15s the patch passed -1 whitespace 0m 0s The patch has 7 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 findbugs 2m 12s the patch passed +1 javadoc 1m 4s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 11s the patch passed with JDK v1.7.0_85 -1 unit 8m 18s hadoop-common in the patch failed with JDK v1.8.0_66. +1 unit 8m 45s hadoop-common in the patch passed with JDK v1.7.0_85. +1 asflicense 0m 26s Patch does not generate ASF License warnings. 81m 16s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.ipc.TestDecayRpcScheduler Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Issue HADOOP-11601 GITHUB PR https://github.com/apache/hadoop/pull/50 Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle uname Linux 599761bf506a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / a6f20d8 findbugs v3.0.0 whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/8120/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8120/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8120/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8120/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Max memory used 75MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8120/console Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8120/console This message was automatically generated.
          Hide
          githubbot ASF GitHub Bot added a comment - - edited

          GitHub user steveloughran opened a pull request:

          <cut>

          Show
          githubbot ASF GitHub Bot added a comment - - edited GitHub user steveloughran opened a pull request: <cut>
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Marking HADOOP-11584 as related, but no "depends-on" relation. That patch has its own tests, which can be pulled up here afterwards

          Show
          stevel@apache.org Steve Loughran added a comment - Marking HADOOP-11584 as related, but no "depends-on" relation. That patch has its own tests, which can be pulled up here afterwards
          Hide
          stevel@apache.org Steve Loughran added a comment -

          There's one thing I quickly want to clarify with the HDFS list: should the blocksize of a file be min(filesize, defaultblocksize)

          Show
          stevel@apache.org Steve Loughran added a comment - There's one thing I quickly want to clarify with the HDFS list: should the blocksize of a file be min(filesize, defaultblocksize)
          Hide
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12699081/HADOOP-11601-001.patch
          against trunk revision ab0b958.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5714//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5714//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12699081/HADOOP-11601-001.patch against trunk revision ab0b958. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5714//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5714//console This message is automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Patch -001.

          1. Updates spec to say MUST be > 0 for a non-empty file
          2. MUST BE >=0 for empty file
          3. tests for these conditions
          Show
          stevel@apache.org Steve Loughran added a comment - Patch -001. Updates spec to say MUST be > 0 for a non-empty file MUST BE >=0 for empty file tests for these conditions

            People

            • Assignee:
              stevel@apache.org Steve Loughran
              Reporter:
              stevel@apache.org Steve Loughran
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 1h
                1h
                Remaining:
                Remaining Estimate - 1h
                1h
                Logged:
                Time Spent - Not Specified
                Not Specified

                  Development