Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9534

Add CLI command to clear storage policy from a path.

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: tools
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The hdfs storagepolicies command has sub-commands for -setStoragePolicy and -getStoragePolicy on a path. However, there is no -removeStoragePolicy to remove a previously set storage policy on a path.

      1. HDFS-9534.001.patch
        24 kB
        Xiaobing Zhou
      2. HDFS-9534.002.patch
        28 kB
        Xiaobing Zhou
      3. HDFS-9534.003.patch
        36 kB
        Xiaobing Zhou
      4. HDFS-9534.004.patch
        38 kB
        Xiaobing Zhou

        Issue Links

          Activity

          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Cherry-picked to 2.8.0.

          Show
          arpitagarwal Arpit Agarwal added a comment - Cherry-picked to 2.8.0.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #9415 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9415/)
          HDFS-9534. Add CLI command to clear storage policy from a path. (arp: rev 27941a1811831e0f2144a2f463d807755cd850b2)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestStoragePolicyCommands.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java
          • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java
          • hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/ClientNamenodeProtocol.proto
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirAttrOp.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FilterFileSystem.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/StoragePolicyAdmin.java
          • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestApplyingStoragePolicy.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ArchivalStorage.md
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestHarFileSystem.java
          • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
          • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolServerSideTranslatorPB.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9415 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9415/ ) HDFS-9534 . Add CLI command to clear storage policy from a path. (arp: rev 27941a1811831e0f2144a2f463d807755cd850b2) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestStoragePolicyCommands.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/ClientNamenodeProtocol.proto hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirAttrOp.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FilterFileSystem.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/StoragePolicyAdmin.java hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestApplyingStoragePolicy.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ArchivalStorage.md hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestHarFileSystem.java hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolServerSideTranslatorPB.java
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Committed to 2.9.0. Thanks for the contribution Xiaobing Zhou and thanks to everyone who reviewed this and provided feedback.

          Show
          arpitagarwal Arpit Agarwal added a comment - Committed to 2.9.0. Thanks for the contribution Xiaobing Zhou and thanks to everyone who reviewed this and provided feedback.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          +1, fairly sure the unit test failures are unrelated. Nice work addressing all the feedback. Thanks for filing HDFS-9894 to address the follow on work.

          I will commit this shortly.

          Show
          arpitagarwal Arpit Agarwal added a comment - +1, fairly sure the unit test failures are unrelated. Nice work addressing all the feedback. Thanks for filing HDFS-9894 to address the follow on work. I will commit this shortly.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 48s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 0m 49s Maven dependency ordering for branch
          +1 mvninstall 11m 9s trunk passed
          +1 compile 14m 28s trunk passed with JDK v1.8.0_72
          +1 compile 11m 15s trunk passed with JDK v1.7.0_95
          +1 checkstyle 1m 40s trunk passed
          +1 mvnsite 3m 44s trunk passed
          +1 mvneclipse 1m 0s trunk passed
          +1 findbugs 6m 56s trunk passed
          +1 javadoc 3m 44s trunk passed with JDK v1.8.0_72
          +1 javadoc 4m 19s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 19s Maven dependency ordering for patch
          +1 mvninstall 2m 52s the patch passed
          +1 compile 13m 46s the patch passed with JDK v1.8.0_72
          -1 cc 16m 57s root-jdk1.8.0_72 with JDK v1.8.0_72 generated 1 new + 9 unchanged - 1 fixed = 10 total (was 10)
          +1 cc 13m 46s the patch passed
          +1 javac 13m 46s the patch passed
          +1 compile 12m 22s the patch passed with JDK v1.7.0_95
          +1 cc 12m 22s the patch passed
          +1 javac 12m 22s the patch passed
          +1 checkstyle 1m 54s root: patch generated 0 new + 560 unchanged - 1 fixed = 560 total (was 561)
          +1 mvnsite 4m 9s the patch passed
          +1 mvneclipse 1m 8s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 8m 23s the patch passed
          +1 javadoc 3m 53s the patch passed with JDK v1.8.0_72
          +1 javadoc 4m 53s the patch passed with JDK v1.7.0_95
          -1 unit 14m 12s hadoop-common in the patch failed with JDK v1.8.0_72.
          +1 unit 1m 49s hadoop-hdfs-client in the patch passed with JDK v1.8.0_72.
          -1 unit 106m 59s hadoop-hdfs in the patch failed with JDK v1.8.0_72.
          -1 unit 11m 5s hadoop-common in the patch failed with JDK v1.7.0_95.
          +1 unit 1m 17s hadoop-hdfs-client in the patch passed with JDK v1.7.0_95.
          -1 unit 97m 20s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 41s Patch does not generate ASF License warnings.
          349m 14s



          Reason Tests
          JDK v1.8.0_72 Failed junit tests hadoop.security.ssl.TestReloadingX509TrustManager
            hadoop.fs.shell.find.TestPrint
            hadoop.fs.shell.find.TestPrint0
            hadoop.fs.shell.find.TestIname
            hadoop.fs.shell.find.TestName
            hadoop.hdfs.web.TestWebHdfsTimeouts
            hadoop.hdfs.TestPersistBlocks
            hadoop.hdfs.security.TestDelegationTokenForProxyUser
            hadoop.hdfs.TestFileAppend
            hadoop.hdfs.server.datanode.TestBlockReplacement
            hadoop.hdfs.TestEncryptionZonesWithKMS
            hadoop.hdfs.server.datanode.TestDirectoryScanner
          JDK v1.7.0_95 Failed junit tests hadoop.fs.shell.find.TestPrint
            hadoop.fs.shell.find.TestPrint0
            hadoop.fs.shell.find.TestIname
            hadoop.hdfs.server.namenode.ha.TestEditLogTailer
            hadoop.hdfs.server.datanode.TestBlockReplacement
            hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot
            hadoop.hdfs.server.datanode.TestDirectoryScanner



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12790983/HDFS-9534.004.patch
          JIRA Issue HDFS-9534
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
          uname Linux 901bbb432a35 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 67880cc
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          cc root-jdk1.8.0_72: https://builds.apache.org/job/PreCommit-HDFS-Build/14691/artifact/patchprocess/diff-compile-cc-root-jdk1.8.0_72.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14691/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_72.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14691/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14691/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14691/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14691/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14691/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14691/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14691/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14691/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14691/console
          Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 48s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 0m 49s Maven dependency ordering for branch +1 mvninstall 11m 9s trunk passed +1 compile 14m 28s trunk passed with JDK v1.8.0_72 +1 compile 11m 15s trunk passed with JDK v1.7.0_95 +1 checkstyle 1m 40s trunk passed +1 mvnsite 3m 44s trunk passed +1 mvneclipse 1m 0s trunk passed +1 findbugs 6m 56s trunk passed +1 javadoc 3m 44s trunk passed with JDK v1.8.0_72 +1 javadoc 4m 19s trunk passed with JDK v1.7.0_95 0 mvndep 0m 19s Maven dependency ordering for patch +1 mvninstall 2m 52s the patch passed +1 compile 13m 46s the patch passed with JDK v1.8.0_72 -1 cc 16m 57s root-jdk1.8.0_72 with JDK v1.8.0_72 generated 1 new + 9 unchanged - 1 fixed = 10 total (was 10) +1 cc 13m 46s the patch passed +1 javac 13m 46s the patch passed +1 compile 12m 22s the patch passed with JDK v1.7.0_95 +1 cc 12m 22s the patch passed +1 javac 12m 22s the patch passed +1 checkstyle 1m 54s root: patch generated 0 new + 560 unchanged - 1 fixed = 560 total (was 561) +1 mvnsite 4m 9s the patch passed +1 mvneclipse 1m 8s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 8m 23s the patch passed +1 javadoc 3m 53s the patch passed with JDK v1.8.0_72 +1 javadoc 4m 53s the patch passed with JDK v1.7.0_95 -1 unit 14m 12s hadoop-common in the patch failed with JDK v1.8.0_72. +1 unit 1m 49s hadoop-hdfs-client in the patch passed with JDK v1.8.0_72. -1 unit 106m 59s hadoop-hdfs in the patch failed with JDK v1.8.0_72. -1 unit 11m 5s hadoop-common in the patch failed with JDK v1.7.0_95. +1 unit 1m 17s hadoop-hdfs-client in the patch passed with JDK v1.7.0_95. -1 unit 97m 20s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 41s Patch does not generate ASF License warnings. 349m 14s Reason Tests JDK v1.8.0_72 Failed junit tests hadoop.security.ssl.TestReloadingX509TrustManager   hadoop.fs.shell.find.TestPrint   hadoop.fs.shell.find.TestPrint0   hadoop.fs.shell.find.TestIname   hadoop.fs.shell.find.TestName   hadoop.hdfs.web.TestWebHdfsTimeouts   hadoop.hdfs.TestPersistBlocks   hadoop.hdfs.security.TestDelegationTokenForProxyUser   hadoop.hdfs.TestFileAppend   hadoop.hdfs.server.datanode.TestBlockReplacement   hadoop.hdfs.TestEncryptionZonesWithKMS   hadoop.hdfs.server.datanode.TestDirectoryScanner JDK v1.7.0_95 Failed junit tests hadoop.fs.shell.find.TestPrint   hadoop.fs.shell.find.TestPrint0   hadoop.fs.shell.find.TestIname   hadoop.hdfs.server.namenode.ha.TestEditLogTailer   hadoop.hdfs.server.datanode.TestBlockReplacement   hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot   hadoop.hdfs.server.datanode.TestDirectoryScanner Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12790983/HDFS-9534.004.patch JIRA Issue HDFS-9534 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux 901bbb432a35 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 67880cc Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 cc root-jdk1.8.0_72: https://builds.apache.org/job/PreCommit-HDFS-Build/14691/artifact/patchprocess/diff-compile-cc-root-jdk1.8.0_72.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14691/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_72.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14691/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14691/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14691/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14691/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14691/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14691/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14691/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14691/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14691/console Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xiaobingo Xiaobing Zhou added a comment -

          004 fixed UT failures. Thanks for review Arpit Agarwal Chris Nauroth.

          Show
          xiaobingo Xiaobing Zhou added a comment - 004 fixed UT failures. Thanks for review Arpit Agarwal Chris Nauroth .
          Hide
          cnauroth Chris Nauroth added a comment -

          +1 from me for patch v003. Thank you, Xiaobing Zhou!

          Show
          cnauroth Chris Nauroth added a comment - +1 from me for patch v003. Thank you, Xiaobing Zhou !
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          The TestHarFileSystem and TestFilterFileSystem failures are related to the patch. Let's fix them and also verify that the remaining failures are unrelated by running them locally.

          2016-02-27 04:34:44,055 ERROR fs.FileSystem (TestFilterFileSystem.java:testFilterFileSystem(160)) - FilterFileSystem MUST implement public void org.apache.hadoop.fs.FileSystem.unsetStoragePolicy(org.apache.hadoop.fs.Path) throws java.io.IOException
          

          +1 once the failures are addressed. Also let's add the new API to FileContext and AbstractFileSystem and derivatives. Okay to do so in a follow up jira.

          Hi Jing Zhao, Chris Nauroth do you have any additional comments?

          Show
          arpitagarwal Arpit Agarwal added a comment - The TestHarFileSystem and TestFilterFileSystem failures are related to the patch. Let's fix them and also verify that the remaining failures are unrelated by running them locally. 2016-02-27 04:34:44,055 ERROR fs.FileSystem (TestFilterFileSystem.java:testFilterFileSystem(160)) - FilterFileSystem MUST implement public void org.apache.hadoop.fs.FileSystem.unsetStoragePolicy(org.apache.hadoop.fs.Path) throws java.io.IOException +1 once the failures are addressed. Also let's add the new API to FileContext and AbstractFileSystem and derivatives. Okay to do so in a follow up jira. Hi Jing Zhao , Chris Nauroth do you have any additional comments?
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 12m 15s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 1m 11s Maven dependency ordering for branch
          +1 mvninstall 7m 7s trunk passed
          +1 compile 7m 40s trunk passed with JDK v1.8.0_72
          +1 compile 7m 25s trunk passed with JDK v1.7.0_95
          +1 checkstyle 1m 9s trunk passed
          +1 mvnsite 2m 37s trunk passed
          +1 mvneclipse 0m 39s trunk passed
          +1 findbugs 5m 14s trunk passed
          +1 javadoc 2m 37s trunk passed with JDK v1.8.0_72
          +1 javadoc 3m 22s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 2m 3s the patch passed
          +1 compile 7m 48s the patch passed with JDK v1.8.0_72
          +1 cc 7m 48s the patch passed
          +1 javac 7m 48s the patch passed
          +1 compile 7m 17s the patch passed with JDK v1.7.0_95
          +1 cc 7m 17s the patch passed
          +1 javac 7m 17s the patch passed
          +1 checkstyle 1m 9s root: patch generated 0 new + 546 unchanged - 1 fixed = 546 total (was 547)
          +1 mvnsite 2m 32s the patch passed
          +1 mvneclipse 0m 37s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 5m 56s the patch passed
          +1 javadoc 2m 35s the patch passed with JDK v1.8.0_72
          +1 javadoc 3m 17s the patch passed with JDK v1.7.0_95
          -1 unit 8m 0s hadoop-common in the patch failed with JDK v1.8.0_72.
          +1 unit 1m 1s hadoop-hdfs-client in the patch passed with JDK v1.8.0_72.
          -1 unit 70m 51s hadoop-hdfs in the patch failed with JDK v1.8.0_72.
          -1 unit 8m 8s hadoop-common in the patch failed with JDK v1.7.0_95.
          +1 unit 1m 2s hadoop-hdfs-client in the patch passed with JDK v1.7.0_95.
          -1 unit 56m 48s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 23s Patch does not generate ASF License warnings.
          232m 30s



          Reason Tests
          JDK v1.8.0_72 Failed junit tests hadoop.fs.TestHarFileSystem
            hadoop.fs.TestFilterFileSystem
            hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation
            hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits
            hadoop.hdfs.TestFileAppend
            hadoop.hdfs.TestRenameWhileOpen
            hadoop.hdfs.server.blockmanagement.TestReconstructStripedBlocksWithRackAwareness
          JDK v1.7.0_95 Failed junit tests hadoop.fs.TestHarFileSystem
            hadoop.fs.TestFilterFileSystem
            hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency
            hadoop.metrics2.sink.TestRollingFileSystemSinkWithSecureHdfs
            hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12790266/HDFS-9534.003.patch
          JIRA Issue HDFS-9534
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
          uname Linux 6f0fed1c80c5 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / d1d4e16
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14636/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_72.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14636/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14636/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14636/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14636/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14636/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14636/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14636/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14636/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14636/console
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 12m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 1m 11s Maven dependency ordering for branch +1 mvninstall 7m 7s trunk passed +1 compile 7m 40s trunk passed with JDK v1.8.0_72 +1 compile 7m 25s trunk passed with JDK v1.7.0_95 +1 checkstyle 1m 9s trunk passed +1 mvnsite 2m 37s trunk passed +1 mvneclipse 0m 39s trunk passed +1 findbugs 5m 14s trunk passed +1 javadoc 2m 37s trunk passed with JDK v1.8.0_72 +1 javadoc 3m 22s trunk passed with JDK v1.7.0_95 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 2m 3s the patch passed +1 compile 7m 48s the patch passed with JDK v1.8.0_72 +1 cc 7m 48s the patch passed +1 javac 7m 48s the patch passed +1 compile 7m 17s the patch passed with JDK v1.7.0_95 +1 cc 7m 17s the patch passed +1 javac 7m 17s the patch passed +1 checkstyle 1m 9s root: patch generated 0 new + 546 unchanged - 1 fixed = 546 total (was 547) +1 mvnsite 2m 32s the patch passed +1 mvneclipse 0m 37s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 5m 56s the patch passed +1 javadoc 2m 35s the patch passed with JDK v1.8.0_72 +1 javadoc 3m 17s the patch passed with JDK v1.7.0_95 -1 unit 8m 0s hadoop-common in the patch failed with JDK v1.8.0_72. +1 unit 1m 1s hadoop-hdfs-client in the patch passed with JDK v1.8.0_72. -1 unit 70m 51s hadoop-hdfs in the patch failed with JDK v1.8.0_72. -1 unit 8m 8s hadoop-common in the patch failed with JDK v1.7.0_95. +1 unit 1m 2s hadoop-hdfs-client in the patch passed with JDK v1.7.0_95. -1 unit 56m 48s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 23s Patch does not generate ASF License warnings. 232m 30s Reason Tests JDK v1.8.0_72 Failed junit tests hadoop.fs.TestHarFileSystem   hadoop.fs.TestFilterFileSystem   hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation   hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits   hadoop.hdfs.TestFileAppend   hadoop.hdfs.TestRenameWhileOpen   hadoop.hdfs.server.blockmanagement.TestReconstructStripedBlocksWithRackAwareness JDK v1.7.0_95 Failed junit tests hadoop.fs.TestHarFileSystem   hadoop.fs.TestFilterFileSystem   hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency   hadoop.metrics2.sink.TestRollingFileSystemSinkWithSecureHdfs   hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12790266/HDFS-9534.003.patch JIRA Issue HDFS-9534 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux 6f0fed1c80c5 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d1d4e16 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/14636/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_72.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14636/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14636/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14636/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14636/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14636/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14636/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14636/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14636/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14636/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xiaobingo Xiaobing Zhou added a comment -

          Thanks Arpit Agarwal. 003 patch addressed 1/2/3/4 except 'one check that unsetPolicy is correctly applied from the edit log on restart'. The reason is that the unset reused setStoragePolicy implementation, whatever edit log works for set works for unset too. Do you agree?

          Show
          xiaobingo Xiaobing Zhou added a comment - Thanks Arpit Agarwal . 003 patch addressed 1/2/3/4 except 'one check that unsetPolicy is correctly applied from the edit log on restart'. The reason is that the unset reused setStoragePolicy implementation, whatever edit log works for set works for unset too. Do you agree?
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Thanks for updating the patch Xiaobing Zhou. My comments on the v2 patch:

          1. I understand why you instantiate a dummy policy in FsDirAttrOp#unsetStoragePolicy to avoid the extra call to getPolicy. Instead can we just pass the storage policy ID to setStoragePolicy? Then we need no change to FsEditLogLoader.java. We will need to fix the isCopyOnCreateFile check in unprotectedSetStoragePolicy.
          2. Update storage policy docs to add a section for unset policy (see ArchivalStorage.md). Also mention here that after the unset command the storage policy of the nearest ancestor will apply and if there is no policy on any ancestor then the default storage policy will apply.
          3. We need more tests, for one check that unsetPolicy is correctly applied from the edit log on restart. I think you have a test case for nested storage policies. Could you split up testUnsetStoragePolicy into multiple smaller tests to make it clearer what is being tested?
          4. Also we should separate the command test from the RPC handling tests. The command tests can check for a couple of simple cases.
          Show
          arpitagarwal Arpit Agarwal added a comment - Thanks for updating the patch Xiaobing Zhou . My comments on the v2 patch: I understand why you instantiate a dummy policy in FsDirAttrOp#unsetStoragePolicy to avoid the extra call to getPolicy. Instead can we just pass the storage policy ID to setStoragePolicy ? Then we need no change to FsEditLogLoader.java. We will need to fix the isCopyOnCreateFile check in unprotectedSetStoragePolicy . Update storage policy docs to add a section for unset policy (see ArchivalStorage.md). Also mention here that after the unset command the storage policy of the nearest ancestor will apply and if there is no policy on any ancestor then the default storage policy will apply. We need more tests, for one check that unsetPolicy is correctly applied from the edit log on restart. I think you have a test case for nested storage policies. Could you split up testUnsetStoragePolicy into multiple smaller tests to make it clearer what is being tested? Also we should separate the command test from the RPC handling tests. The command tests can check for a couple of simple cases.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 11s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 1m 3s Maven dependency ordering for branch
          +1 mvninstall 6m 52s trunk passed
          +1 compile 5m 49s trunk passed with JDK v1.8.0_72
          +1 compile 6m 38s trunk passed with JDK v1.7.0_95
          +1 checkstyle 1m 13s trunk passed
          +1 mvnsite 2m 35s trunk passed
          +1 mvneclipse 0m 41s trunk passed
          +1 findbugs 5m 8s trunk passed
          +1 javadoc 2m 19s trunk passed with JDK v1.8.0_72
          +1 javadoc 3m 11s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 27s Maven dependency ordering for patch
          -1 mvninstall 0m 31s hadoop-hdfs-client in the patch failed.
          -1 mvninstall 0m 22s hadoop-hdfs in the patch failed.
          +1 compile 5m 43s the patch passed with JDK v1.8.0_72
          +1 cc 5m 43s the patch passed
          +1 javac 5m 43s the patch passed
          +1 compile 6m 40s the patch passed with JDK v1.7.0_95
          +1 cc 6m 40s the patch passed
          +1 javac 6m 40s the patch passed
          -1 checkstyle 1m 15s root: patch generated 4 new + 638 unchanged - 2 fixed = 642 total (was 640)
          -1 mvnsite 0m 25s hadoop-hdfs in the patch failed.
          +1 mvneclipse 0m 40s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          -1 findbugs 0m 23s hadoop-hdfs in the patch failed.
          -1 javadoc 4m 32s hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72 with JDK v1.8.0_72 generated 5 new + 7 unchanged - 0 fixed = 12 total (was 7)
          +1 javadoc 2m 18s the patch passed with JDK v1.8.0_72
          -1 javadoc 8m 4s hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95 with JDK v1.7.0_95 generated 5 new + 7 unchanged - 0 fixed = 12 total (was 7)
          +1 javadoc 3m 12s the patch passed with JDK v1.7.0_95
          -1 unit 16m 29s hadoop-common in the patch failed with JDK v1.8.0_72.
          +1 unit 0m 52s hadoop-hdfs-client in the patch passed with JDK v1.8.0_72.
          -1 unit 0m 23s hadoop-hdfs in the patch failed with JDK v1.8.0_72.
          -1 unit 6m 39s hadoop-common in the patch failed with JDK v1.7.0_95.
          +1 unit 0m 54s hadoop-hdfs-client in the patch passed with JDK v1.7.0_95.
          -1 unit 0m 22s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 23s Patch does not generate ASF License warnings.
          91m 4s



          Reason Tests
          JDK v1.8.0_72 Failed junit tests hadoop.fs.TestFilterFileSystem
            hadoop.fs.TestHarFileSystem
          JDK v1.8.0_72 Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle
          JDK v1.7.0_95 Failed junit tests hadoop.fs.TestFilterFileSystem
            hadoop.fs.TestHarFileSystem
            hadoop.net.TestClusterTopology
            hadoop.security.ssl.TestReloadingX509TrustManager



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12787360/HDFS-9534.002.patch
          JIRA Issue HDFS-9534
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
          uname Linux 256b2dc2d137 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / aeb13ef
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          mvninstall https://builds.apache.org/job/PreCommit-HDFS-Build/14447/artifact/patchprocess/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs-client.txt
          mvninstall https://builds.apache.org/job/PreCommit-HDFS-Build/14447/artifact/patchprocess/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14447/artifact/patchprocess/diff-checkstyle-root.txt
          mvnsite https://builds.apache.org/job/PreCommit-HDFS-Build/14447/artifact/patchprocess/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/14447/artifact/patchprocess/patch-findbugs-hadoop-hdfs-project_hadoop-hdfs.txt
          javadoc hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72: https://builds.apache.org/job/PreCommit-HDFS-Build/14447/artifact/patchprocess/diff-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt
          javadoc hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95: https://builds.apache.org/job/PreCommit-HDFS-Build/14447/artifact/patchprocess/diff-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14447/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_72.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14447/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14447/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14447/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14447/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14447/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14447/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: .
          Max memory used 76MB
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14447/console
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 1m 3s Maven dependency ordering for branch +1 mvninstall 6m 52s trunk passed +1 compile 5m 49s trunk passed with JDK v1.8.0_72 +1 compile 6m 38s trunk passed with JDK v1.7.0_95 +1 checkstyle 1m 13s trunk passed +1 mvnsite 2m 35s trunk passed +1 mvneclipse 0m 41s trunk passed +1 findbugs 5m 8s trunk passed +1 javadoc 2m 19s trunk passed with JDK v1.8.0_72 +1 javadoc 3m 11s trunk passed with JDK v1.7.0_95 0 mvndep 0m 27s Maven dependency ordering for patch -1 mvninstall 0m 31s hadoop-hdfs-client in the patch failed. -1 mvninstall 0m 22s hadoop-hdfs in the patch failed. +1 compile 5m 43s the patch passed with JDK v1.8.0_72 +1 cc 5m 43s the patch passed +1 javac 5m 43s the patch passed +1 compile 6m 40s the patch passed with JDK v1.7.0_95 +1 cc 6m 40s the patch passed +1 javac 6m 40s the patch passed -1 checkstyle 1m 15s root: patch generated 4 new + 638 unchanged - 2 fixed = 642 total (was 640) -1 mvnsite 0m 25s hadoop-hdfs in the patch failed. +1 mvneclipse 0m 40s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. -1 findbugs 0m 23s hadoop-hdfs in the patch failed. -1 javadoc 4m 32s hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72 with JDK v1.8.0_72 generated 5 new + 7 unchanged - 0 fixed = 12 total (was 7) +1 javadoc 2m 18s the patch passed with JDK v1.8.0_72 -1 javadoc 8m 4s hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95 with JDK v1.7.0_95 generated 5 new + 7 unchanged - 0 fixed = 12 total (was 7) +1 javadoc 3m 12s the patch passed with JDK v1.7.0_95 -1 unit 16m 29s hadoop-common in the patch failed with JDK v1.8.0_72. +1 unit 0m 52s hadoop-hdfs-client in the patch passed with JDK v1.8.0_72. -1 unit 0m 23s hadoop-hdfs in the patch failed with JDK v1.8.0_72. -1 unit 6m 39s hadoop-common in the patch failed with JDK v1.7.0_95. +1 unit 0m 54s hadoop-hdfs-client in the patch passed with JDK v1.7.0_95. -1 unit 0m 22s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 23s Patch does not generate ASF License warnings. 91m 4s Reason Tests JDK v1.8.0_72 Failed junit tests hadoop.fs.TestFilterFileSystem   hadoop.fs.TestHarFileSystem JDK v1.8.0_72 Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle JDK v1.7.0_95 Failed junit tests hadoop.fs.TestFilterFileSystem   hadoop.fs.TestHarFileSystem   hadoop.net.TestClusterTopology   hadoop.security.ssl.TestReloadingX509TrustManager Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12787360/HDFS-9534.002.patch JIRA Issue HDFS-9534 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux 256b2dc2d137 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / aeb13ef Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 mvninstall https://builds.apache.org/job/PreCommit-HDFS-Build/14447/artifact/patchprocess/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs-client.txt mvninstall https://builds.apache.org/job/PreCommit-HDFS-Build/14447/artifact/patchprocess/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14447/artifact/patchprocess/diff-checkstyle-root.txt mvnsite https://builds.apache.org/job/PreCommit-HDFS-Build/14447/artifact/patchprocess/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/14447/artifact/patchprocess/patch-findbugs-hadoop-hdfs-project_hadoop-hdfs.txt javadoc hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72: https://builds.apache.org/job/PreCommit-HDFS-Build/14447/artifact/patchprocess/diff-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt javadoc hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95: https://builds.apache.org/job/PreCommit-HDFS-Build/14447/artifact/patchprocess/diff-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14447/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_72.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14447/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14447/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14447/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14447/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14447/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14447/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . Max memory used 76MB Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14447/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xiaobingo Xiaobing Zhou added a comment -

          Thanks Chris Nauroth Walter Su Jing Zhao Arpit Agarwal Vinayakumar B all for the reviews. I posted patch V002 that addressed your comments. Please kindly review.

          Show
          xiaobingo Xiaobing Zhou added a comment - Thanks Chris Nauroth Walter Su Jing Zhao Arpit Agarwal Vinayakumar B all for the reviews. I posted patch V002 that addressed your comments. Please kindly review.
          Hide
          jingzhao Jing Zhao added a comment -

          Yeah, for the semantic of the new op, I agree with the comments from Chris Nauroth and Arpit Agarwal. +1 to call the op (reset/unset)StoragePolicy.

          Show
          jingzhao Jing Zhao added a comment - Yeah, for the semantic of the new op, I agree with the comments from Chris Nauroth and Arpit Agarwal . +1 to call the op (reset/unset)StoragePolicy.
          Hide
          vinayrpet Vinayakumar B added a comment -

          1.

             /**
          +   * Remove the storage policy st for a given file or directory,
          +   * and set it to unspecified storage policy.
          +   * @param src file or directory path.
          +   * @throws IOException
          +   */
          

          There is a nit, storage policy st for a.
          Also I think, set it to unspecified storage policy this sentence is not required. As chris said, let it be transparent to user.
          Similar changes in other places as well.

          2. In FSDirAttrOp#setDirStoragePolicy, in case of removal, we should actually remove the xattr, instead of setting with UNSPECIFIED. It will be extra xattr, which adds to memory.
          May be following would work.

          @@ -485,11 +489,18 @@ private static void setDirStoragePolicy(
                 int latestSnapshotId) throws IOException {
               List<XAttr> existingXAttrs = XAttrStorage.readINodeXAttrs(inode);
               XAttr xAttr = BlockStoragePolicySuite.buildXAttr(policyId);
          -    List<XAttr> newXAttrs = FSDirXAttrOp.setINodeXAttrs(fsd, existingXAttrs,
          -                                                        Arrays.asList(xAttr),
          -                                                        EnumSet.of(
          -                                                            XAttrSetFlag.CREATE,
          -                                                            XAttrSetFlag.REPLACE));
          +    List<XAttr> newXAttrs = null;
          +    if (policyId == BLOCK_STORAGE_POLICY_ID_UNSPECIFIED) {
          +      List<XAttr> toRemove = Lists.newArrayList();
          +      toRemove.add(xAttr);
          +      List<XAttr> removed = Lists.newArrayList();
          +      newXAttrs =
          +          FSDirXAttrOp.filterINodeXAttrs(existingXAttrs, toRemove, removed);
          +    } else {
          +      newXAttrs =
          +          FSDirXAttrOp.setINodeXAttrs(fsd, existingXAttrs, Arrays.asList(xAttr),
          +              EnumSet.of(XAttrSetFlag.CREATE, XAttrSetFlag.REPLACE));
          +    }
               XAttrStorage.updateINodeXAttrs(inode, newXAttrs, latestSnapshotId);
             }
          

          Agree?

          Show
          vinayrpet Vinayakumar B added a comment - 1. /** + * Remove the storage policy st for a given file or directory, + * and set it to unspecified storage policy. + * @param src file or directory path. + * @ throws IOException + */ There is a nit, storage policy st for a . Also I think, set it to unspecified storage policy this sentence is not required. As chris said, let it be transparent to user. Similar changes in other places as well. 2. In FSDirAttrOp#setDirStoragePolicy , in case of removal, we should actually remove the xattr, instead of setting with UNSPECIFIED. It will be extra xattr, which adds to memory. May be following would work. @@ -485,11 +489,18 @@ private static void setDirStoragePolicy( int latestSnapshotId) throws IOException { List<XAttr> existingXAttrs = XAttrStorage.readINodeXAttrs(inode); XAttr xAttr = BlockStoragePolicySuite.buildXAttr(policyId); - List<XAttr> newXAttrs = FSDirXAttrOp.setINodeXAttrs(fsd, existingXAttrs, - Arrays.asList(xAttr), - EnumSet.of( - XAttrSetFlag.CREATE, - XAttrSetFlag.REPLACE)); + List<XAttr> newXAttrs = null ; + if (policyId == BLOCK_STORAGE_POLICY_ID_UNSPECIFIED) { + List<XAttr> toRemove = Lists.newArrayList(); + toRemove.add(xAttr); + List<XAttr> removed = Lists.newArrayList(); + newXAttrs = + FSDirXAttrOp.filterINodeXAttrs(existingXAttrs, toRemove, removed); + } else { + newXAttrs = + FSDirXAttrOp.setINodeXAttrs(fsd, existingXAttrs, Arrays.asList(xAttr), + EnumSet.of(XAttrSetFlag.CREATE, XAttrSetFlag.REPLACE)); + } XAttrStorage.updateINodeXAttrs(inode, newXAttrs, latestSnapshotId); } Agree?
          Hide
          cnauroth Chris Nauroth added a comment -

          Also, would you please update documentation to mention the new command? Thanks!

          Show
          cnauroth Chris Nauroth added a comment - Also, would you please update documentation to mention the new command? Thanks!
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          I had been thinking that the remove operation applies just to the specific named inode, and it would not directly alter anything lower in the tree. (Of course, it's possible that removing the storage policy from an inode causes its children to start inheriting different storage policy settings from higher in the tree.)

          Yes I had the same assumption. An unspecified storage policy would result in checking policies on ancestors. We should add a couple of unit tests for nested policy removal.

          Also it's better to call it resetStoragePolicy or clearStoragePolicy rather than removeStoragePolicy. That makes it explicit the operation is idempotent.

          Show
          arpitagarwal Arpit Agarwal added a comment - I had been thinking that the remove operation applies just to the specific named inode, and it would not directly alter anything lower in the tree. (Of course, it's possible that removing the storage policy from an inode causes its children to start inheriting different storage policy settings from higher in the tree.) Yes I had the same assumption. An unspecified storage policy would result in checking policies on ancestors. We should add a couple of unit tests for nested policy removal. Also it's better to call it resetStoragePolicy or clearStoragePolicy rather than removeStoragePolicy . That makes it explicit the operation is idempotent.
          Hide
          cnauroth Chris Nauroth added a comment -

          Here are more details on the user scenario that prompted me to file this issue. Someone set a storage policy on a directory by mistake. They were looking for a way to remove it, but we don't currently have an easy way to do that. Current workarounds are either to create a whole new directory and move everything under that new directory, or explicitly set it back to the parent directory's effective storage policy.

          Note that the latter workaround is not quite the same semantics as removing the storage policy. In that case, storage policy changes higher in the tree would no longer propagate automatically down to the child, because the child would still have an explicit policy of its own acting as an override. I think the effect of this new command should be to revert the storage policy on the inode, not explicitly override it to the default, so that it restores the previous behavior of that inode inheriting its policy from higher in the tree. That would satisfy the "undo" requirement for this user.

          Thus I do not think we need an explicit policy named "UNSPECIFIED".

          I agree. The patch as currently written would expose UNSPECIFIED in the API footprint, so that callers could write code like fileSystem.setStoragePolicy(new Path("/foo"), "UNSPECIFIED");. I don't think that's desired. Instead, I've always thought of UNSPECIFIED as an internal implementation detail. It appears that the only reason the name was added to BlockStoragePolicySuite is so that it would be easier to reuse existing methods in FSDirAttrOp. (Let me know if I'm missing something.) I'd prefer to avoid exposing UNSPECIFIED to clients, even if it means we need a bit of extra code in FSDirAttrOp to support setting the policy without access to a name defined in BlockStoragePolicySuite.

          If we have set a storage policy on "/foo", then should we allow the user to apply the command on "/foo/bar" if bar is not associated with any explicit policy?

          I think it's OK to let this happen and return success from the RPC, but it's effectively a no-op. That makes idempotence easy.

          Or what if we have nested policy settings?

          I had been thinking that the remove operation applies just to the specific named inode, and it would not directly alter anything lower in the tree. (Of course, it's possible that removing the storage policy from an inode causes its children to start inheriting different storage policy settings from higher in the tree.)

          Show
          cnauroth Chris Nauroth added a comment - Here are more details on the user scenario that prompted me to file this issue. Someone set a storage policy on a directory by mistake. They were looking for a way to remove it, but we don't currently have an easy way to do that. Current workarounds are either to create a whole new directory and move everything under that new directory, or explicitly set it back to the parent directory's effective storage policy. Note that the latter workaround is not quite the same semantics as removing the storage policy. In that case, storage policy changes higher in the tree would no longer propagate automatically down to the child, because the child would still have an explicit policy of its own acting as an override. I think the effect of this new command should be to revert the storage policy on the inode, not explicitly override it to the default, so that it restores the previous behavior of that inode inheriting its policy from higher in the tree. That would satisfy the "undo" requirement for this user. Thus I do not think we need an explicit policy named "UNSPECIFIED". I agree. The patch as currently written would expose UNSPECIFIED in the API footprint, so that callers could write code like fileSystem.setStoragePolicy(new Path("/foo"), "UNSPECIFIED"); . I don't think that's desired. Instead, I've always thought of UNSPECIFIED as an internal implementation detail. It appears that the only reason the name was added to BlockStoragePolicySuite is so that it would be easier to reuse existing methods in FSDirAttrOp . (Let me know if I'm missing something.) I'd prefer to avoid exposing UNSPECIFIED to clients, even if it means we need a bit of extra code in FSDirAttrOp to support setting the policy without access to a name defined in BlockStoragePolicySuite . If we have set a storage policy on "/foo", then should we allow the user to apply the command on "/foo/bar" if bar is not associated with any explicit policy? I think it's OK to let this happen and return success from the RPC, but it's effectively a no-op. That makes idempotence easy. Or what if we have nested policy settings? I had been thinking that the remove operation applies just to the specific named inode, and it would not directly alter anything lower in the tree. (Of course, it's possible that removing the storage policy from an inode causes its children to start inheriting different storage policy settings from higher in the tree.)
          Hide
          jingzhao Jing Zhao added a comment -

          Currently unspecified policy means using the default policy. Thus I do not think we need an explicit policy named "UNSPECIFIED".

          In the meanwhile, how should the semantic of the remove op look like? If we have set a storage policy on "/foo", then should we allow the user to apply the command on "/foo/bar" if bar is not associated with any explicit policy? Or what if we have nested policy settings? I think we may need to list all the scenarios and clearly define their semantic first.

          Show
          jingzhao Jing Zhao added a comment - Currently unspecified policy means using the default policy. Thus I do not think we need an explicit policy named "UNSPECIFIED". In the meanwhile, how should the semantic of the remove op look like? If we have set a storage policy on "/foo", then should we allow the user to apply the command on "/foo/bar" if bar is not associated with any explicit policy? Or what if we have nested policy settings? I think we may need to list all the scenarios and clearly define their semantic first.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Hi Walter, Xiaobing, this approach looks fine to me. I see no harm in exposing UNSPECIFIED_STORAGE_POLICY_NAME. It does simplify the implementation a bit.

          Tsz Wo Nicholas Sze/Jing Zhao, do you have any opinion on this approach?

          Show
          arpitagarwal Arpit Agarwal added a comment - Hi Walter, Xiaobing, this approach looks fine to me. I see no harm in exposing UNSPECIFIED_STORAGE_POLICY_NAME. It does simplify the implementation a bit. Tsz Wo Nicholas Sze / Jing Zhao , do you have any opinion on this approach?
          Hide
          xiaobingo Xiaobing Zhou added a comment -

          Thanks Walter Su, do you think what semantic of 'removeStoragePolicy' should be? My understanding is to set it to UNSPECIFIED_STORAGE_POLICY_ID in a way like the dir/file is newly created without storage policy specified. I was also thinking of setting it to default storage policy, but the former is better IMO.

          Show
          xiaobingo Xiaobing Zhou added a comment - Thanks Walter Su , do you think what semantic of 'removeStoragePolicy' should be? My understanding is to set it to UNSPECIFIED_STORAGE_POLICY_ID in a way like the dir/file is newly created without storage policy specified. I was also thinking of setting it to default storage policy, but the former is better IMO.
          Hide
          walter.k.su Walter Su added a comment -

          Thanks Xiaobing Zhou.
          1. I think the original design doesn't mean to make UNSPECIFIED_STORAGE_POLICY_ID a policy. So it's not in policy suite.
          2. In FSDirAttrOp.java, you can pass policyId instead of policyName to setStoragePolicy(..).

          Show
          walter.k.su Walter Su added a comment - Thanks Xiaobing Zhou . 1. I think the original design doesn't mean to make UNSPECIFIED_STORAGE_POLICY_ID a policy. So it's not in policy suite. 2. In FSDirAttrOp.java, you can pass policyId instead of policyName to setStoragePolicy(..) .
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 30s trunk passed
          +1 compile 7m 32s trunk passed with JDK v1.8.0_66
          +1 compile 8m 26s trunk passed with JDK v1.7.0_91
          +1 checkstyle 1m 5s trunk passed
          +1 mvnsite 2m 26s trunk passed
          +1 mvneclipse 0m 41s trunk passed
          +1 findbugs 5m 25s trunk passed
          +1 javadoc 2m 14s trunk passed with JDK v1.8.0_66
          +1 javadoc 3m 10s trunk passed with JDK v1.7.0_91
          +1 mvninstall 2m 57s the patch passed
          +1 compile 7m 28s the patch passed with JDK v1.8.0_66
          +1 cc 7m 28s the patch passed
          +1 javac 7m 28s the patch passed
          +1 compile 8m 25s the patch passed with JDK v1.7.0_91
          +1 cc 8m 25s the patch passed
          +1 javac 8m 25s the patch passed
          -1 checkstyle 1m 5s Patch generated 7 new checkstyle issues in root (total was 577, now 579).
          +1 mvnsite 2m 27s the patch passed
          +1 mvneclipse 0m 41s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 5m 56s the patch passed
          +1 javadoc 2m 14s the patch passed with JDK v1.8.0_66
          +1 javadoc 3m 13s the patch passed with JDK v1.7.0_91
          -1 unit 6m 6s hadoop-common in the patch failed with JDK v1.8.0_66.
          +1 unit 0m 50s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66.
          -1 unit 54m 17s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          -1 unit 6m 21s hadoop-common in the patch failed with JDK v1.7.0_91.
          +1 unit 0m 57s hadoop-hdfs-client in the patch passed with JDK v1.7.0_91.
          -1 unit 49m 15s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
          +1 asflicense 0m 27s Patch does not generate ASF License warnings.
          192m 46s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.fs.TestFilterFileSystem
            hadoop.fs.TestHarFileSystem
            hadoop.hdfs.server.namenode.TestStoragePolicySummary
            hadoop.hdfs.server.namenode.TestFsck
          JDK v1.8.0_66 Timed out junit tests org.apache.hadoop.hdfs.server.namenode.ha.TestDNFencing
          JDK v1.7.0_91 Failed junit tests hadoop.fs.TestFilterFileSystem
            hadoop.fs.TestHarFileSystem
            hadoop.hdfs.server.namenode.TestStoragePolicySummary
            hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks
            hadoop.hdfs.server.namenode.TestFsck



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12781339/HDFS-9534.001.patch
          JIRA Issue HDFS-9534
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
          uname Linux f5a62657ef87 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 109e528
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14078/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14078/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14078/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14078/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_91.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14078/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14078/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14078/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14078/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_91.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14078/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14078/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-client U: .
          Max memory used 76MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14078/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 30s trunk passed +1 compile 7m 32s trunk passed with JDK v1.8.0_66 +1 compile 8m 26s trunk passed with JDK v1.7.0_91 +1 checkstyle 1m 5s trunk passed +1 mvnsite 2m 26s trunk passed +1 mvneclipse 0m 41s trunk passed +1 findbugs 5m 25s trunk passed +1 javadoc 2m 14s trunk passed with JDK v1.8.0_66 +1 javadoc 3m 10s trunk passed with JDK v1.7.0_91 +1 mvninstall 2m 57s the patch passed +1 compile 7m 28s the patch passed with JDK v1.8.0_66 +1 cc 7m 28s the patch passed +1 javac 7m 28s the patch passed +1 compile 8m 25s the patch passed with JDK v1.7.0_91 +1 cc 8m 25s the patch passed +1 javac 8m 25s the patch passed -1 checkstyle 1m 5s Patch generated 7 new checkstyle issues in root (total was 577, now 579). +1 mvnsite 2m 27s the patch passed +1 mvneclipse 0m 41s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 5m 56s the patch passed +1 javadoc 2m 14s the patch passed with JDK v1.8.0_66 +1 javadoc 3m 13s the patch passed with JDK v1.7.0_91 -1 unit 6m 6s hadoop-common in the patch failed with JDK v1.8.0_66. +1 unit 0m 50s hadoop-hdfs-client in the patch passed with JDK v1.8.0_66. -1 unit 54m 17s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 6m 21s hadoop-common in the patch failed with JDK v1.7.0_91. +1 unit 0m 57s hadoop-hdfs-client in the patch passed with JDK v1.7.0_91. -1 unit 49m 15s hadoop-hdfs in the patch failed with JDK v1.7.0_91. +1 asflicense 0m 27s Patch does not generate ASF License warnings. 192m 46s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.fs.TestFilterFileSystem   hadoop.fs.TestHarFileSystem   hadoop.hdfs.server.namenode.TestStoragePolicySummary   hadoop.hdfs.server.namenode.TestFsck JDK v1.8.0_66 Timed out junit tests org.apache.hadoop.hdfs.server.namenode.ha.TestDNFencing JDK v1.7.0_91 Failed junit tests hadoop.fs.TestFilterFileSystem   hadoop.fs.TestHarFileSystem   hadoop.hdfs.server.namenode.TestStoragePolicySummary   hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks   hadoop.hdfs.server.namenode.TestFsck Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12781339/HDFS-9534.001.patch JIRA Issue HDFS-9534 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux f5a62657ef87 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 109e528 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14078/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14078/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14078/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14078/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_91.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14078/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14078/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14078/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14078/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_91.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14078/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14078/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-client U: . Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14078/console This message was automatically generated.
          Hide
          xiaobingo Xiaobing Zhou added a comment -

          The high level impl is the added removeStoragePolicy RPC call that sets the storage policy to newly added UNSPECIFIED_STORAGE_POLICY.

          Show
          xiaobingo Xiaobing Zhou added a comment - The high level impl is the added removeStoragePolicy RPC call that sets the storage policy to newly added UNSPECIFIED_STORAGE_POLICY.
          Hide
          xiaobingo Xiaobing Zhou added a comment -

          Posted patch V001, kindly review, thanks.

          Show
          xiaobingo Xiaobing Zhou added a comment - Posted patch V001, kindly review, thanks.

            People

            • Assignee:
              xiaobingo Xiaobing Zhou
              Reporter:
              cnauroth Chris Nauroth
            • Votes:
              0 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development