Details

    • Type: Sub-task
    • Status: Patch Available
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: HDFS-10467
    • Component/s: fs
    • Labels:
      None

      Description

      Add the Web UI to the Router to expose the status of the federated cluster. It includes the federation metrics.

      1. federationUI-1.png
        65 kB
        Íñigo Goiri
      2. federationUI-2.png
        72 kB
        Íñigo Goiri
      3. federationUI-3.png
        33 kB
        Íñigo Goiri
      4. HDFS-12273-HDFS-10467-000.patch
        133 kB
        Íñigo Goiri
      5. HDFS-12273-HDFS-10467-001.patch
        138 kB
        Íñigo Goiri
      6. HDFS-12273-HDFS-10467-002.patch
        138 kB
        Íñigo Goiri
      7. HDFS-12273-HDFS-10467-003.patch
        45 kB
        Íñigo Goiri
      8. HDFS-12273-HDFS-10467-004.patch
        46 kB
        Íñigo Goiri

        Issue Links

          Activity

          Hide
          elgoiri Íñigo Goiri added a comment -

          Web UI with metrics.

          Show
          elgoiri Íñigo Goiri added a comment - Web UI with metrics.
          Hide
          chris.douglas Chris Douglas added a comment -

          Really minor:

          • If the comparator isn't used except for JSON output, it can move out of MountTable

          +0, overall. I can offer no valuable input on web UIs. Ravi Prakash, do you want to take a look?

          Show
          chris.douglas Chris Douglas added a comment - Really minor: If the comparator isn't used except for JSON output, it can move out of MountTable +0, overall. I can offer no valuable input on web UIs. Ravi Prakash , do you want to take a look?
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 20s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.
                HDFS-10467 Compile Tests
          +1 mvninstall 19m 38s HDFS-10467 passed
          +1 compile 0m 55s HDFS-10467 passed
          +1 checkstyle 0m 46s HDFS-10467 passed
          +1 mvnsite 1m 4s HDFS-10467 passed
          -1 findbugs 2m 1s hadoop-hdfs-project/hadoop-hdfs in HDFS-10467 has 10 extant Findbugs warnings.
          +1 javadoc 0m 49s HDFS-10467 passed
                Patch Compile Tests
          +1 mvninstall 0m 56s the patch passed
          +1 compile 0m 56s the patch passed
          -1 javac 0m 56s hadoop-hdfs-project_hadoop-hdfs generated 11 new + 400 unchanged - 9 fixed = 411 total (was 409)
          -0 checkstyle 0m 44s hadoop-hdfs-project/hadoop-hdfs: The patch generated 13 new + 403 unchanged - 0 fixed = 416 total (was 403)
          +1 mvnsite 1m 2s the patch passed
          -1 whitespace 0m 0s The patch 36 line(s) with tabs.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          -1 findbugs 2m 9s hadoop-hdfs-project/hadoop-hdfs generated 3 new + 10 unchanged - 0 fixed = 13 total (was 10)
          +1 javadoc 0m 50s the patch passed
                Other Tests
          -1 unit 75m 22s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          109m 26s



          Reason Tests
          FindBugs module:hadoop-hdfs-project/hadoop-hdfs
            Exception is caught when Exception is not thrown in org.apache.hadoop.hdfs.server.federation.metrics.FederationMetrics.getNamespaceInfo(String) At FederationMetrics.java:is not thrown in org.apache.hadoop.hdfs.server.federation.metrics.FederationMetrics.getNamespaceInfo(String) At FederationMetrics.java:[line 509]
            Call to method of static java.text.DateFormat in org.apache.hadoop.hdfs.server.federation.metrics.FederationMetrics.getDateString(long) At FederationMetrics.java:java.text.DateFormat in org.apache.hadoop.hdfs.server.federation.metrics.FederationMetrics.getDateString(long) At FederationMetrics.java:[line 644]
            Exception is caught when Exception is not thrown in org.apache.hadoop.hdfs.server.federation.metrics.NamenodeBeanMetrics.getNamespaceInfo(String) At NamenodeBeanMetrics.java:is not thrown in org.apache.hadoop.hdfs.server.federation.metrics.NamenodeBeanMetrics.getNamespaceInfo(String) At NamenodeBeanMetrics.java:[line 387]
          Failed junit tests hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
            hadoop.tools.TestHdfsConfigFields



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue HDFS-12273
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12880929/HDFS-12273-HDFS-10467-000.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux c6bd3419a89e 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision HDFS-10467 / 96ce127
          Default Java 1.8.0_131
          findbugs v3.1.0-RC1
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/20606/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
          javac https://builds.apache.org/job/PreCommit-HDFS-Build/20606/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project_hadoop-hdfs.txt
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/20606/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/20606/artifact/patchprocess/whitespace-tabs.txt
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/20606/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/20606/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20606/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20606/console
          Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 20s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.       HDFS-10467 Compile Tests +1 mvninstall 19m 38s HDFS-10467 passed +1 compile 0m 55s HDFS-10467 passed +1 checkstyle 0m 46s HDFS-10467 passed +1 mvnsite 1m 4s HDFS-10467 passed -1 findbugs 2m 1s hadoop-hdfs-project/hadoop-hdfs in HDFS-10467 has 10 extant Findbugs warnings. +1 javadoc 0m 49s HDFS-10467 passed       Patch Compile Tests +1 mvninstall 0m 56s the patch passed +1 compile 0m 56s the patch passed -1 javac 0m 56s hadoop-hdfs-project_hadoop-hdfs generated 11 new + 400 unchanged - 9 fixed = 411 total (was 409) -0 checkstyle 0m 44s hadoop-hdfs-project/hadoop-hdfs: The patch generated 13 new + 403 unchanged - 0 fixed = 416 total (was 403) +1 mvnsite 1m 2s the patch passed -1 whitespace 0m 0s The patch 36 line(s) with tabs. +1 xml 0m 1s The patch has no ill-formed XML file. -1 findbugs 2m 9s hadoop-hdfs-project/hadoop-hdfs generated 3 new + 10 unchanged - 0 fixed = 13 total (was 10) +1 javadoc 0m 50s the patch passed       Other Tests -1 unit 75m 22s hadoop-hdfs in the patch failed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 109m 26s Reason Tests FindBugs module:hadoop-hdfs-project/hadoop-hdfs   Exception is caught when Exception is not thrown in org.apache.hadoop.hdfs.server.federation.metrics.FederationMetrics.getNamespaceInfo(String) At FederationMetrics.java:is not thrown in org.apache.hadoop.hdfs.server.federation.metrics.FederationMetrics.getNamespaceInfo(String) At FederationMetrics.java: [line 509]   Call to method of static java.text.DateFormat in org.apache.hadoop.hdfs.server.federation.metrics.FederationMetrics.getDateString(long) At FederationMetrics.java:java.text.DateFormat in org.apache.hadoop.hdfs.server.federation.metrics.FederationMetrics.getDateString(long) At FederationMetrics.java: [line 644]   Exception is caught when Exception is not thrown in org.apache.hadoop.hdfs.server.federation.metrics.NamenodeBeanMetrics.getNamespaceInfo(String) At NamenodeBeanMetrics.java:is not thrown in org.apache.hadoop.hdfs.server.federation.metrics.NamenodeBeanMetrics.getNamespaceInfo(String) At NamenodeBeanMetrics.java: [line 387] Failed junit tests hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.tools.TestHdfsConfigFields Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-12273 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12880929/HDFS-12273-HDFS-10467-000.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux c6bd3419a89e 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision HDFS-10467 / 96ce127 Default Java 1.8.0_131 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/20606/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html javac https://builds.apache.org/job/PreCommit-HDFS-Build/20606/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project_hadoop-hdfs.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/20606/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/20606/artifact/patchprocess/whitespace-tabs.txt findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/20606/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html unit https://builds.apache.org/job/PreCommit-HDFS-Build/20606/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20606/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20606/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          elgoiri Íñigo Goiri added a comment -

          Fixed most Jenkins comments.
          I'm leaving the comparator because it will be used by the Rebalancer output.

          Show
          elgoiri Íñigo Goiri added a comment - Fixed most Jenkins comments. I'm leaving the comparator because it will be used by the Rebalancer output.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 13s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 6 new or modified test files.
                HDFS-10467 Compile Tests
          0 mvndep 1m 43s Maven dependency ordering for branch
          +1 mvninstall 17m 15s HDFS-10467 passed
          +1 compile 17m 52s HDFS-10467 passed
          +1 checkstyle 2m 20s HDFS-10467 passed
          +1 mvnsite 3m 11s HDFS-10467 passed
          -1 findbugs 2m 18s hadoop-hdfs-project/hadoop-hdfs in HDFS-10467 has 10 extant Findbugs warnings.
          +1 javadoc 1m 59s HDFS-10467 passed
                Patch Compile Tests
          0 mvndep 0m 18s Maven dependency ordering for patch
          +1 mvninstall 1m 57s the patch passed
          +1 compile 13m 2s the patch passed
          -1 javac 13m 2s root generated 9 new + 1329 unchanged - 9 fixed = 1338 total (was 1338)
          -0 checkstyle 2m 21s root: The patch generated 4 new + 403 unchanged - 0 fixed = 407 total (was 403)
          +1 mvnsite 2m 56s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 5s The patch has no ill-formed XML file.
          -1 findbugs 2m 36s hadoop-hdfs-project/hadoop-hdfs generated 2 new + 10 unchanged - 0 fixed = 12 total (was 10)
          +1 javadoc 2m 1s the patch passed
                Other Tests
          +1 unit 8m 38s hadoop-common in the patch passed.
          -1 unit 67m 50s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 28s The patch does not generate ASF License warnings.
          154m 17s



          Reason Tests
          FindBugs module:hadoop-hdfs-project/hadoop-hdfs
            Incorrect lazy initialization of static field org.apache.hadoop.hdfs.server.federation.store.StateStoreService.metrics in org.apache.hadoop.hdfs.server.federation.store.StateStoreService.serviceInit(Configuration) At StateStoreService.java:field org.apache.hadoop.hdfs.server.federation.store.StateStoreService.metrics in org.apache.hadoop.hdfs.server.federation.store.StateStoreService.serviceInit(Configuration) At StateStoreService.java:[lines 167-168]
            Incorrect lazy initialization of static field org.apache.hadoop.hdfs.server.federation.store.StateStoreService.metrics in org.apache.hadoop.hdfs.server.federation.store.StateStoreService.serviceStop() At StateStoreService.java:field org.apache.hadoop.hdfs.server.federation.store.StateStoreService.metrics in org.apache.hadoop.hdfs.server.federation.store.StateStoreService.serviceStop() At StateStoreService.java:[lines 197-199]
          Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090
            hadoop.hdfs.TestFileCorruption
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue HDFS-12273
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12880948/HDFS-12273-HDFS-10467-001.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux ab24477b79af 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision HDFS-10467 / 96ce127
          Default Java 1.8.0_131
          findbugs v3.1.0-RC1
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/20611/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
          javac https://builds.apache.org/job/PreCommit-HDFS-Build/20611/artifact/patchprocess/diff-compile-javac-root.txt
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/20611/artifact/patchprocess/diff-checkstyle-root.txt
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/20611/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/20611/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20611/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20611/console
          Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 6 new or modified test files.       HDFS-10467 Compile Tests 0 mvndep 1m 43s Maven dependency ordering for branch +1 mvninstall 17m 15s HDFS-10467 passed +1 compile 17m 52s HDFS-10467 passed +1 checkstyle 2m 20s HDFS-10467 passed +1 mvnsite 3m 11s HDFS-10467 passed -1 findbugs 2m 18s hadoop-hdfs-project/hadoop-hdfs in HDFS-10467 has 10 extant Findbugs warnings. +1 javadoc 1m 59s HDFS-10467 passed       Patch Compile Tests 0 mvndep 0m 18s Maven dependency ordering for patch +1 mvninstall 1m 57s the patch passed +1 compile 13m 2s the patch passed -1 javac 13m 2s root generated 9 new + 1329 unchanged - 9 fixed = 1338 total (was 1338) -0 checkstyle 2m 21s root: The patch generated 4 new + 403 unchanged - 0 fixed = 407 total (was 403) +1 mvnsite 2m 56s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 5s The patch has no ill-formed XML file. -1 findbugs 2m 36s hadoop-hdfs-project/hadoop-hdfs generated 2 new + 10 unchanged - 0 fixed = 12 total (was 10) +1 javadoc 2m 1s the patch passed       Other Tests +1 unit 8m 38s hadoop-common in the patch passed. -1 unit 67m 50s hadoop-hdfs in the patch failed. +1 asflicense 0m 28s The patch does not generate ASF License warnings. 154m 17s Reason Tests FindBugs module:hadoop-hdfs-project/hadoop-hdfs   Incorrect lazy initialization of static field org.apache.hadoop.hdfs.server.federation.store.StateStoreService.metrics in org.apache.hadoop.hdfs.server.federation.store.StateStoreService.serviceInit(Configuration) At StateStoreService.java:field org.apache.hadoop.hdfs.server.federation.store.StateStoreService.metrics in org.apache.hadoop.hdfs.server.federation.store.StateStoreService.serviceInit(Configuration) At StateStoreService.java: [lines 167-168]   Incorrect lazy initialization of static field org.apache.hadoop.hdfs.server.federation.store.StateStoreService.metrics in org.apache.hadoop.hdfs.server.federation.store.StateStoreService.serviceStop() At StateStoreService.java:field org.apache.hadoop.hdfs.server.federation.store.StateStoreService.metrics in org.apache.hadoop.hdfs.server.federation.store.StateStoreService.serviceStop() At StateStoreService.java: [lines 197-199] Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090   hadoop.hdfs.TestFileCorruption   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-12273 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12880948/HDFS-12273-HDFS-10467-001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux ab24477b79af 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision HDFS-10467 / 96ce127 Default Java 1.8.0_131 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/20611/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html javac https://builds.apache.org/job/PreCommit-HDFS-Build/20611/artifact/patchprocess/diff-compile-javac-root.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/20611/artifact/patchprocess/diff-checkstyle-root.txt findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/20611/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html unit https://builds.apache.org/job/PreCommit-HDFS-Build/20611/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20611/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20611/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          raviprak Ravi Prakash added a comment -

          I can try to take a look but honestly I'm sorry it won't have very high priority.

          Show
          raviprak Ravi Prakash added a comment - I can try to take a look but honestly I'm sorry it won't have very high priority.
          Hide
          raviprak Ravi Prakash added a comment -

          I didn't realize this was in the Federation branch. This is way out of my field. Sorry. Could some branch committers please review it?

          Show
          raviprak Ravi Prakash added a comment - I didn't realize this was in the Federation branch. This is way out of my field. Sorry. Could some branch committers please review it?
          Hide
          elgoiri Íñigo Goiri added a comment -

          Ravi Prakash, we would appreciate feedback on the pure UI side that tries to mimic the Namenode UI.

          Show
          elgoiri Íñigo Goiri added a comment - Ravi Prakash , we would appreciate feedback on the pure UI side that tries to mimic the Namenode UI.
          Hide
          giovanni.fumarola Giovanni Matteo Fumarola added a comment - - edited

          Íñigo Goiri it would be great if you add some screenshots of the UI.

          Show
          giovanni.fumarola Giovanni Matteo Fumarola added a comment - - edited Íñigo Goiri it would be great if you add some screenshots of the UI.
          Hide
          elgoiri Íñigo Goiri added a comment -

          Screenshots from development cluster.
          Removed some potentially confidential information.

          Show
          elgoiri Íñigo Goiri added a comment - Screenshots from development cluster. Removed some potentially confidential information.
          Hide
          elgoiri Íñigo Goiri added a comment -

          Rebased to HDFS-10467 and fixed a couple minor issues.
          There are a bunch of warnings/checkstyles/findbugs that are the same for NameNode.
          Open to ideas on how to fix those.

          Show
          elgoiri Íñigo Goiri added a comment - Rebased to HDFS-10467 and fixed a couple minor issues. There are a bunch of warnings/checkstyles/findbugs that are the same for NameNode . Open to ideas on how to fix those.
          Hide
          giovanni.fumarola Giovanni Matteo Fumarola added a comment - - edited

          Thanks Íñigo Goiri. Can we split this patch in 2 Jiras - UI and metrics?
          I have already created HDFS-12335 for metrics.

          Show
          giovanni.fumarola Giovanni Matteo Fumarola added a comment - - edited Thanks Íñigo Goiri . Can we split this patch in 2 Jiras - UI and metrics? I have already created HDFS-12335 for metrics.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 17s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 6 new or modified test files.
                HDFS-10467 Compile Tests
          -1 mvninstall 15m 47s root in HDFS-10467 failed.
          +1 compile 0m 53s HDFS-10467 passed
          +1 checkstyle 0m 42s HDFS-10467 passed
          +1 mvnsite 0m 59s HDFS-10467 passed
          +1 findbugs 1m 52s HDFS-10467 passed
          +1 javadoc 0m 48s HDFS-10467 passed
                Patch Compile Tests
          +1 mvninstall 0m 54s the patch passed
          +1 compile 0m 51s the patch passed
          -1 javac 0m 51s hadoop-hdfs-project_hadoop-hdfs generated 10 new + 413 unchanged - 10 fixed = 423 total (was 423)
          -0 checkstyle 0m 39s hadoop-hdfs-project/hadoop-hdfs: The patch generated 3 new + 403 unchanged - 0 fixed = 406 total (was 403)
          +1 mvnsite 0m 58s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 3s The patch has no ill-formed XML file.
          -1 findbugs 1m 59s hadoop-hdfs-project/hadoop-hdfs generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)
          +1 javadoc 0m 53s the patch passed
                Other Tests
          -1 unit 84m 31s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 23s The patch does not generate ASF License warnings.
          114m 6s



          Reason Tests
          FindBugs module:hadoop-hdfs-project/hadoop-hdfs
            Incorrect lazy initialization of static field org.apache.hadoop.hdfs.server.federation.store.StateStoreService.metrics in org.apache.hadoop.hdfs.server.federation.store.StateStoreService.serviceInit(Configuration) At StateStoreService.java:field org.apache.hadoop.hdfs.server.federation.store.StateStoreService.metrics in org.apache.hadoop.hdfs.server.federation.store.StateStoreService.serviceInit(Configuration) At StateStoreService.java:[lines 167-168]
            Incorrect lazy initialization of static field org.apache.hadoop.hdfs.server.federation.store.StateStoreService.metrics in org.apache.hadoop.hdfs.server.federation.store.StateStoreService.serviceStop() At StateStoreService.java:field org.apache.hadoop.hdfs.server.federation.store.StateStoreService.metrics in org.apache.hadoop.hdfs.server.federation.store.StateStoreService.serviceStop() At StateStoreService.java:[lines 197-199]
          Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
            hadoop.hdfs.TestDFSShell



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue HDFS-12273
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12882976/HDFS-12273-HDFS-10467-002.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux cc54d3cd8081 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision HDFS-10467 / fc2c254
          Default Java 1.8.0_144
          mvninstall https://builds.apache.org/job/PreCommit-HDFS-Build/20789/artifact/patchprocess/branch-mvninstall-root.txt
          findbugs v3.1.0-RC1
          javac https://builds.apache.org/job/PreCommit-HDFS-Build/20789/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project_hadoop-hdfs.txt
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/20789/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/20789/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/20789/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20789/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20789/console
          Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 6 new or modified test files.       HDFS-10467 Compile Tests -1 mvninstall 15m 47s root in HDFS-10467 failed. +1 compile 0m 53s HDFS-10467 passed +1 checkstyle 0m 42s HDFS-10467 passed +1 mvnsite 0m 59s HDFS-10467 passed +1 findbugs 1m 52s HDFS-10467 passed +1 javadoc 0m 48s HDFS-10467 passed       Patch Compile Tests +1 mvninstall 0m 54s the patch passed +1 compile 0m 51s the patch passed -1 javac 0m 51s hadoop-hdfs-project_hadoop-hdfs generated 10 new + 413 unchanged - 10 fixed = 423 total (was 423) -0 checkstyle 0m 39s hadoop-hdfs-project/hadoop-hdfs: The patch generated 3 new + 403 unchanged - 0 fixed = 406 total (was 403) +1 mvnsite 0m 58s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 3s The patch has no ill-formed XML file. -1 findbugs 1m 59s hadoop-hdfs-project/hadoop-hdfs generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) +1 javadoc 0m 53s the patch passed       Other Tests -1 unit 84m 31s hadoop-hdfs in the patch failed. +1 asflicense 0m 23s The patch does not generate ASF License warnings. 114m 6s Reason Tests FindBugs module:hadoop-hdfs-project/hadoop-hdfs   Incorrect lazy initialization of static field org.apache.hadoop.hdfs.server.federation.store.StateStoreService.metrics in org.apache.hadoop.hdfs.server.federation.store.StateStoreService.serviceInit(Configuration) At StateStoreService.java:field org.apache.hadoop.hdfs.server.federation.store.StateStoreService.metrics in org.apache.hadoop.hdfs.server.federation.store.StateStoreService.serviceInit(Configuration) At StateStoreService.java: [lines 167-168]   Incorrect lazy initialization of static field org.apache.hadoop.hdfs.server.federation.store.StateStoreService.metrics in org.apache.hadoop.hdfs.server.federation.store.StateStoreService.serviceStop() At StateStoreService.java:field org.apache.hadoop.hdfs.server.federation.store.StateStoreService.metrics in org.apache.hadoop.hdfs.server.federation.store.StateStoreService.serviceStop() At StateStoreService.java: [lines 197-199] Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.hdfs.TestDFSShell Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-12273 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12882976/HDFS-12273-HDFS-10467-002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux cc54d3cd8081 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision HDFS-10467 / fc2c254 Default Java 1.8.0_144 mvninstall https://builds.apache.org/job/PreCommit-HDFS-Build/20789/artifact/patchprocess/branch-mvninstall-root.txt findbugs v3.1.0-RC1 javac https://builds.apache.org/job/PreCommit-HDFS-Build/20789/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project_hadoop-hdfs.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/20789/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/20789/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html unit https://builds.apache.org/job/PreCommit-HDFS-Build/20789/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20789/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20789/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          elgoiri Íñigo Goiri added a comment -

          Giovanni Matteo Fumarola, thank you for the comments.
          I pushed the metrics to HDFS-12335.

          Show
          elgoiri Íñigo Goiri added a comment - Giovanni Matteo Fumarola , thank you for the comments. I pushed the metrics to HDFS-12335 .
          Hide
          elgoiri Íñigo Goiri added a comment -

          Rebased after HDFS-12335.

          Show
          elgoiri Íñigo Goiri added a comment - Rebased after HDFS-12335 .
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 17s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
                HDFS-10467 Compile Tests
          +1 mvninstall 15m 1s HDFS-10467 passed
          +1 compile 0m 52s HDFS-10467 passed
          +1 checkstyle 0m 44s HDFS-10467 passed
          +1 mvnsite 0m 57s HDFS-10467 passed
          +1 findbugs 1m 55s HDFS-10467 passed
          +1 javadoc 0m 48s HDFS-10467 passed
                Patch Compile Tests
          +1 mvninstall 0m 51s the patch passed
          +1 compile 0m 50s the patch passed
          -1 javac 0m 50s hadoop-hdfs-project_hadoop-hdfs generated 5 new + 403 unchanged - 5 fixed = 408 total (was 408)
          +1 checkstyle 0m 40s the patch passed
          +1 mvnsite 0m 53s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 2s The patch has no ill-formed XML file.
          +1 findbugs 1m 53s the patch passed
          +1 javadoc 0m 44s the patch passed
                Other Tests
          -1 unit 94m 12s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          122m 21s



          Reason Tests
          Failed junit tests hadoop.hdfs.TestLeaseRecoveryStriped
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure110
            hadoop.hdfs.TestParallelShortCircuitReadUnCached
            hadoop.hdfs.server.namenode.TestDecommissioningStatus
            hadoop.tools.TestHdfsConfigFields
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure170
            hadoop.hdfs.TestListFilesInFileContext
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070
          Timed out junit tests org.apache.hadoop.hdfs.TestWriteReadStripedFile



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:71bbb86
          JIRA Issue HDFS-12273
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12886116/HDFS-12273-HDFS-10467-003.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux 5437e609bbe6 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision HDFS-10467 / d522007
          Default Java 1.8.0_144
          findbugs v3.1.0-RC1
          javac https://builds.apache.org/job/PreCommit-HDFS-Build/21054/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/21054/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/21054/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/21054/console
          Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.       HDFS-10467 Compile Tests +1 mvninstall 15m 1s HDFS-10467 passed +1 compile 0m 52s HDFS-10467 passed +1 checkstyle 0m 44s HDFS-10467 passed +1 mvnsite 0m 57s HDFS-10467 passed +1 findbugs 1m 55s HDFS-10467 passed +1 javadoc 0m 48s HDFS-10467 passed       Patch Compile Tests +1 mvninstall 0m 51s the patch passed +1 compile 0m 50s the patch passed -1 javac 0m 50s hadoop-hdfs-project_hadoop-hdfs generated 5 new + 403 unchanged - 5 fixed = 408 total (was 408) +1 checkstyle 0m 40s the patch passed +1 mvnsite 0m 53s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. +1 findbugs 1m 53s the patch passed +1 javadoc 0m 44s the patch passed       Other Tests -1 unit 94m 12s hadoop-hdfs in the patch failed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 122m 21s Reason Tests Failed junit tests hadoop.hdfs.TestLeaseRecoveryStriped   hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure110   hadoop.hdfs.TestParallelShortCircuitReadUnCached   hadoop.hdfs.server.namenode.TestDecommissioningStatus   hadoop.tools.TestHdfsConfigFields   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure170   hadoop.hdfs.TestListFilesInFileContext   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070 Timed out junit tests org.apache.hadoop.hdfs.TestWriteReadStripedFile Subsystem Report/Notes Docker Image:yetus/hadoop:71bbb86 JIRA Issue HDFS-12273 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12886116/HDFS-12273-HDFS-10467-003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 5437e609bbe6 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision HDFS-10467 / d522007 Default Java 1.8.0_144 findbugs v3.1.0-RC1 javac https://builds.apache.org/job/PreCommit-HDFS-Build/21054/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/21054/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/21054/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/21054/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          elgoiri Íñigo Goiri added a comment -

          The javac errors are not related to this commit; I've rebased HDFS-10467 to get rid of some of the unit tests.
          Giovanni Matteo Fumarola, do you mind taking a look?

          Show
          elgoiri Íñigo Goiri added a comment - The javac errors are not related to this commit; I've rebased HDFS-10467 to get rid of some of the unit tests. Giovanni Matteo Fumarola , do you mind taking a look?
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
                HDFS-10467 Compile Tests
          +1 mvninstall 17m 3s HDFS-10467 passed
          +1 compile 1m 1s HDFS-10467 passed
          +1 checkstyle 0m 46s HDFS-10467 passed
          +1 mvnsite 1m 0s HDFS-10467 passed
          +1 findbugs 2m 4s HDFS-10467 passed
          +1 javadoc 0m 47s HDFS-10467 passed
                Patch Compile Tests
          +1 mvninstall 1m 4s the patch passed
          +1 compile 0m 57s the patch passed
          -1 javac 0m 57s hadoop-hdfs-project_hadoop-hdfs generated 5 new + 403 unchanged - 5 fixed = 408 total (was 408)
          +1 checkstyle 0m 42s the patch passed
          +1 mvnsite 0m 58s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 2s The patch has no ill-formed XML file.
          +1 findbugs 2m 24s the patch passed
          +1 javadoc 0m 57s the patch passed
                Other Tests
          -1 unit 91m 20s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 16s The patch does not generate ASF License warnings.
          123m 42s



          Reason Tests
          Failed junit tests hadoop.hdfs.TestClientProtocolForPipelineRecovery
            hadoop.hdfs.TestLeaseRecoveryStriped
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure170
            hadoop.hdfs.server.federation.router.TestNamenodeHeartbeat
            hadoop.hdfs.TestReconstructStripedFile
          Timed out junit tests org.apache.hadoop.hdfs.TestWriteReadStripedFile



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:71bbb86
          JIRA Issue HDFS-12273
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12886991/HDFS-12273-HDFS-10467-004.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux ea1d8202a746 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision HDFS-10467 / 679e31a
          Default Java 1.8.0_144
          findbugs v3.1.0-RC1
          javac https://builds.apache.org/job/PreCommit-HDFS-Build/21128/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/21128/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/21128/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/21128/console
          Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.       HDFS-10467 Compile Tests +1 mvninstall 17m 3s HDFS-10467 passed +1 compile 1m 1s HDFS-10467 passed +1 checkstyle 0m 46s HDFS-10467 passed +1 mvnsite 1m 0s HDFS-10467 passed +1 findbugs 2m 4s HDFS-10467 passed +1 javadoc 0m 47s HDFS-10467 passed       Patch Compile Tests +1 mvninstall 1m 4s the patch passed +1 compile 0m 57s the patch passed -1 javac 0m 57s hadoop-hdfs-project_hadoop-hdfs generated 5 new + 403 unchanged - 5 fixed = 408 total (was 408) +1 checkstyle 0m 42s the patch passed +1 mvnsite 0m 58s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. +1 findbugs 2m 24s the patch passed +1 javadoc 0m 57s the patch passed       Other Tests -1 unit 91m 20s hadoop-hdfs in the patch failed. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 123m 42s Reason Tests Failed junit tests hadoop.hdfs.TestClientProtocolForPipelineRecovery   hadoop.hdfs.TestLeaseRecoveryStriped   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure170   hadoop.hdfs.server.federation.router.TestNamenodeHeartbeat   hadoop.hdfs.TestReconstructStripedFile Timed out junit tests org.apache.hadoop.hdfs.TestWriteReadStripedFile Subsystem Report/Notes Docker Image:yetus/hadoop:71bbb86 JIRA Issue HDFS-12273 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12886991/HDFS-12273-HDFS-10467-004.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux ea1d8202a746 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision HDFS-10467 / 679e31a Default Java 1.8.0_144 findbugs v3.1.0-RC1 javac https://builds.apache.org/job/PreCommit-HDFS-Build/21128/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/21128/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/21128/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/21128/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          elgoiri Íñigo Goiri added a comment -

          The failure in TestNamenodeHeartbeat is not related to this change.
          I opened HDFS-12450 to fix it.

          Show
          elgoiri Íñigo Goiri added a comment - The failure in TestNamenodeHeartbeat is not related to this change. I opened HDFS-12450 to fix it.
          Hide
          chris.douglas Chris Douglas added a comment -

          Skimming the smaller patch, most of it looks straightforward.

          I'm not familiar with how the new UIs protect against XSS vulnerabilities. The HtmlQuoting class used to take care of this for the old UI, IIRC Hamlet has its own strategy... Haohui Mai?

          Show
          chris.douglas Chris Douglas added a comment - Skimming the smaller patch, most of it looks straightforward. I'm not familiar with how the new UIs protect against XSS vulnerabilities. The HtmlQuoting class used to take care of this for the old UI, IIRC Hamlet has its own strategy... Haohui Mai ?
          Hide
          chris.douglas Chris Douglas added a comment -

          XSS handling could also be part of HDFS-12284. If we defer that, then it might make sense to add a note to the docs, so users know that security is not supported and not audited.

          Show
          chris.douglas Chris Douglas added a comment - XSS handling could also be part of HDFS-12284 . If we defer that, then it might make sense to add a note to the docs, so users know that security is not supported and not audited.
          Hide
          elgoiri Íñigo Goiri added a comment -

          Thanks Chris Douglas for the check.
          I'm OK with handling security issues related to the Web UI in HDFS-12284 or maybe a follow-up JIRA.
          xiangguang zheng, Zhe Zhang, does it make sense to do it in HDFS-12284?

          Regarding the note in the doc, I could add it here or HDFS-12381 as a general comment on security and not only about the Web UI.

          Show
          elgoiri Íñigo Goiri added a comment - Thanks Chris Douglas for the check. I'm OK with handling security issues related to the Web UI in HDFS-12284 or maybe a follow-up JIRA. xiangguang zheng , Zhe Zhang , does it make sense to do it in HDFS-12284 ? Regarding the note in the doc, I could add it here or HDFS-12381 as a general comment on security and not only about the Web UI.
          Hide
          raviprak Ravi Prakash added a comment -

          Sorry I've since swapped out all front-end and security knowledge from my brain. But for the new UI, I had to add https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/webhdfs/WebHdfsHandler.java#L315 . My guide were the things done for onOpen / onAppend etc. XSS isn't quite related to HDFS-12284, so if at all you want to postpone the analysis, would it make sense to file a different JIRA?

          Show
          raviprak Ravi Prakash added a comment - Sorry I've since swapped out all front-end and security knowledge from my brain. But for the new UI, I had to add https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/webhdfs/WebHdfsHandler.java#L315 . My guide were the things done for onOpen / onAppend etc. XSS isn't quite related to HDFS-12284 , so if at all you want to postpone the analysis, would it make sense to file a different JIRA?
          Hide
          chris.douglas Chris Douglas added a comment -

          Regarding the note in the doc, I could add it here or HDFS-12381 as a general comment on security and not only about the Web UI.

          Adding it to HDFS-12381 is simplest.

          XSS isn't quite related to HDFS-12284, so if at all you want to postpone the analysis, would it make sense to file a different JIRA?

          Sure, fair enough. Even if "harden the federation UI" is closed without requiring any code, it'd be useful for tracking.

          If we defer hardening the UI to after the merge, the current patch seems fine to me.

          Show
          chris.douglas Chris Douglas added a comment - Regarding the note in the doc, I could add it here or HDFS-12381 as a general comment on security and not only about the Web UI. Adding it to HDFS-12381 is simplest. XSS isn't quite related to HDFS-12284 , so if at all you want to postpone the analysis, would it make sense to file a different JIRA? Sure, fair enough. Even if "harden the federation UI" is closed without requiring any code, it'd be useful for tracking. If we defer hardening the UI to after the merge, the current patch seems fine to me.
          Hide
          elgoiri Íñigo Goiri added a comment -

          Thanks Ravi Prakash, actually, one thing is missing is the WebHDFS interface.
          Do you guys think it should go in HDFS-10467?
          In any case, that would require the web app from this JIRA.

          Show
          elgoiri Íñigo Goiri added a comment - Thanks Ravi Prakash , actually, one thing is missing is the WebHDFS interface. Do you guys think it should go in HDFS-10467 ? In any case, that would require the web app from this JIRA.
          Hide
          chris.douglas Chris Douglas added a comment -

          one thing is missing is the WebHDFS interface. Do you guys think it should go in HDFS-10467?

          That could go in after the merge pretty easily. Certainly not a prerequisite.

          Ravi Prakash, if it's not too traumatic for you to review/relive the front-end and security bits, then we can wait on your review. Otherwise we make a concerted effort to harden this code later, after HDFS-12284 is in.

          Show
          chris.douglas Chris Douglas added a comment - one thing is missing is the WebHDFS interface. Do you guys think it should go in HDFS-10467 ? That could go in after the merge pretty easily. Certainly not a prerequisite. Ravi Prakash , if it's not too traumatic for you to review/relive the front-end and security bits, then we can wait on your review. Otherwise we make a concerted effort to harden this code later, after HDFS-12284 is in.
          Hide
          raviprak Ravi Prakash added a comment -

          I eyeballed the much smaller patch and it seems fine to me. Let's please file the JIRA any way, and we can pore over this and the trunk code altogether just once.

          Show
          raviprak Ravi Prakash added a comment - I eyeballed the much smaller patch and it seems fine to me. Let's please file the JIRA any way, and we can pore over this and the trunk code altogether just once.
          Hide
          elgoiri Íñigo Goiri added a comment -

          Ravi Prakash, sounds good.
          I'll create those JIRAs outside of the HDFS-10467 umbrella for the future merge.
          Thinking on following the naming RBF.

          Show
          elgoiri Íñigo Goiri added a comment - Ravi Prakash , sounds good. I'll create those JIRAs outside of the HDFS-10467 umbrella for the future merge. Thinking on following the naming RBF.
          Hide
          elgoiri Íñigo Goiri added a comment -

          I created HDFS-12510 for adding security to the web server and HDFS-12512 to add support for WebHDFS.

          Show
          elgoiri Íñigo Goiri added a comment - I created HDFS-12510 for adding security to the web server and HDFS-12512 to add support for WebHDFS.

            People

            • Assignee:
              elgoiri Íñigo Goiri
              Reporter:
              elgoiri Íñigo Goiri
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:

                Development