Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-10101

Update guava dependency to the latest version

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-alpha2
    • Fix Version/s: 3.0.0-alpha4
    • Component/s: None
    • Labels:
    • Target Version/s:
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Hide
      Guava is updated to version 21.0.

      In the background of merging this patch into trunk, there is a work, shaded Hadoop client artifacts and minicluster, on HADOOP-11804. hadoop-client has its own Guava which is shaded, so we can update dependency with minimum effect compare to previous HADOOP-11804.

      See also HADOOP-14238 as related problem.
      Show
      Guava is updated to version 21.0. In the background of merging this patch into trunk, there is a work, shaded Hadoop client artifacts and minicluster, on HADOOP-11804 . hadoop-client has its own Guava which is shaded, so we can update dependency with minimum effect compare to previous HADOOP-11804 . See also HADOOP-14238 as related problem.

      Description

      The existing guava version is 11.0.2 which is quite old. This issue tries to update the version to as latest version as possible.

      1. HADOOP-10101.018.patch
        23 kB
        Tsuyoshi Ozawa
      2. HADOOP-10101.017.patch
        22 kB
        Tsuyoshi Ozawa
      3. HADOOP-10101.016.patch
        15 kB
        Tsuyoshi Ozawa
      4. HADOOP-10101.015.patch
        15 kB
        Tsuyoshi Ozawa
      5. HADOOP-10101.014.patch
        15 kB
        Tsuyoshi Ozawa
      6. HADOOP-10101.013.patch
        15 kB
        Tsuyoshi Ozawa
      7. HADOOP-10101.012.patch
        0.4 kB
        Tsuyoshi Ozawa
      8. HADOOP-10101-011.patch
        12 kB
        Tsuyoshi Ozawa
      9. HADOOP-10101-010.patch
        138 kB
        Tsuyoshi Ozawa
      10. HADOOP-10101-010.patch
        138 kB
        Tsuyoshi Ozawa
      11. HADOOP-10101-009.patch
        138 kB
        Tsuyoshi Ozawa
      12. HADOOP-10101-009.patch
        138 kB
        Tsuyoshi Ozawa
      13. HADOOP-10101-008.patch
        128 kB
        Tsuyoshi Ozawa
      14. HADOOP-10101-007.patch
        5 kB
        Tsuyoshi Ozawa
      15. HADOOP-10101-006.patch
        2 kB
        Tsuyoshi Ozawa
      16. HADOOP-10101-005.patch
        1 kB
        Tsuyoshi Ozawa
      17. HADOOP-10101-004.patch
        11 kB
        Steve Loughran
      18. HADOOP-10101.patch
        10 kB
        Vinayakumar B
      19. HADOOP-10101-002.patch
        2 kB
        Steve Loughran
      20. HADOOP-10101.patch
        0.4 kB
        Rakesh R

        Issue Links

          Activity

          Hide
          stevel@apache.org Steve Loughran added a comment -

          A. Soroka Ben Podgursky : you should know that even though Hadoop ships a woefully out of date Guava version, it does run against later ones, up to 18, AFAIK; every time they pull something we've had to jump though hoops (HADOOP-10961) to implicitly keep up.

          Shading is going to be the Java 8 solution, until java 9 gives us jlink and we can adopt it (how? when?); IMO we'll have to shade not just guava, but if we can include Avro, then, along with protobuf (done) we've got the high-brittleness and IPC-critical serialization libs sorted

          As far as we are aware, there are no APIs marked as public which export guava classes. This is a good thing, and does at least give us some freedom.

          Show
          stevel@apache.org Steve Loughran added a comment - A. Soroka Ben Podgursky : you should know that even though Hadoop ships a woefully out of date Guava version, it does run against later ones, up to 18, AFAIK; every time they pull something we've had to jump though hoops ( HADOOP-10961 ) to implicitly keep up. Shading is going to be the Java 8 solution, until java 9 gives us jlink and we can adopt it (how? when?); IMO we'll have to shade not just guava, but if we can include Avro, then, along with protobuf (done) we've got the high-brittleness and IPC-critical serialization libs sorted As far as we are aware, there are no APIs marked as public which export guava classes. This is a good thing, and does at least give us some freedom.
          Hide
          public@ecopatz.de Olaf Krische added a comment -

          I am not familiar with the code base, so i do not know, what exactly is exposed to the public (and also used by downstream versions like Spark). But it is possible to expose Guava stuff, so that is compatible to the old guava version, but internally using the latest, but shaded version. For example, internally you can use ListenableFuture from the shaded version, but having a delegate to the ListenableFuture from the public one.

          Show
          public@ecopatz.de Olaf Krische added a comment - I am not familiar with the code base, so i do not know, what exactly is exposed to the public (and also used by downstream versions like Spark). But it is possible to expose Guava stuff, so that is compatible to the old guava version, but internally using the latest, but shaded version. For example, internally you can use ListenableFuture from the shaded version, but having a delegate to the ListenableFuture from the public one.
          Hide
          djp Junping Du added a comment - - edited

          Offline conversations don't count for much in ASF projects.

          Yes. But at least it convince me that not everyone from every project are welcome this guava upgrade.

          Speaking here as a member of the HBase PMC I will be super happy to upgrade our Guava dependency.

          That's still count as your individual opinion. Any public jira or email discussion in HBase community to reflect any consensus on "happy upgrade"?

          FWIW I'm pretty sure we'll just switch to Hadoop's shaded artifacts to allow us to ignore the version used here.

          I am -1 on shading guava everywhere approach (HADOOP-14284) unless someone convince me that is right approach.

          Show
          djp Junping Du added a comment - - edited Offline conversations don't count for much in ASF projects. Yes. But at least it convince me that not everyone from every project are welcome this guava upgrade. Speaking here as a member of the HBase PMC I will be super happy to upgrade our Guava dependency. That's still count as your individual opinion. Any public jira or email discussion in HBase community to reflect any consensus on "happy upgrade"? FWIW I'm pretty sure we'll just switch to Hadoop's shaded artifacts to allow us to ignore the version used here. I am -1 on shading guava everywhere approach ( HADOOP-14284 ) unless someone convince me that is right approach.
          Hide
          busbey Sean Busbey added a comment -

          I doubt this. I talked with several guys from HBase and Tez community offline - none of them are happy with doing the same change. That means our change are pushing them to create/maintain branches for different version of hadoop (if no shading work in HADOOP-14284 which has side effect though). It also means the releases need to get synchronized, otherwise no downstream project release can work with Hadoop GA release will be a bigger problem.

          Offline conversations don't count for much in ASF projects. Speaking here as a member of the HBase PMC I will be super happy to upgrade our Guava dependency. FWIW I'm pretty sure we'll just switch to Hadoop's shaded artifacts to allow us to ignore the version used here.

          Show
          busbey Sean Busbey added a comment - I doubt this. I talked with several guys from HBase and Tez community offline - none of them are happy with doing the same change. That means our change are pushing them to create/maintain branches for different version of hadoop (if no shading work in HADOOP-14284 which has side effect though). It also means the releases need to get synchronized, otherwise no downstream project release can work with Hadoop GA release will be a bigger problem. Offline conversations don't count for much in ASF projects. Speaking here as a member of the HBase PMC I will be super happy to upgrade our Guava dependency. FWIW I'm pretty sure we'll just switch to Hadoop's shaded artifacts to allow us to ignore the version used here.
          Hide
          bpodgursky Ben Podgursky added a comment -

          Thanks Junping Du. Sorry if my reply came off as criticizing Hadoop, I totally recognize that this is a difficult problem. I understand the problems are fundamentally caused by Guava breaking changes, not by Hadoop. But I don't think I'm going to be able to convince Guava to stop doing major releases, and at the end of the day it's a super widely used library, so I figured I should try here instead

          This isn't really a scientific analysis, but my anecdotal experience is that there aren't * that * many breaking changes between each Guava release, and the odds are that you can run Guava 21 against something compiled against 19, and you'll likely be fine. There are some specific interface-to-abstract-class changes between 11 and 13 which are the big problem (IIRC, might be wrong on the versions).

          I agree that classpath isolation is the "real" fix here. I've been following HADOOP-11656 as well, ("Classpath isolation for downstream clients") but there's a ton of debate there about the costs of shading etc, so I'm not sure how close that is to being pushed.

          Anyway, thanks again for your work on this. I'm really hoping we can get a newer version, but I understand the release synchronization costs. Just wanted to mention that this causes user pain that might not be reflected on JIRA too often, since it's something we can generally work around.

          Show
          bpodgursky Ben Podgursky added a comment - Thanks Junping Du . Sorry if my reply came off as criticizing Hadoop, I totally recognize that this is a difficult problem. I understand the problems are fundamentally caused by Guava breaking changes, not by Hadoop. But I don't think I'm going to be able to convince Guava to stop doing major releases, and at the end of the day it's a super widely used library, so I figured I should try here instead This isn't really a scientific analysis, but my anecdotal experience is that there aren't * that * many breaking changes between each Guava release, and the odds are that you can run Guava 21 against something compiled against 19, and you'll likely be fine. There are some specific interface-to-abstract-class changes between 11 and 13 which are the big problem (IIRC, might be wrong on the versions). I agree that classpath isolation is the "real" fix here. I've been following HADOOP-11656 as well, ("Classpath isolation for downstream clients") but there's a ton of debate there about the costs of shading etc, so I'm not sure how close that is to being pushed. Anyway, thanks again for your work on this. I'm really hoping we can get a newer version, but I understand the release synchronization costs. Just wanted to mention that this causes user pain that might not be reflected on JIRA too often, since it's something we can generally work around.
          Hide
          ajs6f A. Soroka added a comment -

          As a committer for Apache Jena (who voted for this ticket and has been watching it hopefully ), I can tell you that the archaic version of Guava in Hadoop is the only reason we must maintain a shaded (modern) version of the same dependency in our project. This is confusing at best for both developers and users. Hadoop moving forward on this would be a consummation devoutly to be wished.

          Show
          ajs6f A. Soroka added a comment - As a committer for Apache Jena (who voted for this ticket and has been watching it hopefully ), I can tell you that the archaic version of Guava in Hadoop is the only reason we must maintain a shaded (modern) version of the same dependency in our project. This is confusing at best for both developers and users. Hadoop moving forward on this would be a consummation devoutly to be wished.
          Hide
          djp Junping Du added a comment -

          Thanks Ben Podgursky for sharing your thoughts. We are glad to hear voices from hadoop users as well as developers from hadoop and other related projects.

          The number of watchers on this ticket should be an indicator in itself, and I'm sure for every person who bothered creating a jira account, there are a couple hundred who were bitten by this.

          The number of watchers doesn't indicate they are supporting this patch but just they pay attention to. They have to because this patch break everything. We can bring this topic to hadoop-user alias to get more audience if needed.

          Has anyone contacted the maintainers on the major affected downstream projects like Spark and HBase? My guess is that they would be more than happy to help work around any breakages this upgrade causes – if they are like us, they would be overjoyed at being able to finally upgrade.

          I doubt this. I talked with several guys from HBase and Tez community offline - none of them are happy with doing the same change. That means our change are pushing them to create/maintain branches for different version of hadoop (if no shading work in HADOOP-14284 which has side effect though). It also means the releases need to get synchronized, otherwise no downstream project release can work with Hadoop GA release will be a bigger problem.

          Hadoop's ancient Guava dependency is the single largest issue we run into when putting any other third party jar on our client or task classpath.

          Please don't blame hadoop for this. The real problem here is the poor incompatibility of Guava across different versions and they released 21 major versions across 7 years. The ancient guava dependency are also suffering from this.

          There are many libraries developed outside the Hadoop ecosystem which (rightfully) assume they can use a newer version of Guava, and we regularly either have to do horrible hacks to get these onto the client classpath, or tell developers not to use them. This is an incredible time-waster that shouldn't exist.

          Is there any protocol for these libraries outside of hadoop to choose an uniform version of guava (like version 21)? If not, bump up guava version here does't help as it will still break these libraries which are using different version. Also, there are also many apps developed within hadoop ecosystem which should be seen as first class citizenship that get totally break by this change. As a system software, we should have some better solution like: application classpath isolation, etc. instead of simply keeping dependency updated.

          I understand the concerns about stability, but this is a major upgrade, no? I don't think it's acceptable to say that Hadoop will be running Guava 11 until Hadoop 4 comes out in 2025.

          Guava is not Java, I don't think any hadoop releases should bind with specific guava version. It is just a third party library - with poor incompatibility - that's it!

          Show
          djp Junping Du added a comment - Thanks Ben Podgursky for sharing your thoughts. We are glad to hear voices from hadoop users as well as developers from hadoop and other related projects. The number of watchers on this ticket should be an indicator in itself, and I'm sure for every person who bothered creating a jira account, there are a couple hundred who were bitten by this. The number of watchers doesn't indicate they are supporting this patch but just they pay attention to. They have to because this patch break everything. We can bring this topic to hadoop-user alias to get more audience if needed. Has anyone contacted the maintainers on the major affected downstream projects like Spark and HBase? My guess is that they would be more than happy to help work around any breakages this upgrade causes – if they are like us, they would be overjoyed at being able to finally upgrade. I doubt this. I talked with several guys from HBase and Tez community offline - none of them are happy with doing the same change. That means our change are pushing them to create/maintain branches for different version of hadoop (if no shading work in HADOOP-14284 which has side effect though). It also means the releases need to get synchronized, otherwise no downstream project release can work with Hadoop GA release will be a bigger problem. Hadoop's ancient Guava dependency is the single largest issue we run into when putting any other third party jar on our client or task classpath. Please don't blame hadoop for this. The real problem here is the poor incompatibility of Guava across different versions and they released 21 major versions across 7 years. The ancient guava dependency are also suffering from this. There are many libraries developed outside the Hadoop ecosystem which (rightfully) assume they can use a newer version of Guava, and we regularly either have to do horrible hacks to get these onto the client classpath, or tell developers not to use them. This is an incredible time-waster that shouldn't exist. Is there any protocol for these libraries outside of hadoop to choose an uniform version of guava (like version 21)? If not, bump up guava version here does't help as it will still break these libraries which are using different version. Also, there are also many apps developed within hadoop ecosystem which should be seen as first class citizenship that get totally break by this change. As a system software, we should have some better solution like: application classpath isolation, etc. instead of simply keeping dependency updated. I understand the concerns about stability, but this is a major upgrade, no? I don't think it's acceptable to say that Hadoop will be running Guava 11 until Hadoop 4 comes out in 2025. Guava is not Java, I don't think any hadoop releases should bind with specific guava version. It is just a third party library - with poor incompatibility - that's it!
          Hide
          vrushalic Vrushali C added a comment - - edited

          Has anyone contacted the maintainers on the major affected downstream projects like Spark and HBase? My guess is that they would be more than happy to help work around any breakages this upgrade causes – if they are like us, they would be overjoyed at being able to finally upgrade.

          There is an open jira in HBase to upgrade to guava 21 HBASE-17908 and I just noticed that Stack mentioned they are looking to shade out guava in a prebuild step.

          HBASE-13517 provided shaded client and server jars. But these shaded jars can't be used inside of coprocessors (which timeline service in yarn currently makes use of).

          For Spark it does look like their users have run into issues as noted in SPARK-16725 and they are hoping hadoop 3.x will upgrade guava.

          Show
          vrushalic Vrushali C added a comment - - edited Has anyone contacted the maintainers on the major affected downstream projects like Spark and HBase? My guess is that they would be more than happy to help work around any breakages this upgrade causes – if they are like us, they would be overjoyed at being able to finally upgrade. There is an open jira in HBase to upgrade to guava 21 HBASE-17908 and I just noticed that Stack mentioned they are looking to shade out guava in a prebuild step. HBASE-13517 provided shaded client and server jars. But these shaded jars can't be used inside of coprocessors (which timeline service in yarn currently makes use of). For Spark it does look like their users have run into issues as noted in SPARK-16725 and they are hoping hadoop 3.x will upgrade guava.
          Hide
          bpodgursky Ben Podgursky added a comment -

          I've been lurking on these Guava tickets for while a while, and I just want to chime in to say I'm going to be incredibly disappointed if this gets reverted. The number of watchers on this ticket should be an indicator in itself, and I'm sure for every person who bothered creating a jira account, there are a couple hundred who were bitten by this.

          Has anyone contacted the maintainers on the major affected downstream projects like Spark and HBase? My guess is that they would be more than happy to help work around any breakages this upgrade causes – if they are like us, they would be overjoyed at being able to finally upgrade.

          Hadoop's ancient Guava dependency is the single largest issue we run into when putting any other third party jar on our client or task classpath. There are many libraries developed outside the Hadoop ecosystem which (rightfully) assume they can use a newer version of Guava, and we regularly either have to do horrible hacks to get these onto the client classpath, or tell developers not to use them. This is an incredible time-waster that shouldn't exist.

          I understand the concerns about stability, but this is a major upgrade, no? I don't think it's acceptable to say that Hadoop will be running Guava 11 until Hadoop 4 comes out in 2025.

          Show
          bpodgursky Ben Podgursky added a comment - I've been lurking on these Guava tickets for while a while, and I just want to chime in to say I'm going to be incredibly disappointed if this gets reverted. The number of watchers on this ticket should be an indicator in itself, and I'm sure for every person who bothered creating a jira account, there are a couple hundred who were bitten by this. Has anyone contacted the maintainers on the major affected downstream projects like Spark and HBase? My guess is that they would be more than happy to help work around any breakages this upgrade causes – if they are like us, they would be overjoyed at being able to finally upgrade. Hadoop's ancient Guava dependency is the single largest issue we run into when putting any other third party jar on our client or task classpath. There are many libraries developed outside the Hadoop ecosystem which (rightfully) assume they can use a newer version of Guava, and we regularly either have to do horrible hacks to get these onto the client classpath, or tell developers not to use them. This is an incredible time-waster that shouldn't exist. I understand the concerns about stability, but this is a major upgrade, no? I don't think it's acceptable to say that Hadoop will be running Guava 11 until Hadoop 4 comes out in 2025.
          Hide
          djp Junping Du added a comment -

          Like my comments in HADOOP-14284, shading guava everywhere failed to convince me that that is a right approach. I think we should revert this patch and stay the same version with downstream projects unless someone convince me why we have to upgrade guava (the poor backward compatibility jar) here.

          Show
          djp Junping Du added a comment - Like my comments in HADOOP-14284 , shading guava everywhere failed to convince me that that is a right approach. I think we should revert this patch and stay the same version with downstream projects unless someone convince me why we have to upgrade guava (the poor backward compatibility jar) here.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          I'm going to create a JIRA so that we can build Hadoop trunk against older versions of Guava; that way andrew can build the release with an older version and we can see what breaks.

          I'm concluding that even if we ship with a newer guava v., we should build with something older (18?), so that the signatures of methods compiled against all match there. When you build with 20, overloads of checkArgument() are enough to stop existing code linking to older guava versions, even if the method call hasn't changed

          Show
          stevel@apache.org Steve Loughran added a comment - I'm going to create a JIRA so that we can build Hadoop trunk against older versions of Guava; that way andrew can build the release with an older version and we can see what breaks. I'm concluding that even if we ship with a newer guava v., we should build with something older (18?), so that the signatures of methods compiled against all match there. When you build with 20, overloads of checkArgument() are enough to stop existing code linking to older guava versions, even if the method call hasn't changed
          Hide
          andrew.wang Andrew Wang added a comment -

          FYI that I've filed HADOOP-14284 to shade Guava everywhere. Shading isn't required for all dep upgrades, but given how many projects are consuming our private artifacts and also use Guava, this is almost sure to break a lot of users.

          If we don't finish the shading in time for 3.0.0-alpha3, I'm going to revert this JIRA (HADOOP-10101) and related out of trunk. As a reminder, alpha3 is currently planned for May 15th.

          Show
          andrew.wang Andrew Wang added a comment - FYI that I've filed HADOOP-14284 to shade Guava everywhere. Shading isn't required for all dep upgrades, but given how many projects are consuming our private artifacts and also use Guava, this is almost sure to break a lot of users. If we don't finish the shading in time for 3.0.0-alpha3, I'm going to revert this JIRA ( HADOOP-10101 ) and related out of trunk. As a reminder, alpha3 is currently planned for May 15th.
          Hide
          vrushalic Vrushali C added a comment -

          This guava change is a potential concern for yarn timeline service v2 as well. It actually broke the tests in org.apache.hadoop.yarn.server.timelineservice since timeline service v2 in yarn uses hbase (which uses hadoop 2.5). YARN-6414 .

          Show
          vrushalic Vrushali C added a comment - This guava change is a potential concern for yarn timeline service v2 as well. It actually broke the tests in org.apache.hadoop.yarn.server.timelineservice since timeline service v2 in yarn uses hbase (which uses hadoop 2.5). YARN-6414 .
          Hide
          andrew.wang Andrew Wang added a comment - - edited

          It would be good to proactively reach out to downstreams who aren't using the shaded clients about this change. I looked at CDH, and found these that seem to consume hadoop-hdfs directly:

          -> % ag -l -G pom.xml "<artifactId>hadoop-hdfs</artifactId>" | cut -d/ -f 1 | sort -u
          crunch
          flume-ng
          hadoop
          hbase
          hbase-indexer
          hive
          impala
          kite
          oozie
          pig
          sentry
          sqoop
          
          Show
          andrew.wang Andrew Wang added a comment - - edited It would be good to proactively reach out to downstreams who aren't using the shaded clients about this change. I looked at CDH, and found these that seem to consume hadoop-hdfs directly: -> % ag -l -G pom.xml "<artifactId>hadoop-hdfs</artifactId>" | cut -d/ -f 1 | sort -u crunch flume-ng hadoop hbase hbase-indexer hive impala kite oozie pig sentry sqoop
          Hide
          busbey Sean Busbey added a comment -

          please, please, please write a release note for this. it is very likely to be disruptive to downstream.

          Show
          busbey Sean Busbey added a comment - please, please, please write a release note for this. it is very likely to be disruptive to downstream.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11461 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11461/)
          HADOOP-10101. Update guava dependency to the latest version. (ozawa) (ozawa: rev 84ddedc0b2d58257d45c16ee5e83b15f94a7ba3a)

          • (edit) hadoop-project/pom.xml
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalSet.java
          • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/curator/ChildReaper.java
          • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MsInfo.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/client/TestQJMWithFaults.java
          • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/lib/MetricsRegistry.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
          • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/util/MetricsCache.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInotifyEventInputStream.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/WebApp.java
          • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/AbstractMetric.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/client/TestQuorumJournalManager.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java
          • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/MetricsTag.java
          • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/ZKFailoverController.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/AclTransformation.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ApplicationSubmissionContextPBImpl.java
          • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/XAttrCommands.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationFileLoaderService.java
          • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/source/JvmMetricsInfo.java
          • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/AbstractMetricsRecord.java
          • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/lib/MetricsInfoImpl.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11461 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11461/ ) HADOOP-10101 . Update guava dependency to the latest version. (ozawa) (ozawa: rev 84ddedc0b2d58257d45c16ee5e83b15f94a7ba3a) (edit) hadoop-project/pom.xml (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalSet.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/curator/ChildReaper.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MsInfo.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/client/TestQJMWithFaults.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/lib/MetricsRegistry.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/util/MetricsCache.java (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInotifyEventInputStream.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/WebApp.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/AbstractMetric.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/client/TestQuorumJournalManager.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/MetricsTag.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/ZKFailoverController.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/AclTransformation.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ApplicationSubmissionContextPBImpl.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/XAttrCommands.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationFileLoaderService.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/source/JvmMetricsInfo.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/AbstractMetricsRecord.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/lib/MetricsInfoImpl.java
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          Committed this to trunk. Thanks Nicholas and Steve for your review, and thanks people who joined this issue for your comments.

          Show
          ozawa Tsuyoshi Ozawa added a comment - Committed this to trunk. Thanks Nicholas and Steve for your review, and thanks people who joined this issue for your comments.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          Thanks Nicholas for taking a look. Checking this in within 2 hours.

          Show
          ozawa Tsuyoshi Ozawa added a comment - Thanks Nicholas for taking a look. Checking this in within 2 hours.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          +1 the 018 patch is perfect, thanks!

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - +1 the 018 patch is perfect, thanks!
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 15s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 0m 14s Maven dependency ordering for branch
          +1 mvninstall 14m 40s trunk passed
          +1 compile 17m 4s trunk passed
          +1 checkstyle 2m 6s trunk passed
          +1 mvnsite 4m 46s trunk passed
          +1 mvneclipse 1m 58s trunk passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project
          +1 findbugs 8m 9s trunk passed
          +1 javadoc 3m 54s trunk passed
          0 mvndep 0m 21s Maven dependency ordering for patch
          +1 mvninstall 3m 51s the patch passed
          +1 compile 16m 20s the patch passed
          +1 javac 16m 20s the patch passed
          -0 checkstyle 2m 5s root: The patch generated 1 new + 388 unchanged - 2 fixed = 389 total (was 390)
          +1 mvnsite 4m 29s the patch passed
          +1 mvneclipse 2m 13s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project
          +1 findbugs 8m 26s the patch passed
          +1 javadoc 3m 52s the patch passed
          +1 unit 0m 18s hadoop-project in the patch passed.
          -1 unit 8m 33s hadoop-common in the patch failed.
          +1 unit 1m 8s hadoop-hdfs-client in the patch passed.
          -1 unit 66m 39s hadoop-hdfs in the patch failed.
          +1 unit 2m 37s hadoop-yarn-common in the patch passed.
          +1 unit 40m 23s hadoop-yarn-server-resourcemanager in the patch passed.
          +1 asflicense 0m 42s The patch does not generate ASF License warnings.
          240m 37s



          Reason Tests
          Failed junit tests hadoop.security.TestKDiag
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-10101
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12860315/HADOOP-10101.018.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 84d71d01fb1b 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / ab759e9
          Default Java 1.8.0_121
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11908/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11908/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11908/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11908/testReport/
          modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11908/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 0m 14s Maven dependency ordering for branch +1 mvninstall 14m 40s trunk passed +1 compile 17m 4s trunk passed +1 checkstyle 2m 6s trunk passed +1 mvnsite 4m 46s trunk passed +1 mvneclipse 1m 58s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project +1 findbugs 8m 9s trunk passed +1 javadoc 3m 54s trunk passed 0 mvndep 0m 21s Maven dependency ordering for patch +1 mvninstall 3m 51s the patch passed +1 compile 16m 20s the patch passed +1 javac 16m 20s the patch passed -0 checkstyle 2m 5s root: The patch generated 1 new + 388 unchanged - 2 fixed = 389 total (was 390) +1 mvnsite 4m 29s the patch passed +1 mvneclipse 2m 13s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project +1 findbugs 8m 26s the patch passed +1 javadoc 3m 52s the patch passed +1 unit 0m 18s hadoop-project in the patch passed. -1 unit 8m 33s hadoop-common in the patch failed. +1 unit 1m 8s hadoop-hdfs-client in the patch passed. -1 unit 66m 39s hadoop-hdfs in the patch failed. +1 unit 2m 37s hadoop-yarn-common in the patch passed. +1 unit 40m 23s hadoop-yarn-server-resourcemanager in the patch passed. +1 asflicense 0m 42s The patch does not generate ASF License warnings. 240m 37s Reason Tests Failed junit tests hadoop.security.TestKDiag   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-10101 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12860315/HADOOP-10101.018.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 84d71d01fb1b 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ab759e9 Default Java 1.8.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11908/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11908/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11908/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11908/testReport/ modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11908/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          ozawa Tsuyoshi Ozawa added a comment - - edited

          Another warning checkstyle raised looks checkstyle's bug - indentation should be 6 at the line, and it is now with the patch. https://github.com/checkstyle/checkstyle/issues/1174

          ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalSet.java:66: return ComparisonChain.start().compareFalseFirst(!elis1.isLocalLog(),: 'method def' child have incorrect indentation level 6, expected level should be one of the following: 8, 10, 12. [Indentation]

          Show
          ozawa Tsuyoshi Ozawa added a comment - - edited Another warning checkstyle raised looks checkstyle's bug - indentation should be 6 at the line, and it is now with the patch. https://github.com/checkstyle/checkstyle/issues/1174 ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalSet.java:66: return ComparisonChain.start().compareFalseFirst(!elis1.isLocalLog(),: 'method def' child have incorrect indentation level 6, expected level should be one of the following: 8, 10, 12. [Indentation]
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          Fixing an additional checkstyle warning - removing unused import.

          Show
          ozawa Tsuyoshi Ozawa added a comment - Fixing an additional checkstyle warning - removing unused import.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 15s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 1m 58s Maven dependency ordering for branch
          +1 mvninstall 14m 6s trunk passed
          +1 compile 15m 44s trunk passed
          +1 checkstyle 1m 58s trunk passed
          +1 mvnsite 4m 13s trunk passed
          +1 mvneclipse 1m 57s trunk passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project
          +1 findbugs 6m 58s trunk passed
          +1 javadoc 3m 29s trunk passed
          0 mvndep 0m 20s Maven dependency ordering for patch
          +1 mvninstall 3m 15s the patch passed
          +1 compile 13m 48s the patch passed
          +1 javac 13m 48s the patch passed
          -0 checkstyle 2m 3s root: The patch generated 2 new + 388 unchanged - 2 fixed = 390 total (was 390)
          +1 mvnsite 4m 45s the patch passed
          +1 mvneclipse 1m 56s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 2s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project
          +1 findbugs 8m 15s the patch passed
          +1 javadoc 3m 44s the patch passed
          +1 unit 0m 18s hadoop-project in the patch passed.
          -1 unit 8m 4s hadoop-common in the patch failed.
          +1 unit 1m 6s hadoop-hdfs-client in the patch passed.
          +1 unit 65m 50s hadoop-hdfs in the patch passed.
          +1 unit 2m 44s hadoop-yarn-common in the patch passed.
          -1 unit 40m 38s hadoop-yarn-server-resourcemanager in the patch failed.
          +1 asflicense 0m 42s The patch does not generate ASF License warnings.
          233m 2s



          Reason Tests
          Failed junit tests hadoop.net.TestDNS
            hadoop.yarn.server.resourcemanager.TestRMRestart



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-10101
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12860272/HADOOP-10101.017.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux db90677ab43a 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 1280155
          Default Java 1.8.0_121
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11905/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11905/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11905/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11905/testReport/
          modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11905/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 1m 58s Maven dependency ordering for branch +1 mvninstall 14m 6s trunk passed +1 compile 15m 44s trunk passed +1 checkstyle 1m 58s trunk passed +1 mvnsite 4m 13s trunk passed +1 mvneclipse 1m 57s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project +1 findbugs 6m 58s trunk passed +1 javadoc 3m 29s trunk passed 0 mvndep 0m 20s Maven dependency ordering for patch +1 mvninstall 3m 15s the patch passed +1 compile 13m 48s the patch passed +1 javac 13m 48s the patch passed -0 checkstyle 2m 3s root: The patch generated 2 new + 388 unchanged - 2 fixed = 390 total (was 390) +1 mvnsite 4m 45s the patch passed +1 mvneclipse 1m 56s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project +1 findbugs 8m 15s the patch passed +1 javadoc 3m 44s the patch passed +1 unit 0m 18s hadoop-project in the patch passed. -1 unit 8m 4s hadoop-common in the patch failed. +1 unit 1m 6s hadoop-hdfs-client in the patch passed. +1 unit 65m 50s hadoop-hdfs in the patch passed. +1 unit 2m 44s hadoop-yarn-common in the patch passed. -1 unit 40m 38s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 42s The patch does not generate ASF License warnings. 233m 2s Reason Tests Failed junit tests hadoop.net.TestDNS   hadoop.yarn.server.resourcemanager.TestRMRestart Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-10101 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12860272/HADOOP-10101.017.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux db90677ab43a 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 1280155 Default Java 1.8.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11905/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11905/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11905/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11905/testReport/ modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11905/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          ozawa Tsuyoshi Ozawa added a comment - - edited

          The above report is with v16, not with v17. v17 includes fixes against javac warnings, so please wait for a report with v17 patch.

          Show
          ozawa Tsuyoshi Ozawa added a comment - - edited The above report is with v16, not with v17. v17 includes fixes against javac warnings, so please wait for a report with v17 patch.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 15s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 1m 58s Maven dependency ordering for branch
          +1 mvninstall 14m 6s trunk passed
          +1 compile 16m 37s trunk passed
          +1 checkstyle 1m 57s trunk passed
          +1 mvnsite 3m 36s trunk passed
          +1 mvneclipse 1m 36s trunk passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project
          +1 findbugs 6m 6s trunk passed
          +1 javadoc 2m 59s trunk passed
          0 mvndep 0m 18s Maven dependency ordering for patch
          +1 mvninstall 3m 25s the patch passed
          +1 compile 13m 46s the patch passed
          -1 javac 13m 46s root generated 8 new + 776 unchanged - 1 fixed = 784 total (was 777)
          +1 checkstyle 2m 0s root: The patch generated 0 new + 220 unchanged - 1 fixed = 220 total (was 221)
          +1 mvnsite 3m 51s the patch passed
          +1 mvneclipse 1m 52s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project
          +1 findbugs 6m 56s the patch passed
          +1 javadoc 3m 8s the patch passed
          +1 unit 0m 16s hadoop-project in the patch passed.
          -1 unit 9m 12s hadoop-common in the patch failed.
          +1 unit 1m 15s hadoop-hdfs-client in the patch passed.
          -1 unit 69m 4s hadoop-hdfs in the patch failed.
          -1 unit 40m 40s hadoop-yarn-server-resourcemanager in the patch failed.
          +1 asflicense 0m 41s The patch does not generate ASF License warnings.
          230m 32s



          Reason Tests
          Failed junit tests hadoop.ha.TestZKFailoverController
            hadoop.security.TestKDiag
            hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation
            hadoop.yarn.server.resourcemanager.TestRMRestart



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-10101
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12860260/HADOOP-10101.016.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 3743f55b7a99 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 1280155
          Default Java 1.8.0_121
          findbugs v3.0.0
          javac https://builds.apache.org/job/PreCommit-HADOOP-Build/11904/artifact/patchprocess/diff-compile-javac-root.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11904/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11904/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11904/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11904/testReport/
          modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11904/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 1m 58s Maven dependency ordering for branch +1 mvninstall 14m 6s trunk passed +1 compile 16m 37s trunk passed +1 checkstyle 1m 57s trunk passed +1 mvnsite 3m 36s trunk passed +1 mvneclipse 1m 36s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project +1 findbugs 6m 6s trunk passed +1 javadoc 2m 59s trunk passed 0 mvndep 0m 18s Maven dependency ordering for patch +1 mvninstall 3m 25s the patch passed +1 compile 13m 46s the patch passed -1 javac 13m 46s root generated 8 new + 776 unchanged - 1 fixed = 784 total (was 777) +1 checkstyle 2m 0s root: The patch generated 0 new + 220 unchanged - 1 fixed = 220 total (was 221) +1 mvnsite 3m 51s the patch passed +1 mvneclipse 1m 52s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project +1 findbugs 6m 56s the patch passed +1 javadoc 3m 8s the patch passed +1 unit 0m 16s hadoop-project in the patch passed. -1 unit 9m 12s hadoop-common in the patch failed. +1 unit 1m 15s hadoop-hdfs-client in the patch passed. -1 unit 69m 4s hadoop-hdfs in the patch failed. -1 unit 40m 40s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 41s The patch does not generate ASF License warnings. 230m 32s Reason Tests Failed junit tests hadoop.ha.TestZKFailoverController   hadoop.security.TestKDiag   hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation   hadoop.yarn.server.resourcemanager.TestRMRestart Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-10101 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12860260/HADOOP-10101.016.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 3743f55b7a99 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 1280155 Default Java 1.8.0_121 findbugs v3.0.0 javac https://builds.apache.org/job/PreCommit-HADOOP-Build/11904/artifact/patchprocess/diff-compile-javac-root.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11904/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11904/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11904/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11904/testReport/ modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11904/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          Fixing javac warnings.

          Show
          ozawa Tsuyoshi Ozawa added a comment - Fixing javac warnings.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          Tsz Wo Nicholas Sze thanks for taking a look! More taking a look at deeper, javac warnings were caused by deprecated warnings after upgrading Guava, as you mentioned.I will update it soon. Canceling a patch.

          Show
          ozawa Tsuyoshi Ozawa added a comment - Tsz Wo Nicholas Sze thanks for taking a look! More taking a look at deeper, javac warnings were caused by deprecated warnings after upgrading Guava, as you mentioned.I will update it soon. Canceling a patch.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          Fixing checkstyle warning again.

          Show
          ozawa Tsuyoshi Ozawa added a comment - Fixing checkstyle warning again.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          Patch looks good. Thanks for working on this well known hard issue – Hadoop is stuck on Guava 11.

          Please take a look the javac/checkstyle warnings. The test failures seem unrelated. Please take a look as well. Thanks.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - Patch looks good. Thanks for working on this well known hard issue – Hadoop is stuck on Guava 11. Please take a look the javac/checkstyle warnings. The test failures seem unrelated. Please take a look as well. Thanks.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 17s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 0m 14s Maven dependency ordering for branch
          +1 mvninstall 13m 47s trunk passed
          +1 compile 21m 20s trunk passed
          +1 checkstyle 1m 57s trunk passed
          +1 mvnsite 3m 45s trunk passed
          +1 mvneclipse 1m 37s trunk passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project
          +1 findbugs 6m 31s trunk passed
          +1 javadoc 3m 13s trunk passed
          0 mvndep 0m 16s Maven dependency ordering for patch
          +1 mvninstall 3m 3s the patch passed
          +1 compile 17m 24s the patch passed
          -1 javac 17m 24s root generated 8 new + 777 unchanged - 1 fixed = 785 total (was 778)
          -0 checkstyle 2m 1s root: The patch generated 1 new + 220 unchanged - 1 fixed = 221 total (was 221)
          +1 mvnsite 3m 52s the patch passed
          +1 mvneclipse 1m 52s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project
          +1 findbugs 6m 42s the patch passed
          +1 javadoc 3m 10s the patch passed
          +1 unit 0m 21s hadoop-project in the patch passed.
          -1 unit 8m 25s hadoop-common in the patch failed.
          +1 unit 1m 8s hadoop-hdfs-client in the patch passed.
          -1 unit 70m 5s hadoop-hdfs in the patch failed.
          -1 unit 40m 14s hadoop-yarn-server-resourcemanager in the patch failed.
          +1 asflicense 0m 43s The patch does not generate ASF License warnings.
          237m 13s



          Reason Tests
          Failed junit tests hadoop.ha.TestZKFailoverController
            hadoop.security.TestKDiag
            hadoop.hdfs.server.namenode.TestUpgradeDomainBlockPlacementPolicy
            hadoop.hdfs.server.balancer.TestBalancer
            hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery
            hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer
            hadoop.yarn.server.resourcemanager.TestRMRestart



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-10101
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12859942/HADOOP-10101.015.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 41034f791bc1 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / f462e1f
          Default Java 1.8.0_121
          findbugs v3.0.0
          javac https://builds.apache.org/job/PreCommit-HADOOP-Build/11878/artifact/patchprocess/diff-compile-javac-root.txt
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11878/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11878/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11878/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11878/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11878/testReport/
          modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11878/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 0m 14s Maven dependency ordering for branch +1 mvninstall 13m 47s trunk passed +1 compile 21m 20s trunk passed +1 checkstyle 1m 57s trunk passed +1 mvnsite 3m 45s trunk passed +1 mvneclipse 1m 37s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project +1 findbugs 6m 31s trunk passed +1 javadoc 3m 13s trunk passed 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 3m 3s the patch passed +1 compile 17m 24s the patch passed -1 javac 17m 24s root generated 8 new + 777 unchanged - 1 fixed = 785 total (was 778) -0 checkstyle 2m 1s root: The patch generated 1 new + 220 unchanged - 1 fixed = 221 total (was 221) +1 mvnsite 3m 52s the patch passed +1 mvneclipse 1m 52s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project +1 findbugs 6m 42s the patch passed +1 javadoc 3m 10s the patch passed +1 unit 0m 21s hadoop-project in the patch passed. -1 unit 8m 25s hadoop-common in the patch failed. +1 unit 1m 8s hadoop-hdfs-client in the patch passed. -1 unit 70m 5s hadoop-hdfs in the patch failed. -1 unit 40m 14s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 43s The patch does not generate ASF License warnings. 237m 13s Reason Tests Failed junit tests hadoop.ha.TestZKFailoverController   hadoop.security.TestKDiag   hadoop.hdfs.server.namenode.TestUpgradeDomainBlockPlacementPolicy   hadoop.hdfs.server.balancer.TestBalancer   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery   hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer   hadoop.yarn.server.resourcemanager.TestRMRestart Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-10101 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12859942/HADOOP-10101.015.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 41034f791bc1 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / f462e1f Default Java 1.8.0_121 findbugs v3.0.0 javac https://builds.apache.org/job/PreCommit-HADOOP-Build/11878/artifact/patchprocess/diff-compile-javac-root.txt checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11878/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11878/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11878/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11878/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11878/testReport/ modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11878/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          ozawa Tsuyoshi Ozawa added a comment - - edited

          Uploading v15 patch. It fixes checkstyle warning. javac warnings and TestKDiag failure are not related to the patch: javac warnings are by using deprecated methods which are not related to the patch, and TestKDiag passes on my local.

          Show
          ozawa Tsuyoshi Ozawa added a comment - - edited Uploading v15 patch. It fixes checkstyle warning. javac warnings and TestKDiag failure are not related to the patch: javac warnings are by using deprecated methods which are not related to the patch, and TestKDiag passes on my local.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 0m 13s Maven dependency ordering for branch
          +1 mvninstall 12m 40s trunk passed
          +1 compile 21m 32s trunk passed
          +1 checkstyle 1m 59s trunk passed
          +1 mvnsite 3m 36s trunk passed
          +1 mvneclipse 1m 37s trunk passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project
          +1 findbugs 5m 58s trunk passed
          +1 javadoc 2m 56s trunk passed
          0 mvndep 0m 25s Maven dependency ordering for patch
          +1 mvninstall 2m 42s the patch passed
          +1 compile 16m 52s the patch passed
          -1 javac 16m 52s root generated 8 new + 777 unchanged - 1 fixed = 785 total (was 778)
          -0 checkstyle 2m 1s root: The patch generated 3 new + 220 unchanged - 1 fixed = 223 total (was 221)
          +1 mvnsite 3m 51s the patch passed
          +1 mvneclipse 1m 53s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project
          +1 findbugs 6m 42s the patch passed
          +1 javadoc 3m 12s the patch passed
          +1 unit 0m 20s hadoop-project in the patch passed.
          -1 unit 8m 8s hadoop-common in the patch failed.
          +1 unit 1m 9s hadoop-hdfs-client in the patch passed.
          +1 unit 65m 1s hadoop-hdfs in the patch passed.
          +1 unit 39m 58s hadoop-yarn-server-resourcemanager in the patch passed.
          +1 asflicense 0m 42s The patch does not generate ASF License warnings.
          228m 57s



          Reason Tests
          Failed junit tests hadoop.security.TestKDiag



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-10101
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12859873/HADOOP-10101.014.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 1dee64063c26 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / f462e1f
          Default Java 1.8.0_121
          findbugs v3.0.0
          javac https://builds.apache.org/job/PreCommit-HADOOP-Build/11873/artifact/patchprocess/diff-compile-javac-root.txt
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11873/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11873/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11873/testReport/
          modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11873/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 0m 13s Maven dependency ordering for branch +1 mvninstall 12m 40s trunk passed +1 compile 21m 32s trunk passed +1 checkstyle 1m 59s trunk passed +1 mvnsite 3m 36s trunk passed +1 mvneclipse 1m 37s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project +1 findbugs 5m 58s trunk passed +1 javadoc 2m 56s trunk passed 0 mvndep 0m 25s Maven dependency ordering for patch +1 mvninstall 2m 42s the patch passed +1 compile 16m 52s the patch passed -1 javac 16m 52s root generated 8 new + 777 unchanged - 1 fixed = 785 total (was 778) -0 checkstyle 2m 1s root: The patch generated 3 new + 220 unchanged - 1 fixed = 223 total (was 221) +1 mvnsite 3m 51s the patch passed +1 mvneclipse 1m 53s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project +1 findbugs 6m 42s the patch passed +1 javadoc 3m 12s the patch passed +1 unit 0m 20s hadoop-project in the patch passed. -1 unit 8m 8s hadoop-common in the patch failed. +1 unit 1m 9s hadoop-hdfs-client in the patch passed. +1 unit 65m 1s hadoop-hdfs in the patch passed. +1 unit 39m 58s hadoop-yarn-server-resourcemanager in the patch passed. +1 asflicense 0m 42s The patch does not generate ASF License warnings. 228m 57s Reason Tests Failed junit tests hadoop.security.TestKDiag Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-10101 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12859873/HADOOP-10101.014.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 1dee64063c26 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / f462e1f Default Java 1.8.0_121 findbugs v3.0.0 javac https://builds.apache.org/job/PreCommit-HADOOP-Build/11873/artifact/patchprocess/diff-compile-javac-root.txt checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11873/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11873/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11873/testReport/ modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11873/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          Hitesh Shah thanks for sharing the information. IIUC, is the problem method AMRMClient#waitFor? It takes a com.google.common.base.Supplier instance as an argument.

          Show
          ozawa Tsuyoshi Ozawa added a comment - Hitesh Shah thanks for sharing the information. IIUC, is the problem method AMRMClient#waitFor? It takes a com.google.common.base.Supplier instance as an argument.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          Since HADOOP-14187 is merged, I'm resubmitting Jenkins CI again.

          Show
          ozawa Tsuyoshi Ozawa added a comment - Since HADOOP-14187 is merged, I'm resubmitting Jenkins CI again.
          Hide
          ozawa Tsuyoshi Ozawa added a comment - - edited

          After working to update Curator for a while, I found that Curator 3.x only supports ZooKeeper 3.5.x. On HADOOP-14187, avoiding the drastic change(YARN-3774) now, I would like to suggest updating ZooKeeper dependency to 3.4.9 and Curator dependency to 2.12.0, which supports ZooKeeper 3.4.x. It's because our current focus here is to update Guava - the blocker of this issue is to introduce Curator which includes shaded Guava.

          I tested with v13 patch on HADOOP-14187 on my local:

          mvn install test -Dtest=TestZKDelegationTokenSecretManager,TestChildReaper,TestKDiag,TestDelegationTokenRenewer,TestSubmitApplicationWithRMHA,TestLeaderElectorService,TestRMHAForNodeLabels,TestZKRMStateStorePerf,TestKillApplicationWithRMHA,TestRMStoreCommands,TestZKRMStateStore,TestRMRestart,TestZKRMStateStoreZKClientConnections,TestReservationSystemWithRMHA
          

          All test passes on my local basically. Flaky tests sometimes fail, but they're not ZK-related. The reason seems to be YARN-5548.

          Show
          ozawa Tsuyoshi Ozawa added a comment - - edited After working to update Curator for a while, I found that Curator 3.x only supports ZooKeeper 3.5.x. On HADOOP-14187 , avoiding the drastic change( YARN-3774 ) now, I would like to suggest updating ZooKeeper dependency to 3.4.9 and Curator dependency to 2.12.0, which supports ZooKeeper 3.4.x. It's because our current focus here is to update Guava - the blocker of this issue is to introduce Curator which includes shaded Guava. I tested with v13 patch on HADOOP-14187 on my local: mvn install test -Dtest=TestZKDelegationTokenSecretManager,TestChildReaper,TestKDiag,TestDelegationTokenRenewer,TestSubmitApplicationWithRMHA,TestLeaderElectorService,TestRMHAForNodeLabels,TestZKRMStateStorePerf,TestKillApplicationWithRMHA,TestRMStoreCommands,TestZKRMStateStore,TestRMRestart,TestZKRMStateStoreZKClientConnections,TestReservationSystemWithRMHA All test passes on my local basically. Flaky tests sometimes fail, but they're not ZK-related. The reason seems to be YARN-5548 .
          Hide
          ozawa Tsuyoshi Ozawa added a comment - - edited

          OIC, we need to upgrade Curator on YARN-3774 at first. I would like to take Curator's update before solving this issue.

          Show
          ozawa Tsuyoshi Ozawa added a comment - - edited OIC, we need to upgrade Curator on YARN-3774 at first. I would like to take Curator's update before solving this issue.
          Hide
          hitesh Hitesh Shah added a comment -

          IIRC, YarnClient has a guava object being used in one of its public APIs.

          Show
          hitesh Hitesh Shah added a comment - IIRC, YarnClient has a guava object being used in one of its public APIs.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 1m 48s Maven dependency ordering for branch
          +1 mvninstall 12m 24s trunk passed
          +1 compile 10m 34s trunk passed
          +1 checkstyle 2m 1s trunk passed
          +1 mvnsite 4m 1s trunk passed
          +1 mvneclipse 2m 1s trunk passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project
          +1 findbugs 6m 16s trunk passed
          +1 javadoc 3m 21s trunk passed
          0 mvndep 0m 19s Maven dependency ordering for patch
          +1 mvninstall 2m 43s the patch passed
          +1 compile 10m 20s the patch passed
          -1 javac 10m 20s root generated 8 new + 754 unchanged - 1 fixed = 762 total (was 755)
          -0 checkstyle 2m 5s root: The patch generated 3 new + 220 unchanged - 1 fixed = 223 total (was 221)
          +1 mvnsite 4m 14s the patch passed
          +1 mvneclipse 2m 18s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project
          +1 findbugs 7m 0s the patch passed
          +1 javadoc 3m 37s the patch passed
          +1 unit 0m 26s hadoop-project in the patch passed.
          -1 unit 7m 56s hadoop-common in the patch failed.
          +1 unit 1m 13s hadoop-hdfs-client in the patch passed.
          +1 unit 64m 3s hadoop-hdfs in the patch passed.
          -1 unit 38m 46s hadoop-yarn-server-resourcemanager in the patch failed.
          +1 asflicense 0m 52s The patch does not generate ASF License warnings.
          214m 6s



          Reason Tests
          Failed junit tests hadoop.security.TestKDiag
            hadoop.security.token.delegation.TestZKDelegationTokenSecretManager
            hadoop.util.curator.TestChildReaper
            hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer
            hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA
            hadoop.yarn.server.resourcemanager.TestLeaderElectorService
            hadoop.yarn.server.resourcemanager.TestRMHAForNodeLabels
            hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStorePerf
            hadoop.yarn.server.resourcemanager.TestKillApplicationWithRMHA
            hadoop.yarn.server.resourcemanager.TestRMStoreCommands
            hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStore
            hadoop.yarn.server.resourcemanager.TestRMRestart
            hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStoreZKClientConnections
            hadoop.yarn.server.resourcemanager.TestReservationSystemWithRMHA



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-10101
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12858877/HADOOP-10101.013.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 9522f9ab4b66 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / bb6a214
          Default Java 1.8.0_121
          findbugs v3.0.0
          javac https://builds.apache.org/job/PreCommit-HADOOP-Build/11828/artifact/patchprocess/diff-compile-javac-root.txt
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11828/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11828/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11828/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11828/testReport/
          modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11828/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 1m 48s Maven dependency ordering for branch +1 mvninstall 12m 24s trunk passed +1 compile 10m 34s trunk passed +1 checkstyle 2m 1s trunk passed +1 mvnsite 4m 1s trunk passed +1 mvneclipse 2m 1s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project +1 findbugs 6m 16s trunk passed +1 javadoc 3m 21s trunk passed 0 mvndep 0m 19s Maven dependency ordering for patch +1 mvninstall 2m 43s the patch passed +1 compile 10m 20s the patch passed -1 javac 10m 20s root generated 8 new + 754 unchanged - 1 fixed = 762 total (was 755) -0 checkstyle 2m 5s root: The patch generated 3 new + 220 unchanged - 1 fixed = 223 total (was 221) +1 mvnsite 4m 14s the patch passed +1 mvneclipse 2m 18s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project +1 findbugs 7m 0s the patch passed +1 javadoc 3m 37s the patch passed +1 unit 0m 26s hadoop-project in the patch passed. -1 unit 7m 56s hadoop-common in the patch failed. +1 unit 1m 13s hadoop-hdfs-client in the patch passed. +1 unit 64m 3s hadoop-hdfs in the patch passed. -1 unit 38m 46s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 52s The patch does not generate ASF License warnings. 214m 6s Reason Tests Failed junit tests hadoop.security.TestKDiag   hadoop.security.token.delegation.TestZKDelegationTokenSecretManager   hadoop.util.curator.TestChildReaper   hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer   hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA   hadoop.yarn.server.resourcemanager.TestLeaderElectorService   hadoop.yarn.server.resourcemanager.TestRMHAForNodeLabels   hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStorePerf   hadoop.yarn.server.resourcemanager.TestKillApplicationWithRMHA   hadoop.yarn.server.resourcemanager.TestRMStoreCommands   hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStore   hadoop.yarn.server.resourcemanager.TestRMRestart   hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStoreZKClientConnections   hadoop.yarn.server.resourcemanager.TestReservationSystemWithRMHA Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-10101 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12858877/HADOOP-10101.013.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 9522f9ab4b66 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / bb6a214 Default Java 1.8.0_121 findbugs v3.0.0 javac https://builds.apache.org/job/PreCommit-HADOOP-Build/11828/artifact/patchprocess/diff-compile-javac-root.txt checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11828/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11828/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11828/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11828/testReport/ modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11828/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          Uploading v13 patch.

          Show
          ozawa Tsuyoshi Ozawa added a comment - Uploading v13 patch.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          On my local, compilation failed with the patch by changes of Guava's interface. Updating it soon.

          Show
          ozawa Tsuyoshi Ozawa added a comment - On my local, compilation failed with the patch by changes of Guava's interface. Updating it soon.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 12m 18s trunk passed
          +1 compile 0m 9s trunk passed
          +1 mvnsite 0m 11s trunk passed
          +1 mvneclipse 0m 9s trunk passed
          +1 javadoc 0m 10s trunk passed
          +1 mvninstall 0m 7s the patch passed
          +1 compile 0m 6s the patch passed
          +1 javac 0m 6s the patch passed
          +1 mvnsite 0m 8s the patch passed
          +1 mvneclipse 0m 6s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 javadoc 0m 7s the patch passed
          +1 unit 0m 6s hadoop-project in the patch passed.
          +1 asflicense 0m 15s The patch does not generate ASF License warnings.
          14m 53s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-10101
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12858811/HADOOP-10101.012.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml
          uname Linux 905876c9c205 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / cc1292e
          Default Java 1.8.0_121
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11827/testReport/
          modules C: hadoop-project U: hadoop-project
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11827/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 12m 18s trunk passed +1 compile 0m 9s trunk passed +1 mvnsite 0m 11s trunk passed +1 mvneclipse 0m 9s trunk passed +1 javadoc 0m 10s trunk passed +1 mvninstall 0m 7s the patch passed +1 compile 0m 6s the patch passed +1 javac 0m 6s the patch passed +1 mvnsite 0m 8s the patch passed +1 mvneclipse 0m 6s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 javadoc 0m 7s the patch passed +1 unit 0m 6s hadoop-project in the patch passed. +1 asflicense 0m 15s The patch does not generate ASF License warnings. 14m 53s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-10101 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12858811/HADOOP-10101.012.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml uname Linux 905876c9c205 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / cc1292e Default Java 1.8.0_121 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11827/testReport/ modules C: hadoop-project U: hadoop-project Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11827/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          I chose version 21.0, the latest version of Guava. It requires jdk 8 or later and fits in hadoop 3.0.

          Show
          ozawa Tsuyoshi Ozawa added a comment - I chose version 21.0, the latest version of Guava. It requires jdk 8 or later and fits in hadoop 3.0.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          Attaching a patch to upgrade guava to run tests.

          Show
          ozawa Tsuyoshi Ozawa added a comment - Attaching a patch to upgrade guava to run tests.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          Thanks Andrew for sharing HADOOP-11804. Now, we can focus on upgrading guava here. The exposed objects of Guava may be a problem as Steve mentioned. I will create a following jira to address the issue.

          Show
          ozawa Tsuyoshi Ozawa added a comment - Thanks Andrew for sharing HADOOP-11804 . Now, we can focus on upgrading guava here. The exposed objects of Guava may be a problem as Steve mentioned. I will create a following jira to address the issue.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          The issue for APIs is that public APIs can't shade things, as then only callers with matching shade declarations can use them (personal experience there with hive & spark & kryo). It's ok privately, but the issue is defining "private" is a hard one, given that things like UGI have for a long time been tagged as private. Essentially: anything down as @LimitedPrivate for MapReduce has to be assumed as "needed by YARN apps" and touched very, very carefully

          Show
          stevel@apache.org Steve Loughran added a comment - The issue for APIs is that public APIs can't shade things, as then only callers with matching shade declarations can use them (personal experience there with hive & spark & kryo). It's ok privately, but the issue is defining "private" is a hard one, given that things like UGI have for a long time been tagged as private. Essentially: anything down as @LimitedPrivate for MapReduce has to be assumed as "needed by YARN apps" and touched very, very carefully
          Hide
          andrew.wang Andrew Wang added a comment -

          Is this addressed by using the shaded client introduced by HADOOP-11804 for Hadoop 3? It shades Guava also.

          Show
          andrew.wang Andrew Wang added a comment - Is this addressed by using the shaded client introduced by HADOOP-11804 for Hadoop 3? It shades Guava also.
          Hide
          rdub Ryan Williams added a comment -

          Need to make sure that Guava objects aren't in use in "public" APIs

          Are there any tools that analyze public vs. private usage of dependencies? Seems important but I've not seen any.

          where public includes those bits of the private APIs which are actually needed to produce working code.

          Can you elaborate on this? My (naive) impression was that anything not "leaked" in method-signatures could be shaded.

          Show
          rdub Ryan Williams added a comment - Need to make sure that Guava objects aren't in use in "public" APIs Are there any tools that analyze public vs. private usage of dependencies? Seems important but I've not seen any. where public includes those bits of the private APIs which are actually needed to produce working code. Can you elaborate on this? My (naive) impression was that anything not "leaked" in method-signatures could be shaded.
          Hide
          ozawa Tsuyoshi Ozawa added a comment - - edited

          I would like to resume working on this jira, based on shading approach as Nicholas and Hitesh mentioned. Please give me to try the approach for a few days.

          why version 18 and not 19? Guava 19 was released in 2015 or so

          It's because Guava 19 had more incompatible change. Now, 21.0, which requires JDK-8 or upper, has been released. I would like to create a patch to update Guava's version to 21.0, targeting branch-3.

          have you confirmed that no deprecated and/or beta apis are being used?

          Not yet.

          I believe there are some places where guava objects are being used in public facing APIs. This might not be something to address in this jira but should definitely be looked at as a blocker before the next release in addition to my initial comment.

          Yeah, let's do this on another jira. I think we should create this wrapper class instead of exposing Guava's objects directly.

          I did try a quick prototype approach on branch-2 and it seems to function without directly pulling in guava ( except of course - the curator guava dependency problem )

          Curator have also shared Guava. If we can move the version to up to date version, we can avoid the problem.

          Show
          ozawa Tsuyoshi Ozawa added a comment - - edited I would like to resume working on this jira, based on shading approach as Nicholas and Hitesh mentioned. Please give me to try the approach for a few days. why version 18 and not 19? Guava 19 was released in 2015 or so It's because Guava 19 had more incompatible change. Now, 21.0, which requires JDK-8 or upper, has been released. I would like to create a patch to update Guava's version to 21.0, targeting branch-3. have you confirmed that no deprecated and/or beta apis are being used? Not yet. I believe there are some places where guava objects are being used in public facing APIs. This might not be something to address in this jira but should definitely be looked at as a blocker before the next release in addition to my initial comment. Yeah, let's do this on another jira. I think we should create this wrapper class instead of exposing Guava's objects directly. I did try a quick prototype approach on branch-2 and it seems to function without directly pulling in guava ( except of course - the curator guava dependency problem ) Curator have also shared Guava. If we can move the version to up to date version, we can avoid the problem.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Need to make sure that Guava objects aren't in use in "public" APIs, where public includes those bits of the private APIs which are actually needed to produce working code. There's a lot in YARN, and historically large bits of Hadoop common

          Lei Guo based on protobuf experience, I woudn't trust any google lib to have backwards compatibility. Think very hard about that before adopting grpc in any of your own code

          Show
          stevel@apache.org Steve Loughran added a comment - Need to make sure that Guava objects aren't in use in "public" APIs, where public includes those bits of the private APIs which are actually needed to produce working code. There's a lot in YARN, and historically large bits of Hadoop common Lei Guo based on protobuf experience, I woudn't trust any google lib to have backwards compatibility. Think very hard about that before adopting grpc in any of your own code
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 patch 0m 9s HADOOP-10101 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



          Subsystem Report/Notes
          JIRA Issue HADOOP-10101
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12699092/HADOOP-10101-011.patch
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11815/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 9s HADOOP-10101 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Issue HADOOP-10101 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12699092/HADOOP-10101-011.patch Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11815/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          > I think it is high time we bump up the dependency however we should look to hiding it too i.e. shading it as needed so that downstream users can use whatever version of guava they want to use.

          I agree with Hitesh Shah that we should shade guava. Then, this JIRA is no longer an incompatible change.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - > I think it is high time we bump up the dependency however we should look to hiding it too i.e. shading it as needed so that downstream users can use whatever version of guava they want to use. I agree with Hitesh Shah that we should shade guava. Then, this JIRA is no longer an incompatible change.
          Hide
          grey Lei Guo added a comment - - edited

          We hit one problem with Guava version 18 for HDFS, The application we built is based on grpc, guava version 18 is minimum required

          Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.base.Enums.valueOfFunction(Ljava/lang/Class;)Lcom/google/common/base/Function;
          at org.apache.hadoop.fs.shell.XAttrCommands$GetfattrCommand.<clinit>(XAttrCommands.java:69)
          at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
          at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
          at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
          at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
          at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:132)
          at org.apache.hadoop.fs.shell.CommandFactory.getInstance(CommandFactory.java:125)
          at org.apache.hadoop.fs.shell.CommandFactory.getInstance(CommandFactory.java:109)
          at org.apache.hadoop.fs.FsShell.run(FsShell.java:290)
          at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
          at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
          at org.apache.hadoop.fs.FsShell.main(FsShell.java:352)

          Show
          grey Lei Guo added a comment - - edited We hit one problem with Guava version 18 for HDFS, The application we built is based on grpc, guava version 18 is minimum required Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.base.Enums.valueOfFunction(Ljava/lang/Class;)Lcom/google/common/base/Function; at org.apache.hadoop.fs.shell.XAttrCommands$GetfattrCommand.<clinit>(XAttrCommands.java:69) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:132) at org.apache.hadoop.fs.shell.CommandFactory.getInstance(CommandFactory.java:125) at org.apache.hadoop.fs.shell.CommandFactory.getInstance(CommandFactory.java:109) at org.apache.hadoop.fs.FsShell.run(FsShell.java:290) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) at org.apache.hadoop.fs.FsShell.main(FsShell.java:352)
          Hide
          stevel@apache.org Steve Loughran added a comment -

          whatever the latest version goes hitesh, that's what the title says. the Hadoop code is designed to work with later versions (i.e we've moved off bits that were taken away), but still ship with an older one. I'd love to bump us up to a new one

          Show
          stevel@apache.org Steve Loughran added a comment - whatever the latest version goes hitesh, that's what the title says. the Hadoop code is designed to work with later versions (i.e we've moved off bits that were taken away), but still ship with an older one. I'd love to bump us up to a new one
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 patch 0m 7s HADOOP-10101 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



          Subsystem Report/Notes
          JIRA Issue HADOOP-10101
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12699092/HADOOP-10101-011.patch
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11244/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 7s HADOOP-10101 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Issue HADOOP-10101 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12699092/HADOOP-10101-011.patch Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11244/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hitesh Hitesh Shah added a comment -

          I think it is high time we bump up the dependency however we should look to hiding it too i.e. shading it as needed so that downstream users can use whatever version of guava they want to use.

          Some questions:

          • why version 18 and not 19? Guava 19 was released in 2015 or so
          • have you confirmed that no deprecated and/or beta apis are being used?
          • I believe there are some places where guava objects are being used in public facing APIs. This might not be something to address in this jira but should definitely be looked at as a blocker before the next release in addition to my initial comment.
          Show
          hitesh Hitesh Shah added a comment - I think it is high time we bump up the dependency however we should look to hiding it too i.e. shading it as needed so that downstream users can use whatever version of guava they want to use. Some questions: why version 18 and not 19? Guava 19 was released in 2015 or so have you confirmed that no deprecated and/or beta apis are being used? I believe there are some places where guava objects are being used in public facing APIs. This might not be something to address in this jira but should definitely be looked at as a blocker before the next release in addition to my initial comment.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 patch 0m 6s HADOOP-10101 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



          Subsystem Report/Notes
          JIRA Issue HADOOP-10101
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12699092/HADOOP-10101-011.patch
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10648/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 6s HADOOP-10101 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Issue HADOOP-10101 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12699092/HADOOP-10101-011.patch Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10648/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          wutaklon@amazon.com Taklon Stephen Wu added a comment -

          Ping, any plan on upgrading Guava in this thread? btw, Guava is on 19.0.

          Show
          wutaklon@amazon.com Taklon Stephen Wu added a comment - Ping, any plan on upgrading Guava in this thread? btw, Guava is on 19.0.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 patch 0m 5s HADOOP-10101 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



          Subsystem Report/Notes
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12699092/HADOOP-10101-011.patch
          JIRA Issue HADOOP-10101
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9842/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 5s HADOOP-10101 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12699092/HADOOP-10101-011.patch JIRA Issue HADOOP-10101 Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9842/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user afs commented on the pull request:

          https://github.com/apache/jena/pull/87#issuecomment-123468596

          org.apache.hadoop:hadoop-common:jar:2.6.0 depends on com.google.guava:guava:jar:11.0.2 (provided).

          Just because Jena tests work isn't evidence Guava 16.0.1 works unfortunately. Jena is a library and may be co-resident with other code. See HADOOP-10101 including the initial attempt at migration to 15.0 which did not go in. At this very late stage, it needs a strong advantage for users for change.

          (and this is why we shade guava for Jena!)

          Show
          githubbot ASF GitHub Bot added a comment - Github user afs commented on the pull request: https://github.com/apache/jena/pull/87#issuecomment-123468596 org.apache.hadoop:hadoop-common:jar:2.6.0 depends on com.google.guava:guava:jar:11.0.2 (provided). Just because Jena tests work isn't evidence Guava 16.0.1 works unfortunately. Jena is a library and may be co-resident with other code. See HADOOP-10101 including the initial attempt at migration to 15.0 which did not go in. At this very late stage, it needs a strong advantage for users for change. (and this is why we shade guava for Jena!)
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 18m 56s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          -1 javac 7m 35s The applied patch generated 1 additional warning messages.
          +1 javadoc 9m 37s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 43s The applied patch generated 3 new checkstyle issues (total was 83, now 86).
          -1 checkstyle 2m 23s The applied patch generated 2 new checkstyle issues (total was 43, now 44).
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 20s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 4m 19s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 common tests 22m 2s Tests passed in hadoop-common.
          -1 hdfs tests 160m 45s Tests failed in hadoop-hdfs.
              227m 59s  



          Reason Tests
          Failed unit tests hadoop.hdfs.TestDistributedFileSystem
            hadoop.hdfs.TestAppendSnapshotTruncate
            hadoop.hdfs.TestLeaseRecovery2



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12699092/HADOOP-10101-011.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 419c51d
          javac https://builds.apache.org/job/PreCommit-HADOOP-Build/7298/artifact/patchprocess/diffJavacWarnings.txt
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/7298/artifact/patchprocess/diffcheckstylehadoop-common.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/7298/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
          hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7298/artifact/patchprocess/testrun_hadoop-common.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7298/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7298/testReport/
          Java 1.7.0_55
          uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7298/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 18m 56s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. -1 javac 7m 35s The applied patch generated 1 additional warning messages. +1 javadoc 9m 37s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 43s The applied patch generated 3 new checkstyle issues (total was 83, now 86). -1 checkstyle 2m 23s The applied patch generated 2 new checkstyle issues (total was 43, now 44). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 20s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 4m 19s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 common tests 22m 2s Tests passed in hadoop-common. -1 hdfs tests 160m 45s Tests failed in hadoop-hdfs.     227m 59s   Reason Tests Failed unit tests hadoop.hdfs.TestDistributedFileSystem   hadoop.hdfs.TestAppendSnapshotTruncate   hadoop.hdfs.TestLeaseRecovery2 Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12699092/HADOOP-10101-011.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 419c51d javac https://builds.apache.org/job/PreCommit-HADOOP-Build/7298/artifact/patchprocess/diffJavacWarnings.txt checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/7298/artifact/patchprocess/diffcheckstylehadoop-common.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/7298/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7298/artifact/patchprocess/testrun_hadoop-common.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7298/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7298/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7298/console This message was automatically generated.
          Hide
          hitesh Hitesh Shah added a comment -

          I hit something similar in tez recently and looking at this jira, I am wondering as to why shading ( relocating all classes ) the guava jar is not an option?

          Consider the following approach:

          • introduce a new hadoop-guava module which depends on guava version X.
          • hadoop-guava relocates all classes in guava to org.apache.hadoop.shaded.guava
          • all code in hadoop depends on hadoop-guava and excludes the google-guava dependency

          I did try a quick prototype approach on branch-2 and it seems to function without directly pulling in guava ( except of course - the curator guava dependency problem )

          Show
          hitesh Hitesh Shah added a comment - I hit something similar in tez recently and looking at this jira, I am wondering as to why shading ( relocating all classes ) the guava jar is not an option? Consider the following approach: introduce a new hadoop-guava module which depends on guava version X. hadoop-guava relocates all classes in guava to org.apache.hadoop.shaded.guava all code in hadoop depends on hadoop-guava and excludes the google-guava dependency I did try a quick prototype approach on branch-2 and it seems to function without directly pulling in guava ( except of course - the curator guava dependency problem )
          Hide
          karel1980 Karel Vervaeke added a comment -

          I think Sean Busbey made a good point. I'm sure there are some practical concerns, but as more libraries start to depend on more recent guava versions it becomes more of a problem.
          For now I've taken the reverse approach for our integration tests (shading dropwizard 0.8.1 w. relocated guava classes) to allow both hadoop and dropwizard services to run in one jvm.

          Show
          karel1980 Karel Vervaeke added a comment - I think Sean Busbey made a good point. I'm sure there are some practical concerns, but as more libraries start to depend on more recent guava versions it becomes more of a problem. For now I've taken the reverse approach for our integration tests (shading dropwizard 0.8.1 w. relocated guava classes) to allow both hadoop and dropwizard services to run in one jvm.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          The javac warning looks not related to the patch. Steve Loughran, could you take a look at v11 patch?

          Show
          ozawa Tsuyoshi Ozawa added a comment - The javac warning looks not related to the patch. Steve Loughran , could you take a look at v11 patch?
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12699092/HADOOP-10101-011.patch
          against trunk revision f0412de.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          -1 javac. The applied patch generated 1152 javac compiler warnings (more than the trunk's current 1151 warnings).

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5727//testReport/
          Javac warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5727//artifact/patchprocess/diffJavacWarnings.txt
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5727//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12699092/HADOOP-10101-011.patch against trunk revision f0412de. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. -1 javac . The applied patch generated 1152 javac compiler warnings (more than the trunk's current 1151 warnings). +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5727//testReport/ Javac warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5727//artifact/patchprocess/diffJavacWarnings.txt Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5727//console This message is automatically generated.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          Submitting v11 patch since HADOOP-11600 was committed.

          Show
          ozawa Tsuyoshi Ozawa added a comment - Submitting v11 patch since HADOOP-11600 was committed.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          This patch is only for trunk and can be applied after HADOOP-11600 is committed.

          Show
          ozawa Tsuyoshi Ozawa added a comment - This patch is only for trunk and can be applied after HADOOP-11600 is committed.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          Talked with Steve offline. The patch is now very large, so we should split them:

          1. Fix up usage of Guava's methods for both branch-2 and trunk on HADOOP-11600.
          2. Locale bug is addressed on HADOOP-11602.
          3. trunk-specific change to upgrade Guava 18.0 on this issue(HADOOP-10101).

          3 depends on 1, so I'll submit a patch after 1 is committed.

          Show
          ozawa Tsuyoshi Ozawa added a comment - Talked with Steve offline. The patch is now very large, so we should split them: 1. Fix up usage of Guava's methods for both branch-2 and trunk on HADOOP-11600 . 2. Locale bug is addressed on HADOOP-11602 . 3. trunk-specific change to upgrade Guava 18.0 on this issue( HADOOP-10101 ). 3 depends on 1, so I'll submit a patch after 1 is committed.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          The test failures are not related to the patch. Steve Loughran, do you mind taking a look? I'll create a patches for branch-2 after trunk code.

          Show
          ozawa Tsuyoshi Ozawa added a comment - The test failures are not related to the patch. Steve Loughran , do you mind taking a look? I'll create a patches for branch-2 after trunk code.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12698912/HADOOP-10101-010.patch
          against trunk revision ef950ea.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 18 new or modified test files.

          -1 javac. The applied patch generated 1152 javac compiler warnings (more than the trunk's current 1151 warnings).

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 19 new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-annotations hadoop-common-project/hadoop-auth hadoop-common-project/hadoop-common hadoop-common-project/hadoop-nfs hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-maven-plugins hadoop-tools/hadoop-azure hadoop-tools/hadoop-distcp hadoop-tools/hadoop-extras hadoop-tools/hadoop-gridmix hadoop-tools/hadoop-rumen hadoop-tools/hadoop-streaming hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:

          org.apache.hadoop.conf.TestJobConf
          org.apache.hadoop.mapred.gridmix.TestHighRamJob
          org.apache.hadoop.fs.http.client.TestHttpFSFWithWebhdfsFileSystem

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5698//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5698//artifact/patchprocess/newPatchFindbugsWarningshadoop-maven-plugins.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5698//artifact/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5698//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
          Javac warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5698//artifact/patchprocess/diffJavacWarnings.txt
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5698//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12698912/HADOOP-10101-010.patch against trunk revision ef950ea. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 18 new or modified test files. -1 javac . The applied patch generated 1152 javac compiler warnings (more than the trunk's current 1151 warnings). +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 19 new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-common-project/hadoop-annotations hadoop-common-project/hadoop-auth hadoop-common-project/hadoop-common hadoop-common-project/hadoop-nfs hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-maven-plugins hadoop-tools/hadoop-azure hadoop-tools/hadoop-distcp hadoop-tools/hadoop-extras hadoop-tools/hadoop-gridmix hadoop-tools/hadoop-rumen hadoop-tools/hadoop-streaming hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: org.apache.hadoop.conf.TestJobConf org.apache.hadoop.mapred.gridmix.TestHighRamJob org.apache.hadoop.fs.http.client.TestHttpFSFWithWebhdfsFileSystem Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5698//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5698//artifact/patchprocess/newPatchFindbugsWarningshadoop-maven-plugins.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5698//artifact/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5698//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html Javac warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5698//artifact/patchprocess/diffJavacWarnings.txt Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5698//console This message is automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12698904/HADOOP-10101-010.patch
          against trunk revision 6804d68.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 18 new or modified test files.

          -1 javac. The applied patch generated 1152 javac compiler warnings (more than the trunk's current 1151 warnings).

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 19 new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-annotations hadoop-common-project/hadoop-auth hadoop-common-project/hadoop-common hadoop-common-project/hadoop-nfs hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-maven-plugins hadoop-tools/hadoop-azure hadoop-tools/hadoop-distcp hadoop-tools/hadoop-extras hadoop-tools/hadoop-gridmix hadoop-tools/hadoop-rumen hadoop-tools/hadoop-streaming hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:

          org.apache.hadoop.conf.TestJobConf
          org.apache.hadoop.mapred.gridmix.TestHighRamJob

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5696//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5696//artifact/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5696//artifact/patchprocess/newPatchFindbugsWarningshadoop-maven-plugins.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5696//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
          Javac warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5696//artifact/patchprocess/diffJavacWarnings.txt
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5696//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12698904/HADOOP-10101-010.patch against trunk revision 6804d68. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 18 new or modified test files. -1 javac . The applied patch generated 1152 javac compiler warnings (more than the trunk's current 1151 warnings). +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 19 new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-common-project/hadoop-annotations hadoop-common-project/hadoop-auth hadoop-common-project/hadoop-common hadoop-common-project/hadoop-nfs hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-maven-plugins hadoop-tools/hadoop-azure hadoop-tools/hadoop-distcp hadoop-tools/hadoop-extras hadoop-tools/hadoop-gridmix hadoop-tools/hadoop-rumen hadoop-tools/hadoop-streaming hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: org.apache.hadoop.conf.TestJobConf org.apache.hadoop.mapred.gridmix.TestHighRamJob Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5696//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5696//artifact/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5696//artifact/patchprocess/newPatchFindbugsWarningshadoop-maven-plugins.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5696//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html Javac warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5696//artifact/patchprocess/diffJavacWarnings.txt Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5696//console This message is automatically generated.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          Refreshed a patch.

          Show
          ozawa Tsuyoshi Ozawa added a comment - Refreshed a patch.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          Refreshed a patch.

          Show
          ozawa Tsuyoshi Ozawa added a comment - Refreshed a patch.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12698468/HADOOP-10101-009.patch
          against trunk revision 3826277.

          -1 patch. Trunk compilation may be broken.

          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5673//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12698468/HADOOP-10101-009.patch against trunk revision 3826277. -1 patch . Trunk compilation may be broken. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5673//console This message is automatically generated.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          Attaching v9 patch again.

          Show
          ozawa Tsuyoshi Ozawa added a comment - Attaching v9 patch again.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          Rakesh's patch was merged on HDFS-7709.

          Show
          ozawa Tsuyoshi Ozawa added a comment - Rakesh's patch was merged on HDFS-7709 .
          Hide
          rakeshr Rakesh R added a comment -

          I also checked findbugs are not related.

          I've attached patch to fix HDFS-7709, kindly review it. Thanks!

          Show
          rakeshr Rakesh R added a comment - I also checked findbugs are not related. I've attached patch to fix HDFS-7709 , kindly review it. Thanks!
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          Steve Loughran I checked Javac warnings: these warnings look not related to the patch. I also checked findbugs are not related. Could you take a look at 009 patch?

          Show
          ozawa Tsuyoshi Ozawa added a comment - Steve Loughran I checked Javac warnings: these warnings look not related to the patch. I also checked findbugs are not related. Could you take a look at 009 patch?
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12696476/HADOOP-10101-009.patch
          against trunk revision 42548f4.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 18 new or modified test files.

          -1 javac. The applied patch generated 1151 javac compiler warnings (more than the trunk's current 1150 warnings).

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 26 new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-annotations hadoop-common-project/hadoop-auth hadoop-common-project/hadoop-common hadoop-common-project/hadoop-nfs hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-maven-plugins hadoop-tools/hadoop-azure hadoop-tools/hadoop-distcp hadoop-tools/hadoop-extras hadoop-tools/hadoop-gridmix hadoop-tools/hadoop-rumen hadoop-tools/hadoop-streaming hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:

          org.apache.hadoop.ipc.TestRPCWaitForProxy
          org.apache.hadoop.mapreduce.TestLargeSort
          org.apache.hadoop.conf.TestJobConf
          org.apache.hadoop.mapred.gridmix.TestHighRamJob

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5579//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5579//artifact/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5579//artifact/patchprocess/newPatchFindbugsWarningshadoop-maven-plugins.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5579//artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs-httpfs.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5579//artifact/patchprocess/newPatchFindbugsWarningshadoop-rumen.html
          Javac warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5579//artifact/patchprocess/diffJavacWarnings.txt
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5579//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12696476/HADOOP-10101-009.patch against trunk revision 42548f4. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 18 new or modified test files. -1 javac . The applied patch generated 1151 javac compiler warnings (more than the trunk's current 1150 warnings). +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 26 new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-common-project/hadoop-annotations hadoop-common-project/hadoop-auth hadoop-common-project/hadoop-common hadoop-common-project/hadoop-nfs hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-maven-plugins hadoop-tools/hadoop-azure hadoop-tools/hadoop-distcp hadoop-tools/hadoop-extras hadoop-tools/hadoop-gridmix hadoop-tools/hadoop-rumen hadoop-tools/hadoop-streaming hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: org.apache.hadoop.ipc.TestRPCWaitForProxy org.apache.hadoop.mapreduce.TestLargeSort org.apache.hadoop.conf.TestJobConf org.apache.hadoop.mapred.gridmix.TestHighRamJob Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5579//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5579//artifact/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5579//artifact/patchprocess/newPatchFindbugsWarningshadoop-maven-plugins.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5579//artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs-httpfs.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5579//artifact/patchprocess/newPatchFindbugsWarningshadoop-rumen.html Javac warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5579//artifact/patchprocess/diffJavacWarnings.txt Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5579//console This message is automatically generated.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          Attaching a patch for trunk. Replaced deprecated methods with proper methods:

          • Objects.toStringHelper-> MoreObjects.toStringHelper
          • MoreExecutors.sameThreadExecutor -> MoreExecutors.newDirectExecutorService
          • Iterators.emptyIterator -> Collections.emptyIterator
          Show
          ozawa Tsuyoshi Ozawa added a comment - Attaching a patch for trunk. Replaced deprecated methods with proper methods: Objects.toStringHelper-> MoreObjects.toStringHelper MoreExecutors.sameThreadExecutor -> MoreExecutors.newDirectExecutorService Iterators.emptyIterator -> Collections.emptyIterator
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          Thanks Steve for your review. I'll check the javac warnings.

          Show
          ozawa Tsuyoshi Ozawa added a comment - Thanks Steve for your review. I'll check the javac warnings.
          Hide
          rakeshr Rakesh R added a comment -

          findbugs warnings all seem unrelated

          Raised HDFS-7709 to address the findbug warnings

          Show
          rakeshr Rakesh R added a comment - findbugs warnings all seem unrelated Raised HDFS-7709 to address the findbug warnings
          Hide
          stevel@apache.org Steve Loughran added a comment -
          • findbugs warnings all seem unrelated
          • javac warnings are a mix of spurious and more deprecated warnings; presumably some are guava 18 telling hadoop of for having source compatible with 11.x. Those warnings should be viewed as a sign that those methods will be going away at some point, which will matter for Hadoop-on-Guava 11.

          I wouldn't worry about them; just the tests

          Show
          stevel@apache.org Steve Loughran added a comment - findbugs warnings all seem unrelated javac warnings are a mix of spurious and more deprecated warnings; presumably some are guava 18 telling hadoop of for having source compatible with 11.x. Those warnings should be viewed as a sign that those methods will be going away at some point, which will matter for Hadoop-on-Guava 11. I wouldn't worry about them; just the tests
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12695405/HADOOP-10101-008.patch
          against trunk revision e36ef3b.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 17 new or modified test files.

          -1 javac. The applied patch generated 1199 javac compiler warnings (more than the trunk's current 1187 warnings).

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 26 new Findbugs (version 2.0.3) warnings.

          -1 release audit. The applied patch generated 1 release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-annotations hadoop-common-project/hadoop-auth hadoop-common-project/hadoop-common hadoop-common-project/hadoop-nfs hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-maven-plugins hadoop-tools/hadoop-azure hadoop-tools/hadoop-distcp hadoop-tools/hadoop-extras hadoop-tools/hadoop-gridmix hadoop-tools/hadoop-rumen hadoop-tools/hadoop-streaming hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:

          org.apache.hadoop.conf.TestJobConf
          org.apache.hadoop.mapreduce.TestLargeSort
          org.apache.hadoop.mapred.gridmix.TestHighRamJob
          org.apache.hadoop.hdfs.server.balancer.TestBalancer

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5542//testReport/
          Release audit warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5542//artifact/patchprocess/patchReleaseAuditProblems.txt
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5542//artifact/patchprocess/newPatchFindbugsWarningshadoop-rumen.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5542//artifact/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5542//artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs-httpfs.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5542//artifact/patchprocess/newPatchFindbugsWarningshadoop-maven-plugins.html
          Javac warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5542//artifact/patchprocess/diffJavacWarnings.txt
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5542//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12695405/HADOOP-10101-008.patch against trunk revision e36ef3b. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 17 new or modified test files. -1 javac . The applied patch generated 1199 javac compiler warnings (more than the trunk's current 1187 warnings). +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 26 new Findbugs (version 2.0.3) warnings. -1 release audit . The applied patch generated 1 release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-common-project/hadoop-annotations hadoop-common-project/hadoop-auth hadoop-common-project/hadoop-common hadoop-common-project/hadoop-nfs hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-maven-plugins hadoop-tools/hadoop-azure hadoop-tools/hadoop-distcp hadoop-tools/hadoop-extras hadoop-tools/hadoop-gridmix hadoop-tools/hadoop-rumen hadoop-tools/hadoop-streaming hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: org.apache.hadoop.conf.TestJobConf org.apache.hadoop.mapreduce.TestLargeSort org.apache.hadoop.mapred.gridmix.TestHighRamJob org.apache.hadoop.hdfs.server.balancer.TestBalancer Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5542//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5542//artifact/patchprocess/patchReleaseAuditProblems.txt Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5542//artifact/patchprocess/newPatchFindbugsWarningshadoop-rumen.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5542//artifact/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5542//artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs-httpfs.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5542//artifact/patchprocess/newPatchFindbugsWarningshadoop-maven-plugins.html Javac warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5542//artifact/patchprocess/diffJavacWarnings.txt Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5542//console This message is automatically generated.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          Attaching a patch for trunk:

          1. Updating Guava to 18.0.
          2. Using org.apache.hadoop.util.LimitInputStream instead of Guava's one.
          3. Using Enums.stringConverter instead of Enums.valueOfFunction.
          4. Using only Locale.ENGLISH for String#toUpperCase/toLowerCase.

          Show
          ozawa Tsuyoshi Ozawa added a comment - Attaching a patch for trunk: 1. Updating Guava to 18.0. 2. Using org.apache.hadoop.util.LimitInputStream instead of Guava's one. 3. Using Enums.stringConverter instead of Enums.valueOfFunction. 4. Using only Locale.ENGLISH for String#toUpperCase/toLowerCase.
          Hide
          tucu00 Alejandro Abdelnur added a comment -

          got it, thx

          Show
          tucu00 Alejandro Abdelnur added a comment - got it, thx
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          I noticed that Enums#stringConverter is supported from 16. This change cannot be applied to branch-2 since Guava is not updated in branch-2.

          Show
          ozawa Tsuyoshi Ozawa added a comment - I noticed that Enums#stringConverter is supported from 16. This change cannot be applied to branch-2 since Guava is not updated in branch-2.
          Hide
          sjlee0 Sangjin Lee added a comment -

          Sorry what I said may sound confusing.

          I understand that we are NOT upgrading guava in branch-2. As such, any change that upgrades guava cannot go into branch-2 as is. But code changes that make it compatible with different versions of guava can go into branch-2. I was suggesting pushing those code changes to branch-2. On trunk, we can also upgrade guava.

          Show
          sjlee0 Sangjin Lee added a comment - Sorry what I said may sound confusing. I understand that we are NOT upgrading guava in branch-2. As such, any change that upgrades guava cannot go into branch-2 as is. But code changes that make it compatible with different versions of guava can go into branch-2. I was suggesting pushing those code changes to branch-2. On trunk, we can also upgrade guava.
          Hide
          tucu00 Alejandro Abdelnur added a comment -

          Am I missing something here? If we push this to branch-2 that would be an incompatible change, we may break applications that that use ByteStreams.limit.

          Show
          tucu00 Alejandro Abdelnur added a comment - Am I missing something here? If we push this to branch-2 that would be an incompatible change, we may break applications that that use ByteStreams.limit.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          Sangjin Lee, Good point, you're correct. We can upgrade guava to 17.0 with hadoop's LimitInputStream instead of ByteStreams.limit.

          Show
          ozawa Tsuyoshi Ozawa added a comment - Sangjin Lee , Good point, you're correct. We can upgrade guava to 17.0 with hadoop's LimitInputStream instead of ByteStreams.limit.
          Hide
          sjlee0 Sangjin Lee added a comment -

          HADOOP-11286 replaced all use of guava's LimitInputStream with Hadoop's own. It appears a new reference to guava's LimitInputStream has crept in since then.

          Since we introduced Hadoop's LimitInputStream, we might as well stick with that approach. I understand it is not perfect (as people can still introduce new references without enforcement), but it still gives us value.

          Could you fix that to use Hadoop's LimitInputStream rather than using ByteStreams.limit()? That way it can be committed to branch-2 as well.

          If we fix that one, then the compatibility is expanded to prior to 18.0, correct?

          Show
          sjlee0 Sangjin Lee added a comment - HADOOP-11286 replaced all use of guava's LimitInputStream with Hadoop's own. It appears a new reference to guava's LimitInputStream has crept in since then. Since we introduced Hadoop's LimitInputStream, we might as well stick with that approach. I understand it is not perfect (as people can still introduce new references without enforcement), but it still gives us value. Could you fix that to use Hadoop's LimitInputStream rather than using ByteStreams.limit()? That way it can be committed to branch-2 as well. If we fix that one, then the compatibility is expanded to prior to 18.0, correct?
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          YARN-3029 addresses locale problem partially. Linking to the issue.

          Show
          ozawa Tsuyoshi Ozawa added a comment - YARN-3029 addresses locale problem partially. Linking to the issue.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          Thanks Steven and Sangjin for your review.

          We can upgrade Guava to 14.0.1 without the patch because:
          1. LimitInputStream is removed from 15.0: http://docs.guava-libraries.googlecode.com/git-history/5e074af6a526af39a4410a836517e57c72aa50bb/javadoc/com/google/common/io/LimitInputStream.html
          2. Enums.valueOfFunction is removed from 18.0: http://docs.guava-libraries.googlecode.com/git-history/v16.0.1/javadoc/com/google/common/base/Enums.html

          My understand is as follows:
          1. For branch-2, we replace deprecated methods and en.toUpperCase() without pom.xml change(, or should we upgrade the version to 14.0.1?).
          2. For trunk, we replace deprecated methods and en.toUpperCase() with pom.xml change(guava will be 18.0?).

          Please let me know if I'm wrong. I'll update the patches soon.

          Show
          ozawa Tsuyoshi Ozawa added a comment - Thanks Steven and Sangjin for your review. We can upgrade Guava to 14.0.1 without the patch because: 1. LimitInputStream is removed from 15.0: http://docs.guava-libraries.googlecode.com/git-history/5e074af6a526af39a4410a836517e57c72aa50bb/javadoc/com/google/common/io/LimitInputStream.html 2. Enums.valueOfFunction is removed from 18.0: http://docs.guava-libraries.googlecode.com/git-history/v16.0.1/javadoc/com/google/common/base/Enums.html My understand is as follows: 1. For branch-2, we replace deprecated methods and en.toUpperCase() without pom.xml change(, or should we upgrade the version to 14.0.1?). 2. For trunk, we replace deprecated methods and en.toUpperCase() with pom.xml change(guava will be 18.0?). Please let me know if I'm wrong. I'll update the patches soon.
          Hide
          sjlee0 Sangjin Lee added a comment -

          Along that line, if we can maximize the guava compatibility within the hadoop code (which several previous HADOOP JIRAs achieved), that is already a win for users. +1 for making code changes to make things compatible.

          As for the patch, note that HADOOP-11286 introduced the Hadoop version of LimitInputStream. It'd be good to use it to remove the dependency on guava's LimitInputStream.

          Show
          sjlee0 Sangjin Lee added a comment - Along that line, if we can maximize the guava compatibility within the hadoop code (which several previous HADOOP JIRAs achieved), that is already a win for users. +1 for making code changes to make things compatible. As for the patch, note that HADOOP-11286 introduced the Hadoop version of LimitInputStream. It'd be good to use it to remove the dependency on guava's LimitInputStream.
          Hide
          stevel@apache.org Steve Loughran added a comment -
          1. without this patch, what is the max guava version Hadoop 2.6+ runs against?
          2. can this patch (without the actual POM change) go into branch-2
          3. ...so then the switch to 18.0 would be the branch-3 change

          also, minor, please set a LOCALE_EN in en.toUpperCase() for consistent behaviour round the world. That's a bug in the existing code that could be fixed here

          Show
          stevel@apache.org Steve Loughran added a comment - without this patch, what is the max guava version Hadoop 2.6+ runs against? can this patch (without the actual POM change) go into branch-2 ...so then the switch to 18.0 would be the branch-3 change also, minor, please set a LOCALE_EN in en.toUpperCase() for consistent behaviour round the world. That's a bug in the existing code that could be fixed here
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12695199/HADOOP-10101-007.patch
          against trunk revision 7882bc0.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 javac. The applied patch generated 1199 javac compiler warnings (more than the trunk's current 1187 warnings).

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5529//testReport/
          Javac warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5529//artifact/patchprocess/diffJavacWarnings.txt
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5529//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12695199/HADOOP-10101-007.patch against trunk revision 7882bc0. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javac . The applied patch generated 1199 javac compiler warnings (more than the trunk's current 1187 warnings). +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5529//testReport/ Javac warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5529//artifact/patchprocess/diffJavacWarnings.txt Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5529//console This message is automatically generated.
          Hide
          sjlee0 Sangjin Lee added a comment -

          Tsuyoshi Ozawa, you might want to take a look at HADOOP-11286 and HADOOP-11470.

          Show
          sjlee0 Sangjin Lee added a comment - Tsuyoshi Ozawa , you might want to take a look at HADOOP-11286 and HADOOP-11470 .
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          Attaching patch for fixing it.

          Show
          ozawa Tsuyoshi Ozawa added a comment - Attaching patch for fixing it.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          We need to replace LimitInpuStream with ByteStreams.limit(java.io.InputStream, long) instead in addition to Enums#valueOfFunction.

          I think we shouldn't use functions annotated as Beta or Deprecated in upstream Guava.

          Show
          ozawa Tsuyoshi Ozawa added a comment - We need to replace LimitInpuStream with ByteStreams.limit(java.io.InputStream, long) instead in addition to Enums#valueOfFunction. I think we shouldn't use functions annotated as Beta or Deprecated in upstream Guava.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12695092/HADOOP-10101-006.patch
          against trunk revision caf7298.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 javac. The patch appears to cause the build to fail.

          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5527//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12695092/HADOOP-10101-006.patch against trunk revision caf7298. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javac . The patch appears to cause the build to fail. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5527//console This message is automatically generated.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          The failure of TestXAttrCommands is related. We need to make error handling consistent before and after the change. Attaching new patch to address the issue.

          Show
          ozawa Tsuyoshi Ozawa added a comment - The failure of TestXAttrCommands is related. We need to make error handling consistent before and after the change. Attaching new patch to address the issue.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12695059/HADOOP-10101-005.patch
          against trunk revision 9850e15.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 javac. The applied patch generated 1199 javac compiler warnings (more than the trunk's current 1187 warnings).

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common:

          org.apache.hadoop.ipc.TestRPCWaitForProxy
          org.apache.hadoop.fs.shell.TestXAttrCommands

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5524//testReport/
          Javac warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5524//artifact/patchprocess/diffJavacWarnings.txt
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5524//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12695059/HADOOP-10101-005.patch against trunk revision 9850e15. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javac . The applied patch generated 1199 javac compiler warnings (more than the trunk's current 1187 warnings). +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-common-project/hadoop-common: org.apache.hadoop.ipc.TestRPCWaitForProxy org.apache.hadoop.fs.shell.TestXAttrCommands Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5524//testReport/ Javac warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5524//artifact/patchprocess/diffJavacWarnings.txt Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5524//console This message is automatically generated.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          The patch needs to be updated because HADOOP-11032 is committed. BTW, guava v18 removes Enums#valueOfFunction. This patch includes the fix.

          Show
          ozawa Tsuyoshi Ozawa added a comment - The patch needs to be updated because HADOOP-11032 is committed. BTW, guava v18 removes Enums#valueOfFunction. This patch includes the fix.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12619314/HADOOP-10101-004.patch
          against trunk revision 5bd048e.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5105//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12619314/HADOOP-10101-004.patch against trunk revision 5bd048e. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5105//console This message is automatically generated.
          Hide
          busbey Sean Busbey added a comment -

          Could Hadoop start using the maven shade plugin to relocate the guava classes it uses into a private package? Then downstream projects that make use of Guava wouldn't be impacted by whatever version Hadoop chooses.

          Show
          busbey Sean Busbey added a comment - Could Hadoop start using the maven shade plugin to relocate the guava classes it uses into a private package? Then downstream projects that make use of Guava wouldn't be impacted by whatever version Hadoop chooses.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12619314/HADOOP-10101-004.patch
          against trunk revision 4be9517.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4698//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12619314/HADOOP-10101-004.patch against trunk revision 4be9517. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4698//console This message is automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          It would have to be on trunk, guava is a troublespot all round. I'm almost tempted to see if we can repimplement the most common uses (checkArguments, visibleForTesting, and some of the collections) and pull it out.

          Some of the fixes for 7u55+ changes are only in more recent versions of Guava, so the update should follow a java 7 migrate in trunk

          Show
          stevel@apache.org Steve Loughran added a comment - It would have to be on trunk, guava is a troublespot all round. I'm almost tempted to see if we can repimplement the most common uses (checkArguments, visibleForTesting, and some of the collections) and pull it out. Some of the fixes for 7u55+ changes are only in more recent versions of Guava, so the update should follow a java 7 migrate in trunk
          Hide
          sjlee0 Sangjin Lee added a comment -

          +1 on going to guava 17 on the trunk. If we stuck with 15, it would be obsolete from the moment when hadoop 3 is released.

          Show
          sjlee0 Sangjin Lee added a comment - +1 on going to guava 17 on the trunk. If we stuck with 15, it would be obsolete from the moment when hadoop 3 is released.
          Hide
          tucu00 Alejandro Abdelnur added a comment -

          ok, gr8 we are sync on this going into trunk.

          There is another JIRA going on to bump up trunk to jdk1.7.

          I think that is related to this as well and we should revive the thread we had in dev@ a while ago to come up with a balance on how to move to newer components without making it too difficult for backporting things to stable releases which cannot move to those new components.

          Show
          tucu00 Alejandro Abdelnur added a comment - ok, gr8 we are sync on this going into trunk. There is another JIRA going on to bump up trunk to jdk1.7. I think that is related to this as well and we should revive the thread we had in dev@ a while ago to come up with a balance on how to move to newer components without making it too difficult for backporting things to stable releases which cannot move to those new components.
          Hide
          wheat9 Haohui Mai added a comment -

          Correct me if I'm wrong, this patch should only be committed to trunk based on Steve Loughran's comments.

          Show
          wheat9 Haohui Mai added a comment - Correct me if I'm wrong, this patch should only be committed to trunk based on Steve Loughran 's comments.
          Hide
          tucu00 Alejandro Abdelnur added a comment - - edited

          Haohui Mai, if we make this change, per prior discussion in this JIRA (se above), it will live only in trunk, correct?

          Show
          tucu00 Alejandro Abdelnur added a comment - - edited Haohui Mai , if we make this change, per prior discussion in this JIRA (se above), it will live only in trunk, correct?
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12619314/HADOOP-10101-004.patch
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4096//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12619314/HADOOP-10101-004.patch against trunk revision . -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4096//console This message is automatically generated.
          Hide
          wheat9 Haohui Mai added a comment -

          This patch needs to be rebased, and it is probably okay to move guava towards v17.

          Show
          wheat9 Haohui Mai added a comment - This patch needs to be rebased, and it is probably okay to move guava towards v17.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Guava dependency injection pre v16 has been broken by Java 7u51; as/when google backport the fix to the 15.x branch we will need to upgrade -as downstream clients will need to do their own fix.

          Show
          stevel@apache.org Steve Loughran added a comment - Guava dependency injection pre v16 has been broken by Java 7u51; as/when google backport the fix to the 15.x branch we will need to upgrade -as downstream clients will need to do their own fix.
          Hide
          stevel@apache.org Steve Loughran added a comment -
          1. yes, this must be a 3.x patch. I'm only putting in low/no risk updates into branch-2.
          2. I would like to identify apps at risk! as otherwise they only know of the problem at ship time.
          3. I mentioned OSGi as you mentioned 'classpath magic'. We don't need more magic, and OSGi enabling branch 2 would be transparent to existing apps, while isolating new ones from change.
          4. Shading has its own flaws! including stopping ops teams from auditing and updating an installation from a security perspective.
          Show
          stevel@apache.org Steve Loughran added a comment - yes, this must be a 3.x patch. I'm only putting in low/no risk updates into branch-2. I would like to identify apps at risk! as otherwise they only know of the problem at ship time. I mentioned OSGi as you mentioned 'classpath magic'. We don't need more magic, and OSGi enabling branch 2 would be transparent to existing apps, while isolating new ones from change. Shading has its own flaws! including stopping ops teams from auditing and updating an installation from a security perspective.
          Hide
          tucu00 Alejandro Abdelnur added a comment -

          You'll only know when you release with Guava 15 and users apps start breaking.

          MR jobs can use classloading isolation, yes. But only on the task side and by default is OFF.

          OSGi requires changes in existing apps.

          My point is, and I think you agree, we cannot do this in 2.x timeframe.

          Show
          tucu00 Alejandro Abdelnur added a comment - You'll only know when you release with Guava 15 and users apps start breaking. MR jobs can use classloading isolation, yes. But only on the task side and by default is OFF. OSGi requires changes in existing apps. My point is, and I think you agree, we cannot do this in 2.x timeframe.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          -yes, but which apps are they? As I don't want us to be constrained fir theoretical reasons.
          -MR jobs can already override classloaders
          -there's some OSGi JIRAs, which won't break compatibility as they only declare what is exported. Once we have them in, downstream apps than want isolation can use OSGi classloaders with having to try to roll their own solutions

          Show
          stevel@apache.org Steve Loughran added a comment - -yes, but which apps are they? As I don't want us to be constrained fir theoretical reasons. -MR jobs can already override classloaders -there's some OSGi JIRAs, which won't break compatibility as they only declare what is exported. Once we have them in, downstream apps than want isolation can use OSGi classloaders with having to try to roll their own solutions
          Hide
          tucu00 Alejandro Abdelnur added a comment -

          #2, if somebody is using the same Guava classes that break HDFS.

          #3, we should figure this out somehow, either using classloader magic or shading to hide all 3rd party dep JARs from the user space. But this is a 3.x timeframe thing.

          Show
          tucu00 Alejandro Abdelnur added a comment - #2, if somebody is using the same Guava classes that break HDFS. #3, we should figure this out somehow, either using classloader magic or shading to hide all 3rd party dep JARs from the user space. But this is a 3.x timeframe thing.
          Hide
          stevel@apache.org Steve Loughran added a comment -
          1. marked the patch as targeting 3.0
          2. are there any specific applications which you know will break! or is this just a theoretical concern?
          3. I did write a section on dependency compatibility for the docs, precisely baca use they are visible. It comes down to "there are no guarantees"

          Pint #3 doesn't mean that we should push this in to branch 2, just that we explicitly state that we don't make any guarantees about dependency compatibility -I'd we did we would never be able to upgrade anything

          Show
          stevel@apache.org Steve Loughran added a comment - marked the patch as targeting 3.0 are there any specific applications which you know will break! or is this just a theoretical concern? I did write a section on dependency compatibility for the docs, precisely baca use they are visible. It comes down to "there are no guarantees" Pint #3 doesn't mean that we should push this in to branch 2, just that we explicitly state that we don't make any guarantees about dependency compatibility -I'd we did we would never be able to upgrade anything
          Hide
          tucu00 Alejandro Abdelnur added a comment -

          Because 3rd party Hadoop dependencies leak to the client and apps, this is an incompatible change (Guava 15 removed some classes). This may affect applications that use those classes avail in older versions of Guava.

          IMO, we have to wait till Hadoop 3 for this change. Plus, we should add this to the compat guidelines.

          Show
          tucu00 Alejandro Abdelnur added a comment - Because 3rd party Hadoop dependencies leak to the client and apps, this is an incompatible change (Guava 15 removed some classes). This may affect applications that use those classes avail in older versions of Guava. IMO, we have to wait till Hadoop 3 for this change. Plus, we should add this to the compat guidelines.
          Hide
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12619314/HADOOP-10101-004.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3368//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3368//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12619314/HADOOP-10101-004.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 4 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3368//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3368//console This message is automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12617772/HADOOP-10101.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3367//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3367//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12617772/HADOOP-10101.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 4 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3367//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3367//console This message is automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          This is the previous patch resynced with trunk and all trailing CR's stripped.

          Vinay, can you make sure that your editor/OS/SCM tool isn't adding the wrong line endings?

          Show
          stevel@apache.org Steve Loughran added a comment - This is the previous patch resynced with trunk and all trailing CR's stripped. Vinay, can you make sure that your editor/OS/SCM tool isn't adding the wrong line endings?
          Hide
          vinayrpet Vinayakumar B added a comment -

          Hi Steve Loughran, Could you take a look at changes...?
          Thanks,

          Show
          vinayrpet Vinayakumar B added a comment - Hi Steve Loughran , Could you take a look at changes...? Thanks,
          Hide
          vinayrpet Vinayakumar B added a comment -

          Attaching the same patch as in HDFS-5518, with Hadoop QA overall +1.

          Show
          vinayrpet Vinayakumar B added a comment - Attaching the same patch as in HDFS-5518 , with Hadoop QA overall +1.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          I'm happy for you to take it up!

          Show
          stevel@apache.org Steve Loughran added a comment - I'm happy for you to take it up!
          Hide
          vinayrpet Vinayakumar B added a comment -

          Any update on this Jira?
          If no objection, I would like to take-up and replace all deprecated with their successors..

          Show
          vinayrpet Vinayakumar B added a comment - Any update on this Jira? If no objection, I would like to take-up and replace all deprecated with their successors..
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Deprecation warnings of

          • com.google.common.collect.ComparisonChain
          • com.google.common.base.Stopwatch

          Replacing them with their successors would change more code but reduce the effort on the next guava update, when perhaps these files go away completely

          291a292
          > [WARNING] /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/VersionUtil.java:[112,6] [deprecation] compare(boolean,boolean) in com.google.common.collect.ComparisonChain has been deprecated
          293a295,296
          > [WARNING] /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/JvmPauseMonitor.java:[156,21] [deprecation] Stopwatch() in com.google.common.base.Stopwatch has been deprecated
          > [WARNING] /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/JvmPauseMonitor.java:[165,32] [deprecation] elapsedMillis() in com.google.common.base.Stopwatch has been deprecated
          432a436,439
          > [WARNING] /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/Journal.java:[372,19] [deprecation] Stopwatch() in com.google.common.base.Stopwatch has been deprecated
          > [WARNING] /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/Journal.java:[377,22] [deprecation] elapsedTime(java.util.concurrent.TimeUnit) in com.google.common.base.Stopwatch has been deprecated
          > [WARNING] /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/Journal.java:[378,10] [deprecation] elapsedTime(java.util.concurrent.TimeUnit) in com.google.common.base.Stopwatch has been deprecated
          > [WARNING] /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/Journal.java:[380,28] [deprecation] elapsedTime(java.util.concurrent.TimeUnit) in com.google.common.base.Stopwatch has been deprecated
          435a443,444
          > [WARNING] /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/IPCLoggerChannel.java:[133,45] [deprecation] Stopwatch() in com.google.common.base.Stopwatch has been deprecated
          > [WARNING] /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/IPCLoggerChannel.java:[437,30] [deprecation] elapsedMillis() in com.google.common.base.Stopwatch has been deprecated
          437a447,448
          > [WARNING] /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/server/TestJournalNode.java:[330,19] [deprecation] Stopwatch() in com.google.common.base.Stopwatch has been deprecated
          > [WARNING] /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/server/TestJournalNode.java:[334,18] [deprecation] elapsedMillis() in com.google.common.base.Stopwatch has been deprecated
          438a450,453
          > [WARNING] /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestMultiThreadedHflush.java:[103,21] [deprecation] Stopwatch() in com.google.common.base.Stopwatch has been deprecated
          > [WARNING] /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestMultiThreadedHflush.java:[106,22] [deprecation] elapsedTime(java.util.concurrent.TimeUnit) in com.google.common.base.Stopwatch has been deprecated
          > [WARNING] /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestMultiThreadedHflush.java:[279,21] [deprecation] Stopwatch() in com.google.common.base.Stopwatch has been deprecated
          > [WARNING] /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestMultiThreadedHflush.java:[284,44] [deprecation] elapsedMillis() in com.google.common.base.Stopwatch has been deprecated
          447a463,466
          > [WARNING] /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/util/TestChunkedArrayList.java:[72,23] [deprecation] Stopwatch() in com.google.common.base.Stopwatch has been deprecated
          > [WARNING] /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/util/TestChunkedArrayList.java:[77,51] [deprecation] elapsedMillis() in com.google.common.base.Stopwatch has been deprecated
          > [WARNING] /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/util/TestChunkedArrayList.java:[84,23] [deprecation] Stopwatch() in com.google.common.base.Stopwatch has been deprecated
          > [WARNING] /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/util/TestChunkedArrayList.java:[89,51] [deprecation] elapsedMillis() in com.google.common.base.Stopwatch has been deprecated
          
          Show
          stevel@apache.org Steve Loughran added a comment - Deprecation warnings of com.google.common.collect.ComparisonChain com.google.common.base.Stopwatch Replacing them with their successors would change more code but reduce the effort on the next guava update, when perhaps these files go away completely 291a292 > [WARNING] /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/VersionUtil.java:[112,6] [deprecation] compare( boolean , boolean ) in com.google.common.collect.ComparisonChain has been deprecated 293a295,296 > [WARNING] /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/JvmPauseMonitor.java:[156,21] [deprecation] Stopwatch() in com.google.common.base.Stopwatch has been deprecated > [WARNING] /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/JvmPauseMonitor.java:[165,32] [deprecation] elapsedMillis() in com.google.common.base.Stopwatch has been deprecated 432a436,439 > [WARNING] /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/Journal.java:[372,19] [deprecation] Stopwatch() in com.google.common.base.Stopwatch has been deprecated > [WARNING] /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/Journal.java:[377,22] [deprecation] elapsedTime(java.util.concurrent.TimeUnit) in com.google.common.base.Stopwatch has been deprecated > [WARNING] /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/Journal.java:[378,10] [deprecation] elapsedTime(java.util.concurrent.TimeUnit) in com.google.common.base.Stopwatch has been deprecated > [WARNING] /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/Journal.java:[380,28] [deprecation] elapsedTime(java.util.concurrent.TimeUnit) in com.google.common.base.Stopwatch has been deprecated 435a443,444 > [WARNING] /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/IPCLoggerChannel.java:[133,45] [deprecation] Stopwatch() in com.google.common.base.Stopwatch has been deprecated > [WARNING] /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/IPCLoggerChannel.java:[437,30] [deprecation] elapsedMillis() in com.google.common.base.Stopwatch has been deprecated 437a447,448 > [WARNING] /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/server/TestJournalNode.java:[330,19] [deprecation] Stopwatch() in com.google.common.base.Stopwatch has been deprecated > [WARNING] /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/server/TestJournalNode.java:[334,18] [deprecation] elapsedMillis() in com.google.common.base.Stopwatch has been deprecated 438a450,453 > [WARNING] /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestMultiThreadedHflush.java:[103,21] [deprecation] Stopwatch() in com.google.common.base.Stopwatch has been deprecated > [WARNING] /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestMultiThreadedHflush.java:[106,22] [deprecation] elapsedTime(java.util.concurrent.TimeUnit) in com.google.common.base.Stopwatch has been deprecated > [WARNING] /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestMultiThreadedHflush.java:[279,21] [deprecation] Stopwatch() in com.google.common.base.Stopwatch has been deprecated > [WARNING] /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestMultiThreadedHflush.java:[284,44] [deprecation] elapsedMillis() in com.google.common.base.Stopwatch has been deprecated 447a463,466 > [WARNING] /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/util/TestChunkedArrayList.java:[72,23] [deprecation] Stopwatch() in com.google.common.base.Stopwatch has been deprecated > [WARNING] /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/util/TestChunkedArrayList.java:[77,51] [deprecation] elapsedMillis() in com.google.common.base.Stopwatch has been deprecated > [WARNING] /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/util/TestChunkedArrayList.java:[84,23] [deprecation] Stopwatch() in com.google.common.base.Stopwatch has been deprecated > [WARNING] /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/util/TestChunkedArrayList.java:[89,51] [deprecation] elapsedMillis() in com.google.common.base.Stopwatch has been deprecated
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Link to HDFS-5518: the test/review side of the HDFS patch

          Show
          stevel@apache.org Steve Loughran added a comment - Link to HDFS-5518 : the test/review side of the HDFS patch
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12613889/HADOOP-10101-002.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          -1 javac. The applied patch generated 1565 javac compiler warnings (more than the trunk's current 1546 warnings).

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3287//testReport/
          Javac warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/3287//artifact/trunk/patchprocess/diffJavacWarnings.txt
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3287//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12613889/HADOOP-10101-002.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. -1 javac . The applied patch generated 1565 javac compiler warnings (more than the trunk's current 1546 warnings). +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3287//testReport/ Javac warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/3287//artifact/trunk/patchprocess/diffJavacWarnings.txt Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3287//console This message is automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          patch fixing two compilation errors after Guava dropped classes

          Show
          stevel@apache.org Steve Loughran added a comment - patch fixing two compilation errors after Guava dropped classes
          Hide
          stevel@apache.org Steve Loughran added a comment -

          problem HBase had that also arose in hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDataTransferKeepalive.java

          Show
          stevel@apache.org Steve Loughran added a comment - problem HBase had that also arose in hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDataTransferKeepalive.java
          Hide
          stevel@apache.org Steve Loughran added a comment -

          hdfs stops compiling with this patch

          hadoop-trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/Journal.java:[68,32] error: cannot find symbol
          [ERROR] symbol:   class Ranges
          
          
          Show
          stevel@apache.org Steve Loughran added a comment - hdfs stops compiling with this patch hadoop-trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/Journal.java:[68,32] error: cannot find symbol [ERROR] symbol: class Ranges
          Hide
          stevel@apache.org Steve Loughran added a comment -

          -though as the only place @Nullable is used in the entire Hadoop codebase is apparently TestMetricsSystemImpl, HADOOP-10067 will have fixed it.

          Show
          stevel@apache.org Steve Loughran added a comment - -though as the only place @Nullable is used in the entire Hadoop codebase is apparently TestMetricsSystemImpl , HADOOP-10067 will have fixed it.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          This update will remove the @Nullable attribute from jsr305, which may propagate through the code

          Show
          stevel@apache.org Steve Loughran added a comment - This update will remove the @Nullable attribute from jsr305, which may propagate through the code

            People

            • Assignee:
              ozawa Tsuyoshi Ozawa
              Reporter:
              rakeshr Rakesh R
            • Votes:
              5 Vote for this issue
              Watchers:
              41 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development