HBase
  1. HBase
  2. HBASE-5955

Guava 11 drops MapEvictionListener and Hadoop 2.0.0-alpha requires it

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.94.0
    • Fix Version/s: 0.94.1
    • Component/s: None
    • Labels:
      None

      Description

      Hadoop 2.0.0-alpha depends on Guava 11.0.2. Updating HBase dependencies to match produces the following compilation errors:

      [ERROR] SingleSizeCache.java:[41,32] cannot find symbol
      [ERROR] symbol  : class MapEvictionListener
      [ERROR] location: package com.google.common.collect
      [ERROR] 
      [ERROR] SingleSizeCache.java:[94,4] cannot find symbol
      [ERROR] symbol  : class MapEvictionListener
      [ERROR] location: class org.apache.hadoop.hbase.io.hfile.slab.SingleSizeCache
      [ERROR] 
      [ERROR] SingleSizeCache.java:[94,69] cannot find symbol
      [ERROR] symbol  : class MapEvictionListener
      [ERROR] location: class org.apache.hadoop.hbase.io.hfile.slab.SingleSizeCache
      
      1. 5955.txt
        3 kB
        Lars Hofhansl

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in HBase-0.94-security #39 (See https://builds.apache.org/job/HBase-0.94-security/39/)
          HBASE-5955 Guava 11 drops MapEvictionListener and Hadoop 2.0.0-alpha requires it (Revision 1356379)

          Result = SUCCESS
          larsh :
          Files :

          • /hbase/branches/0.94/pom.xml
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
          Show
          Hudson added a comment - Integrated in HBase-0.94-security #39 (See https://builds.apache.org/job/HBase-0.94-security/39/ ) HBASE-5955 Guava 11 drops MapEvictionListener and Hadoop 2.0.0-alpha requires it (Revision 1356379) Result = SUCCESS larsh : Files : /hbase/branches/0.94/pom.xml /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
          Hide
          Hudson added a comment -

          Integrated in HBase-0.94 #290 (See https://builds.apache.org/job/HBase-0.94/290/)
          HBASE-5955 Guava 11 drops MapEvictionListener and Hadoop 2.0.0-alpha requires it (Revision 1356379)

          Result = SUCCESS
          larsh :
          Files :

          • /hbase/branches/0.94/pom.xml
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
          Show
          Hudson added a comment - Integrated in HBase-0.94 #290 (See https://builds.apache.org/job/HBase-0.94/290/ ) HBASE-5955 Guava 11 drops MapEvictionListener and Hadoop 2.0.0-alpha requires it (Revision 1356379) Result = SUCCESS larsh : Files : /hbase/branches/0.94/pom.xml /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
          Hide
          Lars Hofhansl added a comment -

          Committed to 0.94

          Show
          Lars Hofhansl added a comment - Committed to 0.94
          Hide
          Lars Hofhansl added a comment -

          0.94 patch

          Show
          Lars Hofhansl added a comment - 0.94 patch
          Hide
          Lars Hofhansl added a comment -

          Unless there are objections I will pull parent into 0.94 (breaking HBase with all hadoop versions: hadoop-0.22 <= version < hadoop-2.0.0)

          Show
          Lars Hofhansl added a comment - Unless there are objections I will pull parent into 0.94 (breaking HBase with all hadoop versions: hadoop-0.22 <= version < hadoop-2.0.0)
          Hide
          Lars Hofhansl added a comment -

          So specifically +1 on updating to Guava 11.0.2 in 0.94.1.
          Please let me know if you disagree (hadoop 0.22 and some later versions before Hadoop 2.0.x would be broken).

          Show
          Lars Hofhansl added a comment - So specifically +1 on updating to Guava 11.0.2 in 0.94.1. Please let me know if you disagree (hadoop 0.22 and some later versions before Hadoop 2.0.x would be broken).
          Hide
          Lars Hofhansl added a comment -

          Looks like guava was only add in hadoop 0.22 (r09 there) - I asked for the exact point of introduction in parent to be sure.

          In that case I am fine with updating to 11.0.2 in 0.94.1 (and hence some versions of Hadoop 1.0.x < version < 2.0.x would not work correctly).

          Show
          Lars Hofhansl added a comment - Looks like guava was only add in hadoop 0.22 (r09 there) - I asked for the exact point of introduction in parent to be sure. In that case I am fine with updating to 11.0.2 in 0.94.1 (and hence some versions of Hadoop 1.0.x < version < 2.0.x would not work correctly).
          Hide
          Andrew Purtell added a comment -

          Are you sure you didn't update this locally to make it work for Hadoop 2?

          You are right of course, I see now that HBASE-5739 only went into trunk.

          Also, I checked whether Guava r09 has RemovalListener (see parent patch), but it does not, so we'd have to reflect this away.

          Yes.

          Show
          Andrew Purtell added a comment - Are you sure you didn't update this locally to make it work for Hadoop 2? You are right of course, I see now that HBASE-5739 only went into trunk. Also, I checked whether Guava r09 has RemovalListener (see parent patch), but it does not, so we'd have to reflect this away. Yes.
          Hide
          Lars Hofhansl added a comment -

          Oh, an the problem here is in the DFSClient as used from HBase, so it "sees" all the HBase libraries (which in this case is Guava r09).
          Also, I checked whether Guava r09 has RemovalListener (see parent patch), but it does not, so we'd have to reflect this away.
          (Because if we use the current EvictionListener it won't work with Guava 11.x, and if we use RemovalListener it won't work Guava 9, and at the same time we have to use the same version of Guava that Hadoop is using)

          Show
          Lars Hofhansl added a comment - Oh, an the problem here is in the DFSClient as used from HBase, so it "sees" all the HBase libraries (which in this case is Guava r09). Also, I checked whether Guava r09 has RemovalListener (see parent patch), but it does not, so we'd have to reflect this away. (Because if we use the current EvictionListener it won't work with Guava 11.x, and if we use RemovalListener it won't work Guava 9, and at the same time we have to use the same version of Guava that Hadoop is using)
          Hide
          Lars Hofhansl added a comment -

          Hmm... I see:

          <guava.version>r09</guava.version>

          in 0.94's pom.xml

          Are you sure you didn't update this locally to make it work for Hadoop 2?

          Show
          Lars Hofhansl added a comment - Hmm... I see: <guava.version>r09</guava.version> in 0.94's pom.xml Are you sure you didn't update this locally to make it work for Hadoop 2?
          Hide
          Andrew Purtell added a comment -

          In the POMs I have on hand for Hadoop 2.0.1 and HBase 0.94, the Guava version for both is 11.0.2 (via HDFS-3187 and HBASE-5739). I'm guessing you have a Hadoop compiled against something older?

          Show
          Andrew Purtell added a comment - In the POMs I have on hand for Hadoop 2.0.1 and HBase 0.94, the Guava version for both is 11.0.2 (via HDFS-3187 and HBASE-5739 ). I'm guessing you have a Hadoop compiled against something older?
          Hide
          Lars Hofhansl added a comment -

          I find that I get runtime errors with Hadoop-2 (but not compile time errrors):

          java.lang.NoSuchMethodError: com.google.common.collect.LinkedListMultimap.values()Ljava/util/List;
                  at org.apache.hadoop.hdfs.SocketCache.clear(SocketCache.java:131)
                  at org.apache.hadoop.hdfs.DFSClient.close(DFSClient.java:604)
                  at org.apache.hadoop.hdfs.DistributedFileSystem.close(DistributedFileSystem.java:506)
                  at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2237)
                  at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2215)
                  at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:302)
                  at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:158)
                  at org.apache.hadoop.hbase.fs.HFileSystem.<init>(HFileSystem.java:62)
                  at org.apache.hadoop.hbase.regionserver.HRegionServer.handleReportForDutyResponse(HRegionServer.java:974)
                  at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:672)
                  at java.lang.Thread.run(Thread.java:619)
          

          The signature of LinkedListMultiMap.values() has changed to return a List rather than a collection. That would imply that switching out Guava would fail with the reverse problem in Hadoop-1 and before. Sigh.
          So this would need to be profile dependent?

          Show
          Lars Hofhansl added a comment - I find that I get runtime errors with Hadoop-2 (but not compile time errrors): java.lang.NoSuchMethodError: com.google.common.collect.LinkedListMultimap.values()Ljava/util/List; at org.apache.hadoop.hdfs.SocketCache.clear(SocketCache.java:131) at org.apache.hadoop.hdfs.DFSClient.close(DFSClient.java:604) at org.apache.hadoop.hdfs.DistributedFileSystem.close(DistributedFileSystem.java:506) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2237) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2215) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:302) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:158) at org.apache.hadoop.hbase.fs.HFileSystem.<init>(HFileSystem.java:62) at org.apache.hadoop.hbase.regionserver.HRegionServer.handleReportForDutyResponse(HRegionServer.java:974) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:672) at java.lang. Thread .run( Thread .java:619) The signature of LinkedListMultiMap.values() has changed to return a List rather than a collection. That would imply that switching out Guava would fail with the reverse problem in Hadoop-1 and before. Sigh. So this would need to be profile dependent?
          Hide
          Andrew Purtell added a comment -

          Does version of hadoop even matter for this issue?

          You can't compile HBase 0.94 against Hadoop 2.0.x without this patch.

          Show
          Andrew Purtell added a comment - Does version of hadoop even matter for this issue? You can't compile HBase 0.94 against Hadoop 2.0.x without this patch.
          Hide
          Andrew Purtell added a comment -

          Those of us who want to run 0.94 on Hadoop 2.0.x can carry around a private patch if you want to be 100% safe Lars.

          Show
          Andrew Purtell added a comment - Those of us who want to run 0.94 on Hadoop 2.0.x can carry around a private patch if you want to be 100% safe Lars.
          Hide
          stack added a comment -

          We want to do same for 0.94? Does version of hadoop even matter for this issue?

          Up on dev list, discussion had it that only 0.96 can require 1.0.0 hadoop as its minimum version.

          Show
          stack added a comment - We want to do same for 0.94? Does version of hadoop even matter for this issue? Up on dev list, discussion had it that only 0.96 can require 1.0.0 hadoop as its minimum version.
          Hide
          Lars Hofhansl added a comment -

          Will this have any negative side effects when using older versions of Hadoop?
          I.e. is it 100% safe for 0.94?

          Show
          Lars Hofhansl added a comment - Will this have any negative side effects when using older versions of Hadoop? I.e. is it 100% safe for 0.94?
          Hide
          Andrew Purtell added a comment -

          This is a dup of HBASE-5739, right?

          Yes, with the variation on which branch is the commit target.

          Show
          Andrew Purtell added a comment - This is a dup of HBASE-5739 , right? Yes, with the variation on which branch is the commit target.
          Hide
          Lars Hofhansl added a comment -

          +1 on backport. This is a dup of HBASE-5739, right?

          Show
          Lars Hofhansl added a comment - +1 on backport. This is a dup of HBASE-5739 , right?
          Hide
          Andrew Purtell added a comment -

          Reassign to Lars as it is not my call.

          Show
          Andrew Purtell added a comment - Reassign to Lars as it is not my call.
          Hide
          Andrew Purtell added a comment -

          @Lars, how about getting HBASE-5739 into 0.94?

          Show
          Andrew Purtell added a comment - @Lars, how about getting HBASE-5739 into 0.94?
          Hide
          Andrew Purtell added a comment -

          FYI, the symptom that brought me here is if one tries to use 0.94 with 2.0.0 then you can't instantiate HDFS filesystems, with a cryptic "No filesystem for scheme: hdfs".

          Show
          Andrew Purtell added a comment - FYI, the symptom that brought me here is if one tries to use 0.94 with 2.0.0 then you can't instantiate HDFS filesystems, with a cryptic "No filesystem for scheme: hdfs".
          Hide
          Andrew Purtell added a comment - - edited

          + 1 on backport to 0.94. I don't think we need support 2.0.0 with 0.92, we can stick with 1.0, 0.22, and 0.23, to varying degrees.

          Edit: Fix accidental use of markup.

          Show
          Andrew Purtell added a comment - - edited + 1 on backport to 0.94. I don't think we need support 2.0.0 with 0.92, we can stick with 1.0, 0.22, and 0.23, to varying degrees. Edit: Fix accidental use of markup.
          Hide
          Todd Lipcon added a comment -

          Oh, wait, it is committed... maybe we need to backport to 0.94 or 0.92? Will that be painful for downstream users? Dare we shade guava?

          Show
          Todd Lipcon added a comment - Oh, wait, it is committed... maybe we need to backport to 0.94 or 0.92? Will that be painful for downstream users? Dare we shade guava?
          Hide
          Todd Lipcon added a comment -

          See HBASE-5739. I'll commit that tomorrow if no one objects.

          Show
          Todd Lipcon added a comment - See HBASE-5739 . I'll commit that tomorrow if no one objects.
          Hide
          Andrew Purtell added a comment -

          Also there's a change in MapMaker:

          [ERROR] SingleSizeCache.java:[104,31] maximumSize(int) is not public in com.google.common.collect.MapMaker; cannot be accessed from outside package
          
          Show
          Andrew Purtell added a comment - Also there's a change in MapMaker: [ERROR] SingleSizeCache.java:[104,31] maximumSize( int ) is not public in com.google.common.collect.MapMaker; cannot be accessed from outside package

            People

            • Assignee:
              Lars Hofhansl
              Reporter:
              Andrew Purtell
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development