Details

    • Type: Sub-task Sub-task
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.8.0
    • Fix Version/s: 0.8.0
    • Component/s: build
    • Labels:
      None

      Description

      flume 1.4.0 build fails with protobuf 2.5.0. I propose to bump flume to 1.4.0 + Flume-2172 or new version of flume
      solution: Flume-2172 can be placed in bigtop-packages/src/common/flume/

      Renaming the ticket as Flume 1.5 is planned to be release around right now

      1. BIGTOP-1244.1.patch
        1 kB
        YoungWoo Kim
      2. BIGTOP-1244.patch
        0.9 kB
        Konstantin Boudnik
      3. BIGTOP-1244.patch
        0.5 kB
        Konstantin Boudnik

        Issue Links

          Activity

          Hide
          Konstantin Boudnik added a comment - - edited

          We normally don't patch released components in Bigtop - e.g. we are not forking/maintaning upstream stuff. I suggest to reach out to Flume community to find out when they are going to have the update release that addresses the issue. I am pretty sure they are working on it as next CDH version is based off at least Hadoop 2.2.0 hence their flume developers have the same problem (which I presume is aready fixed internally).

          Show
          Konstantin Boudnik added a comment - - edited We normally don't patch released components in Bigtop - e.g. we are not forking/maintaning upstream stuff. I suggest to reach out to Flume community to find out when they are going to have the update release that addresses the issue. I am pretty sure they are working on it as next CDH version is based off at least Hadoop 2.2.0 hence their flume developers have the same problem (which I presume is aready fixed internally).
          Hide
          Konstantin Boudnik added a comment -

          1.5 release of Flume is tracked by FLUME-2366

          Show
          Konstantin Boudnik added a comment - 1.5 release of Flume is tracked by FLUME-2366
          Hide
          Hari Shreedharan added a comment -

          I am working on a release for Flume. I will put an RC out to vote later today.

          Show
          Hari Shreedharan added a comment - I am working on a release for Flume. I will put an RC out to vote later today.
          Hide
          Konstantin Boudnik added a comment -

          Thank you very much!

          Show
          Konstantin Boudnik added a comment - Thank you very much!
          Hide
          Konstantin Boudnik added a comment -

          Flume 1.5 RC is in the final vote right now. Can anyone familiar with this component take a stab at the patch?

          Show
          Konstantin Boudnik added a comment - Flume 1.5 RC is in the final vote right now. Can anyone familiar with this component take a stab at the patch?
          Hide
          Konstantin Boudnik added a comment -

          Perhaps this will suffice?

          Show
          Konstantin Boudnik added a comment - Perhaps this will suffice?
          Hide
          YoungWoo Kim added a comment -

          Bump up HBase version from BOM does not work for me. HBase's artifactId has been changed.

          Show
          YoungWoo Kim added a comment - Bump up HBase version from BOM does not work for me. HBase's artifactId has been changed.
          Hide
          YoungWoo Kim added a comment -

          Updated patch

          Show
          YoungWoo Kim added a comment - Updated patch
          Hide
          Mark Grover added a comment -

          Taking a quick look now.

          Show
          Mark Grover added a comment - Taking a quick look now.
          Hide
          Mark Grover added a comment -

          It's looking fine to me, have you been able to build and test it?

          Show
          Mark Grover added a comment - It's looking fine to me, have you been able to build and test it?
          Hide
          Konstantin Boudnik added a comment -

          HBase's artifactId has been changed

          I am not sure what are you referring to? And how Flume would know which HBase version it needs to be built against?

          Show
          Konstantin Boudnik added a comment - HBase's artifactId has been changed I am not sure what are you referring to? And how Flume would know which HBase version it needs to be built against?
          Hide
          Konstantin Boudnik added a comment -

          Ok, I've tried the patch and I see that comment meant to say. Flume doesn't reflect the singularity point of HBase. E.g. Flume still expect org.apache.hbase:hbase:jar to be present in 0.98.2. Hence, I don't think the later patch is adequate as it will be pulling in incorrect version of the flume.

          I can do in the flight-mod of the flume poms to start using hbase-client artifact instead, but then the whole bunch of compile errors is kicking in. E.g.

          [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.3.2:compile (default-compile) on project flume-ng-hbase-sink: Compilation failure: Compilation failure:
          [ERROR] /home/cos/work/bigtop/build/flume/rpm/BUILD/apache-flume-1.5.0-src/flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/AsyncHBaseSink.java:[44,30] cannot find symbol
          [ERROR] symbol  : class HBaseConfiguration
          [ERROR] location: package org.apache.hadoop.hbase
          [ERROR] /home/cos/work/bigtop/build/flume/rpm/BUILD/apache-flume-1.5.0-src/flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/AsyncHBaseSink.java:[45,30] cannot find symbol
          [ERROR] symbol  : class HConstants
          [ERROR] location: package org.apache.hadoop.hbase
          [ERROR] /home/cos/work/bigtop/build/flume/rpm/BUILD/apache-flume-1.5.0-src/flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/HBaseSink.java:[43,30] cannot find symbol
          [ERROR] symbol  : class HBaseConfiguration
          [ERROR] location: package org.apache.hadoop.hbase
          [ERROR] /home/cos/work/bigtop/build/flume/rpm/BUILD/apache-flume-1.5.0-src/flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/HBaseSink.java:[44,30] cannot find symbol
          [ERROR] symbol  : class HConstants
          [ERROR] location: package org.apache.hadoop.hbase
          [ERROR] /home/cos/work/bigtop/build/flume/rpm/BUILD/apache-flume-1.5.0-src/flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/HBaseSink.java:[49,35] cannot find symbol
          [ERROR] symbol  : class Bytes
          [ERROR] location: package org.apache.hadoop.hbase.util
          [ERROR] /home/cos/work/bigtop/build/flume/rpm/BUILD/apache-flume-1.5.0-src/flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/HBaseSink.java:[57,39] cannot find symbol
          [ERROR] symbol  : class User
          [ERROR] location: package org.apache.hadoop.hbase.security
          [ERROR] /home/cos/work/bigtop/build/flume/rpm/BUILD/apache-flume-1.5.0-src/flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/HBaseSink.java:[106,10] cannot find symbol
          [ERROR] symbol  : class User
          [ERROR] location: class org.apache.flume.sink.hbase.HBaseSink
          [ERROR] /home/cos/work/bigtop/build/flume/rpm/BUILD/apache-flume-1.5.0-src/flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/HBaseSinkConfigurationConstants.java:[21,30] cannot find symbol
          [ERROR] symbol  : class HConstants
          [ERROR] location: package org.apache.hadoop.hbase
          [ERROR] /home/cos/work/bigtop/build/flume/rpm/BUILD/apache-flume-1.5.0-src/flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/HBaseSinkSecurityManager.java:[27,39] cannot find symbol
          [ERROR] symbol  : class User
          [ERROR] location: package org.apache.hadoop.hbase.security
          [ERROR] /home/cos/work/bigtop/build/flume/rpm/BUILD/apache-flume-1.5.0-src/flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/HBaseSinkSecurityManager.java:[71,29] cannot find symbol
          [ERROR] symbol  : class User
          [ERROR] location: class org.apache.flume.sink.hbase.HBaseSinkSecurityManager
          [ERROR] /home/cos/work/bigtop/build/flume/rpm/BUILD/apache-flume-1.5.0-src/flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/RegexHbaseEventSerializer.java:[193,13] cannot access org.apache.hadoop.hbase.Cell
          [ERROR] class file for org.apache.hadoop.hbase.Cell not found
          [ERROR] put.add(cf, colNames.get(i), m.group(i + 1).getBytes(Charsets.UTF_8));
          [ERROR] /home/cos/work/bigtop/build/flume/rpm/BUILD/apache-flume-1.5.0-src/flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/HBaseSinkConfigurationConstants.java:[67,6] cannot find symbol
          [ERROR] symbol  : variable HConstants
          [ERROR] location: class org.apache.flume.sink.hbase.HBaseSinkConfigurationConstants
          [ERROR] /home/cos/work/bigtop/build/flume/rpm/BUILD/apache-flume-1.5.0-src/flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/AsyncHBaseSink.java:[368,15] cannot find symbol
          [ERROR] symbol  : variable HBaseConfiguration
          [ERROR] location: class org.apache.flume.sink.hbase.AsyncHBaseSink
          [ERROR] /home/cos/work/bigtop/build/flume/rpm/BUILD/apache-flume-1.5.0-src/flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/AsyncHBaseSink.java:[371,27] cannot find symbol
          [ERROR] symbol  : variable HConstants
          [ERROR] location: class org.apache.flume.sink.hbase.AsyncHBaseSink
          [ERROR] /home/cos/work/bigtop/build/flume/rpm/BUILD/apache-flume-1.5.0-src/flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/AsyncHBaseSink.java:[372,8] cannot find symbol
          [ERROR] symbol  : variable HConstants
          [ERROR] location: class org.apache.flume.sink.hbase.AsyncHBaseSink
          [ERROR] /home/cos/work/bigtop/build/flume/rpm/BUILD/apache-flume-1.5.0-src/flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/HBaseSink.java:[116,9] cannot find symbol
          [ERROR] symbol  : variable HBaseConfiguration
          [ERROR] location: class org.apache.flume.sink.hbase.HBaseSink
          [ERROR] /home/cos/work/bigtop/build/flume/rpm/BUILD/apache-flume-1.5.0-src/flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/HBaseSink.java:[148,25] cannot access org.apache.hadoop.hbase.TableName
          [ERROR] class file for org.apache.hadoop.hbase.TableName not found
          [ERROR] HTable table = new HTable(config, tableName);
          [ERROR] /home/cos/work/bigtop/build/flume/rpm/BUILD/apache-flume-1.5.0-src/flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/HBaseSink.java:[170,50] cannot find symbol
          [ERROR] symbol  : variable Bytes
          [ERROR] location: class org.apache.flume.sink.hbase.HBaseSink
          [ERROR] /home/cos/work/bigtop/build/flume/rpm/BUILD/apache-flume-1.5.0-src/flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/HBaseSink.java:[178,16] cannot find symbol
          [ERROR] symbol  : variable Bytes
          [ERROR] location: class org.apache.flume.sink.hbase.HBaseSink
          [ERROR] /home/cos/work/bigtop/build/flume/rpm/BUILD/apache-flume-1.5.0-src/flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/HBaseSink.java:[291,15] cannot find symbol
          [ERROR] symbol  : variable HConstants
          [ERROR] location: class org.apache.flume.sink.hbase.HBaseSink
          [ERROR] /home/cos/work/bigtop/build/flume/rpm/BUILD/apache-flume-1.5.0-src/flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/HBaseSink.java:[293,22] cannot find symbol
          [ERROR] symbol  : variable HConstants
          [ERROR] location: class org.apache.flume.sink.hbase.HBaseSink
          [ERROR] /home/cos/work/bigtop/build/flume/rpm/BUILD/apache-flume-1.5.0-src/flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/HBaseSink.java:[294,25] cannot find symbol
          [ERROR] symbol  : variable HConstants
          [ERROR] location: class org.apache.flume.sink.hbase.HBaseSink
          [ERROR] /home/cos/work/bigtop/build/flume/rpm/BUILD/apache-flume-1.5.0-src/flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/HBaseSink.java:[299,22] cannot find symbol
          [ERROR] symbol  : variable HConstants
          [ERROR] location: class org.apache.flume.sink.hbase.HBaseSink
          [ERROR] /home/cos/work/bigtop/build/flume/rpm/BUILD/apache-flume-1.5.0-src/flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/HBaseSink.java:[491,24] cannot find symbol
          [ERROR] symbol  : variable Bytes
          [ERROR] location: class org.apache.flume.sink.hbase.HBaseSink
          [ERROR] /home/cos/work/bigtop/build/flume/rpm/BUILD/apache-flume-1.5.0-src/flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/HBaseSink.java:[542,33] cannot find symbol
          [ERROR] symbol  : variable Bytes
          [ERROR] location: class org.apache.flume.sink.hbase.HBaseSink
          [ERROR] /home/cos/work/bigtop/build/flume/rpm/BUILD/apache-flume-1.5.0-src/flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/HBaseSink.java:[548,35] cannot find symbol
          [ERROR] symbol  : variable Bytes
          [ERROR] location: class org.apache.flume.sink.hbase.HBaseSink
          [ERROR] /home/cos/work/bigtop/build/flume/rpm/BUILD/apache-flume-1.5.0-src/flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/HBaseSinkSecurityManager.java:[56,11] cannot find symbol
          [ERROR] symbol  : variable User
          [ERROR] location: class org.apache.flume.sink.hbase.HBaseSinkSecurityManager
          [ERROR] /home/cos/work/bigtop/build/flume/rpm/BUILD/apache-flume-1.5.0-src/flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/HBaseSinkSecurityManager.java:[56,39] cannot find symbol
          [ERROR] symbol  : variable User
          [ERROR] location: class org.apache.flume.sink.hbase.HBaseSinkSecurityManager
          [ERROR] /home/cos/work/bigtop/build/flume/rpm/BUILD/apache-flume-1.5.0-src/flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/HBaseSinkSecurityManager.java:[106,4] cannot find symbol
          [ERROR] symbol  : class User
          [ERROR] location: class org.apache.flume.sink.hbase.HBaseSinkSecurityManager
          [ERROR] /home/cos/work/bigtop/build/flume/rpm/BUILD/apache-flume-1.5.0-src/flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/HBaseSinkSecurityManager.java:[113,20] cannot find symbol
          [ERROR] symbol  : variable User
          [ERROR] location: class org.apache.flume.sink.hbase.HBaseSinkSecurityManager
          [ERROR] /home/cos/work/bigtop/build/flume/rpm/BUILD/apache-flume-1.5.0-src/flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/HBaseSinkSecurityManager.java:[126,6] cannot find symbol
          [ERROR] symbol  : variable User
          [ERROR] location: class org.apache.flume.sink.hbase.HBaseSinkSecurityManager
          [ERROR] /home/cos/work/bigtop/build/flume/rpm/BUILD/apache-flume-1.5.0-src/flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/HBaseSinkSecurityManager.java:[127,18] cannot find symbol
          [ERROR] symbol  : variable User
          [ERROR] location: class org.apache.flume.sink.hbase.HBaseSinkSecurityManager
          [ERROR] -> [Help 1]
          [ERROR]
          

          Is it me or Flume 1.5 isn't ready for HBase 0.98?

          Show
          Konstantin Boudnik added a comment - Ok, I've tried the patch and I see that comment meant to say. Flume doesn't reflect the singularity point of HBase. E.g. Flume still expect org.apache.hbase:hbase:jar to be present in 0.98.2. Hence, I don't think the later patch is adequate as it will be pulling in incorrect version of the flume. I can do in the flight-mod of the flume poms to start using hbase-client artifact instead, but then the whole bunch of compile errors is kicking in. E.g. [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.3.2:compile (default-compile) on project flume-ng-hbase-sink: Compilation failure: Compilation failure: [ERROR] /home/cos/work/bigtop/build/flume/rpm/BUILD/apache-flume-1.5.0-src/flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/AsyncHBaseSink.java:[44,30] cannot find symbol [ERROR] symbol : class HBaseConfiguration [ERROR] location: package org.apache.hadoop.hbase [ERROR] /home/cos/work/bigtop/build/flume/rpm/BUILD/apache-flume-1.5.0-src/flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/AsyncHBaseSink.java:[45,30] cannot find symbol [ERROR] symbol : class HConstants [ERROR] location: package org.apache.hadoop.hbase [ERROR] /home/cos/work/bigtop/build/flume/rpm/BUILD/apache-flume-1.5.0-src/flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/HBaseSink.java:[43,30] cannot find symbol [ERROR] symbol : class HBaseConfiguration [ERROR] location: package org.apache.hadoop.hbase [ERROR] /home/cos/work/bigtop/build/flume/rpm/BUILD/apache-flume-1.5.0-src/flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/HBaseSink.java:[44,30] cannot find symbol [ERROR] symbol : class HConstants [ERROR] location: package org.apache.hadoop.hbase [ERROR] /home/cos/work/bigtop/build/flume/rpm/BUILD/apache-flume-1.5.0-src/flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/HBaseSink.java:[49,35] cannot find symbol [ERROR] symbol : class Bytes [ERROR] location: package org.apache.hadoop.hbase.util [ERROR] /home/cos/work/bigtop/build/flume/rpm/BUILD/apache-flume-1.5.0-src/flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/HBaseSink.java:[57,39] cannot find symbol [ERROR] symbol : class User [ERROR] location: package org.apache.hadoop.hbase.security [ERROR] /home/cos/work/bigtop/build/flume/rpm/BUILD/apache-flume-1.5.0-src/flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/HBaseSink.java:[106,10] cannot find symbol [ERROR] symbol : class User [ERROR] location: class org.apache.flume.sink.hbase.HBaseSink [ERROR] /home/cos/work/bigtop/build/flume/rpm/BUILD/apache-flume-1.5.0-src/flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/HBaseSinkConfigurationConstants.java:[21,30] cannot find symbol [ERROR] symbol : class HConstants [ERROR] location: package org.apache.hadoop.hbase [ERROR] /home/cos/work/bigtop/build/flume/rpm/BUILD/apache-flume-1.5.0-src/flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/HBaseSinkSecurityManager.java:[27,39] cannot find symbol [ERROR] symbol : class User [ERROR] location: package org.apache.hadoop.hbase.security [ERROR] /home/cos/work/bigtop/build/flume/rpm/BUILD/apache-flume-1.5.0-src/flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/HBaseSinkSecurityManager.java:[71,29] cannot find symbol [ERROR] symbol : class User [ERROR] location: class org.apache.flume.sink.hbase.HBaseSinkSecurityManager [ERROR] /home/cos/work/bigtop/build/flume/rpm/BUILD/apache-flume-1.5.0-src/flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/RegexHbaseEventSerializer.java:[193,13] cannot access org.apache.hadoop.hbase.Cell [ERROR] class file for org.apache.hadoop.hbase.Cell not found [ERROR] put.add(cf, colNames.get(i), m.group(i + 1).getBytes(Charsets.UTF_8)); [ERROR] /home/cos/work/bigtop/build/flume/rpm/BUILD/apache-flume-1.5.0-src/flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/HBaseSinkConfigurationConstants.java:[67,6] cannot find symbol [ERROR] symbol : variable HConstants [ERROR] location: class org.apache.flume.sink.hbase.HBaseSinkConfigurationConstants [ERROR] /home/cos/work/bigtop/build/flume/rpm/BUILD/apache-flume-1.5.0-src/flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/AsyncHBaseSink.java:[368,15] cannot find symbol [ERROR] symbol : variable HBaseConfiguration [ERROR] location: class org.apache.flume.sink.hbase.AsyncHBaseSink [ERROR] /home/cos/work/bigtop/build/flume/rpm/BUILD/apache-flume-1.5.0-src/flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/AsyncHBaseSink.java:[371,27] cannot find symbol [ERROR] symbol : variable HConstants [ERROR] location: class org.apache.flume.sink.hbase.AsyncHBaseSink [ERROR] /home/cos/work/bigtop/build/flume/rpm/BUILD/apache-flume-1.5.0-src/flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/AsyncHBaseSink.java:[372,8] cannot find symbol [ERROR] symbol : variable HConstants [ERROR] location: class org.apache.flume.sink.hbase.AsyncHBaseSink [ERROR] /home/cos/work/bigtop/build/flume/rpm/BUILD/apache-flume-1.5.0-src/flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/HBaseSink.java:[116,9] cannot find symbol [ERROR] symbol : variable HBaseConfiguration [ERROR] location: class org.apache.flume.sink.hbase.HBaseSink [ERROR] /home/cos/work/bigtop/build/flume/rpm/BUILD/apache-flume-1.5.0-src/flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/HBaseSink.java:[148,25] cannot access org.apache.hadoop.hbase.TableName [ERROR] class file for org.apache.hadoop.hbase.TableName not found [ERROR] HTable table = new HTable(config, tableName); [ERROR] /home/cos/work/bigtop/build/flume/rpm/BUILD/apache-flume-1.5.0-src/flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/HBaseSink.java:[170,50] cannot find symbol [ERROR] symbol : variable Bytes [ERROR] location: class org.apache.flume.sink.hbase.HBaseSink [ERROR] /home/cos/work/bigtop/build/flume/rpm/BUILD/apache-flume-1.5.0-src/flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/HBaseSink.java:[178,16] cannot find symbol [ERROR] symbol : variable Bytes [ERROR] location: class org.apache.flume.sink.hbase.HBaseSink [ERROR] /home/cos/work/bigtop/build/flume/rpm/BUILD/apache-flume-1.5.0-src/flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/HBaseSink.java:[291,15] cannot find symbol [ERROR] symbol : variable HConstants [ERROR] location: class org.apache.flume.sink.hbase.HBaseSink [ERROR] /home/cos/work/bigtop/build/flume/rpm/BUILD/apache-flume-1.5.0-src/flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/HBaseSink.java:[293,22] cannot find symbol [ERROR] symbol : variable HConstants [ERROR] location: class org.apache.flume.sink.hbase.HBaseSink [ERROR] /home/cos/work/bigtop/build/flume/rpm/BUILD/apache-flume-1.5.0-src/flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/HBaseSink.java:[294,25] cannot find symbol [ERROR] symbol : variable HConstants [ERROR] location: class org.apache.flume.sink.hbase.HBaseSink [ERROR] /home/cos/work/bigtop/build/flume/rpm/BUILD/apache-flume-1.5.0-src/flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/HBaseSink.java:[299,22] cannot find symbol [ERROR] symbol : variable HConstants [ERROR] location: class org.apache.flume.sink.hbase.HBaseSink [ERROR] /home/cos/work/bigtop/build/flume/rpm/BUILD/apache-flume-1.5.0-src/flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/HBaseSink.java:[491,24] cannot find symbol [ERROR] symbol : variable Bytes [ERROR] location: class org.apache.flume.sink.hbase.HBaseSink [ERROR] /home/cos/work/bigtop/build/flume/rpm/BUILD/apache-flume-1.5.0-src/flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/HBaseSink.java:[542,33] cannot find symbol [ERROR] symbol : variable Bytes [ERROR] location: class org.apache.flume.sink.hbase.HBaseSink [ERROR] /home/cos/work/bigtop/build/flume/rpm/BUILD/apache-flume-1.5.0-src/flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/HBaseSink.java:[548,35] cannot find symbol [ERROR] symbol : variable Bytes [ERROR] location: class org.apache.flume.sink.hbase.HBaseSink [ERROR] /home/cos/work/bigtop/build/flume/rpm/BUILD/apache-flume-1.5.0-src/flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/HBaseSinkSecurityManager.java:[56,11] cannot find symbol [ERROR] symbol : variable User [ERROR] location: class org.apache.flume.sink.hbase.HBaseSinkSecurityManager [ERROR] /home/cos/work/bigtop/build/flume/rpm/BUILD/apache-flume-1.5.0-src/flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/HBaseSinkSecurityManager.java:[56,39] cannot find symbol [ERROR] symbol : variable User [ERROR] location: class org.apache.flume.sink.hbase.HBaseSinkSecurityManager [ERROR] /home/cos/work/bigtop/build/flume/rpm/BUILD/apache-flume-1.5.0-src/flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/HBaseSinkSecurityManager.java:[106,4] cannot find symbol [ERROR] symbol : class User [ERROR] location: class org.apache.flume.sink.hbase.HBaseSinkSecurityManager [ERROR] /home/cos/work/bigtop/build/flume/rpm/BUILD/apache-flume-1.5.0-src/flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/HBaseSinkSecurityManager.java:[113,20] cannot find symbol [ERROR] symbol : variable User [ERROR] location: class org.apache.flume.sink.hbase.HBaseSinkSecurityManager [ERROR] /home/cos/work/bigtop/build/flume/rpm/BUILD/apache-flume-1.5.0-src/flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/HBaseSinkSecurityManager.java:[126,6] cannot find symbol [ERROR] symbol : variable User [ERROR] location: class org.apache.flume.sink.hbase.HBaseSinkSecurityManager [ERROR] /home/cos/work/bigtop/build/flume/rpm/BUILD/apache-flume-1.5.0-src/flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/HBaseSinkSecurityManager.java:[127,18] cannot find symbol [ERROR] symbol : variable User [ERROR] location: class org.apache.flume.sink.hbase.HBaseSinkSecurityManager [ERROR] -> [Help 1] [ERROR] Is it me or Flume 1.5 isn't ready for HBase 0.98?
          Hide
          Hari Shreedharan added a comment -

          Currently, no. There is an open jira for this one - FLUME-1618. We need to add a new profile for 0.96+ (currently the hadoop-2 profile reflects hbase-0.94).

          Show
          Hari Shreedharan added a comment - Currently, no. There is an open jira for this one - FLUME-1618 . We need to add a new profile for 0.96+ (currently the hadoop-2 profile reflects hbase-0.94).
          Hide
          Konstantin Boudnik added a comment -

          Thanks for the reference. I have commented on the ticket wrt the timeline of the fix. I think our only option is to remove Flume 1.5.0 from 0.8.0 BOM unless the fix and subsequent release can happen in the next a couple of weeks.

          Other thoughts from the community here?

          Show
          Konstantin Boudnik added a comment - Thanks for the reference. I have commented on the ticket wrt the timeline of the fix. I think our only option is to remove Flume 1.5.0 from 0.8.0 BOM unless the fix and subsequent release can happen in the next a couple of weeks. Other thoughts from the community here?
          Hide
          Hari Shreedharan added a comment -

          I can try and do a release 1.5.0.1 if that can unblock this issue. Do you need a specific profile that is built against hbase-98 (we don't package any hbase libs ourselves) or would something built against hbase-96 work?

          Show
          Hari Shreedharan added a comment - I can try and do a release 1.5.0.1 if that can unblock this issue. Do you need a specific profile that is built against hbase-98 (we don't package any hbase libs ourselves) or would something built against hbase-96 work?
          Hide
          Konstantin Boudnik added a comment -

          Thanks a lot Hari - most appreciated! If we can have a profile that will build against Hadoop 2.3.0 (which current hadoop2 profile does) and Hbase 0.98+ - it'd be sufficient: Bigtop takes care about libraries inclusions, etc. Please let me know if we can help!

          Show
          Konstantin Boudnik added a comment - Thanks a lot Hari - most appreciated! If we can have a profile that will build against Hadoop 2.3.0 (which current hadoop2 profile does) and Hbase 0.98+ - it'd be sufficient: Bigtop takes care about libraries inclusions, etc. Please let me know if we can help!
          Hide
          Hari Shreedharan added a comment -

          Flume's Hadoop-2 profile depends on Hadoop 2.4.0, so I will have the new profiles depend on that as well, but since we don't bundle Hadoop - that should not be an issue.Hopefully, the jira will be committed tomorrow and I will spin the RC up tomorrow - though I may not be able to close the vote for a few days as I might be out.

          Show
          Hari Shreedharan added a comment - Flume's Hadoop-2 profile depends on Hadoop 2.4.0, so I will have the new profiles depend on that as well, but since we don't bundle Hadoop - that should not be an issue.Hopefully, the jira will be committed tomorrow and I will spin the RC up tomorrow - though I may not be able to close the vote for a few days as I might be out.
          Hide
          Konstantin Boudnik added a comment -

          When we build bigtop stack we do something like this:

          mvn -DskipTests -Dhadoop.profile=2  \
            -Dhadoop.version=$HADOOP_VERSION  \
            -Dhbase.version=$HBASE_VERSION    \
            clean install "$@"
          

          as far as this external specification works - I do not worry at all Thank you!

          Show
          Konstantin Boudnik added a comment - When we build bigtop stack we do something like this: mvn -DskipTests -Dhadoop.profile=2 \ -Dhadoop.version=$HADOOP_VERSION \ -Dhbase.version=$HBASE_VERSION \ clean install "$@" as far as this external specification works - I do not worry at all Thank you!
          Hide
          Hari Shreedharan added a comment -

          Flume 1.5.0.1 is done. It is available on dist. I have not yet updated the webpages but the release is on dist and maven central.

          Show
          Hari Shreedharan added a comment - Flume 1.5.0.1 is done. It is available on dist. I have not yet updated the webpages but the release is on dist and maven central.
          Hide
          Hari Shreedharan added a comment -

          You'd need to use -Dhadoop.profile=hbase-98 to build.

          Show
          Hari Shreedharan added a comment - You'd need to use -Dhadoop.profile=hbase-98 to build.
          Hide
          Konstantin Boudnik added a comment -

          It makes things better, thank you! I guess we are unblocked on the Flume now. Posting the patch.
          I have an issue with the HBase POM that we are building but this is different from the current one. Namely I see this:

          [INFO] Building Flume NG HBase Sink 1.5.0.1
          [INFO] ------------------------------------------------------------------------
          [WARNING] The POM for org.apache.hbase:hbase-client:jar:0.98.2 is invalid, transitive dependencies (if any) will not be available, enable debug logging for more details
          [WARNING] The POM for org.apache.hbase:hbase-client:jar:tests:0.98.2 is invalid, transitive dependencies (if any) will not be available, enable debug logging for more details
          [WARNING] The POM for org.apache.hbase:hbase-server:jar:0.98.2 is invalid, transitive dependencies (if any) will not be available, enable debug logging for more details
          [WARNING] The POM for org.apache.hbase:hbase-server:jar:tests:0.98.2 is invalid, transitive dependencies (if any) will not be available, enable debug logging for more details
          [WARNING] The POM for org.apache.hbase:hbase-common:jar:0.98.2 is invalid, transitive dependencies (if any) will not be available, enable debug logging for more details
          [WARNING] The POM for org.apache.hbase:hbase-testing-util:jar:0.98.2 is invalid, transitive dependencies (if any) will not be available, enable debug logging for more details
          

          I will look into this tomorrow.

          Show
          Konstantin Boudnik added a comment - It makes things better, thank you! I guess we are unblocked on the Flume now. Posting the patch. I have an issue with the HBase POM that we are building but this is different from the current one. Namely I see this: [INFO] Building Flume NG HBase Sink 1.5.0.1 [INFO] ------------------------------------------------------------------------ [WARNING] The POM for org.apache.hbase:hbase-client:jar:0.98.2 is invalid, transitive dependencies (if any) will not be available, enable debug logging for more details [WARNING] The POM for org.apache.hbase:hbase-client:jar:tests:0.98.2 is invalid, transitive dependencies (if any) will not be available, enable debug logging for more details [WARNING] The POM for org.apache.hbase:hbase-server:jar:0.98.2 is invalid, transitive dependencies (if any) will not be available, enable debug logging for more details [WARNING] The POM for org.apache.hbase:hbase-server:jar:tests:0.98.2 is invalid, transitive dependencies (if any) will not be available, enable debug logging for more details [WARNING] The POM for org.apache.hbase:hbase-common:jar:0.98.2 is invalid, transitive dependencies (if any) will not be available, enable debug logging for more details [WARNING] The POM for org.apache.hbase:hbase-testing-util:jar:0.98.2 is invalid, transitive dependencies (if any) will not be available, enable debug logging for more details I will look into this tomorrow.
          Hide
          Konstantin Boudnik added a comment -

          Patch to add Flume 1.5.0.1 into the stack. Validated to be produce correct bits with Hbase 0.98.2-hadoop2. The version of Hbase we produce during the stack build has some issues, that require additional investigation.

          Show
          Konstantin Boudnik added a comment - Patch to add Flume 1.5.0.1 into the stack. Validated to be produce correct bits with Hbase 0.98.2-hadoop2. The version of Hbase we produce during the stack build has some issues, that require additional investigation.
          Hide
          Konstantin Boudnik added a comment -

          Anyone to review?

          Show
          Konstantin Boudnik added a comment - Anyone to review?
          Hide
          Mark Grover added a comment -

          +1
          Thanks Cos and Hari!

          Show
          Mark Grover added a comment - +1 Thanks Cos and Hari!
          Hide
          Konstantin Boudnik added a comment -

          Pushed to the master. Thanks for the review Mark

          Show
          Konstantin Boudnik added a comment - Pushed to the master. Thanks for the review Mark

            People

            • Assignee:
              Konstantin Boudnik
              Reporter:
              Guo Ruijing
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development