Apache Drill
  1. Apache Drill
  2. DRILL-15

Build HBase storage engine implementation

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    1. DRILL-15.metastore.patch.txt
      192 kB
      Lisen Mu
    2. DRILL-15-03.patch
      73 kB
      David Alves
    3. DRILL-15-02.patch
      74 kB
      David Alves
    4. DRILL-15.patch
      71 kB
      David Alves

      Issue Links

        Activity

        Hide
        goog cheng added a comment - - edited

        read records from Hbase which is a NoSQL, how to know the schemaless schema

        Show
        goog cheng added a comment - - edited read records from Hbase which is a NoSQL, how to know the schemaless schema
        Hide
        Henry Saputra added a comment -

        Related to DRILL-16

        Show
        Henry Saputra added a comment - Related to DRILL-16
        Hide
        David Alves added a comment -

        Here's a first stab at an hbase scanner against the reference implementation https://github.com/apache/incubator-drill/pull/12.patch together with some tests.

        I know that probably reference impl is temporary but I wanted to get a feel of what would be a nice interface for the storage engine.

        Show
        David Alves added a comment - Here's a first stab at an hbase scanner against the reference implementation https://github.com/apache/incubator-drill/pull/12.patch together with some tests. I know that probably reference impl is temporary but I wanted to get a feel of what would be a nice interface for the storage engine.
        Hide
        Ted Yu added a comment -
        +        <dependency>
        +            <groupId>org.apache.hbase</groupId>
        +            <artifactId>hbase</artifactId>
        +            <version>0.94.1</version>
        

        0.94.5 is the latest release of HBase. Can you use that ?

        Show
        Ted Yu added a comment - + <dependency> + <groupId>org.apache.hbase</groupId> + <artifactId>hbase</artifactId> + <version>0.94.1</version> 0.94.5 is the latest release of HBase. Can you use that ?
        Hide
        David Alves added a comment -

        upgraded it to HBase 0.94.5

        Show
        David Alves added a comment - upgraded it to HBase 0.94.5
        Hide
        David Alves added a comment -

        Complete patch that implements the table version of the record reader (i.e. using a table scanner, a more powerful version version that uses a region scanner is next.).

        Note: this version only supports READS.

        This version scans the table defined in the scan operator and returns all CF's and all column version (some limited filtering and projection on CF/column names should be possible even without the region scanning version).

        An identity plan with console as the out operator yields data in this format:

        {
          "rowKey" : "MDAwMQ==",
          "batters" : {
            "1001" : {
              "timestamp" : 1364113347095,
              "value" : "U3RyaW5nU2NhbGFyIFtzZXE9UmVndWxhcl0="
            },
            "1002" : {
              "timestamp" : 1364113347095,
              "value" : "U3RyaW5nU2NhbGFyIFtzZXE9Q2hvY29sYXRlXQ=="
            }
          },
          "metadata" : {
            "name" : {
              "timestamp" : 1364113347095,
              "value" : "U3RyaW5nU2NhbGFyIFtzZXE9Q2FrZV0="
            },
            "ppu" : {
              "timestamp" : 1364113347095,
              "value" : "P+GZmZmZmZo="
            },
            "sales" : {
              "timestamp" : 1364113347095,
              "value" : "AAAAIw=="
            },
            "type" : {
              "timestamp" : 1364113347095,
              "value" : "U3RyaW5nU2NhbGFyIFtzZXE9ZG9udXRd"
            }
          }
        } 
        ...
        

        The region based version of the record reader is the one that is going to support filter/projection/partial aggregation/local join pushdown, but this one should ba useful already.

        Please try/review!

        Show
        David Alves added a comment - Complete patch that implements the table version of the record reader (i.e. using a table scanner, a more powerful version version that uses a region scanner is next.). Note: this version only supports READS. This version scans the table defined in the scan operator and returns all CF's and all column version (some limited filtering and projection on CF/column names should be possible even without the region scanning version). An identity plan with console as the out operator yields data in this format: { "rowKey" : "MDAwMQ==" , "batters" : { "1001" : { "timestamp" : 1364113347095, "value" : "U3RyaW5nU2NhbGFyIFtzZXE9UmVndWxhcl0=" }, "1002" : { "timestamp" : 1364113347095, "value" : "U3RyaW5nU2NhbGFyIFtzZXE9Q2hvY29sYXRlXQ==" } }, "metadata" : { "name" : { "timestamp" : 1364113347095, "value" : "U3RyaW5nU2NhbGFyIFtzZXE9Q2FrZV0=" }, "ppu" : { "timestamp" : 1364113347095, "value" : "P+GZmZmZmZo=" }, "sales" : { "timestamp" : 1364113347095, "value" : "AAAAIw==" }, "type" : { "timestamp" : 1364113347095, "value" : "U3RyaW5nU2NhbGFyIFtzZXE9ZG9udXRd" } } } ... The region based version of the record reader is the one that is going to support filter/projection/partial aggregation/local join pushdown, but this one should ba useful already. Please try/review!
        Hide
        David Alves added a comment -

        removed overlapping code from DRILL-55

        Show
        David Alves added a comment - removed overlapping code from DRILL-55
        Hide
        Ted Yu added a comment -

        The following feature would be useful for future enhancement:
        HBASE-5416 Improve performance of scans with some kind of filters

        + * A DataValue corresponding the whole row.
        + */
        +public class HBaseResultValue extends ImmutableHBaseMapValue {

        The above class duplicates information from hbase.client.Result
        It would be nice to document in class javadoc why this is needed.

        + for (NavigableMap.Entry<byte[], NavigableMap<byte[], NavigableMap<Long, byte[]>>> family : result.getMap().entrySet()) {
        + this.families.put(HbaseUtils.nameFromBytes(family.getKey()), new HBaseFamilyValue(family.getValue()));

        Does Drill have limitation on length of one line ?

        Show
        Ted Yu added a comment - The following feature would be useful for future enhancement: HBASE-5416 Improve performance of scans with some kind of filters + * A DataValue corresponding the whole row. + */ +public class HBaseResultValue extends ImmutableHBaseMapValue { The above class duplicates information from hbase.client.Result It would be nice to document in class javadoc why this is needed. + for (NavigableMap.Entry<byte[], NavigableMap<byte[], NavigableMap<Long, byte[]>>> family : result.getMap().entrySet()) { + this.families.put(HbaseUtils.nameFromBytes(family.getKey()), new HBaseFamilyValue(family.getValue())); Does Drill have limitation on length of one line ?
        Hide
        David Alves added a comment -

        HBaseResultValue is a thin wrapper around an Result (from HBase) that complies with DataValue from Drill.

        I'll be working on filters and projections at the RegionScanner level because that the priority for my particular project but I'll add them to the table scanner as soon as possible.

        Show
        David Alves added a comment - HBaseResultValue is a thin wrapper around an Result (from HBase) that complies with DataValue from Drill. I'll be working on filters and projections at the RegionScanner level because that the priority for my particular project but I'll add them to the table scanner as soon as possible.
        Hide
        Ted Yu added a comment -

        Thanks for the quick response, David.

        You can consider using review board to facilitate review:
        https://reviews.apache.org/r/new/

        There is drill-git Repository.

        Show
        Ted Yu added a comment - Thanks for the quick response, David. You can consider using review board to facilitate review: https://reviews.apache.org/r/new/ There is drill-git Repository.
        Hide
        Ted Dunning added a comment -

        + for (NavigableMap.Entry<byte[], NavigableMap<byte[], NavigableMap<Long, byte[]>>> family : result.getMap().entrySet()) {
        + this.families.put(HbaseUtils.nameFromBytes(family.getKey()), new HBaseFamilyValue(family.getValue()));

        Does Drill have limitation on length of one line ?

        I believe we settled on 120 characters. The first of these lines, however, would be hard to split well. I would tend to make an exception for the first line.

        Show
        Ted Dunning added a comment - + for (NavigableMap.Entry<byte[], NavigableMap<byte[], NavigableMap<Long, byte[]>>> family : result.getMap().entrySet()) { + this.families.put(HbaseUtils.nameFromBytes(family.getKey()), new HBaseFamilyValue(family.getValue())); Does Drill have limitation on length of one line ? I believe we settled on 120 characters. The first of these lines, however, would be hard to split well. I would tend to make an exception for the first line.
        Hide
        David Alves added a comment -

        Running the live test depends on fixing the RSE registry bug.

        Show
        David Alves added a comment - Running the live test depends on fixing the RSE registry bug.
        Hide
        David Alves added a comment -

        This patch consolidates the Iterators in a single impl, avoids map copies by using transformers over the raw (HBase) Result maps and allows to fetch timestamp and value of a column by name.

        Show
        David Alves added a comment - This patch consolidates the Iterators in a single impl, avoids map copies by using transformers over the raw (HBase) Result maps and allows to fetch timestamp and value of a column by name.
        Hide
        David Alves added a comment -

        review request can be found at: https://reviews.apache.org/r/10099/

        Show
        David Alves added a comment - review request can be found at: https://reviews.apache.org/r/10099/
        Hide
        David Alves added a comment -

        addressed ted's review comments.
        new version is also available in reviewboard.

        Show
        David Alves added a comment - addressed ted's review comments. new version is also available in reviewboard.
        Hide
        David Alves added a comment -

        If everyone agrees I'd like to constrain the scope of this issue by making it first iteration on the hbase storage engine which is "usable" from drill for querying hbase from a single node (i.e. acts at a table level) and create a new one for parallel/distributed execution that will work a the region level.

        Show
        David Alves added a comment - If everyone agrees I'd like to constrain the scope of this issue by making it first iteration on the hbase storage engine which is "usable" from drill for querying hbase from a single node (i.e. acts at a table level) and create a new one for parallel/distributed execution that will work a the region level.
        Hide
        Jacques Nadeau added a comment -

        I agree with that goal. There is review feedback pending on this I believe.

        Show
        Jacques Nadeau added a comment - I agree with that goal. There is review feedback pending on this I believe.
        Hide
        Lisen Mu added a comment - - edited

        cut related part form our code base. this patch is pretty far from ready-to-merge state, but I guess it won't be merged anyway just show the idea and the problem.

        We are using hive metastore to represent data schema information of HTables. Metastore has 3 kind of information:

        • WHAT the logical fields look like. From user's perspective, a record may have different fields like integer, date, boolean or text etc. This also indicates how drill would process fields in memory.
        • WHERE the logical fields is stored in HTable. there are many places in HTable in which information can be stored: rowkey; qualifier name of a particular CF; value under a particular CF:qualifier; or even version number of a particular cell. The value of a logical field can be stored as any of above. Further, rowkey may contain multiple logical fields. This highly depends on how user design their storage schema.
        • HOW logical fields is stored. HBase basically provides a storage for byte[]. So HTable scanner needs to know how the fields like integer, date, boolean are serialized as byte[]. For example, 255 would be serialized to \xFF as BINARY:1byte, or [FF 00 00 00] as BINARY:4byte, or "255" as TEXT(with variable length), or "00000255" as TEXT(with fixed length:8). Another example would be logical DATE to (first integer 1375483564 then) [AC 36 FC 51] as BINARY:4byte, or "20130803" as TEXT(with fixed length:8).

        The meta definition is in com.xingcloud.meta.HBaseFieldInfo.java

        These information will be used in HBase scanner to generate most effective scan (mapping logical Filter to HBase's filter class, and deciding startKey and endKey to scan least data), and in conversion from LogicalPlan to PhysicalPlan, to generate the correct ReadEntry for HBase.

        I do understand that strong schema is not drill's primary concern, however I think other approaches to HBase scanner also have to solve the problems above to work correctly. plus: I guess most HBase users do have a carefully designed schema in mind...

        Show
        Lisen Mu added a comment - - edited cut related part form our code base. this patch is pretty far from ready-to-merge state, but I guess it won't be merged anyway just show the idea and the problem. We are using hive metastore to represent data schema information of HTables. Metastore has 3 kind of information: WHAT the logical fields look like. From user's perspective, a record may have different fields like integer, date, boolean or text etc. This also indicates how drill would process fields in memory. WHERE the logical fields is stored in HTable. there are many places in HTable in which information can be stored: rowkey; qualifier name of a particular CF; value under a particular CF:qualifier; or even version number of a particular cell. The value of a logical field can be stored as any of above. Further, rowkey may contain multiple logical fields. This highly depends on how user design their storage schema. HOW logical fields is stored. HBase basically provides a storage for byte[]. So HTable scanner needs to know how the fields like integer, date, boolean are serialized as byte[]. For example, 255 would be serialized to \xFF as BINARY:1byte, or [FF 00 00 00] as BINARY:4byte, or "255" as TEXT(with variable length), or "00000255" as TEXT(with fixed length:8). Another example would be logical DATE to (first integer 1375483564 then) [AC 36 FC 51] as BINARY:4byte, or "20130803" as TEXT(with fixed length:8). The meta definition is in com.xingcloud.meta.HBaseFieldInfo.java These information will be used in HBase scanner to generate most effective scan (mapping logical Filter to HBase's filter class, and deciding startKey and endKey to scan least data), and in conversion from LogicalPlan to PhysicalPlan, to generate the correct ReadEntry for HBase. I do understand that strong schema is not drill's primary concern, however I think other approaches to HBase scanner also have to solve the problems above to work correctly. plus: I guess most HBase users do have a carefully designed schema in mind...
        Hide
        Jacques Nadeau added a comment -

        New version merged in later JIRA

        Show
        Jacques Nadeau added a comment - New version merged in later JIRA

          People

          • Assignee:
            David Alves
            Reporter:
            Ted Yu
          • Votes:
            0 Vote for this issue
            Watchers:
            13 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development