Hive
  1. Hive
  2. HIVE-3823

Performance issue while retrieving the Result objects in HiveHBaseTableInputFormat

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Trivial Trivial
    • Resolution: Unresolved
    • Affects Version/s: 0.9.0, 0.9.1, 0.10.0
    • Fix Version/s: None
    • Component/s: HBase Handler
    • Labels:
      None

      Description

      In HiveHBaseTableInputFormat.java, the Result objects retrieving has performance issue.

      HiveHBaseTableInputFormat.java
            @Override
            public boolean next(ImmutableBytesWritable rowKey, Result value) throws IOException {
      
              boolean next = false;
      
              try {
                next = recordReader.nextKeyValue();
      
                if (next) {
                  rowKey.set(recordReader.getCurrentValue().getRow());
                  // performance issue here, as the copyWritable
                  // is Serialization - Bytes Copying - Deserialization.
                  Writables.copyWritable(recordReader.getCurrentValue(), value);
                }
              } catch (InterruptedException e) {
                throw new IOException(e);
              }
      
              return next;
            }
      

      In HBASE 0.94.4 & 0.96.0, the Result provides a new method "copyFrom(Result from)", will solve the issue.

      See HBASE-7381

        Issue Links

          Activity

          Cheng Hao created issue -
          Cheng Hao made changes -
          Field Original Value New Value
          Description In HiveHBaseTableInputFormat.java, the Result objects retrieving has performance issue.
          {code:title:HiveHBaseTableInputFormat}
                @Override
                public boolean next(ImmutableBytesWritable rowKey, Result value) throws IOException {

                  boolean next = false;

                  try {
                    next = recordReader.nextKeyValue();

                    if (next) {
                      rowKey.set(recordReader.getCurrentValue().getRow());
                      // performance issue here, as the copyWritable
                      // is Serialization - Bytes Copying - Deserialization.
                      Writables.copyWritable(recordReader.getCurrentValue(), value);
                    }
                  } catch (InterruptedException e) {
                    throw new IOException(e);
                  }

                  return next;
                }
          {code}

          In HBASE 0.94.4 & 0.96.0, the Result provides a new method copyFrom, would solve the issue.

          See [HBASE-7381|https://issues.apache.org/jira/browse/HBASE-7381]
          In HiveHBaseTableInputFormat.java, the Result objects retrieving has performance issue.
          {code:title:HiveHBaseTableInputFormat.java}
                @Override
                public boolean next(ImmutableBytesWritable rowKey, Result value) throws IOException {

                  boolean next = false;

                  try {
                    next = recordReader.nextKeyValue();

                    if (next) {
                      rowKey.set(recordReader.getCurrentValue().getRow());
                      // performance issue here, as the copyWritable
                      // is Serialization - Bytes Copying - Deserialization.
                      Writables.copyWritable(recordReader.getCurrentValue(), value);
                    }
                  } catch (InterruptedException e) {
                    throw new IOException(e);
                  }

                  return next;
                }
          {code}

          In HBASE 0.94.4 & 0.96.0, the Result provides a new method copyFrom, would solve the issue.

          See [HBASE-7381|https://issues.apache.org/jira/browse/HBASE-7381]
          Cheng Hao made changes -
          Description In HiveHBaseTableInputFormat.java, the Result objects retrieving has performance issue.
          {code:title:HiveHBaseTableInputFormat.java}
                @Override
                public boolean next(ImmutableBytesWritable rowKey, Result value) throws IOException {

                  boolean next = false;

                  try {
                    next = recordReader.nextKeyValue();

                    if (next) {
                      rowKey.set(recordReader.getCurrentValue().getRow());
                      // performance issue here, as the copyWritable
                      // is Serialization - Bytes Copying - Deserialization.
                      Writables.copyWritable(recordReader.getCurrentValue(), value);
                    }
                  } catch (InterruptedException e) {
                    throw new IOException(e);
                  }

                  return next;
                }
          {code}

          In HBASE 0.94.4 & 0.96.0, the Result provides a new method copyFrom, would solve the issue.

          See [HBASE-7381|https://issues.apache.org/jira/browse/HBASE-7381]
          In HiveHBaseTableInputFormat.java, the Result objects retrieving has performance issue.
          {code:title=HiveHBaseTableInputFormat.java}
                @Override
                public boolean next(ImmutableBytesWritable rowKey, Result value) throws IOException {

                  boolean next = false;

                  try {
                    next = recordReader.nextKeyValue();

                    if (next) {
                      rowKey.set(recordReader.getCurrentValue().getRow());
                      // performance issue here, as the copyWritable
                      // is Serialization - Bytes Copying - Deserialization.
                      Writables.copyWritable(recordReader.getCurrentValue(), value);
                    }
                  } catch (InterruptedException e) {
                    throw new IOException(e);
                  }

                  return next;
                }
          {code}

          In HBASE 0.94.4 & 0.96.0, the Result provides a new method copyFrom, would solve the issue.

          See [HBASE-7381|https://issues.apache.org/jira/browse/HBASE-7381]
          Cheng Hao made changes -
          Description In HiveHBaseTableInputFormat.java, the Result objects retrieving has performance issue.
          {code:title=HiveHBaseTableInputFormat.java}
                @Override
                public boolean next(ImmutableBytesWritable rowKey, Result value) throws IOException {

                  boolean next = false;

                  try {
                    next = recordReader.nextKeyValue();

                    if (next) {
                      rowKey.set(recordReader.getCurrentValue().getRow());
                      // performance issue here, as the copyWritable
                      // is Serialization - Bytes Copying - Deserialization.
                      Writables.copyWritable(recordReader.getCurrentValue(), value);
                    }
                  } catch (InterruptedException e) {
                    throw new IOException(e);
                  }

                  return next;
                }
          {code}

          In HBASE 0.94.4 & 0.96.0, the Result provides a new method copyFrom, would solve the issue.

          See [HBASE-7381|https://issues.apache.org/jira/browse/HBASE-7381]
          In HiveHBaseTableInputFormat.java, the Result objects retrieving has performance issue.
          {code:title=HiveHBaseTableInputFormat.java}
                @Override
                public boolean next(ImmutableBytesWritable rowKey, Result value) throws IOException {

                  boolean next = false;

                  try {
                    next = recordReader.nextKeyValue();

                    if (next) {
                      rowKey.set(recordReader.getCurrentValue().getRow());
                      // performance issue here, as the copyWritable
                      // is Serialization - Bytes Copying - Deserialization.
                      Writables.copyWritable(recordReader.getCurrentValue(), value);
                    }
                  } catch (InterruptedException e) {
                    throw new IOException(e);
                  }

                  return next;
                }
          {code}

          In HBASE 0.94.4 & 0.96.0, the Result provides a new method copyFrom(Result from), would solve the issue.

          See [HBASE-7381|https://issues.apache.org/jira/browse/HBASE-7381]
          Cheng Hao made changes -
          Description In HiveHBaseTableInputFormat.java, the Result objects retrieving has performance issue.
          {code:title=HiveHBaseTableInputFormat.java}
                @Override
                public boolean next(ImmutableBytesWritable rowKey, Result value) throws IOException {

                  boolean next = false;

                  try {
                    next = recordReader.nextKeyValue();

                    if (next) {
                      rowKey.set(recordReader.getCurrentValue().getRow());
                      // performance issue here, as the copyWritable
                      // is Serialization - Bytes Copying - Deserialization.
                      Writables.copyWritable(recordReader.getCurrentValue(), value);
                    }
                  } catch (InterruptedException e) {
                    throw new IOException(e);
                  }

                  return next;
                }
          {code}

          In HBASE 0.94.4 & 0.96.0, the Result provides a new method copyFrom(Result from), would solve the issue.

          See [HBASE-7381|https://issues.apache.org/jira/browse/HBASE-7381]
          In HiveHBaseTableInputFormat.java, the Result objects retrieving has performance issue.
          {code:title=HiveHBaseTableInputFormat.java}
                @Override
                public boolean next(ImmutableBytesWritable rowKey, Result value) throws IOException {

                  boolean next = false;

                  try {
                    next = recordReader.nextKeyValue();

                    if (next) {
                      rowKey.set(recordReader.getCurrentValue().getRow());
                      // performance issue here, as the copyWritable
                      // is Serialization - Bytes Copying - Deserialization.
                      Writables.copyWritable(recordReader.getCurrentValue(), value);
                    }
                  } catch (InterruptedException e) {
                    throw new IOException(e);
                  }

                  return next;
                }
          {code}

          In HBASE 0.94.4 & 0.96.0, the Result provides a new method "copyFrom(Result from)", will solve the issue.

          See [HBASE-7381|https://issues.apache.org/jira/browse/HBASE-7381]
          Cheng Hao made changes -
          Link This issue depends on HBASE-7381 [ HBASE-7381 ]
          Gavin made changes -
          Link This issue depends on HBASE-7381 [ HBASE-7381 ]
          Gavin made changes -
          Link This issue depends upon HBASE-7381 [ HBASE-7381 ]

            People

            • Assignee:
              Unassigned
              Reporter:
              Cheng Hao
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:

                Development