[HIVE-12827] Vectorization: VectorCopyRow/VectorAssignRow/VectorDeserializeRow assign needs explicit isNull[offset] modification

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.0.0

Release Note

commit 9cab4414caf1bba2eb1852536a9d3676ba7eab21
Author: Gopal V <[e-mail]>
Date: Mon Jan 18 16:03:40 2016 -0800

    Vectorization: VectorCopyRow/VectorAssignRow/VectorDeserializeRow assign needs explicit isNull[offset] modification (Gopal V, reviewed by Sergey Shelukhin)
    
    Signed-off-by: Gopal V <[e-mail]>

• Assignee: Gopal Vijayaraghavan
• Reporter: Gopal Vijayaraghavan
• Created: 09/Jan/16 13:41
• Updated: 02/Jun/16 10:17
• Resolved: 19/Jan/16 00:04

Description

Some scenarios do set Double.NaN instead of isNull=true, but all types aren't consistent.

Examples of un-set isNull for the valid values are

  private class FloatReader extends AbstractDoubleReader {

    FloatReader(int columnIndex) {
      super(columnIndex);
    }

    @Override
    void apply(VectorizedRowBatch batch, int batchIndex) throws IOException {
      DoubleColumnVector colVector = (DoubleColumnVector) batch.cols[columnIndex];

      if (deserializeRead.readCheckNull()) {
        VectorizedBatchUtil.setNullColIsNullValue(colVector, batchIndex);
      } else {
        float value = deserializeRead.readFloat();
        colVector.vector[batchIndex] = (double) value;
      }
    }
  }

  private class DoubleCopyRow extends CopyRow {

    DoubleCopyRow(int inColumnIndex, int outColumnIndex) {
      super(inColumnIndex, outColumnIndex);
    }

    @Override
    void copy(VectorizedRowBatch inBatch, int inBatchIndex, VectorizedRowBatch outBatch, int outBatchIndex) {
      DoubleColumnVector inColVector = (DoubleColumnVector) inBatch.cols[inColumnIndex];
      DoubleColumnVector outColVector = (DoubleColumnVector) outBatch.cols[outColumnIndex];

      if (inColVector.isRepeating) {
        if (inColVector.noNulls || !inColVector.isNull[0]) {
          outColVector.vector[outBatchIndex] = inColVector.vector[0];
        } else {
          VectorizedBatchUtil.setNullColIsNullValue(outColVector, outBatchIndex);
        }
      } else {
        if (inColVector.noNulls || !inColVector.isNull[inBatchIndex]) {
          outColVector.vector[outBatchIndex] = inColVector.vector[inBatchIndex];
        } else {
          VectorizedBatchUtil.setNullColIsNullValue(outColVector, outBatchIndex);
        }
      }
    }
  }

 private static abstract class VectorDoubleColumnAssign
    extends VectorColumnAssignVectorBase<DoubleColumnVector> {

    protected void assignDouble(double value, int destIndex) {
      outCol.vector[destIndex] = value;
    }
  }

The pattern to imitate would be the earlier code from VectorBatchUtil

    case DOUBLE: {
      DoubleColumnVector dcv = (DoubleColumnVector) batch.cols[offset + colIndex];
      if (writableCol != null) {
        dcv.vector[rowIndex] = ((DoubleWritable) writableCol).get();
        dcv.isNull[rowIndex] = false;
      } else {
        dcv.vector[rowIndex] = Double.NaN;
        setNullColIsNullValue(dcv, rowIndex);
      }
    }
      break;

Issue Links

• is required by: HIVE-12750 Vectorization logic not maintaining noNulls flag and isNull array properly causing wrong query results
• relates to: HIVE-11704 Create errata.txt file