Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.8.0
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      New datatype BINARY can now be used in column definitions.

      Description

      Add bytearray as a primitive data type.

      1. hive-2380_1.patch
        129 kB
        Ashutosh Chauhan
      2. hive-2380_2.patch
        135 kB
        Ashutosh Chauhan
      3. hive-2380_3.patch
        133 kB
        Ashutosh Chauhan
      4. hive-2380_4.patch
        148 kB
        Ashutosh Chauhan
      5. hive-2380.patch
        127 kB
        Ashutosh Chauhan

        Issue Links

          Activity

          Hide
          Ashutosh Chauhan added a comment -

          This is WIP. Attaching it early to illustrate the design to get feedback.

          Show
          Ashutosh Chauhan added a comment - This is WIP. Attaching it early to illustrate the design to get feedback.
          Hide
          Ashutosh Chauhan added a comment -

          Complete patch. It is ready for review.

          Show
          Ashutosh Chauhan added a comment - Complete patch. It is ready for review.
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/1586/
          -----------------------------------------------------------

          Review request for hive.

          Summary
          -------

          This patch adds bytearray as a new datatype in Hive.

          This addresses bug HIVE-2380.
          https://issues.apache.org/jira/browse/HIVE-2380

          Diffs


          trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/SettableByteArrayObjectInspector.java PRE-CREATION
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableByteArrayObjectInspector.java PRE-CREATION
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableVoidObjectInspector.java 1159452
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/typeinfo/TypeInfoFactory.java 1159452
          trunk/serde/src/test/org/apache/hadoop/hive/serde2/TestStatsSerde.java 1159452
          trunk/serde/src/test/org/apache/hadoop/hive/serde2/binarysortable/MyTestClass.java 1159452
          trunk/serde/src/test/org/apache/hadoop/hive/serde2/binarysortable/TestBinarySortableSerDe.java 1159452
          trunk/serde/src/test/org/apache/hadoop/hive/serde2/columnar/TestLazyBinaryColumnarSerDe.java 1159452
          trunk/serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazyPrimitive.java 1159452
          trunk/serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazySimpleSerDe.java 1159452
          trunk/serde/src/test/org/apache/hadoop/hive/serde2/lazybinary/MyTestClassBigger.java 1159452
          trunk/serde/src/test/org/apache/hadoop/hive/serde2/lazybinary/TestLazyBinarySerDe.java 1159452
          trunk/serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestObjectInspectorConverters.java 1159452
          trunk/serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestStandardObjectInspectors.java 1159452
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorConverter.java 1159452
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorFactory.java 1159452
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorUtils.java 1159452
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/PrimitiveObjectInspector.java 1159452
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/ByteArrayObjectInspector.java PRE-CREATION
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/JavaByteArrayObjectInspector.java PRE-CREATION
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java 1159452
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinarySerDe.java 1159452
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryUtils.java 1159452
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorConverters.java 1159452
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryByteArray.java PRE-CREATION
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryFactory.java 1159452
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/primitive/LazyByteArrayObjectInspector.java PRE-CREATION
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/primitive/LazyPrimitiveObjectInspectorFactory.java 1159452
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyByteArray.java PRE-CREATION
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyFactory.java 1159452
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyUtils.java 1159452
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/columnar/LazyBinaryColumnarSerDe.java 1159452
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/columnar/LazyBinaryColumnarStruct.java 1159452
          trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 1159452
          trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g 1159452
          trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ParseDriver.java 1159452
          trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java 1159452
          trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToBoolean.java 1159452
          trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToByte.java 1159452
          trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToDouble.java 1159452
          trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToFloat.java 1159452
          trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToInteger.java 1159452
          trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToLong.java 1159452
          trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToShort.java 1159452
          trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToString.java 1159452
          trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/UDFToByteArray.java PRE-CREATION
          trunk/ql/src/test/queries/clientpositive/ba_table1.q PRE-CREATION
          trunk/ql/src/test/queries/clientpositive/ba_table2.q PRE-CREATION
          trunk/ql/src/test/results/clientpositive/ba_table1.q.out PRE-CREATION
          trunk/ql/src/test/results/clientpositive/ba_table2.q.out PRE-CREATION
          trunk/serde/if/serde.thrift 1159452
          trunk/serde/src/gen/thrift/gen-cpp/serde_constants.h 1159452
          trunk/serde/src/gen/thrift/gen-cpp/serde_constants.cpp 1159452
          trunk/serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/Constants.java 1159452
          trunk/serde/src/gen/thrift/gen-php/serde/serde_constants.php 1159452
          trunk/serde/src/gen/thrift/gen-py/org_apache_hadoop_hive_serde/constants.py 1159452
          trunk/serde/src/gen/thrift/gen-rb/serde_constants.rb 1159452
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/SerDeUtils.java 1159452
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/binarysortable/BinarySortableSerDe.java 1159452
          trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 1159452
          trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 1159452

          Diff: https://reviews.apache.org/r/1586/diff

          Testing
          -------

          Added and updated unit tests.
          Added new system tests.

          Thanks,

          Ashutosh

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1586/ ----------------------------------------------------------- Review request for hive. Summary ------- This patch adds bytearray as a new datatype in Hive. This addresses bug HIVE-2380 . https://issues.apache.org/jira/browse/HIVE-2380 Diffs trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/SettableByteArrayObjectInspector.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableByteArrayObjectInspector.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableVoidObjectInspector.java 1159452 trunk/serde/src/java/org/apache/hadoop/hive/serde2/typeinfo/TypeInfoFactory.java 1159452 trunk/serde/src/test/org/apache/hadoop/hive/serde2/TestStatsSerde.java 1159452 trunk/serde/src/test/org/apache/hadoop/hive/serde2/binarysortable/MyTestClass.java 1159452 trunk/serde/src/test/org/apache/hadoop/hive/serde2/binarysortable/TestBinarySortableSerDe.java 1159452 trunk/serde/src/test/org/apache/hadoop/hive/serde2/columnar/TestLazyBinaryColumnarSerDe.java 1159452 trunk/serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazyPrimitive.java 1159452 trunk/serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazySimpleSerDe.java 1159452 trunk/serde/src/test/org/apache/hadoop/hive/serde2/lazybinary/MyTestClassBigger.java 1159452 trunk/serde/src/test/org/apache/hadoop/hive/serde2/lazybinary/TestLazyBinarySerDe.java 1159452 trunk/serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestObjectInspectorConverters.java 1159452 trunk/serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestStandardObjectInspectors.java 1159452 trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorConverter.java 1159452 trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorFactory.java 1159452 trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorUtils.java 1159452 trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/PrimitiveObjectInspector.java 1159452 trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/ByteArrayObjectInspector.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/JavaByteArrayObjectInspector.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java 1159452 trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinarySerDe.java 1159452 trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryUtils.java 1159452 trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorConverters.java 1159452 trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryByteArray.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryFactory.java 1159452 trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/primitive/LazyByteArrayObjectInspector.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/primitive/LazyPrimitiveObjectInspectorFactory.java 1159452 trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyByteArray.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyFactory.java 1159452 trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyUtils.java 1159452 trunk/serde/src/java/org/apache/hadoop/hive/serde2/columnar/LazyBinaryColumnarSerDe.java 1159452 trunk/serde/src/java/org/apache/hadoop/hive/serde2/columnar/LazyBinaryColumnarStruct.java 1159452 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 1159452 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g 1159452 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ParseDriver.java 1159452 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java 1159452 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToBoolean.java 1159452 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToByte.java 1159452 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToDouble.java 1159452 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToFloat.java 1159452 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToInteger.java 1159452 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToLong.java 1159452 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToShort.java 1159452 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToString.java 1159452 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/UDFToByteArray.java PRE-CREATION trunk/ql/src/test/queries/clientpositive/ba_table1.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/ba_table2.q PRE-CREATION trunk/ql/src/test/results/clientpositive/ba_table1.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/ba_table2.q.out PRE-CREATION trunk/serde/if/serde.thrift 1159452 trunk/serde/src/gen/thrift/gen-cpp/serde_constants.h 1159452 trunk/serde/src/gen/thrift/gen-cpp/serde_constants.cpp 1159452 trunk/serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/Constants.java 1159452 trunk/serde/src/gen/thrift/gen-php/serde/serde_constants.php 1159452 trunk/serde/src/gen/thrift/gen-py/org_apache_hadoop_hive_serde/constants.py 1159452 trunk/serde/src/gen/thrift/gen-rb/serde_constants.rb 1159452 trunk/serde/src/java/org/apache/hadoop/hive/serde2/SerDeUtils.java 1159452 trunk/serde/src/java/org/apache/hadoop/hive/serde2/binarysortable/BinarySortableSerDe.java 1159452 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 1159452 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 1159452 Diff: https://reviews.apache.org/r/1586/diff Testing ------- Added and updated unit tests. Added new system tests. Thanks, Ashutosh
          Hide
          John Sichi added a comment -

          Taking a look.

          Show
          John Sichi added a comment - Taking a look.
          Hide
          John Sichi added a comment -

          Ashutosh, maybe we can discuss this one at the contributor meetup next week (and then record the conclusions here).

          A few questions that I've heard so far:

          • Is there a design doc somewhere?
          • Since Hive already has an array type, but this feature is independent, we probably want a different type name than bytearray.
          • For conversions, is going through string for all types a good default behavior? An alternative would be to prevent implicit conversions altogether, and force users to pick the UDF with the desired behavior. E.g. for string/binary conversion, it's a good idea to be able to specify an encoding rather than always using the JVM default.
          • How does the new type work with TRANSFORM scripts, UDF's, saving to textfile, etc?
          • Don't we need more accessor functions (e.g. making the existing string functions such as LENGTH work)?
          Show
          John Sichi added a comment - Ashutosh, maybe we can discuss this one at the contributor meetup next week (and then record the conclusions here). A few questions that I've heard so far: Is there a design doc somewhere? Since Hive already has an array type, but this feature is independent, we probably want a different type name than bytearray. For conversions, is going through string for all types a good default behavior? An alternative would be to prevent implicit conversions altogether, and force users to pick the UDF with the desired behavior. E.g. for string/binary conversion, it's a good idea to be able to specify an encoding rather than always using the JVM default. How does the new type work with TRANSFORM scripts, UDF's, saving to textfile, etc? Don't we need more accessor functions (e.g. making the existing string functions such as LENGTH work)?
          Hide
          Ashutosh Chauhan added a comment -

          Is there a design doc somewhere?

          Not yet. I can put together some design notes quickly over here. Is there an example of design docs for the features done previously for Hive which I can look at to get an idea what shall I cover in it?

          Since Hive already has an array type, but this feature is independent, we probably want a different type name than bytearray.

          OK. Binary ?

          For conversions, is going through string for all types a good default behavior? An alternative would be to prevent implicit conversions altogether, and force users to pick the UDF with the desired behavior. E.g. for string/binary conversion, it's a good idea to be able to specify an encoding rather than always using the JVM default.

          I also thought about casting and wasn't inclined for adding implicit casting. But I went with it so as to make easy things easier. This way users can use this type easily in scripts without needing to insert casts every time and in cases this doesnt work they can always write udfs. Further, in many cases, JVM encoding is a good default. But, if you think thats not a good idea, I can take away implicit casting.

          How does the new type work with

          TRANSFORM scripts,

          I am assuming providing toString() will be good enough to make sure we can send data in string form and after receiving can convert into bytearray. Is there anything else ?

          UDF's,

          Like other types. Do I need to think about anything here?

          saving to textfile, etc?

          I assume you mean file containing text data in other columns, if user does so, it will be his responsibility to escape and format data appropriately so that he can load it later, potentially with the serde which does understand the format and escaping.

          Don't we need more accessor functions (e.g. making the existing string functions such as LENGTH work)?

          Length should be possible. Any other accessor functions?

          Show
          Ashutosh Chauhan added a comment - Is there a design doc somewhere? Not yet. I can put together some design notes quickly over here. Is there an example of design docs for the features done previously for Hive which I can look at to get an idea what shall I cover in it? Since Hive already has an array type, but this feature is independent, we probably want a different type name than bytearray. OK. Binary ? For conversions, is going through string for all types a good default behavior? An alternative would be to prevent implicit conversions altogether, and force users to pick the UDF with the desired behavior. E.g. for string/binary conversion, it's a good idea to be able to specify an encoding rather than always using the JVM default. I also thought about casting and wasn't inclined for adding implicit casting. But I went with it so as to make easy things easier. This way users can use this type easily in scripts without needing to insert casts every time and in cases this doesnt work they can always write udfs. Further, in many cases, JVM encoding is a good default. But, if you think thats not a good idea, I can take away implicit casting. How does the new type work with TRANSFORM scripts, I am assuming providing toString() will be good enough to make sure we can send data in string form and after receiving can convert into bytearray. Is there anything else ? UDF's, Like other types. Do I need to think about anything here? saving to textfile, etc? I assume you mean file containing text data in other columns, if user does so, it will be his responsibility to escape and format data appropriately so that he can load it later, potentially with the serde which does understand the format and escaping. Don't we need more accessor functions (e.g. making the existing string functions such as LENGTH work)? Length should be possible. Any other accessor functions?
          Hide
          John Sichi added a comment -

          You can find design doc examples here:

          https://cwiki.apache.org/confluence/display/Hive/DesignDocs

          Show
          John Sichi added a comment - You can find design doc examples here: https://cwiki.apache.org/confluence/display/Hive/DesignDocs
          Hide
          Ashutosh Chauhan added a comment -

          Initial design proposal is here:
          https://cwiki.apache.org/confluence/display/Hive/Binary+DataType+Proposal

          Comments welcome.

          Show
          Ashutosh Chauhan added a comment - Initial design proposal is here: https://cwiki.apache.org/confluence/display/Hive/Binary+DataType+Proposal Comments welcome.
          Hide
          John Sichi added a comment -

          For accessor functions:

          • length
          • substring
          • concat

          We can follow up later with search capabilities.

          For conversions:

          • to/from hex string
          • to/from string using a specific encoding (or default JVM encoding if not specified)
          • to/from base64 string

          We can follow up later with more interesting conversions for non-string types.

          Show
          John Sichi added a comment - For accessor functions: length substring concat We can follow up later with search capabilities. For conversions: to/from hex string to/from string using a specific encoding (or default JVM encoding if not specified) to/from base64 string We can follow up later with more interesting conversions for non-string types.
          Hide
          Ashutosh Chauhan added a comment -

          Is name binary acceptable?

          Show
          Ashutosh Chauhan added a comment - Is name binary acceptable?
          Hide
          Ashutosh Chauhan added a comment -

          @John,
          For conversions, in this jira I am planning to provide functionality for to/from string using default JVM encoding, rest of these conversions I will take up in a follow-up jira. Motivation is to avoid having one giant patch and have multiple smaller patches. Sounds reasonable?

          Show
          Ashutosh Chauhan added a comment - @John, For conversions, in this jira I am planning to provide functionality for to/from string using default JVM encoding, rest of these conversions I will take up in a follow-up jira. Motivation is to avoid having one giant patch and have multiple smaller patches. Sounds reasonable?
          Hide
          John Sichi added a comment -

          Just checked, and the name BINARY in JDBC and standard SQL corresponds to a fixed-length binary type with a specified precision:

          http://download.oracle.com/javase/1.3/docs/guide/jdbc/getstart/mapping.html

          But I'm fine with us using it to mean something slightly different, and matching thrift.

          Regarding the conversions, it's fine to break it out into multiple patches; open follow-up JIRA's linked to this one. I think the hex one will be very useful for debugging.

          Show
          John Sichi added a comment - Just checked, and the name BINARY in JDBC and standard SQL corresponds to a fixed-length binary type with a specified precision: http://download.oracle.com/javase/1.3/docs/guide/jdbc/getstart/mapping.html But I'm fine with us using it to mean something slightly different, and matching thrift. Regarding the conversions, it's fine to break it out into multiple patches; open follow-up JIRA's linked to this one. I think the hex one will be very useful for debugging.
          Hide
          Ashutosh Chauhan added a comment -

          Turned out binary is already reserved KW in hive grammar. But, looks like its not used anywhere? Can someone confirm. If so, I will use that KW, else we need to find a different name.

          Show
          Ashutosh Chauhan added a comment - Turned out binary is already reserved KW in hive grammar. But, looks like its not used anywhere? Can someone confirm. If so, I will use that KW, else we need to find a different name.
          Hide
          John Sichi added a comment -

          I don't see any references to it, so I think you're free to use it.

          Show
          John Sichi added a comment - I don't see any references to it, so I think you're free to use it.
          Hide
          Ashutosh Chauhan added a comment -

          Updated patch incorporating feedback.

          Show
          Ashutosh Chauhan added a comment - Updated patch incorporating feedback.
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/1586/
          -----------------------------------------------------------

          (Updated 2011-09-18 05:52:41.197110)

          Review request for hive.

          Changes
          -------

          Updated patch incorporating feedback

          Summary
          -------

          This patch adds bytearray as a new datatype in Hive.

          This addresses bug HIVE-2380.
          https://issues.apache.org/jira/browse/HIVE-2380

          Diffs (updated)


          trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 1172168
          trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 1172168
          trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 1172168
          trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g 1172168
          trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ParseDriver.java 1172168
          trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java 1172168
          trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFConcat.java 1172168
          trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFLength.java 1172168
          trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFSubstr.java 1172168
          trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToString.java 1172168
          trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFToBinary.java PRE-CREATION
          trunk/ql/src/test/queries/clientpositive/ba_table1.q PRE-CREATION
          trunk/ql/src/test/queries/clientpositive/ba_table2.q PRE-CREATION
          trunk/ql/src/test/queries/clientpositive/ba_table_udfs.q PRE-CREATION
          trunk/ql/src/test/results/clientpositive/ba_table1.q.out PRE-CREATION
          trunk/ql/src/test/results/clientpositive/ba_table2.q.out PRE-CREATION
          trunk/ql/src/test/results/clientpositive/ba_table_udfs.q.out PRE-CREATION
          trunk/serde/if/serde.thrift 1172168
          trunk/serde/src/gen/thrift/gen-cpp/serde_constants.h 1172168
          trunk/serde/src/gen/thrift/gen-cpp/serde_constants.cpp 1172168
          trunk/serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/Constants.java 1172168
          trunk/serde/src/gen/thrift/gen-php/serde/serde_constants.php 1172168
          trunk/serde/src/gen/thrift/gen-py/org_apache_hadoop_hive_serde/constants.py 1172168
          trunk/serde/src/gen/thrift/gen-rb/serde_constants.rb 1172168
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/SerDeUtils.java 1172168
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/binarysortable/BinarySortableSerDe.java 1172168
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/columnar/LazyBinaryColumnarSerDe.java 1172168
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/columnar/LazyBinaryColumnarStruct.java 1172168
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyBinary.java PRE-CREATION
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyFactory.java 1172168
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyUtils.java 1172168
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/primitive/LazyBinaryObjectInspector.java PRE-CREATION
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/primitive/LazyPrimitiveObjectInspectorFactory.java 1172168
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryBinary.java PRE-CREATION
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryFactory.java 1172168
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinarySerDe.java 1172168
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryUtils.java 1172168
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorConverters.java 1172168
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java 1172168
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/PrimitiveObjectInspector.java 1172168
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/BinaryObjectInspector.java PRE-CREATION
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/JavaBinaryObjectInspector.java PRE-CREATION
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorConverter.java 1172168
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorFactory.java 1172168
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorUtils.java 1172168
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/SettableBinaryObjectInspector.java PRE-CREATION
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableBinaryObjectInspector.java PRE-CREATION
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/typeinfo/TypeInfoFactory.java 1172168
          trunk/serde/src/test/org/apache/hadoop/hive/serde2/TestStatsSerde.java 1172168
          trunk/serde/src/test/org/apache/hadoop/hive/serde2/binarysortable/MyTestClass.java 1172168
          trunk/serde/src/test/org/apache/hadoop/hive/serde2/binarysortable/TestBinarySortableSerDe.java 1172168
          trunk/serde/src/test/org/apache/hadoop/hive/serde2/columnar/TestLazyBinaryColumnarSerDe.java 1172168
          trunk/serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazyPrimitive.java 1172168
          trunk/serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazySimpleSerDe.java 1172168
          trunk/serde/src/test/org/apache/hadoop/hive/serde2/lazybinary/MyTestClassBigger.java 1172168
          trunk/serde/src/test/org/apache/hadoop/hive/serde2/lazybinary/TestLazyBinarySerDe.java 1172168
          trunk/serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestObjectInspectorConverters.java 1172168
          trunk/serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestStandardObjectInspectors.java 1172168

          Diff: https://reviews.apache.org/r/1586/diff

          Testing
          -------

          Added and updated unit tests.
          Added new system tests.

          Thanks,

          Ashutosh

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1586/ ----------------------------------------------------------- (Updated 2011-09-18 05:52:41.197110) Review request for hive. Changes ------- Updated patch incorporating feedback Summary ------- This patch adds bytearray as a new datatype in Hive. This addresses bug HIVE-2380 . https://issues.apache.org/jira/browse/HIVE-2380 Diffs (updated) trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 1172168 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 1172168 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 1172168 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g 1172168 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ParseDriver.java 1172168 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java 1172168 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFConcat.java 1172168 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFLength.java 1172168 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFSubstr.java 1172168 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToString.java 1172168 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFToBinary.java PRE-CREATION trunk/ql/src/test/queries/clientpositive/ba_table1.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/ba_table2.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/ba_table_udfs.q PRE-CREATION trunk/ql/src/test/results/clientpositive/ba_table1.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/ba_table2.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/ba_table_udfs.q.out PRE-CREATION trunk/serde/if/serde.thrift 1172168 trunk/serde/src/gen/thrift/gen-cpp/serde_constants.h 1172168 trunk/serde/src/gen/thrift/gen-cpp/serde_constants.cpp 1172168 trunk/serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/Constants.java 1172168 trunk/serde/src/gen/thrift/gen-php/serde/serde_constants.php 1172168 trunk/serde/src/gen/thrift/gen-py/org_apache_hadoop_hive_serde/constants.py 1172168 trunk/serde/src/gen/thrift/gen-rb/serde_constants.rb 1172168 trunk/serde/src/java/org/apache/hadoop/hive/serde2/SerDeUtils.java 1172168 trunk/serde/src/java/org/apache/hadoop/hive/serde2/binarysortable/BinarySortableSerDe.java 1172168 trunk/serde/src/java/org/apache/hadoop/hive/serde2/columnar/LazyBinaryColumnarSerDe.java 1172168 trunk/serde/src/java/org/apache/hadoop/hive/serde2/columnar/LazyBinaryColumnarStruct.java 1172168 trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyBinary.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyFactory.java 1172168 trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyUtils.java 1172168 trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/primitive/LazyBinaryObjectInspector.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/primitive/LazyPrimitiveObjectInspectorFactory.java 1172168 trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryBinary.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryFactory.java 1172168 trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinarySerDe.java 1172168 trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryUtils.java 1172168 trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorConverters.java 1172168 trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java 1172168 trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/PrimitiveObjectInspector.java 1172168 trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/BinaryObjectInspector.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/JavaBinaryObjectInspector.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorConverter.java 1172168 trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorFactory.java 1172168 trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorUtils.java 1172168 trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/SettableBinaryObjectInspector.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableBinaryObjectInspector.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/typeinfo/TypeInfoFactory.java 1172168 trunk/serde/src/test/org/apache/hadoop/hive/serde2/TestStatsSerde.java 1172168 trunk/serde/src/test/org/apache/hadoop/hive/serde2/binarysortable/MyTestClass.java 1172168 trunk/serde/src/test/org/apache/hadoop/hive/serde2/binarysortable/TestBinarySortableSerDe.java 1172168 trunk/serde/src/test/org/apache/hadoop/hive/serde2/columnar/TestLazyBinaryColumnarSerDe.java 1172168 trunk/serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazyPrimitive.java 1172168 trunk/serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazySimpleSerDe.java 1172168 trunk/serde/src/test/org/apache/hadoop/hive/serde2/lazybinary/MyTestClassBigger.java 1172168 trunk/serde/src/test/org/apache/hadoop/hive/serde2/lazybinary/TestLazyBinarySerDe.java 1172168 trunk/serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestObjectInspectorConverters.java 1172168 trunk/serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestStandardObjectInspectors.java 1172168 Diff: https://reviews.apache.org/r/1586/diff Testing ------- Added and updated unit tests. Added new system tests. Thanks, Ashutosh
          Hide
          John Sichi added a comment -

          I'm planning to review this one next week.

          Show
          John Sichi added a comment - I'm planning to review this one next week.
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/1586/#review2076
          -----------------------------------------------------------

          trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java
          <https://reviews.apache.org/r/1586/#comment4695>

          I thought we decided an explicit cast would be required even for string<-->binary?

          trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFConcat.java
          <https://reviews.apache.org/r/1586/#comment4699>

          @Description annotation for this class needs to be updated.

          trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFLength.java
          <https://reviews.apache.org/r/1586/#comment4698>

          @Description annotation for this class needs to be updated.

          trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFSubstr.java
          <https://reviews.apache.org/r/1586/#comment4700>

          @Description annotation for this class needs to be updated.

          trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFToBinary.java
          <https://reviews.apache.org/r/1586/#comment4697>

          This UDF class needs an @Description annotation.

          trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFToBinary.java
          <https://reviews.apache.org/r/1586/#comment4696>

          The CAST should have parentheses, so

          CAST(x AS BINARY)

          (this applies to the existing code in GenericUDFTimestamp too)

          trunk/ql/src/test/queries/clientpositive/ba_table1.q
          <https://reviews.apache.org/r/1586/#comment4701>

          Use ORDER BY on a key for all queries to guarantee test determinism. Also, since test output keeps getting bigger and bigger, it's best to filter down to just a few rows to keep it easier to review and manage.

          (This comment applies to all tests in this patch.)

          trunk/serde/src/java/org/apache/hadoop/hive/serde2/SerDeUtils.java
          <https://reviews.apache.org/r/1586/#comment4708>

          Is there a test for this somewhere? It seems like a case where we need base64.

          trunk/serde/src/java/org/apache/hadoop/hive/serde2/columnar/LazyBinaryColumnarSerDe.java
          <https://reviews.apache.org/r/1586/#comment4715>

          I'm not sure about the original connection between UTF and strings here (was something forcing a UTF encoding), but is it valid for arbitrary binary data?

          trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorUtils.java
          <https://reviews.apache.org/r/1586/#comment4716>

          Since we decided we're not going to support implicit conversions, these don't belong here, right?

          • John

          On 2011-09-18 05:52:41, Ashutosh Chauhan wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/1586/

          -----------------------------------------------------------

          (Updated 2011-09-18 05:52:41)

          Review request for hive.

          Summary

          -------

          This patch adds bytearray as a new datatype in Hive.

          This addresses bug HIVE-2380.

          https://issues.apache.org/jira/browse/HIVE-2380

          Diffs

          -----

          trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 1172168

          trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 1172168

          trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 1172168

          trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g 1172168

          trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ParseDriver.java 1172168

          trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java 1172168

          trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFConcat.java 1172168

          trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFLength.java 1172168

          trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFSubstr.java 1172168

          trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToString.java 1172168

          trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFToBinary.java PRE-CREATION

          trunk/ql/src/test/queries/clientpositive/ba_table1.q PRE-CREATION

          trunk/ql/src/test/queries/clientpositive/ba_table2.q PRE-CREATION

          trunk/ql/src/test/queries/clientpositive/ba_table_udfs.q PRE-CREATION

          trunk/ql/src/test/results/clientpositive/ba_table1.q.out PRE-CREATION

          trunk/ql/src/test/results/clientpositive/ba_table2.q.out PRE-CREATION

          trunk/ql/src/test/results/clientpositive/ba_table_udfs.q.out PRE-CREATION

          trunk/serde/if/serde.thrift 1172168

          trunk/serde/src/gen/thrift/gen-cpp/serde_constants.h 1172168

          trunk/serde/src/gen/thrift/gen-cpp/serde_constants.cpp 1172168

          trunk/serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/Constants.java 1172168

          trunk/serde/src/gen/thrift/gen-php/serde/serde_constants.php 1172168

          trunk/serde/src/gen/thrift/gen-py/org_apache_hadoop_hive_serde/constants.py 1172168

          trunk/serde/src/gen/thrift/gen-rb/serde_constants.rb 1172168

          trunk/serde/src/java/org/apache/hadoop/hive/serde2/SerDeUtils.java 1172168

          trunk/serde/src/java/org/apache/hadoop/hive/serde2/binarysortable/BinarySortableSerDe.java 1172168

          trunk/serde/src/java/org/apache/hadoop/hive/serde2/columnar/LazyBinaryColumnarSerDe.java 1172168

          trunk/serde/src/java/org/apache/hadoop/hive/serde2/columnar/LazyBinaryColumnarStruct.java 1172168

          trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyBinary.java PRE-CREATION

          trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyFactory.java 1172168

          trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyUtils.java 1172168

          trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/primitive/LazyBinaryObjectInspector.java PRE-CREATION

          trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/primitive/LazyPrimitiveObjectInspectorFactory.java 1172168

          trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryBinary.java PRE-CREATION

          trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryFactory.java 1172168

          trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinarySerDe.java 1172168

          trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryUtils.java 1172168

          trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorConverters.java 1172168

          trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java 1172168

          trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/PrimitiveObjectInspector.java 1172168

          trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/BinaryObjectInspector.java PRE-CREATION

          trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/JavaBinaryObjectInspector.java PRE-CREATION

          trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorConverter.java 1172168

          trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorFactory.java 1172168

          trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorUtils.java 1172168

          trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/SettableBinaryObjectInspector.java PRE-CREATION

          trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableBinaryObjectInspector.java PRE-CREATION

          trunk/serde/src/java/org/apache/hadoop/hive/serde2/typeinfo/TypeInfoFactory.java 1172168

          trunk/serde/src/test/org/apache/hadoop/hive/serde2/TestStatsSerde.java 1172168

          trunk/serde/src/test/org/apache/hadoop/hive/serde2/binarysortable/MyTestClass.java 1172168

          trunk/serde/src/test/org/apache/hadoop/hive/serde2/binarysortable/TestBinarySortableSerDe.java 1172168

          trunk/serde/src/test/org/apache/hadoop/hive/serde2/columnar/TestLazyBinaryColumnarSerDe.java 1172168

          trunk/serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazyPrimitive.java 1172168

          trunk/serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazySimpleSerDe.java 1172168

          trunk/serde/src/test/org/apache/hadoop/hive/serde2/lazybinary/MyTestClassBigger.java 1172168

          trunk/serde/src/test/org/apache/hadoop/hive/serde2/lazybinary/TestLazyBinarySerDe.java 1172168

          trunk/serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestObjectInspectorConverters.java 1172168

          trunk/serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestStandardObjectInspectors.java 1172168

          Diff: https://reviews.apache.org/r/1586/diff

          Testing

          -------

          Added and updated unit tests.

          Added new system tests.

          Thanks,

          Ashutosh

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1586/#review2076 ----------------------------------------------------------- trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java < https://reviews.apache.org/r/1586/#comment4695 > I thought we decided an explicit cast would be required even for string<-->binary? trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFConcat.java < https://reviews.apache.org/r/1586/#comment4699 > @Description annotation for this class needs to be updated. trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFLength.java < https://reviews.apache.org/r/1586/#comment4698 > @Description annotation for this class needs to be updated. trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFSubstr.java < https://reviews.apache.org/r/1586/#comment4700 > @Description annotation for this class needs to be updated. trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFToBinary.java < https://reviews.apache.org/r/1586/#comment4697 > This UDF class needs an @Description annotation. trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFToBinary.java < https://reviews.apache.org/r/1586/#comment4696 > The CAST should have parentheses, so CAST(x AS BINARY) (this applies to the existing code in GenericUDFTimestamp too) trunk/ql/src/test/queries/clientpositive/ba_table1.q < https://reviews.apache.org/r/1586/#comment4701 > Use ORDER BY on a key for all queries to guarantee test determinism. Also, since test output keeps getting bigger and bigger, it's best to filter down to just a few rows to keep it easier to review and manage. (This comment applies to all tests in this patch.) trunk/serde/src/java/org/apache/hadoop/hive/serde2/SerDeUtils.java < https://reviews.apache.org/r/1586/#comment4708 > Is there a test for this somewhere? It seems like a case where we need base64. trunk/serde/src/java/org/apache/hadoop/hive/serde2/columnar/LazyBinaryColumnarSerDe.java < https://reviews.apache.org/r/1586/#comment4715 > I'm not sure about the original connection between UTF and strings here (was something forcing a UTF encoding), but is it valid for arbitrary binary data? trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorUtils.java < https://reviews.apache.org/r/1586/#comment4716 > Since we decided we're not going to support implicit conversions, these don't belong here, right? John On 2011-09-18 05:52:41, Ashutosh Chauhan wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1586/ ----------------------------------------------------------- (Updated 2011-09-18 05:52:41) Review request for hive. Summary ------- This patch adds bytearray as a new datatype in Hive. This addresses bug HIVE-2380 . https://issues.apache.org/jira/browse/HIVE-2380 Diffs ----- trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 1172168 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 1172168 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 1172168 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g 1172168 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ParseDriver.java 1172168 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java 1172168 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFConcat.java 1172168 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFLength.java 1172168 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFSubstr.java 1172168 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToString.java 1172168 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFToBinary.java PRE-CREATION trunk/ql/src/test/queries/clientpositive/ba_table1.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/ba_table2.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/ba_table_udfs.q PRE-CREATION trunk/ql/src/test/results/clientpositive/ba_table1.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/ba_table2.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/ba_table_udfs.q.out PRE-CREATION trunk/serde/if/serde.thrift 1172168 trunk/serde/src/gen/thrift/gen-cpp/serde_constants.h 1172168 trunk/serde/src/gen/thrift/gen-cpp/serde_constants.cpp 1172168 trunk/serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/Constants.java 1172168 trunk/serde/src/gen/thrift/gen-php/serde/serde_constants.php 1172168 trunk/serde/src/gen/thrift/gen-py/org_apache_hadoop_hive_serde/constants.py 1172168 trunk/serde/src/gen/thrift/gen-rb/serde_constants.rb 1172168 trunk/serde/src/java/org/apache/hadoop/hive/serde2/SerDeUtils.java 1172168 trunk/serde/src/java/org/apache/hadoop/hive/serde2/binarysortable/BinarySortableSerDe.java 1172168 trunk/serde/src/java/org/apache/hadoop/hive/serde2/columnar/LazyBinaryColumnarSerDe.java 1172168 trunk/serde/src/java/org/apache/hadoop/hive/serde2/columnar/LazyBinaryColumnarStruct.java 1172168 trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyBinary.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyFactory.java 1172168 trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyUtils.java 1172168 trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/primitive/LazyBinaryObjectInspector.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/primitive/LazyPrimitiveObjectInspectorFactory.java 1172168 trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryBinary.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryFactory.java 1172168 trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinarySerDe.java 1172168 trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryUtils.java 1172168 trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorConverters.java 1172168 trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java 1172168 trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/PrimitiveObjectInspector.java 1172168 trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/BinaryObjectInspector.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/JavaBinaryObjectInspector.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorConverter.java 1172168 trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorFactory.java 1172168 trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorUtils.java 1172168 trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/SettableBinaryObjectInspector.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableBinaryObjectInspector.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/typeinfo/TypeInfoFactory.java 1172168 trunk/serde/src/test/org/apache/hadoop/hive/serde2/TestStatsSerde.java 1172168 trunk/serde/src/test/org/apache/hadoop/hive/serde2/binarysortable/MyTestClass.java 1172168 trunk/serde/src/test/org/apache/hadoop/hive/serde2/binarysortable/TestBinarySortableSerDe.java 1172168 trunk/serde/src/test/org/apache/hadoop/hive/serde2/columnar/TestLazyBinaryColumnarSerDe.java 1172168 trunk/serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazyPrimitive.java 1172168 trunk/serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazySimpleSerDe.java 1172168 trunk/serde/src/test/org/apache/hadoop/hive/serde2/lazybinary/MyTestClassBigger.java 1172168 trunk/serde/src/test/org/apache/hadoop/hive/serde2/lazybinary/TestLazyBinarySerDe.java 1172168 trunk/serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestObjectInspectorConverters.java 1172168 trunk/serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestStandardObjectInspectors.java 1172168 Diff: https://reviews.apache.org/r/1586/diff Testing ------- Added and updated unit tests. Added new system tests. Thanks, Ashutosh
          Hide
          John Sichi added a comment -

          Some comments added on Review Board. Also:

          • I would be a lot more comfortable with this patch if it included testing of real binary data (not just strings encoded as binary) at the QL level end-to-end, getting stored in a table, queried, passed through UDF's, etc. (not just the Java unit test level of serdes) since that is the intended use case
          • could you open followup JIRA issues for any items which you intend to address as immediate followups to this patch?
          • need negative tests for attempting illegal conversions to/from binary
          • please rename this issue since we're going with BINARY instead of BYTEARRAY
          Show
          John Sichi added a comment - Some comments added on Review Board. Also: I would be a lot more comfortable with this patch if it included testing of real binary data (not just strings encoded as binary) at the QL level end-to-end, getting stored in a table, queried, passed through UDF's, etc. (not just the Java unit test level of serdes) since that is the intended use case could you open followup JIRA issues for any items which you intend to address as immediate followups to this patch? need negative tests for attempting illegal conversions to/from binary please rename this issue since we're going with BINARY instead of BYTEARRAY
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/1586/
          -----------------------------------------------------------

          (Updated 2011-09-30 17:37:25.293739)

          Review request for hive.

          Changes
          -------

          Incorporating John's feedback + lot of tests.

          Summary
          -------

          This patch adds bytearray as a new datatype in Hive.

          This addresses bug HIVE-2380.
          https://issues.apache.org/jira/browse/HIVE-2380

          Diffs (updated)


          trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 1177528
          trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 1177528
          trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 1177528
          trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g 1177528
          trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ParseDriver.java 1177528
          trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java 1177528
          trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFConcat.java 1177528
          trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFLength.java 1177528
          trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFSubstr.java 1177528
          trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToString.java 1177528
          trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFTimestamp.java 1177528
          trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFToBinary.java PRE-CREATION
          trunk/ql/src/test/queries/clientnegative/invalid_cast_from_binary_1.q PRE-CREATION
          trunk/ql/src/test/queries/clientnegative/invalid_cast_from_binary_2.q PRE-CREATION
          trunk/ql/src/test/queries/clientnegative/invalid_cast_from_binary_3.q PRE-CREATION
          trunk/ql/src/test/queries/clientnegative/invalid_cast_from_binary_4.q PRE-CREATION
          trunk/ql/src/test/queries/clientnegative/invalid_cast_from_binary_5.q PRE-CREATION
          trunk/ql/src/test/queries/clientnegative/invalid_cast_from_binary_6.q PRE-CREATION
          trunk/ql/src/test/queries/clientnegative/invalid_cast_to_binary_1.q PRE-CREATION
          trunk/ql/src/test/queries/clientnegative/invalid_cast_to_binary_2.q PRE-CREATION
          trunk/ql/src/test/queries/clientnegative/invalid_cast_to_binary_3.q PRE-CREATION
          trunk/ql/src/test/queries/clientnegative/invalid_cast_to_binary_4.q PRE-CREATION
          trunk/ql/src/test/queries/clientnegative/invalid_cast_to_binary_5.q PRE-CREATION
          trunk/ql/src/test/queries/clientnegative/invalid_cast_to_binary_6.q PRE-CREATION
          trunk/ql/src/test/queries/clientpositive/ba_table1.q PRE-CREATION
          trunk/ql/src/test/queries/clientpositive/ba_table2.q PRE-CREATION
          trunk/ql/src/test/queries/clientpositive/ba_table3.q PRE-CREATION
          trunk/ql/src/test/queries/clientpositive/ba_table_udfs.q PRE-CREATION
          trunk/ql/src/test/queries/clientpositive/ba_table_union.q PRE-CREATION
          trunk/ql/src/test/queries/clientpositive/binary_table_bincolserde.q PRE-CREATION
          trunk/ql/src/test/queries/clientpositive/binary_table_colserde.q PRE-CREATION
          trunk/ql/src/test/queries/clientpositive/load_binary_data.q PRE-CREATION
          trunk/ql/src/test/results/clientnegative/invalid_cast_from_binary_1.q.out PRE-CREATION
          trunk/ql/src/test/results/clientnegative/invalid_cast_from_binary_2.q.out PRE-CREATION
          trunk/ql/src/test/results/clientnegative/invalid_cast_from_binary_3.q.out PRE-CREATION
          trunk/ql/src/test/results/clientnegative/invalid_cast_from_binary_4.q.out PRE-CREATION
          trunk/ql/src/test/results/clientnegative/invalid_cast_from_binary_5.q.out PRE-CREATION
          trunk/ql/src/test/results/clientnegative/invalid_cast_from_binary_6.q.out PRE-CREATION
          trunk/ql/src/test/results/clientnegative/invalid_cast_to_binary_1.q.out PRE-CREATION
          trunk/ql/src/test/results/clientnegative/invalid_cast_to_binary_2.q.out PRE-CREATION
          trunk/ql/src/test/results/clientnegative/invalid_cast_to_binary_3.q.out PRE-CREATION
          trunk/ql/src/test/results/clientnegative/invalid_cast_to_binary_4.q.out PRE-CREATION
          trunk/ql/src/test/results/clientnegative/invalid_cast_to_binary_5.q.out PRE-CREATION
          trunk/ql/src/test/results/clientnegative/invalid_cast_to_binary_6.q.out PRE-CREATION
          trunk/ql/src/test/results/clientpositive/ba_table1.q.out PRE-CREATION
          trunk/ql/src/test/results/clientpositive/ba_table2.q.out PRE-CREATION
          trunk/ql/src/test/results/clientpositive/ba_table3.q.out PRE-CREATION
          trunk/ql/src/test/results/clientpositive/ba_table_udfs.q.out PRE-CREATION
          trunk/ql/src/test/results/clientpositive/ba_table_union.q.out PRE-CREATION
          trunk/ql/src/test/results/clientpositive/binary_table_bincolserde.q.out PRE-CREATION
          trunk/ql/src/test/results/clientpositive/binary_table_colserde.q.out PRE-CREATION
          trunk/ql/src/test/results/clientpositive/load_binary_data.q.out PRE-CREATION
          trunk/serde/if/serde.thrift 1177528
          trunk/serde/src/gen/thrift/gen-cpp/serde_constants.h 1177528
          trunk/serde/src/gen/thrift/gen-cpp/serde_constants.cpp 1177528
          trunk/serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/Constants.java 1177528
          trunk/serde/src/gen/thrift/gen-php/serde/serde_constants.php 1177528
          trunk/serde/src/gen/thrift/gen-py/org_apache_hadoop_hive_serde/constants.py 1177528
          trunk/serde/src/gen/thrift/gen-rb/serde_constants.rb 1177528
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/DelimitedJSONSerDe.java 1177528
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/SerDeUtils.java 1177528
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/binarysortable/BinarySortableSerDe.java 1177528
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/columnar/LazyBinaryColumnarSerDe.java 1177528
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/columnar/LazyBinaryColumnarStruct.java 1177528
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyBinary.java PRE-CREATION
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyFactory.java 1177528
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyUtils.java 1177528
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/primitive/LazyBinaryObjectInspector.java PRE-CREATION
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/primitive/LazyPrimitiveObjectInspectorFactory.java 1177528
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryBinary.java PRE-CREATION
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryFactory.java 1177528
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinarySerDe.java 1177528
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryUtils.java 1177528
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorConverters.java 1177528
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java 1177528
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/PrimitiveObjectInspector.java 1177528
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/BinaryObjectInspector.java PRE-CREATION
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/JavaBinaryObjectInspector.java PRE-CREATION
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorConverter.java 1177528
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorFactory.java 1177528
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorUtils.java 1177528
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/SettableBinaryObjectInspector.java PRE-CREATION
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableBinaryObjectInspector.java PRE-CREATION
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/typeinfo/TypeInfoFactory.java 1177528
          trunk/serde/src/test/org/apache/hadoop/hive/serde2/TestStatsSerde.java 1177528
          trunk/serde/src/test/org/apache/hadoop/hive/serde2/binarysortable/MyTestClass.java 1177528
          trunk/serde/src/test/org/apache/hadoop/hive/serde2/binarysortable/TestBinarySortableSerDe.java 1177528
          trunk/serde/src/test/org/apache/hadoop/hive/serde2/columnar/TestLazyBinaryColumnarSerDe.java 1177528
          trunk/serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazyPrimitive.java 1177528
          trunk/serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazySimpleSerDe.java 1177528
          trunk/serde/src/test/org/apache/hadoop/hive/serde2/lazybinary/MyTestClassBigger.java 1177528
          trunk/serde/src/test/org/apache/hadoop/hive/serde2/lazybinary/TestLazyBinarySerDe.java 1177528
          trunk/serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestObjectInspectorConverters.java 1177528
          trunk/serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestStandardObjectInspectors.java 1177528

          Diff: https://reviews.apache.org/r/1586/diff

          Testing
          -------

          Added and updated unit tests.
          Added new system tests.

          Thanks,

          Ashutosh

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1586/ ----------------------------------------------------------- (Updated 2011-09-30 17:37:25.293739) Review request for hive. Changes ------- Incorporating John's feedback + lot of tests. Summary ------- This patch adds bytearray as a new datatype in Hive. This addresses bug HIVE-2380 . https://issues.apache.org/jira/browse/HIVE-2380 Diffs (updated) trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 1177528 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 1177528 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 1177528 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g 1177528 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ParseDriver.java 1177528 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java 1177528 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFConcat.java 1177528 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFLength.java 1177528 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFSubstr.java 1177528 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToString.java 1177528 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFTimestamp.java 1177528 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFToBinary.java PRE-CREATION trunk/ql/src/test/queries/clientnegative/invalid_cast_from_binary_1.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/invalid_cast_from_binary_2.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/invalid_cast_from_binary_3.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/invalid_cast_from_binary_4.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/invalid_cast_from_binary_5.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/invalid_cast_from_binary_6.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/invalid_cast_to_binary_1.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/invalid_cast_to_binary_2.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/invalid_cast_to_binary_3.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/invalid_cast_to_binary_4.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/invalid_cast_to_binary_5.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/invalid_cast_to_binary_6.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/ba_table1.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/ba_table2.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/ba_table3.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/ba_table_udfs.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/ba_table_union.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/binary_table_bincolserde.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/binary_table_colserde.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/load_binary_data.q PRE-CREATION trunk/ql/src/test/results/clientnegative/invalid_cast_from_binary_1.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/invalid_cast_from_binary_2.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/invalid_cast_from_binary_3.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/invalid_cast_from_binary_4.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/invalid_cast_from_binary_5.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/invalid_cast_from_binary_6.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/invalid_cast_to_binary_1.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/invalid_cast_to_binary_2.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/invalid_cast_to_binary_3.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/invalid_cast_to_binary_4.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/invalid_cast_to_binary_5.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/invalid_cast_to_binary_6.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/ba_table1.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/ba_table2.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/ba_table3.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/ba_table_udfs.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/ba_table_union.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/binary_table_bincolserde.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/binary_table_colserde.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/load_binary_data.q.out PRE-CREATION trunk/serde/if/serde.thrift 1177528 trunk/serde/src/gen/thrift/gen-cpp/serde_constants.h 1177528 trunk/serde/src/gen/thrift/gen-cpp/serde_constants.cpp 1177528 trunk/serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/Constants.java 1177528 trunk/serde/src/gen/thrift/gen-php/serde/serde_constants.php 1177528 trunk/serde/src/gen/thrift/gen-py/org_apache_hadoop_hive_serde/constants.py 1177528 trunk/serde/src/gen/thrift/gen-rb/serde_constants.rb 1177528 trunk/serde/src/java/org/apache/hadoop/hive/serde2/DelimitedJSONSerDe.java 1177528 trunk/serde/src/java/org/apache/hadoop/hive/serde2/SerDeUtils.java 1177528 trunk/serde/src/java/org/apache/hadoop/hive/serde2/binarysortable/BinarySortableSerDe.java 1177528 trunk/serde/src/java/org/apache/hadoop/hive/serde2/columnar/LazyBinaryColumnarSerDe.java 1177528 trunk/serde/src/java/org/apache/hadoop/hive/serde2/columnar/LazyBinaryColumnarStruct.java 1177528 trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyBinary.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyFactory.java 1177528 trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyUtils.java 1177528 trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/primitive/LazyBinaryObjectInspector.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/primitive/LazyPrimitiveObjectInspectorFactory.java 1177528 trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryBinary.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryFactory.java 1177528 trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinarySerDe.java 1177528 trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryUtils.java 1177528 trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorConverters.java 1177528 trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java 1177528 trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/PrimitiveObjectInspector.java 1177528 trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/BinaryObjectInspector.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/JavaBinaryObjectInspector.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorConverter.java 1177528 trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorFactory.java 1177528 trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorUtils.java 1177528 trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/SettableBinaryObjectInspector.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableBinaryObjectInspector.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/typeinfo/TypeInfoFactory.java 1177528 trunk/serde/src/test/org/apache/hadoop/hive/serde2/TestStatsSerde.java 1177528 trunk/serde/src/test/org/apache/hadoop/hive/serde2/binarysortable/MyTestClass.java 1177528 trunk/serde/src/test/org/apache/hadoop/hive/serde2/binarysortable/TestBinarySortableSerDe.java 1177528 trunk/serde/src/test/org/apache/hadoop/hive/serde2/columnar/TestLazyBinaryColumnarSerDe.java 1177528 trunk/serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazyPrimitive.java 1177528 trunk/serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazySimpleSerDe.java 1177528 trunk/serde/src/test/org/apache/hadoop/hive/serde2/lazybinary/MyTestClassBigger.java 1177528 trunk/serde/src/test/org/apache/hadoop/hive/serde2/lazybinary/TestLazyBinarySerDe.java 1177528 trunk/serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestObjectInspectorConverters.java 1177528 trunk/serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestStandardObjectInspectors.java 1177528 Diff: https://reviews.apache.org/r/1586/diff Testing ------- Added and updated unit tests. Added new system tests. Thanks, Ashutosh
          Hide
          jiraposter@reviews.apache.org added a comment -

          On 2011-09-27 00:43:03, John Sichi wrote:

          > trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java, line 675

          > <https://reviews.apache.org/r/1586/diff/2/?file=41673#file41673line675>

          >

          > I thought we decided an explicit cast would be required even for string<-->binary?

          Correct. Accidentally left. Will remove it in the new patch.

          On 2011-09-27 00:43:03, John Sichi wrote:

          > trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFConcat.java, line 36

          > <https://reviews.apache.org/r/1586/diff/2/?file=41678#file41678line36>

          >

          > @Description annotation for this class needs to be updated.

          Will update in new patch.

          On 2011-09-27 00:43:03, John Sichi wrote:

          > trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFLength.java, line 55

          > <https://reviews.apache.org/r/1586/diff/2/?file=41679#file41679line55>

          >

          > @Description annotation for this class needs to be updated.

          Will update in new patch.

          On 2011-09-27 00:43:03, John Sichi wrote:

          > trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFToBinary.java, line 29

          > <https://reviews.apache.org/r/1586/diff/2/?file=41682#file41682line29>

          >

          > This UDF class needs an @Description annotation.

          Will update in new patch.

          On 2011-09-27 00:43:03, John Sichi wrote:

          > trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFSubstr.java, line 43

          > <https://reviews.apache.org/r/1586/diff/2/?file=41680#file41680line43>

          >

          > @Description annotation for this class needs to be updated.

          Will update in new patch.

          On 2011-09-27 00:43:03, John Sichi wrote:

          > trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFToBinary.java, line 67

          > <https://reviews.apache.org/r/1586/diff/2/?file=41682#file41682line67>

          >

          > The CAST should have parentheses, so

          >

          > CAST(x AS BINARY)

          >

          > (this applies to the existing code in GenericUDFTimestamp too)

          >

          Will update in new patch.

          On 2011-09-27 00:43:03, John Sichi wrote:

          > trunk/ql/src/test/queries/clientpositive/ba_table1.q, line 9

          > <https://reviews.apache.org/r/1586/diff/2/?file=41683#file41683line9>

          >

          > Use ORDER BY on a key for all queries to guarantee test determinism. Also, since test output keeps getting bigger and bigger, it's best to filter down to just a few rows to keep it easier to review and manage.

          >

          > (This comment applies to all tests in this patch.)

          Added order-by to guarantee determinism and limit to keep number of rows small in all the tests.

          On 2011-09-27 00:43:03, John Sichi wrote:

          > trunk/serde/src/java/org/apache/hadoop/hive/serde2/SerDeUtils.java, line 267

          > <https://reviews.apache.org/r/1586/diff/2/?file=41696#file41696line267>

          >

          > Is there a test for this somewhere? It seems like a case where we need base64.

          Missed the changes in DelimitedJSONserde so it was not getting used. DelimitedJSONserde now uses this method to print, so its gets used by all the tests.

          On 2011-09-27 00:43:03, John Sichi wrote:

          > trunk/serde/src/java/org/apache/hadoop/hive/serde2/columnar/LazyBinaryColumnarSerDe.java, line 83

          > <https://reviews.apache.org/r/1586/diff/2/?file=41698#file41698line83>

          >

          > I'm not sure about the original connection between UTF and strings here (was something forcing a UTF encoding), but is it valid for arbitrary binary data?

          You are correct, this may break when there is a binary data of exactly one byte long and with that bit pattern, it will return null instead of that bit pattern. But thats a pretty corner-case. We can document something along the line "If there is a possibility that your binary data may contain exactly one byte of 10111111, don't use LazyBinaryColumnarSerde with binary data type."

          On 2011-09-27 00:43:03, John Sichi wrote:

          > trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorUtils.java, line 393

          > <https://reviews.apache.org/r/1586/diff/2/?file=41716#file41716line393>

          >

          > Since we decided we're not going to support implicit conversions, these don't belong here, right?

          Correct. Removed.

          • Ashutosh

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/1586/#review2076
          -----------------------------------------------------------

          On 2011-09-18 05:52:41, Ashutosh Chauhan wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/1586/

          -----------------------------------------------------------

          (Updated 2011-09-18 05:52:41)

          Review request for hive.

          Summary

          -------

          This patch adds bytearray as a new datatype in Hive.

          This addresses bug HIVE-2380.

          https://issues.apache.org/jira/browse/HIVE-2380

          Diffs

          -----

          trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 1172168

          trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 1172168

          trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 1172168

          trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g 1172168

          trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ParseDriver.java 1172168

          trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java 1172168

          trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFConcat.java 1172168

          trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFLength.java 1172168

          trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFSubstr.java 1172168

          trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToString.java 1172168

          trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFToBinary.java PRE-CREATION

          trunk/ql/src/test/queries/clientpositive/ba_table1.q PRE-CREATION

          trunk/ql/src/test/queries/clientpositive/ba_table2.q PRE-CREATION

          trunk/ql/src/test/queries/clientpositive/ba_table_udfs.q PRE-CREATION

          trunk/ql/src/test/results/clientpositive/ba_table1.q.out PRE-CREATION

          trunk/ql/src/test/results/clientpositive/ba_table2.q.out PRE-CREATION

          trunk/ql/src/test/results/clientpositive/ba_table_udfs.q.out PRE-CREATION

          trunk/serde/if/serde.thrift 1172168

          trunk/serde/src/gen/thrift/gen-cpp/serde_constants.h 1172168

          trunk/serde/src/gen/thrift/gen-cpp/serde_constants.cpp 1172168

          trunk/serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/Constants.java 1172168

          trunk/serde/src/gen/thrift/gen-php/serde/serde_constants.php 1172168

          trunk/serde/src/gen/thrift/gen-py/org_apache_hadoop_hive_serde/constants.py 1172168

          trunk/serde/src/gen/thrift/gen-rb/serde_constants.rb 1172168

          trunk/serde/src/java/org/apache/hadoop/hive/serde2/SerDeUtils.java 1172168

          trunk/serde/src/java/org/apache/hadoop/hive/serde2/binarysortable/BinarySortableSerDe.java 1172168

          trunk/serde/src/java/org/apache/hadoop/hive/serde2/columnar/LazyBinaryColumnarSerDe.java 1172168

          trunk/serde/src/java/org/apache/hadoop/hive/serde2/columnar/LazyBinaryColumnarStruct.java 1172168

          trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyBinary.java PRE-CREATION

          trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyFactory.java 1172168

          trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyUtils.java 1172168

          trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/primitive/LazyBinaryObjectInspector.java PRE-CREATION

          trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/primitive/LazyPrimitiveObjectInspectorFactory.java 1172168

          trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryBinary.java PRE-CREATION

          trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryFactory.java 1172168

          trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinarySerDe.java 1172168

          trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryUtils.java 1172168

          trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorConverters.java 1172168

          trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java 1172168

          trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/PrimitiveObjectInspector.java 1172168

          trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/BinaryObjectInspector.java PRE-CREATION

          trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/JavaBinaryObjectInspector.java PRE-CREATION

          trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorConverter.java 1172168

          trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorFactory.java 1172168

          trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorUtils.java 1172168

          trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/SettableBinaryObjectInspector.java PRE-CREATION

          trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableBinaryObjectInspector.java PRE-CREATION

          trunk/serde/src/java/org/apache/hadoop/hive/serde2/typeinfo/TypeInfoFactory.java 1172168

          trunk/serde/src/test/org/apache/hadoop/hive/serde2/TestStatsSerde.java 1172168

          trunk/serde/src/test/org/apache/hadoop/hive/serde2/binarysortable/MyTestClass.java 1172168

          trunk/serde/src/test/org/apache/hadoop/hive/serde2/binarysortable/TestBinarySortableSerDe.java 1172168

          trunk/serde/src/test/org/apache/hadoop/hive/serde2/columnar/TestLazyBinaryColumnarSerDe.java 1172168

          trunk/serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazyPrimitive.java 1172168

          trunk/serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazySimpleSerDe.java 1172168

          trunk/serde/src/test/org/apache/hadoop/hive/serde2/lazybinary/MyTestClassBigger.java 1172168

          trunk/serde/src/test/org/apache/hadoop/hive/serde2/lazybinary/TestLazyBinarySerDe.java 1172168

          trunk/serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestObjectInspectorConverters.java 1172168

          trunk/serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestStandardObjectInspectors.java 1172168

          Diff: https://reviews.apache.org/r/1586/diff

          Testing

          -------

          Added and updated unit tests.

          Added new system tests.

          Thanks,

          Ashutosh

          Show
          jiraposter@reviews.apache.org added a comment - On 2011-09-27 00:43:03, John Sichi wrote: > trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java, line 675 > < https://reviews.apache.org/r/1586/diff/2/?file=41673#file41673line675 > > > I thought we decided an explicit cast would be required even for string<-->binary? Correct. Accidentally left. Will remove it in the new patch. On 2011-09-27 00:43:03, John Sichi wrote: > trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFConcat.java, line 36 > < https://reviews.apache.org/r/1586/diff/2/?file=41678#file41678line36 > > > @Description annotation for this class needs to be updated. Will update in new patch. On 2011-09-27 00:43:03, John Sichi wrote: > trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFLength.java, line 55 > < https://reviews.apache.org/r/1586/diff/2/?file=41679#file41679line55 > > > @Description annotation for this class needs to be updated. Will update in new patch. On 2011-09-27 00:43:03, John Sichi wrote: > trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFToBinary.java, line 29 > < https://reviews.apache.org/r/1586/diff/2/?file=41682#file41682line29 > > > This UDF class needs an @Description annotation. Will update in new patch. On 2011-09-27 00:43:03, John Sichi wrote: > trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFSubstr.java, line 43 > < https://reviews.apache.org/r/1586/diff/2/?file=41680#file41680line43 > > > @Description annotation for this class needs to be updated. Will update in new patch. On 2011-09-27 00:43:03, John Sichi wrote: > trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFToBinary.java, line 67 > < https://reviews.apache.org/r/1586/diff/2/?file=41682#file41682line67 > > > The CAST should have parentheses, so > > CAST(x AS BINARY) > > (this applies to the existing code in GenericUDFTimestamp too) > Will update in new patch. On 2011-09-27 00:43:03, John Sichi wrote: > trunk/ql/src/test/queries/clientpositive/ba_table1.q, line 9 > < https://reviews.apache.org/r/1586/diff/2/?file=41683#file41683line9 > > > Use ORDER BY on a key for all queries to guarantee test determinism. Also, since test output keeps getting bigger and bigger, it's best to filter down to just a few rows to keep it easier to review and manage. > > (This comment applies to all tests in this patch.) Added order-by to guarantee determinism and limit to keep number of rows small in all the tests. On 2011-09-27 00:43:03, John Sichi wrote: > trunk/serde/src/java/org/apache/hadoop/hive/serde2/SerDeUtils.java, line 267 > < https://reviews.apache.org/r/1586/diff/2/?file=41696#file41696line267 > > > Is there a test for this somewhere? It seems like a case where we need base64. Missed the changes in DelimitedJSONserde so it was not getting used. DelimitedJSONserde now uses this method to print, so its gets used by all the tests. On 2011-09-27 00:43:03, John Sichi wrote: > trunk/serde/src/java/org/apache/hadoop/hive/serde2/columnar/LazyBinaryColumnarSerDe.java, line 83 > < https://reviews.apache.org/r/1586/diff/2/?file=41698#file41698line83 > > > I'm not sure about the original connection between UTF and strings here (was something forcing a UTF encoding), but is it valid for arbitrary binary data? You are correct, this may break when there is a binary data of exactly one byte long and with that bit pattern, it will return null instead of that bit pattern. But thats a pretty corner-case. We can document something along the line "If there is a possibility that your binary data may contain exactly one byte of 10111111, don't use LazyBinaryColumnarSerde with binary data type." On 2011-09-27 00:43:03, John Sichi wrote: > trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorUtils.java, line 393 > < https://reviews.apache.org/r/1586/diff/2/?file=41716#file41716line393 > > > Since we decided we're not going to support implicit conversions, these don't belong here, right? Correct. Removed. Ashutosh ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1586/#review2076 ----------------------------------------------------------- On 2011-09-18 05:52:41, Ashutosh Chauhan wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1586/ ----------------------------------------------------------- (Updated 2011-09-18 05:52:41) Review request for hive. Summary ------- This patch adds bytearray as a new datatype in Hive. This addresses bug HIVE-2380 . https://issues.apache.org/jira/browse/HIVE-2380 Diffs ----- trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 1172168 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 1172168 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 1172168 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g 1172168 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ParseDriver.java 1172168 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java 1172168 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFConcat.java 1172168 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFLength.java 1172168 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFSubstr.java 1172168 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToString.java 1172168 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFToBinary.java PRE-CREATION trunk/ql/src/test/queries/clientpositive/ba_table1.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/ba_table2.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/ba_table_udfs.q PRE-CREATION trunk/ql/src/test/results/clientpositive/ba_table1.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/ba_table2.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/ba_table_udfs.q.out PRE-CREATION trunk/serde/if/serde.thrift 1172168 trunk/serde/src/gen/thrift/gen-cpp/serde_constants.h 1172168 trunk/serde/src/gen/thrift/gen-cpp/serde_constants.cpp 1172168 trunk/serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/Constants.java 1172168 trunk/serde/src/gen/thrift/gen-php/serde/serde_constants.php 1172168 trunk/serde/src/gen/thrift/gen-py/org_apache_hadoop_hive_serde/constants.py 1172168 trunk/serde/src/gen/thrift/gen-rb/serde_constants.rb 1172168 trunk/serde/src/java/org/apache/hadoop/hive/serde2/SerDeUtils.java 1172168 trunk/serde/src/java/org/apache/hadoop/hive/serde2/binarysortable/BinarySortableSerDe.java 1172168 trunk/serde/src/java/org/apache/hadoop/hive/serde2/columnar/LazyBinaryColumnarSerDe.java 1172168 trunk/serde/src/java/org/apache/hadoop/hive/serde2/columnar/LazyBinaryColumnarStruct.java 1172168 trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyBinary.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyFactory.java 1172168 trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyUtils.java 1172168 trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/primitive/LazyBinaryObjectInspector.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/primitive/LazyPrimitiveObjectInspectorFactory.java 1172168 trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryBinary.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryFactory.java 1172168 trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinarySerDe.java 1172168 trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryUtils.java 1172168 trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorConverters.java 1172168 trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java 1172168 trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/PrimitiveObjectInspector.java 1172168 trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/BinaryObjectInspector.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/JavaBinaryObjectInspector.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorConverter.java 1172168 trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorFactory.java 1172168 trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorUtils.java 1172168 trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/SettableBinaryObjectInspector.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableBinaryObjectInspector.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/typeinfo/TypeInfoFactory.java 1172168 trunk/serde/src/test/org/apache/hadoop/hive/serde2/TestStatsSerde.java 1172168 trunk/serde/src/test/org/apache/hadoop/hive/serde2/binarysortable/MyTestClass.java 1172168 trunk/serde/src/test/org/apache/hadoop/hive/serde2/binarysortable/TestBinarySortableSerDe.java 1172168 trunk/serde/src/test/org/apache/hadoop/hive/serde2/columnar/TestLazyBinaryColumnarSerDe.java 1172168 trunk/serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazyPrimitive.java 1172168 trunk/serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazySimpleSerDe.java 1172168 trunk/serde/src/test/org/apache/hadoop/hive/serde2/lazybinary/MyTestClassBigger.java 1172168 trunk/serde/src/test/org/apache/hadoop/hive/serde2/lazybinary/TestLazyBinarySerDe.java 1172168 trunk/serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestObjectInspectorConverters.java 1172168 trunk/serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestStandardObjectInspectors.java 1172168 Diff: https://reviews.apache.org/r/1586/diff Testing ------- Added and updated unit tests. Added new system tests. Thanks, Ashutosh
          Hide
          Ashutosh Chauhan added a comment -

          @John

          I would be a lot more comfortable with this patch if it included testing of real binary data (not just strings encoded as binary) at the QL level end-to-end, getting stored in a table, queried, passed through UDF's, etc. (not just the Java unit test level of serdes) since that is the intended use case

          Added following tests:

          • ba_table1.q basic test which tests a) binary type works correctly in grammar b) string can be cast into binary c) binary can be stored in a table d) binary data can be loaded back again and queried d) order-by on a binary key
          • ba_table2.q All the test in ba_test1.q + using LazyBinarySerde instead of LazySimpleSerde
          • ba_table3.q All the tests of ba_table1.q + test for a group-by and aggregation on a binary key.
          • ba_table_udfs.q this query tests all the udfs provided to work with binary works.
          • ba_table_union.q this query tests ba_table1.q + nested queries with multiple operations on binary data types + union on binary types
          • binary_table_colserde.q - Everything in ba_table1.q + columnar serde in RCFILE.
          • binary_table_bincolserde.q Tests everything in binary_table_colserde.q + uses LazyBinaryColumnarSerde
          • load_binary_data.q this query loads native binary data, stores in a table and then queries it. Note that string.txt contains binary data. Also uses transform clause and then length udf on binary fields.

          need negative tests for attempting illegal conversions to/from binary

          Added whole bunch of negative tests of form invalid_cast_to_binary_[1..6].q and invalid_cast_from_binary_[1..6].q to tests illegal conversion attempts.

          please rename this issue since we're going with BINARY instead of BYTEARRAY

          Done.

          Show
          Ashutosh Chauhan added a comment - @John I would be a lot more comfortable with this patch if it included testing of real binary data (not just strings encoded as binary) at the QL level end-to-end, getting stored in a table, queried, passed through UDF's, etc. (not just the Java unit test level of serdes) since that is the intended use case Added following tests: ba_table1.q basic test which tests a) binary type works correctly in grammar b) string can be cast into binary c) binary can be stored in a table d) binary data can be loaded back again and queried d) order-by on a binary key ba_table2.q All the test in ba_test1.q + using LazyBinarySerde instead of LazySimpleSerde ba_table3.q All the tests of ba_table1.q + test for a group-by and aggregation on a binary key. ba_table_udfs.q this query tests all the udfs provided to work with binary works. ba_table_union.q this query tests ba_table1.q + nested queries with multiple operations on binary data types + union on binary types binary_table_colserde.q - Everything in ba_table1.q + columnar serde in RCFILE. binary_table_bincolserde.q Tests everything in binary_table_colserde.q + uses LazyBinaryColumnarSerde load_binary_data.q this query loads native binary data, stores in a table and then queries it. Note that string.txt contains binary data. Also uses transform clause and then length udf on binary fields. need negative tests for attempting illegal conversions to/from binary Added whole bunch of negative tests of form invalid_cast_to_binary_ [1..6] .q and invalid_cast_from_binary_ [1..6] .q to tests illegal conversion attempts. please rename this issue since we're going with BINARY instead of BYTEARRAY Done.
          Hide
          Ashutosh Chauhan added a comment -

          Same patch as on RB.

          Show
          Ashutosh Chauhan added a comment - Same patch as on RB.
          Hide
          Ashutosh Chauhan added a comment -

          Marking as Patch Available as its ready for review.

          Show
          Ashutosh Chauhan added a comment - Marking as Patch Available as its ready for review.
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/1586/#review2239
          -----------------------------------------------------------

          Looking good (added a note for one missing ORDER BY remaining).

          But regarding "10111111", I don't think we should have any corner cases at all in the set of binary strings which can be stored. That would just be a land mine. Can you think of a solution?

          trunk/ql/src/test/queries/clientpositive/ba_table_union.q
          <https://reviews.apache.org/r/1586/#comment5195>

          Need ORDER BY.

          • John

          On 2011-09-30 17:37:25, Ashutosh Chauhan wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/1586/

          -----------------------------------------------------------

          (Updated 2011-09-30 17:37:25)

          Review request for hive.

          Summary

          -------

          This patch adds bytearray as a new datatype in Hive.

          This addresses bug HIVE-2380.

          https://issues.apache.org/jira/browse/HIVE-2380

          Diffs

          -----

          trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 1177528

          trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 1177528

          trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 1177528

          trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g 1177528

          trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ParseDriver.java 1177528

          trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java 1177528

          trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFConcat.java 1177528

          trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFLength.java 1177528

          trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFSubstr.java 1177528

          trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToString.java 1177528

          trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFTimestamp.java 1177528

          trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFToBinary.java PRE-CREATION

          trunk/ql/src/test/queries/clientnegative/invalid_cast_from_binary_1.q PRE-CREATION

          trunk/ql/src/test/queries/clientnegative/invalid_cast_from_binary_2.q PRE-CREATION

          trunk/ql/src/test/queries/clientnegative/invalid_cast_from_binary_3.q PRE-CREATION

          trunk/ql/src/test/queries/clientnegative/invalid_cast_from_binary_4.q PRE-CREATION

          trunk/ql/src/test/queries/clientnegative/invalid_cast_from_binary_5.q PRE-CREATION

          trunk/ql/src/test/queries/clientnegative/invalid_cast_from_binary_6.q PRE-CREATION

          trunk/ql/src/test/queries/clientnegative/invalid_cast_to_binary_1.q PRE-CREATION

          trunk/ql/src/test/queries/clientnegative/invalid_cast_to_binary_2.q PRE-CREATION

          trunk/ql/src/test/queries/clientnegative/invalid_cast_to_binary_3.q PRE-CREATION

          trunk/ql/src/test/queries/clientnegative/invalid_cast_to_binary_4.q PRE-CREATION

          trunk/ql/src/test/queries/clientnegative/invalid_cast_to_binary_5.q PRE-CREATION

          trunk/ql/src/test/queries/clientnegative/invalid_cast_to_binary_6.q PRE-CREATION

          trunk/ql/src/test/queries/clientpositive/ba_table1.q PRE-CREATION

          trunk/ql/src/test/queries/clientpositive/ba_table2.q PRE-CREATION

          trunk/ql/src/test/queries/clientpositive/ba_table3.q PRE-CREATION

          trunk/ql/src/test/queries/clientpositive/ba_table_udfs.q PRE-CREATION

          trunk/ql/src/test/queries/clientpositive/ba_table_union.q PRE-CREATION

          trunk/ql/src/test/queries/clientpositive/binary_table_bincolserde.q PRE-CREATION

          trunk/ql/src/test/queries/clientpositive/binary_table_colserde.q PRE-CREATION

          trunk/ql/src/test/queries/clientpositive/load_binary_data.q PRE-CREATION

          trunk/ql/src/test/results/clientnegative/invalid_cast_from_binary_1.q.out PRE-CREATION

          trunk/ql/src/test/results/clientnegative/invalid_cast_from_binary_2.q.out PRE-CREATION

          trunk/ql/src/test/results/clientnegative/invalid_cast_from_binary_3.q.out PRE-CREATION

          trunk/ql/src/test/results/clientnegative/invalid_cast_from_binary_4.q.out PRE-CREATION

          trunk/ql/src/test/results/clientnegative/invalid_cast_from_binary_5.q.out PRE-CREATION

          trunk/ql/src/test/results/clientnegative/invalid_cast_from_binary_6.q.out PRE-CREATION

          trunk/ql/src/test/results/clientnegative/invalid_cast_to_binary_1.q.out PRE-CREATION

          trunk/ql/src/test/results/clientnegative/invalid_cast_to_binary_2.q.out PRE-CREATION

          trunk/ql/src/test/results/clientnegative/invalid_cast_to_binary_3.q.out PRE-CREATION

          trunk/ql/src/test/results/clientnegative/invalid_cast_to_binary_4.q.out PRE-CREATION

          trunk/ql/src/test/results/clientnegative/invalid_cast_to_binary_5.q.out PRE-CREATION

          trunk/ql/src/test/results/clientnegative/invalid_cast_to_binary_6.q.out PRE-CREATION

          trunk/ql/src/test/results/clientpositive/ba_table1.q.out PRE-CREATION

          trunk/ql/src/test/results/clientpositive/ba_table2.q.out PRE-CREATION

          trunk/ql/src/test/results/clientpositive/ba_table3.q.out PRE-CREATION

          trunk/ql/src/test/results/clientpositive/ba_table_udfs.q.out PRE-CREATION

          trunk/ql/src/test/results/clientpositive/ba_table_union.q.out PRE-CREATION

          trunk/ql/src/test/results/clientpositive/binary_table_bincolserde.q.out PRE-CREATION

          trunk/ql/src/test/results/clientpositive/binary_table_colserde.q.out PRE-CREATION

          trunk/ql/src/test/results/clientpositive/load_binary_data.q.out PRE-CREATION

          trunk/serde/if/serde.thrift 1177528

          trunk/serde/src/gen/thrift/gen-cpp/serde_constants.h 1177528

          trunk/serde/src/gen/thrift/gen-cpp/serde_constants.cpp 1177528

          trunk/serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/Constants.java 1177528

          trunk/serde/src/gen/thrift/gen-php/serde/serde_constants.php 1177528

          trunk/serde/src/gen/thrift/gen-py/org_apache_hadoop_hive_serde/constants.py 1177528

          trunk/serde/src/gen/thrift/gen-rb/serde_constants.rb 1177528

          trunk/serde/src/java/org/apache/hadoop/hive/serde2/DelimitedJSONSerDe.java 1177528

          trunk/serde/src/java/org/apache/hadoop/hive/serde2/SerDeUtils.java 1177528

          trunk/serde/src/java/org/apache/hadoop/hive/serde2/binarysortable/BinarySortableSerDe.java 1177528

          trunk/serde/src/java/org/apache/hadoop/hive/serde2/columnar/LazyBinaryColumnarSerDe.java 1177528

          trunk/serde/src/java/org/apache/hadoop/hive/serde2/columnar/LazyBinaryColumnarStruct.java 1177528

          trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyBinary.java PRE-CREATION

          trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyFactory.java 1177528

          trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyUtils.java 1177528

          trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/primitive/LazyBinaryObjectInspector.java PRE-CREATION

          trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/primitive/LazyPrimitiveObjectInspectorFactory.java 1177528

          trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryBinary.java PRE-CREATION

          trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryFactory.java 1177528

          trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinarySerDe.java 1177528

          trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryUtils.java 1177528

          trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorConverters.java 1177528

          trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java 1177528

          trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/PrimitiveObjectInspector.java 1177528

          trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/BinaryObjectInspector.java PRE-CREATION

          trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/JavaBinaryObjectInspector.java PRE-CREATION

          trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorConverter.java 1177528

          trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorFactory.java 1177528

          trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorUtils.java 1177528

          trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/SettableBinaryObjectInspector.java PRE-CREATION

          trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableBinaryObjectInspector.java PRE-CREATION

          trunk/serde/src/java/org/apache/hadoop/hive/serde2/typeinfo/TypeInfoFactory.java 1177528

          trunk/serde/src/test/org/apache/hadoop/hive/serde2/TestStatsSerde.java 1177528

          trunk/serde/src/test/org/apache/hadoop/hive/serde2/binarysortable/MyTestClass.java 1177528

          trunk/serde/src/test/org/apache/hadoop/hive/serde2/binarysortable/TestBinarySortableSerDe.java 1177528

          trunk/serde/src/test/org/apache/hadoop/hive/serde2/columnar/TestLazyBinaryColumnarSerDe.java 1177528

          trunk/serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazyPrimitive.java 1177528

          trunk/serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazySimpleSerDe.java 1177528

          trunk/serde/src/test/org/apache/hadoop/hive/serde2/lazybinary/MyTestClassBigger.java 1177528

          trunk/serde/src/test/org/apache/hadoop/hive/serde2/lazybinary/TestLazyBinarySerDe.java 1177528

          trunk/serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestObjectInspectorConverters.java 1177528

          trunk/serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestStandardObjectInspectors.java 1177528

          Diff: https://reviews.apache.org/r/1586/diff

          Testing

          -------

          Added and updated unit tests.

          Added new system tests.

          Thanks,

          Ashutosh

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1586/#review2239 ----------------------------------------------------------- Looking good (added a note for one missing ORDER BY remaining). But regarding "10111111", I don't think we should have any corner cases at all in the set of binary strings which can be stored. That would just be a land mine. Can you think of a solution? trunk/ql/src/test/queries/clientpositive/ba_table_union.q < https://reviews.apache.org/r/1586/#comment5195 > Need ORDER BY. John On 2011-09-30 17:37:25, Ashutosh Chauhan wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1586/ ----------------------------------------------------------- (Updated 2011-09-30 17:37:25) Review request for hive. Summary ------- This patch adds bytearray as a new datatype in Hive. This addresses bug HIVE-2380 . https://issues.apache.org/jira/browse/HIVE-2380 Diffs ----- trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 1177528 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 1177528 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 1177528 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g 1177528 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ParseDriver.java 1177528 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java 1177528 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFConcat.java 1177528 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFLength.java 1177528 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFSubstr.java 1177528 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToString.java 1177528 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFTimestamp.java 1177528 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFToBinary.java PRE-CREATION trunk/ql/src/test/queries/clientnegative/invalid_cast_from_binary_1.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/invalid_cast_from_binary_2.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/invalid_cast_from_binary_3.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/invalid_cast_from_binary_4.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/invalid_cast_from_binary_5.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/invalid_cast_from_binary_6.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/invalid_cast_to_binary_1.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/invalid_cast_to_binary_2.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/invalid_cast_to_binary_3.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/invalid_cast_to_binary_4.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/invalid_cast_to_binary_5.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/invalid_cast_to_binary_6.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/ba_table1.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/ba_table2.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/ba_table3.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/ba_table_udfs.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/ba_table_union.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/binary_table_bincolserde.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/binary_table_colserde.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/load_binary_data.q PRE-CREATION trunk/ql/src/test/results/clientnegative/invalid_cast_from_binary_1.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/invalid_cast_from_binary_2.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/invalid_cast_from_binary_3.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/invalid_cast_from_binary_4.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/invalid_cast_from_binary_5.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/invalid_cast_from_binary_6.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/invalid_cast_to_binary_1.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/invalid_cast_to_binary_2.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/invalid_cast_to_binary_3.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/invalid_cast_to_binary_4.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/invalid_cast_to_binary_5.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/invalid_cast_to_binary_6.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/ba_table1.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/ba_table2.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/ba_table3.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/ba_table_udfs.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/ba_table_union.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/binary_table_bincolserde.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/binary_table_colserde.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/load_binary_data.q.out PRE-CREATION trunk/serde/if/serde.thrift 1177528 trunk/serde/src/gen/thrift/gen-cpp/serde_constants.h 1177528 trunk/serde/src/gen/thrift/gen-cpp/serde_constants.cpp 1177528 trunk/serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/Constants.java 1177528 trunk/serde/src/gen/thrift/gen-php/serde/serde_constants.php 1177528 trunk/serde/src/gen/thrift/gen-py/org_apache_hadoop_hive_serde/constants.py 1177528 trunk/serde/src/gen/thrift/gen-rb/serde_constants.rb 1177528 trunk/serde/src/java/org/apache/hadoop/hive/serde2/DelimitedJSONSerDe.java 1177528 trunk/serde/src/java/org/apache/hadoop/hive/serde2/SerDeUtils.java 1177528 trunk/serde/src/java/org/apache/hadoop/hive/serde2/binarysortable/BinarySortableSerDe.java 1177528 trunk/serde/src/java/org/apache/hadoop/hive/serde2/columnar/LazyBinaryColumnarSerDe.java 1177528 trunk/serde/src/java/org/apache/hadoop/hive/serde2/columnar/LazyBinaryColumnarStruct.java 1177528 trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyBinary.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyFactory.java 1177528 trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyUtils.java 1177528 trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/primitive/LazyBinaryObjectInspector.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/primitive/LazyPrimitiveObjectInspectorFactory.java 1177528 trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryBinary.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryFactory.java 1177528 trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinarySerDe.java 1177528 trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryUtils.java 1177528 trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorConverters.java 1177528 trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java 1177528 trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/PrimitiveObjectInspector.java 1177528 trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/BinaryObjectInspector.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/JavaBinaryObjectInspector.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorConverter.java 1177528 trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorFactory.java 1177528 trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorUtils.java 1177528 trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/SettableBinaryObjectInspector.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableBinaryObjectInspector.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/typeinfo/TypeInfoFactory.java 1177528 trunk/serde/src/test/org/apache/hadoop/hive/serde2/TestStatsSerde.java 1177528 trunk/serde/src/test/org/apache/hadoop/hive/serde2/binarysortable/MyTestClass.java 1177528 trunk/serde/src/test/org/apache/hadoop/hive/serde2/binarysortable/TestBinarySortableSerDe.java 1177528 trunk/serde/src/test/org/apache/hadoop/hive/serde2/columnar/TestLazyBinaryColumnarSerDe.java 1177528 trunk/serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazyPrimitive.java 1177528 trunk/serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazySimpleSerDe.java 1177528 trunk/serde/src/test/org/apache/hadoop/hive/serde2/lazybinary/MyTestClassBigger.java 1177528 trunk/serde/src/test/org/apache/hadoop/hive/serde2/lazybinary/TestLazyBinarySerDe.java 1177528 trunk/serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestObjectInspectorConverters.java 1177528 trunk/serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestStandardObjectInspectors.java 1177528 Diff: https://reviews.apache.org/r/1586/diff Testing ------- Added and updated unit tests. Added new system tests. Thanks, Ashutosh
          Hide
          Ashutosh Chauhan added a comment -

          Attaching new patch which addresses John's two concerns. Added order-by in union query and for

          But regarding "10111111", I don't think we should have any corner cases at all in the set of binary strings which can be stored. That would just be a land mine. Can you think of a solution?

          Instead of special casing for the case of serializing zero length binary throwing an exception. With that, if there ever is a case when user tries to serialize a 0-length binary through BinaryColumnarSerde, we will throw an exception. Now, there is no need of special casing in LazyBinaryColumnarSerde and LazyBinaryColumnarSerde, so rolled back changes in those. Also, added a unit test case testLazyBinaryColumnarSerdeWithEmpyBinary()to exercise this code path.

          Show
          Ashutosh Chauhan added a comment - Attaching new patch which addresses John's two concerns. Added order-by in union query and for But regarding "10111111", I don't think we should have any corner cases at all in the set of binary strings which can be stored. That would just be a land mine. Can you think of a solution? Instead of special casing for the case of serializing zero length binary throwing an exception. With that, if there ever is a case when user tries to serialize a 0-length binary through BinaryColumnarSerde, we will throw an exception. Now, there is no need of special casing in LazyBinaryColumnarSerde and LazyBinaryColumnarSerde, so rolled back changes in those. Also, added a unit test case testLazyBinaryColumnarSerdeWithEmpyBinary()to exercise this code path.
          Hide
          John Sichi added a comment -

          Can you update review board with latest?

          Show
          John Sichi added a comment - Can you update review board with latest?
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/1586/
          -----------------------------------------------------------

          (Updated 2011-10-04 01:10:59.091568)

          Review request for hive.

          Summary
          -------

          This patch adds bytearray as a new datatype in Hive.

          This addresses bug HIVE-2380.
          https://issues.apache.org/jira/browse/HIVE-2380

          Diffs (updated)


          trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 1177528
          trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 1177528
          trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 1177528
          trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g 1177528
          trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ParseDriver.java 1177528
          trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java 1177528
          trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFConcat.java 1177528
          trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFLength.java 1177528
          trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFSubstr.java 1177528
          trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToString.java 1177528
          trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFTimestamp.java 1177528
          trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFToBinary.java PRE-CREATION
          trunk/ql/src/test/queries/clientnegative/invalid_cast_from_binary_1.q PRE-CREATION
          trunk/ql/src/test/queries/clientnegative/invalid_cast_from_binary_2.q PRE-CREATION
          trunk/ql/src/test/queries/clientnegative/invalid_cast_from_binary_3.q PRE-CREATION
          trunk/ql/src/test/queries/clientnegative/invalid_cast_from_binary_4.q PRE-CREATION
          trunk/ql/src/test/queries/clientnegative/invalid_cast_from_binary_5.q PRE-CREATION
          trunk/ql/src/test/queries/clientnegative/invalid_cast_from_binary_6.q PRE-CREATION
          trunk/ql/src/test/queries/clientnegative/invalid_cast_to_binary_1.q PRE-CREATION
          trunk/ql/src/test/queries/clientnegative/invalid_cast_to_binary_2.q PRE-CREATION
          trunk/ql/src/test/queries/clientnegative/invalid_cast_to_binary_3.q PRE-CREATION
          trunk/ql/src/test/queries/clientnegative/invalid_cast_to_binary_4.q PRE-CREATION
          trunk/ql/src/test/queries/clientnegative/invalid_cast_to_binary_5.q PRE-CREATION
          trunk/ql/src/test/queries/clientnegative/invalid_cast_to_binary_6.q PRE-CREATION
          trunk/ql/src/test/queries/clientpositive/ba_table1.q PRE-CREATION
          trunk/ql/src/test/queries/clientpositive/ba_table2.q PRE-CREATION
          trunk/ql/src/test/queries/clientpositive/ba_table3.q PRE-CREATION
          trunk/ql/src/test/queries/clientpositive/ba_table_udfs.q PRE-CREATION
          trunk/ql/src/test/queries/clientpositive/ba_table_union.q PRE-CREATION
          trunk/ql/src/test/queries/clientpositive/binary_table_bincolserde.q PRE-CREATION
          trunk/ql/src/test/queries/clientpositive/binary_table_colserde.q PRE-CREATION
          trunk/ql/src/test/queries/clientpositive/load_binary_data.q PRE-CREATION
          trunk/ql/src/test/results/clientnegative/invalid_cast_from_binary_1.q.out PRE-CREATION
          trunk/ql/src/test/results/clientnegative/invalid_cast_from_binary_2.q.out PRE-CREATION
          trunk/ql/src/test/results/clientnegative/invalid_cast_from_binary_3.q.out PRE-CREATION
          trunk/ql/src/test/results/clientnegative/invalid_cast_from_binary_4.q.out PRE-CREATION
          trunk/ql/src/test/results/clientnegative/invalid_cast_from_binary_5.q.out PRE-CREATION
          trunk/ql/src/test/results/clientnegative/invalid_cast_from_binary_6.q.out PRE-CREATION
          trunk/ql/src/test/results/clientnegative/invalid_cast_to_binary_1.q.out PRE-CREATION
          trunk/ql/src/test/results/clientnegative/invalid_cast_to_binary_2.q.out PRE-CREATION
          trunk/ql/src/test/results/clientnegative/invalid_cast_to_binary_3.q.out PRE-CREATION
          trunk/ql/src/test/results/clientnegative/invalid_cast_to_binary_4.q.out PRE-CREATION
          trunk/ql/src/test/results/clientnegative/invalid_cast_to_binary_5.q.out PRE-CREATION
          trunk/ql/src/test/results/clientnegative/invalid_cast_to_binary_6.q.out PRE-CREATION
          trunk/ql/src/test/results/clientpositive/ba_table1.q.out PRE-CREATION
          trunk/ql/src/test/results/clientpositive/ba_table2.q.out PRE-CREATION
          trunk/ql/src/test/results/clientpositive/ba_table3.q.out PRE-CREATION
          trunk/ql/src/test/results/clientpositive/ba_table_udfs.q.out PRE-CREATION
          trunk/ql/src/test/results/clientpositive/ba_table_union.q.out PRE-CREATION
          trunk/ql/src/test/results/clientpositive/binary_table_bincolserde.q.out PRE-CREATION
          trunk/ql/src/test/results/clientpositive/binary_table_colserde.q.out PRE-CREATION
          trunk/ql/src/test/results/clientpositive/load_binary_data.q.out PRE-CREATION
          trunk/serde/if/serde.thrift 1177528
          trunk/serde/src/gen/thrift/gen-cpp/serde_constants.h 1177528
          trunk/serde/src/gen/thrift/gen-cpp/serde_constants.cpp 1177528
          trunk/serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/Constants.java 1177528
          trunk/serde/src/gen/thrift/gen-php/serde/serde_constants.php 1177528
          trunk/serde/src/gen/thrift/gen-py/org_apache_hadoop_hive_serde/constants.py 1177528
          trunk/serde/src/gen/thrift/gen-rb/serde_constants.rb 1177528
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/DelimitedJSONSerDe.java 1177528
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/SerDeUtils.java 1177528
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/binarysortable/BinarySortableSerDe.java 1177528
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyBinary.java PRE-CREATION
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyFactory.java 1177528
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyUtils.java 1177528
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/primitive/LazyBinaryObjectInspector.java PRE-CREATION
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/primitive/LazyPrimitiveObjectInspectorFactory.java 1177528
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryBinary.java PRE-CREATION
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryFactory.java 1177528
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinarySerDe.java 1177528
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryUtils.java 1177528
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorConverters.java 1177528
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java 1177528
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/PrimitiveObjectInspector.java 1177528
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/BinaryObjectInspector.java PRE-CREATION
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/JavaBinaryObjectInspector.java PRE-CREATION
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorConverter.java 1177528
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorFactory.java 1177528
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorUtils.java 1177528
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/SettableBinaryObjectInspector.java PRE-CREATION
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableBinaryObjectInspector.java PRE-CREATION
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/typeinfo/TypeInfoFactory.java 1177528
          trunk/serde/src/test/org/apache/hadoop/hive/serde2/TestStatsSerde.java 1177528
          trunk/serde/src/test/org/apache/hadoop/hive/serde2/binarysortable/MyTestClass.java 1177528
          trunk/serde/src/test/org/apache/hadoop/hive/serde2/binarysortable/TestBinarySortableSerDe.java 1177528
          trunk/serde/src/test/org/apache/hadoop/hive/serde2/columnar/TestLazyBinaryColumnarSerDe.java 1177528
          trunk/serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazyPrimitive.java 1177528
          trunk/serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazySimpleSerDe.java 1177528
          trunk/serde/src/test/org/apache/hadoop/hive/serde2/lazybinary/MyTestClassBigger.java 1177528
          trunk/serde/src/test/org/apache/hadoop/hive/serde2/lazybinary/TestLazyBinarySerDe.java 1177528
          trunk/serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestObjectInspectorConverters.java 1177528
          trunk/serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestStandardObjectInspectors.java 1177528

          Diff: https://reviews.apache.org/r/1586/diff

          Testing
          -------

          Added and updated unit tests.
          Added new system tests.

          Thanks,

          Ashutosh

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1586/ ----------------------------------------------------------- (Updated 2011-10-04 01:10:59.091568) Review request for hive. Summary ------- This patch adds bytearray as a new datatype in Hive. This addresses bug HIVE-2380 . https://issues.apache.org/jira/browse/HIVE-2380 Diffs (updated) trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 1177528 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 1177528 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 1177528 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g 1177528 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ParseDriver.java 1177528 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java 1177528 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFConcat.java 1177528 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFLength.java 1177528 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFSubstr.java 1177528 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToString.java 1177528 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFTimestamp.java 1177528 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFToBinary.java PRE-CREATION trunk/ql/src/test/queries/clientnegative/invalid_cast_from_binary_1.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/invalid_cast_from_binary_2.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/invalid_cast_from_binary_3.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/invalid_cast_from_binary_4.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/invalid_cast_from_binary_5.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/invalid_cast_from_binary_6.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/invalid_cast_to_binary_1.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/invalid_cast_to_binary_2.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/invalid_cast_to_binary_3.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/invalid_cast_to_binary_4.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/invalid_cast_to_binary_5.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/invalid_cast_to_binary_6.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/ba_table1.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/ba_table2.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/ba_table3.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/ba_table_udfs.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/ba_table_union.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/binary_table_bincolserde.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/binary_table_colserde.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/load_binary_data.q PRE-CREATION trunk/ql/src/test/results/clientnegative/invalid_cast_from_binary_1.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/invalid_cast_from_binary_2.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/invalid_cast_from_binary_3.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/invalid_cast_from_binary_4.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/invalid_cast_from_binary_5.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/invalid_cast_from_binary_6.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/invalid_cast_to_binary_1.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/invalid_cast_to_binary_2.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/invalid_cast_to_binary_3.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/invalid_cast_to_binary_4.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/invalid_cast_to_binary_5.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/invalid_cast_to_binary_6.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/ba_table1.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/ba_table2.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/ba_table3.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/ba_table_udfs.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/ba_table_union.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/binary_table_bincolserde.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/binary_table_colserde.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/load_binary_data.q.out PRE-CREATION trunk/serde/if/serde.thrift 1177528 trunk/serde/src/gen/thrift/gen-cpp/serde_constants.h 1177528 trunk/serde/src/gen/thrift/gen-cpp/serde_constants.cpp 1177528 trunk/serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/Constants.java 1177528 trunk/serde/src/gen/thrift/gen-php/serde/serde_constants.php 1177528 trunk/serde/src/gen/thrift/gen-py/org_apache_hadoop_hive_serde/constants.py 1177528 trunk/serde/src/gen/thrift/gen-rb/serde_constants.rb 1177528 trunk/serde/src/java/org/apache/hadoop/hive/serde2/DelimitedJSONSerDe.java 1177528 trunk/serde/src/java/org/apache/hadoop/hive/serde2/SerDeUtils.java 1177528 trunk/serde/src/java/org/apache/hadoop/hive/serde2/binarysortable/BinarySortableSerDe.java 1177528 trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyBinary.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyFactory.java 1177528 trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyUtils.java 1177528 trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/primitive/LazyBinaryObjectInspector.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/primitive/LazyPrimitiveObjectInspectorFactory.java 1177528 trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryBinary.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryFactory.java 1177528 trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinarySerDe.java 1177528 trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryUtils.java 1177528 trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorConverters.java 1177528 trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java 1177528 trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/PrimitiveObjectInspector.java 1177528 trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/BinaryObjectInspector.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/JavaBinaryObjectInspector.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorConverter.java 1177528 trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorFactory.java 1177528 trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorUtils.java 1177528 trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/SettableBinaryObjectInspector.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableBinaryObjectInspector.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/typeinfo/TypeInfoFactory.java 1177528 trunk/serde/src/test/org/apache/hadoop/hive/serde2/TestStatsSerde.java 1177528 trunk/serde/src/test/org/apache/hadoop/hive/serde2/binarysortable/MyTestClass.java 1177528 trunk/serde/src/test/org/apache/hadoop/hive/serde2/binarysortable/TestBinarySortableSerDe.java 1177528 trunk/serde/src/test/org/apache/hadoop/hive/serde2/columnar/TestLazyBinaryColumnarSerDe.java 1177528 trunk/serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazyPrimitive.java 1177528 trunk/serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazySimpleSerDe.java 1177528 trunk/serde/src/test/org/apache/hadoop/hive/serde2/lazybinary/MyTestClassBigger.java 1177528 trunk/serde/src/test/org/apache/hadoop/hive/serde2/lazybinary/TestLazyBinarySerDe.java 1177528 trunk/serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestObjectInspectorConverters.java 1177528 trunk/serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestStandardObjectInspectors.java 1177528 Diff: https://reviews.apache.org/r/1586/diff Testing ------- Added and updated unit tests. Added new system tests. Thanks, Ashutosh
          Hide
          Ashutosh Chauhan added a comment -

          Updated RB with latest patch.

          Show
          Ashutosh Chauhan added a comment - Updated RB with latest patch.
          Hide
          John Sichi added a comment -

          +1. Will commit when tests pass.

          Ashutosh, could you please open and link these followups:

          • UDF's to convert to/from hex string
          • UDF's to convert to/from string using a specific encoding
          • UDF's to convert to/from base64 string
          • UDF's to convert to/from non-string types using a particular serde
          • make LazyBinaryColumnarSerDe support 0-length binary strings, assigned to Krishna Kumar (see HIVE-956) in case he wants to work on a way to remove the limitation
          Show
          John Sichi added a comment - +1. Will commit when tests pass. Ashutosh, could you please open and link these followups: UDF's to convert to/from hex string UDF's to convert to/from string using a specific encoding UDF's to convert to/from base64 string UDF's to convert to/from non-string types using a particular serde make LazyBinaryColumnarSerDe support 0-length binary strings, assigned to Krishna Kumar (see HIVE-956 ) in case he wants to work on a way to remove the limitation
          Hide
          Ashutosh Chauhan added a comment -

          @John created and linked new jiras for follow-up work.

          Show
          Ashutosh Chauhan added a comment - @John created and linked new jiras for follow-up work.
          Hide
          John Sichi added a comment -

          ant test hit the following failures:

          TestCliDriver:
          show_describe_func_quotes
          show_functions
          udf_concat
          udf_length
          udf_substr
          udf_substring

          TestParseNegative:
          invalid_function_param2

          Show
          John Sichi added a comment - ant test hit the following failures: TestCliDriver: show_describe_func_quotes show_functions udf_concat udf_length udf_substr udf_substring TestParseNegative: invalid_function_param2
          Hide
          Ashutosh Chauhan added a comment -

          Sorry about that. Those test cases needs to be updated to match the updated description of those udfs. Attaching latest patch which includes updated *.q.out files for those udfs.

          Show
          Ashutosh Chauhan added a comment - Sorry about that. Those test cases needs to be updated to match the updated description of those udfs. Attaching latest patch which includes updated *.q.out files for those udfs.
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/1586/
          -----------------------------------------------------------

          (Updated 2011-10-05 18:05:13.442838)

          Review request for hive.

          Changes
          -------

          Uploading hive-2380_4.patch which includes updated *.q.out files for udfs whose description got updated.

          Summary
          -------

          This patch adds bytearray as a new datatype in Hive.

          This addresses bug HIVE-2380.
          https://issues.apache.org/jira/browse/HIVE-2380

          Diffs (updated)


          trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 1179305
          trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 1179305
          trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 1179305
          trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g 1179305
          trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ParseDriver.java 1179305
          trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java 1179305
          trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFConcat.java 1179305
          trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFLength.java 1179305
          trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFSubstr.java 1179305
          trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToString.java 1179305
          trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFTimestamp.java 1179305
          trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFToBinary.java PRE-CREATION
          trunk/ql/src/test/queries/clientnegative/invalid_cast_from_binary_1.q PRE-CREATION
          trunk/ql/src/test/queries/clientnegative/invalid_cast_from_binary_2.q PRE-CREATION
          trunk/ql/src/test/queries/clientnegative/invalid_cast_from_binary_3.q PRE-CREATION
          trunk/ql/src/test/queries/clientnegative/invalid_cast_from_binary_4.q PRE-CREATION
          trunk/ql/src/test/queries/clientnegative/invalid_cast_from_binary_5.q PRE-CREATION
          trunk/ql/src/test/queries/clientnegative/invalid_cast_from_binary_6.q PRE-CREATION
          trunk/ql/src/test/queries/clientnegative/invalid_cast_to_binary_1.q PRE-CREATION
          trunk/ql/src/test/queries/clientnegative/invalid_cast_to_binary_2.q PRE-CREATION
          trunk/ql/src/test/queries/clientnegative/invalid_cast_to_binary_3.q PRE-CREATION
          trunk/ql/src/test/queries/clientnegative/invalid_cast_to_binary_4.q PRE-CREATION
          trunk/ql/src/test/queries/clientnegative/invalid_cast_to_binary_5.q PRE-CREATION
          trunk/ql/src/test/queries/clientnegative/invalid_cast_to_binary_6.q PRE-CREATION
          trunk/ql/src/test/queries/clientpositive/ba_table1.q PRE-CREATION
          trunk/ql/src/test/queries/clientpositive/ba_table2.q PRE-CREATION
          trunk/ql/src/test/queries/clientpositive/ba_table3.q PRE-CREATION
          trunk/ql/src/test/queries/clientpositive/ba_table_udfs.q PRE-CREATION
          trunk/ql/src/test/queries/clientpositive/ba_table_union.q PRE-CREATION
          trunk/ql/src/test/queries/clientpositive/binary_table_bincolserde.q PRE-CREATION
          trunk/ql/src/test/queries/clientpositive/binary_table_colserde.q PRE-CREATION
          trunk/ql/src/test/queries/clientpositive/load_binary_data.q PRE-CREATION
          trunk/ql/src/test/results/clientnegative/invalid_cast_from_binary_1.q.out PRE-CREATION
          trunk/ql/src/test/results/clientnegative/invalid_cast_from_binary_2.q.out PRE-CREATION
          trunk/ql/src/test/results/clientnegative/invalid_cast_from_binary_3.q.out PRE-CREATION
          trunk/ql/src/test/results/clientnegative/invalid_cast_from_binary_4.q.out PRE-CREATION
          trunk/ql/src/test/results/clientnegative/invalid_cast_from_binary_5.q.out PRE-CREATION
          trunk/ql/src/test/results/clientnegative/invalid_cast_from_binary_6.q.out PRE-CREATION
          trunk/ql/src/test/results/clientnegative/invalid_cast_to_binary_1.q.out PRE-CREATION
          trunk/ql/src/test/results/clientnegative/invalid_cast_to_binary_2.q.out PRE-CREATION
          trunk/ql/src/test/results/clientnegative/invalid_cast_to_binary_3.q.out PRE-CREATION
          trunk/ql/src/test/results/clientnegative/invalid_cast_to_binary_4.q.out PRE-CREATION
          trunk/ql/src/test/results/clientnegative/invalid_cast_to_binary_5.q.out PRE-CREATION
          trunk/ql/src/test/results/clientnegative/invalid_cast_to_binary_6.q.out PRE-CREATION
          trunk/ql/src/test/results/clientpositive/ba_table1.q.out PRE-CREATION
          trunk/ql/src/test/results/clientpositive/ba_table2.q.out PRE-CREATION
          trunk/ql/src/test/results/clientpositive/ba_table3.q.out PRE-CREATION
          trunk/ql/src/test/results/clientpositive/ba_table_udfs.q.out PRE-CREATION
          trunk/ql/src/test/results/clientpositive/ba_table_union.q.out PRE-CREATION
          trunk/ql/src/test/results/clientpositive/binary_table_bincolserde.q.out PRE-CREATION
          trunk/ql/src/test/results/clientpositive/binary_table_colserde.q.out PRE-CREATION
          trunk/ql/src/test/results/clientpositive/load_binary_data.q.out PRE-CREATION
          trunk/ql/src/test/results/clientpositive/show_describe_func_quotes.q.out 1179305
          trunk/ql/src/test/results/clientpositive/show_functions.q.out 1179305
          trunk/ql/src/test/results/clientpositive/udf_concat.q.out 1179305
          trunk/ql/src/test/results/clientpositive/udf_length.q.out 1179305
          trunk/ql/src/test/results/clientpositive/udf_substr.q.out 1179305
          trunk/ql/src/test/results/clientpositive/udf_substring.q.out 1179305
          trunk/ql/src/test/results/compiler/errors/invalid_function_param2.q.out 1179305
          trunk/serde/if/serde.thrift 1179305
          trunk/serde/src/gen/thrift/gen-cpp/serde_constants.h 1179305
          trunk/serde/src/gen/thrift/gen-cpp/serde_constants.cpp 1179305
          trunk/serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/Constants.java 1179305
          trunk/serde/src/gen/thrift/gen-php/serde/serde_constants.php 1179305
          trunk/serde/src/gen/thrift/gen-py/org_apache_hadoop_hive_serde/constants.py 1179305
          trunk/serde/src/gen/thrift/gen-rb/serde_constants.rb 1179305
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/DelimitedJSONSerDe.java 1179305
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/SerDeUtils.java 1179305
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/binarysortable/BinarySortableSerDe.java 1179305
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyBinary.java PRE-CREATION
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyFactory.java 1179305
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyUtils.java 1179305
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/primitive/LazyBinaryObjectInspector.java PRE-CREATION
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/primitive/LazyPrimitiveObjectInspectorFactory.java 1179305
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryBinary.java PRE-CREATION
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryFactory.java 1179305
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinarySerDe.java 1179305
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryUtils.java 1179305
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorConverters.java 1179305
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java 1179305
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/PrimitiveObjectInspector.java 1179305
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/BinaryObjectInspector.java PRE-CREATION
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/JavaBinaryObjectInspector.java PRE-CREATION
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorConverter.java 1179305
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorFactory.java 1179305
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorUtils.java 1179305
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/SettableBinaryObjectInspector.java PRE-CREATION
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableBinaryObjectInspector.java PRE-CREATION
          trunk/serde/src/java/org/apache/hadoop/hive/serde2/typeinfo/TypeInfoFactory.java 1179305
          trunk/serde/src/test/org/apache/hadoop/hive/serde2/TestStatsSerde.java 1179305
          trunk/serde/src/test/org/apache/hadoop/hive/serde2/binarysortable/MyTestClass.java 1179305
          trunk/serde/src/test/org/apache/hadoop/hive/serde2/binarysortable/TestBinarySortableSerDe.java 1179305
          trunk/serde/src/test/org/apache/hadoop/hive/serde2/columnar/TestLazyBinaryColumnarSerDe.java 1179305
          trunk/serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazyPrimitive.java 1179305
          trunk/serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazySimpleSerDe.java 1179305
          trunk/serde/src/test/org/apache/hadoop/hive/serde2/lazybinary/MyTestClassBigger.java 1179305
          trunk/serde/src/test/org/apache/hadoop/hive/serde2/lazybinary/TestLazyBinarySerDe.java 1179305
          trunk/serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestObjectInspectorConverters.java 1179305
          trunk/serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestStandardObjectInspectors.java 1179305

          Diff: https://reviews.apache.org/r/1586/diff

          Testing
          -------

          Added and updated unit tests.
          Added new system tests.

          Thanks,

          Ashutosh

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1586/ ----------------------------------------------------------- (Updated 2011-10-05 18:05:13.442838) Review request for hive. Changes ------- Uploading hive-2380_4.patch which includes updated *.q.out files for udfs whose description got updated. Summary ------- This patch adds bytearray as a new datatype in Hive. This addresses bug HIVE-2380 . https://issues.apache.org/jira/browse/HIVE-2380 Diffs (updated) trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 1179305 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 1179305 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 1179305 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g 1179305 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ParseDriver.java 1179305 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java 1179305 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFConcat.java 1179305 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFLength.java 1179305 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFSubstr.java 1179305 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToString.java 1179305 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFTimestamp.java 1179305 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFToBinary.java PRE-CREATION trunk/ql/src/test/queries/clientnegative/invalid_cast_from_binary_1.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/invalid_cast_from_binary_2.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/invalid_cast_from_binary_3.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/invalid_cast_from_binary_4.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/invalid_cast_from_binary_5.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/invalid_cast_from_binary_6.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/invalid_cast_to_binary_1.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/invalid_cast_to_binary_2.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/invalid_cast_to_binary_3.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/invalid_cast_to_binary_4.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/invalid_cast_to_binary_5.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/invalid_cast_to_binary_6.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/ba_table1.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/ba_table2.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/ba_table3.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/ba_table_udfs.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/ba_table_union.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/binary_table_bincolserde.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/binary_table_colserde.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/load_binary_data.q PRE-CREATION trunk/ql/src/test/results/clientnegative/invalid_cast_from_binary_1.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/invalid_cast_from_binary_2.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/invalid_cast_from_binary_3.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/invalid_cast_from_binary_4.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/invalid_cast_from_binary_5.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/invalid_cast_from_binary_6.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/invalid_cast_to_binary_1.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/invalid_cast_to_binary_2.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/invalid_cast_to_binary_3.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/invalid_cast_to_binary_4.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/invalid_cast_to_binary_5.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/invalid_cast_to_binary_6.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/ba_table1.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/ba_table2.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/ba_table3.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/ba_table_udfs.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/ba_table_union.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/binary_table_bincolserde.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/binary_table_colserde.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/load_binary_data.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/show_describe_func_quotes.q.out 1179305 trunk/ql/src/test/results/clientpositive/show_functions.q.out 1179305 trunk/ql/src/test/results/clientpositive/udf_concat.q.out 1179305 trunk/ql/src/test/results/clientpositive/udf_length.q.out 1179305 trunk/ql/src/test/results/clientpositive/udf_substr.q.out 1179305 trunk/ql/src/test/results/clientpositive/udf_substring.q.out 1179305 trunk/ql/src/test/results/compiler/errors/invalid_function_param2.q.out 1179305 trunk/serde/if/serde.thrift 1179305 trunk/serde/src/gen/thrift/gen-cpp/serde_constants.h 1179305 trunk/serde/src/gen/thrift/gen-cpp/serde_constants.cpp 1179305 trunk/serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/Constants.java 1179305 trunk/serde/src/gen/thrift/gen-php/serde/serde_constants.php 1179305 trunk/serde/src/gen/thrift/gen-py/org_apache_hadoop_hive_serde/constants.py 1179305 trunk/serde/src/gen/thrift/gen-rb/serde_constants.rb 1179305 trunk/serde/src/java/org/apache/hadoop/hive/serde2/DelimitedJSONSerDe.java 1179305 trunk/serde/src/java/org/apache/hadoop/hive/serde2/SerDeUtils.java 1179305 trunk/serde/src/java/org/apache/hadoop/hive/serde2/binarysortable/BinarySortableSerDe.java 1179305 trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyBinary.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyFactory.java 1179305 trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyUtils.java 1179305 trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/primitive/LazyBinaryObjectInspector.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/primitive/LazyPrimitiveObjectInspectorFactory.java 1179305 trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryBinary.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryFactory.java 1179305 trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinarySerDe.java 1179305 trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryUtils.java 1179305 trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorConverters.java 1179305 trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java 1179305 trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/PrimitiveObjectInspector.java 1179305 trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/BinaryObjectInspector.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/JavaBinaryObjectInspector.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorConverter.java 1179305 trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorFactory.java 1179305 trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorUtils.java 1179305 trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/SettableBinaryObjectInspector.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableBinaryObjectInspector.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/typeinfo/TypeInfoFactory.java 1179305 trunk/serde/src/test/org/apache/hadoop/hive/serde2/TestStatsSerde.java 1179305 trunk/serde/src/test/org/apache/hadoop/hive/serde2/binarysortable/MyTestClass.java 1179305 trunk/serde/src/test/org/apache/hadoop/hive/serde2/binarysortable/TestBinarySortableSerDe.java 1179305 trunk/serde/src/test/org/apache/hadoop/hive/serde2/columnar/TestLazyBinaryColumnarSerDe.java 1179305 trunk/serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazyPrimitive.java 1179305 trunk/serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazySimpleSerDe.java 1179305 trunk/serde/src/test/org/apache/hadoop/hive/serde2/lazybinary/MyTestClassBigger.java 1179305 trunk/serde/src/test/org/apache/hadoop/hive/serde2/lazybinary/TestLazyBinarySerDe.java 1179305 trunk/serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestObjectInspectorConverters.java 1179305 trunk/serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestStandardObjectInspectors.java 1179305 Diff: https://reviews.apache.org/r/1586/diff Testing ------- Added and updated unit tests. Added new system tests. Thanks, Ashutosh
          Hide
          John Sichi added a comment -

          I'll rerun with latest. Please remember to click "Submit Patch" to get it back into the queue.

          Show
          John Sichi added a comment - I'll rerun with latest. Please remember to click "Submit Patch" to get it back into the queue.
          Hide
          John Sichi added a comment -

          Committed to trunk. Thanks Ashutosh!

          Show
          John Sichi added a comment - Committed to trunk. Thanks Ashutosh!
          Hide
          Hudson added a comment -

          Integrated in Hive-trunk-h0.21 #997 (See https://builds.apache.org/job/Hive-trunk-h0.21/997/)
          HIVE-2380. Add Binary Datatype in Hive
          (Ashutosh Chauhan via jvs)

          jvs : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1179784
          Files :

          • /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ParseDriver.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFConcat.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFLength.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFSubstr.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToString.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFTimestamp.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFToBinary.java
          • /hive/trunk/ql/src/test/queries/clientnegative/invalid_cast_from_binary_1.q
          • /hive/trunk/ql/src/test/queries/clientnegative/invalid_cast_from_binary_2.q
          • /hive/trunk/ql/src/test/queries/clientnegative/invalid_cast_from_binary_3.q
          • /hive/trunk/ql/src/test/queries/clientnegative/invalid_cast_from_binary_4.q
          • /hive/trunk/ql/src/test/queries/clientnegative/invalid_cast_from_binary_5.q
          • /hive/trunk/ql/src/test/queries/clientnegative/invalid_cast_from_binary_6.q
          • /hive/trunk/ql/src/test/queries/clientnegative/invalid_cast_to_binary_1.q
          • /hive/trunk/ql/src/test/queries/clientnegative/invalid_cast_to_binary_2.q
          • /hive/trunk/ql/src/test/queries/clientnegative/invalid_cast_to_binary_3.q
          • /hive/trunk/ql/src/test/queries/clientnegative/invalid_cast_to_binary_4.q
          • /hive/trunk/ql/src/test/queries/clientnegative/invalid_cast_to_binary_5.q
          • /hive/trunk/ql/src/test/queries/clientnegative/invalid_cast_to_binary_6.q
          • /hive/trunk/ql/src/test/queries/clientpositive/ba_table1.q
          • /hive/trunk/ql/src/test/queries/clientpositive/ba_table2.q
          • /hive/trunk/ql/src/test/queries/clientpositive/ba_table3.q
          • /hive/trunk/ql/src/test/queries/clientpositive/ba_table_udfs.q
          • /hive/trunk/ql/src/test/queries/clientpositive/ba_table_union.q
          • /hive/trunk/ql/src/test/queries/clientpositive/binary_table_bincolserde.q
          • /hive/trunk/ql/src/test/queries/clientpositive/binary_table_colserde.q
          • /hive/trunk/ql/src/test/queries/clientpositive/load_binary_data.q
          • /hive/trunk/ql/src/test/results/clientnegative/invalid_cast_from_binary_1.q.out
          • /hive/trunk/ql/src/test/results/clientnegative/invalid_cast_from_binary_2.q.out
          • /hive/trunk/ql/src/test/results/clientnegative/invalid_cast_from_binary_3.q.out
          • /hive/trunk/ql/src/test/results/clientnegative/invalid_cast_from_binary_4.q.out
          • /hive/trunk/ql/src/test/results/clientnegative/invalid_cast_from_binary_5.q.out
          • /hive/trunk/ql/src/test/results/clientnegative/invalid_cast_from_binary_6.q.out
          • /hive/trunk/ql/src/test/results/clientnegative/invalid_cast_to_binary_1.q.out
          • /hive/trunk/ql/src/test/results/clientnegative/invalid_cast_to_binary_2.q.out
          • /hive/trunk/ql/src/test/results/clientnegative/invalid_cast_to_binary_3.q.out
          • /hive/trunk/ql/src/test/results/clientnegative/invalid_cast_to_binary_4.q.out
          • /hive/trunk/ql/src/test/results/clientnegative/invalid_cast_to_binary_5.q.out
          • /hive/trunk/ql/src/test/results/clientnegative/invalid_cast_to_binary_6.q.out
          • /hive/trunk/ql/src/test/results/clientpositive/ba_table1.q.out
          • /hive/trunk/ql/src/test/results/clientpositive/ba_table2.q.out
          • /hive/trunk/ql/src/test/results/clientpositive/ba_table3.q.out
          • /hive/trunk/ql/src/test/results/clientpositive/ba_table_udfs.q.out
          • /hive/trunk/ql/src/test/results/clientpositive/ba_table_union.q.out
          • /hive/trunk/ql/src/test/results/clientpositive/binary_table_bincolserde.q.out
          • /hive/trunk/ql/src/test/results/clientpositive/binary_table_colserde.q.out
          • /hive/trunk/ql/src/test/results/clientpositive/load_binary_data.q.out
          • /hive/trunk/ql/src/test/results/clientpositive/show_describe_func_quotes.q.out
          • /hive/trunk/ql/src/test/results/clientpositive/show_functions.q.out
          • /hive/trunk/ql/src/test/results/clientpositive/udf_concat.q.out
          • /hive/trunk/ql/src/test/results/clientpositive/udf_length.q.out
          • /hive/trunk/ql/src/test/results/clientpositive/udf_substr.q.out
          • /hive/trunk/ql/src/test/results/clientpositive/udf_substring.q.out
          • /hive/trunk/ql/src/test/results/compiler/errors/invalid_function_param2.q.out
          • /hive/trunk/serde/if/serde.thrift
          • /hive/trunk/serde/src/gen/thrift/gen-cpp/serde_constants.cpp
          • /hive/trunk/serde/src/gen/thrift/gen-cpp/serde_constants.h
          • /hive/trunk/serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/Constants.java
          • /hive/trunk/serde/src/gen/thrift/gen-php/serde/serde_constants.php
          • /hive/trunk/serde/src/gen/thrift/gen-py/org_apache_hadoop_hive_serde/constants.py
          • /hive/trunk/serde/src/gen/thrift/gen-rb/serde_constants.rb
          • /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/DelimitedJSONSerDe.java
          • /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/SerDeUtils.java
          • /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/binarysortable/BinarySortableSerDe.java
          • /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyBinary.java
          • /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyFactory.java
          • /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyUtils.java
          • /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/primitive/LazyBinaryObjectInspector.java
          • /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/primitive/LazyPrimitiveObjectInspectorFactory.java
          • /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryBinary.java
          • /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryFactory.java
          • /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinarySerDe.java
          • /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryUtils.java
          • /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorConverters.java
          • /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java
          • /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/PrimitiveObjectInspector.java
          • /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/BinaryObjectInspector.java
          • /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/JavaBinaryObjectInspector.java
          • /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorConverter.java
          • /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorFactory.java
          • /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorUtils.java
          • /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/SettableBinaryObjectInspector.java
          • /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableBinaryObjectInspector.java
          • /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/typeinfo/TypeInfoFactory.java
          • /hive/trunk/serde/src/test/org/apache/hadoop/hive/serde2/TestStatsSerde.java
          • /hive/trunk/serde/src/test/org/apache/hadoop/hive/serde2/binarysortable/MyTestClass.java
          • /hive/trunk/serde/src/test/org/apache/hadoop/hive/serde2/binarysortable/TestBinarySortableSerDe.java
          • /hive/trunk/serde/src/test/org/apache/hadoop/hive/serde2/columnar/TestLazyBinaryColumnarSerDe.java
          • /hive/trunk/serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazyPrimitive.java
          • /hive/trunk/serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazySimpleSerDe.java
          • /hive/trunk/serde/src/test/org/apache/hadoop/hive/serde2/lazybinary/MyTestClassBigger.java
          • /hive/trunk/serde/src/test/org/apache/hadoop/hive/serde2/lazybinary/TestLazyBinarySerDe.java
          • /hive/trunk/serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestObjectInspectorConverters.java
          • /hive/trunk/serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestStandardObjectInspectors.java
          Show
          Hudson added a comment - Integrated in Hive-trunk-h0.21 #997 (See https://builds.apache.org/job/Hive-trunk-h0.21/997/ ) HIVE-2380 . Add Binary Datatype in Hive (Ashutosh Chauhan via jvs) jvs : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1179784 Files : /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ParseDriver.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFConcat.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFLength.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFSubstr.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToString.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFTimestamp.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFToBinary.java /hive/trunk/ql/src/test/queries/clientnegative/invalid_cast_from_binary_1.q /hive/trunk/ql/src/test/queries/clientnegative/invalid_cast_from_binary_2.q /hive/trunk/ql/src/test/queries/clientnegative/invalid_cast_from_binary_3.q /hive/trunk/ql/src/test/queries/clientnegative/invalid_cast_from_binary_4.q /hive/trunk/ql/src/test/queries/clientnegative/invalid_cast_from_binary_5.q /hive/trunk/ql/src/test/queries/clientnegative/invalid_cast_from_binary_6.q /hive/trunk/ql/src/test/queries/clientnegative/invalid_cast_to_binary_1.q /hive/trunk/ql/src/test/queries/clientnegative/invalid_cast_to_binary_2.q /hive/trunk/ql/src/test/queries/clientnegative/invalid_cast_to_binary_3.q /hive/trunk/ql/src/test/queries/clientnegative/invalid_cast_to_binary_4.q /hive/trunk/ql/src/test/queries/clientnegative/invalid_cast_to_binary_5.q /hive/trunk/ql/src/test/queries/clientnegative/invalid_cast_to_binary_6.q /hive/trunk/ql/src/test/queries/clientpositive/ba_table1.q /hive/trunk/ql/src/test/queries/clientpositive/ba_table2.q /hive/trunk/ql/src/test/queries/clientpositive/ba_table3.q /hive/trunk/ql/src/test/queries/clientpositive/ba_table_udfs.q /hive/trunk/ql/src/test/queries/clientpositive/ba_table_union.q /hive/trunk/ql/src/test/queries/clientpositive/binary_table_bincolserde.q /hive/trunk/ql/src/test/queries/clientpositive/binary_table_colserde.q /hive/trunk/ql/src/test/queries/clientpositive/load_binary_data.q /hive/trunk/ql/src/test/results/clientnegative/invalid_cast_from_binary_1.q.out /hive/trunk/ql/src/test/results/clientnegative/invalid_cast_from_binary_2.q.out /hive/trunk/ql/src/test/results/clientnegative/invalid_cast_from_binary_3.q.out /hive/trunk/ql/src/test/results/clientnegative/invalid_cast_from_binary_4.q.out /hive/trunk/ql/src/test/results/clientnegative/invalid_cast_from_binary_5.q.out /hive/trunk/ql/src/test/results/clientnegative/invalid_cast_from_binary_6.q.out /hive/trunk/ql/src/test/results/clientnegative/invalid_cast_to_binary_1.q.out /hive/trunk/ql/src/test/results/clientnegative/invalid_cast_to_binary_2.q.out /hive/trunk/ql/src/test/results/clientnegative/invalid_cast_to_binary_3.q.out /hive/trunk/ql/src/test/results/clientnegative/invalid_cast_to_binary_4.q.out /hive/trunk/ql/src/test/results/clientnegative/invalid_cast_to_binary_5.q.out /hive/trunk/ql/src/test/results/clientnegative/invalid_cast_to_binary_6.q.out /hive/trunk/ql/src/test/results/clientpositive/ba_table1.q.out /hive/trunk/ql/src/test/results/clientpositive/ba_table2.q.out /hive/trunk/ql/src/test/results/clientpositive/ba_table3.q.out /hive/trunk/ql/src/test/results/clientpositive/ba_table_udfs.q.out /hive/trunk/ql/src/test/results/clientpositive/ba_table_union.q.out /hive/trunk/ql/src/test/results/clientpositive/binary_table_bincolserde.q.out /hive/trunk/ql/src/test/results/clientpositive/binary_table_colserde.q.out /hive/trunk/ql/src/test/results/clientpositive/load_binary_data.q.out /hive/trunk/ql/src/test/results/clientpositive/show_describe_func_quotes.q.out /hive/trunk/ql/src/test/results/clientpositive/show_functions.q.out /hive/trunk/ql/src/test/results/clientpositive/udf_concat.q.out /hive/trunk/ql/src/test/results/clientpositive/udf_length.q.out /hive/trunk/ql/src/test/results/clientpositive/udf_substr.q.out /hive/trunk/ql/src/test/results/clientpositive/udf_substring.q.out /hive/trunk/ql/src/test/results/compiler/errors/invalid_function_param2.q.out /hive/trunk/serde/if/serde.thrift /hive/trunk/serde/src/gen/thrift/gen-cpp/serde_constants.cpp /hive/trunk/serde/src/gen/thrift/gen-cpp/serde_constants.h /hive/trunk/serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/Constants.java /hive/trunk/serde/src/gen/thrift/gen-php/serde/serde_constants.php /hive/trunk/serde/src/gen/thrift/gen-py/org_apache_hadoop_hive_serde/constants.py /hive/trunk/serde/src/gen/thrift/gen-rb/serde_constants.rb /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/DelimitedJSONSerDe.java /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/SerDeUtils.java /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/binarysortable/BinarySortableSerDe.java /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyBinary.java /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyFactory.java /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyUtils.java /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/primitive/LazyBinaryObjectInspector.java /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/primitive/LazyPrimitiveObjectInspectorFactory.java /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryBinary.java /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryFactory.java /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinarySerDe.java /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryUtils.java /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorConverters.java /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/PrimitiveObjectInspector.java /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/BinaryObjectInspector.java /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/JavaBinaryObjectInspector.java /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorConverter.java /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorFactory.java /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorUtils.java /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/SettableBinaryObjectInspector.java /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableBinaryObjectInspector.java /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/typeinfo/TypeInfoFactory.java /hive/trunk/serde/src/test/org/apache/hadoop/hive/serde2/TestStatsSerde.java /hive/trunk/serde/src/test/org/apache/hadoop/hive/serde2/binarysortable/MyTestClass.java /hive/trunk/serde/src/test/org/apache/hadoop/hive/serde2/binarysortable/TestBinarySortableSerDe.java /hive/trunk/serde/src/test/org/apache/hadoop/hive/serde2/columnar/TestLazyBinaryColumnarSerDe.java /hive/trunk/serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazyPrimitive.java /hive/trunk/serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazySimpleSerDe.java /hive/trunk/serde/src/test/org/apache/hadoop/hive/serde2/lazybinary/MyTestClassBigger.java /hive/trunk/serde/src/test/org/apache/hadoop/hive/serde2/lazybinary/TestLazyBinarySerDe.java /hive/trunk/serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestObjectInspectorConverters.java /hive/trunk/serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestStandardObjectInspectors.java
          Hide
          Hojjat Jafarpour added a comment -

          Is there an example for defining a table with binary and timestamp columns and loading daata into the table and querying it?

          Show
          Hojjat Jafarpour added a comment - Is there an example for defining a table with binary and timestamp columns and loading daata into the table and querying it?
          Hide
          Chuck Connell added a comment -

          I am trying to use this feature (BINARY columns) and I believe I have the perfect use-case for it, but I am missing something.

          Here is the background... I have some files that each contain just one logical field, which is a binary object. (The files are Google Protobuf format.) I want to put these binary files into a larger file, where each protobuf is a logical record. Then I want to define a Hive table that stores each protobuf as one row, with the entire protobuf object in one BINARY column. Then I will use a custom UDF to select/query the binary object.

          This is about as simple as can be for putting binary data into Hive. But all of the test cases for this jira seem to draw the binary columns from another existing table and CAST them. I want to load the files from disk.

          What file format should I use to package the binary rows? What should the Hive table definition be? I cannot use TEXTFILE, since the binary may contain newlines. Many of my attempts have choked on the newlines.

          Thanks very much,
          Chuck Connell
          Nuance
          Burlington, MA

          Show
          Chuck Connell added a comment - I am trying to use this feature (BINARY columns) and I believe I have the perfect use-case for it, but I am missing something. Here is the background... I have some files that each contain just one logical field, which is a binary object. (The files are Google Protobuf format.) I want to put these binary files into a larger file, where each protobuf is a logical record. Then I want to define a Hive table that stores each protobuf as one row, with the entire protobuf object in one BINARY column. Then I will use a custom UDF to select/query the binary object. This is about as simple as can be for putting binary data into Hive. But all of the test cases for this jira seem to draw the binary columns from another existing table and CAST them. I want to load the files from disk. What file format should I use to package the binary rows? What should the Hive table definition be? I cannot use TEXTFILE, since the binary may contain newlines. Many of my attempts have choked on the newlines. Thanks very much, Chuck Connell Nuance Burlington, MA
          Hide
          Chuck Connell added a comment -

          Let me state my earlier comment another way... If the BINARY column type "works" in narrow sense, but it is impossible to get binary data from a file into Hive, then this feature is essentially useless. So we need some explanation for how to use the existing DDL features to support BINARY columns, or a new file format that allows it.

          Show
          Chuck Connell added a comment - Let me state my earlier comment another way... If the BINARY column type "works" in narrow sense, but it is impossible to get binary data from a file into Hive, then this feature is essentially useless. So we need some explanation for how to use the existing DDL features to support BINARY columns, or a new file format that allows it.

            People

            • Assignee:
              Ashutosh Chauhan
              Reporter:
              Ashutosh Chauhan
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development