Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.10.0
    • Component/s: SQL
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      This feature is incomplete and has been disabled until HIVE-3073 (DML support for list bucketing) is finished and committed.
      Show
      This feature is incomplete and has been disabled until HIVE-3073 (DML support for list bucketing) is finished and committed.

      Description

      If a hive table column has skewed keys, query performance on non-skewed key is always impacted. Hive List Bucketing feature will address it:

      https://cwiki.apache.org/Hive/listbucketing.html

      This jira issue will track DDL change for the feature. It's for both single skewed column and multiple columns.

      1. HIVE-3072.patch
        504 kB
        Gang Tim Liu
      2. HIVE-3072.patch.1
        506 kB
        Gang Tim Liu
      3. HIVE-3072.patch.2
        513 kB
        Gang Tim Liu
      4. HIVE-3072.patch.3
        513 kB
        Gang Tim Liu
      5. HIVE-3072.patch.4
        526 kB
        Gang Tim Liu
      6. HIVE-3072.patch.5
        605 kB
        Gang Tim Liu
      7. HIVE-3072.patch.6
        606 kB
        Gang Tim Liu
      8. HIVE-3072.patch.7
        607 kB
        Gang Tim Liu

        Issue Links

          Activity

          Hide
          gangtimliu Gang Tim Liu added a comment -

          making progress on DML. The following syntax started to work:
          create table T (c1 string, c2 string) list bucketed by (c1) with skew ('x1');
          create table T (c1 string, c2 string, c3 string) list bucketed by (c1, c2) with skew (('x1', 'x2'), ('y1', 'y2'));

          Show
          gangtimliu Gang Tim Liu added a comment - making progress on DML. The following syntax started to work: create table T (c1 string, c2 string) list bucketed by (c1) with skew ('x1'); create table T (c1 string, c2 string, c3 string) list bucketed by (c1, c2) with skew (('x1', 'x2'), ('y1', 'y2'));
          Hide
          gangtimliu Gang Tim Liu added a comment -

          We rethink "release approach". We can deliver DDL and DML as separate patches or a single patch. Either has pros and cons. not perfect. Separate patch approach can make release more manageable. A single patch makes release make more sense because with DDL but no DML you can't experience list bucketing.

          We have to pick up one. We choose a single patch approach. It reduces overhead of multiple-patch release, gives community more time to review proposal and reserves room for us to adjust according to proposal review.

          I will call proposal review again today.

          Show
          gangtimliu Gang Tim Liu added a comment - We rethink "release approach". We can deliver DDL and DML as separate patches or a single patch. Either has pros and cons. not perfect. Separate patch approach can make release more manageable. A single patch makes release make more sense because with DDL but no DML you can't experience list bucketing. We have to pick up one. We choose a single patch approach. It reduces overhead of multiple-patch release, gives community more time to review proposal and reserves room for us to adjust according to proposal review. I will call proposal review again today.
          Hide
          cwsteinbach Carl Steinbach added a comment -

          If this feature requires metastore changes then I'd like to request that the first patch contain only changes to the metastore schema and metastore Thrift API. I would also prefer that the DML and DDL changes go in as a single patch since it a) prevents half-implemented features from showing up in releases and b) demonstrates that the feature actually works.

          Show
          cwsteinbach Carl Steinbach added a comment - If this feature requires metastore changes then I'd like to request that the first patch contain only changes to the metastore schema and metastore Thrift API. I would also prefer that the DML and DDL changes go in as a single patch since it a) prevents half-implemented features from showing up in releases and b) demonstrates that the feature actually works.
          Hide
          gangtimliu Gang Tim Liu added a comment -

          Yes, we are heading to a single patch approach.

          Yes, this feature requires metastore change.

          Show
          gangtimliu Gang Tim Liu added a comment - Yes, we are heading to a single patch approach. Yes, this feature requires metastore change.
          Show
          gangtimliu Gang Tim Liu added a comment - https://reviews.facebook.net/D4599
          Hide
          gangtimliu Gang Tim Liu added a comment -

          There are a few reasons to release a DDL as a patch:
          1. the content is getting bigger. might be better to get it review so that we can capture issues earlier.
          2. the skewed grammar will not only benefit list bucketing feature but also other feature like skewed join. releasing grammar patch will unblock others' development.

          Show
          gangtimliu Gang Tim Liu added a comment - There are a few reasons to release a DDL as a patch: 1. the content is getting bigger. might be better to get it review so that we can capture issues earlier. 2. the skewed grammar will not only benefit list bucketing feature but also other feature like skewed join. releasing grammar patch will unblock others' development.
          Hide
          namit Namit Jain added a comment -

          Also, the DML support would require complete support for running tests on hadoop 23 (or some version of hadoop
          which has support for nested directories). I know, there are a couple of jiras for making it work, like HIVE-3341,
          but they are not done yet.
          Also, these are independent and we should reduce dependencies on HIVE-3341 as far as possible.

          Show
          namit Namit Jain added a comment - Also, the DML support would require complete support for running tests on hadoop 23 (or some version of hadoop which has support for nested directories). I know, there are a couple of jiras for making it work, like HIVE-3341 , but they are not done yet. Also, these are independent and we should reduce dependencies on HIVE-3341 as far as possible.
          Hide
          cwsteinbach Carl Steinbach added a comment -

          Are you planning to submit this patch for review?

          Show
          cwsteinbach Carl Steinbach added a comment - Are you planning to submit this patch for review?
          Hide
          gangtimliu Gang Tim Liu added a comment -

          yes, I am.

          Show
          gangtimliu Gang Tim Liu added a comment - yes, I am.
          Hide
          gangtimliu Gang Tim Liu added a comment -

          Patch is available for review.

          Show
          gangtimliu Gang Tim Liu added a comment - Patch is available for review.
          Hide
          namit Namit Jain added a comment -

          comments on phabricator

          Show
          namit Namit Jain added a comment - comments on phabricator
          Hide
          gangtimliu Gang Tim Liu added a comment -

          Patch is in both jira and phabricator.

          Show
          gangtimliu Gang Tim Liu added a comment - Patch is in both jira and phabricator.
          Hide
          cwsteinbach Carl Steinbach added a comment -

          How far away is HIVE-3073 from completion? Is it possible to post a work-in-progress patch for HIVE-3073 so we can see how this all fits together?

          Show
          cwsteinbach Carl Steinbach added a comment - How far away is HIVE-3073 from completion? Is it possible to post a work-in-progress patch for HIVE-3073 so we can see how this all fits together?
          Hide
          gangtimliu Gang Tim Liu added a comment -

          Good points. Work-in-progress patch might not be available within days. But will post to HIVE-3073 once available. should be soon.

          In the meanwhile, interface between DDL and DML is data structure in metastore. DML takes information saved by DDL and uses it. Hope the data structure can help to see how all fit together.

          Show
          gangtimliu Gang Tim Liu added a comment - Good points. Work-in-progress patch might not be available within days. But will post to HIVE-3073 once available. should be soon. In the meanwhile, interface between DDL and DML is data structure in metastore. DML takes information saved by DDL and uses it. Hope the data structure can help to see how all fit together.
          Hide
          namit Namit Jain added a comment -

          The changes look good to me.
          Will start tests

          Show
          namit Namit Jain added a comment - The changes look good to me. Will start tests
          Hide
          cwsteinbach Carl Steinbach added a comment -

          I left comments on phabricator. Thanks.

          Show
          cwsteinbach Carl Steinbach added a comment - I left comments on phabricator. Thanks.
          Hide
          gangtimliu Gang Tim Liu added a comment -

          thank you for providing comments. will address them and provide patch. thanks

          Show
          gangtimliu Gang Tim Liu added a comment - thank you for providing comments. will address them and provide patch. thanks
          Hide
          gangtimliu Gang Tim Liu added a comment -

          Patch available in phabricator and jira.

          Show
          gangtimliu Gang Tim Liu added a comment - Patch available in phabricator and jira.
          Hide
          cwsteinbach Carl Steinbach added a comment -

          @Tim: please see my comments on phabricator. Thanks.

          Show
          cwsteinbach Carl Steinbach added a comment - @Tim: please see my comments on phabricator. Thanks.
          Hide
          gangtimliu Gang Tim Liu added a comment -

          Patch is available on both jira and phabricator.

          Show
          gangtimliu Gang Tim Liu added a comment - Patch is available on both jira and phabricator.
          Hide
          cwsteinbach Carl Steinbach added a comment -

          @Tim: Please see my comments on phabricator. Thanks.

          Show
          cwsteinbach Carl Steinbach added a comment - @Tim: Please see my comments on phabricator. Thanks.
          Hide
          gangtimliu Gang Tim Liu added a comment -

          @Carl, thank you very much for quick review. will address them right now and get a patch tonight. thanks

          Show
          gangtimliu Gang Tim Liu added a comment - @Carl, thank you very much for quick review. will address them right now and get a patch tonight. thanks
          Hide
          gangtimliu Gang Tim Liu added a comment -

          patch is available on both jira and phabricator. thanks

          Show
          gangtimliu Gang Tim Liu added a comment - patch is available on both jira and phabricator. thanks
          Hide
          namit Namit Jain added a comment -

          Comments on phabricator.

          I think, we should limit this jira to skew join DDL and not try to club in
          hive code cleanup here.

          Show
          namit Namit Jain added a comment - Comments on phabricator. I think, we should limit this jira to skew join DDL and not try to club in hive code cleanup here.
          Hide
          gangtimliu Gang Tim Liu added a comment -

          Patch is ready for review. It's in both jira and phabricator.

          Show
          gangtimliu Gang Tim Liu added a comment - Patch is ready for review. It's in both jira and phabricator.
          Hide
          namit Namit Jain added a comment -

          some minor comments.

          Show
          namit Namit Jain added a comment - some minor comments.
          Hide
          namit Namit Jain added a comment -

          I had some minor comments on the patch.
          Otherwise, it looks good to me.

          @Carl, do you have any additional comments ?
          Otherwise, I will start testing once Tim has addressed the new comments.

          Show
          namit Namit Jain added a comment - I had some minor comments on the patch. Otherwise, it looks good to me. @Carl, do you have any additional comments ? Otherwise, I will start testing once Tim has addressed the new comments.
          Hide
          cwsteinbach Carl Steinbach added a comment -

          @Namit: I'll make another pass through the patch later today. One thing I'd like to request is that we add an internal configuration property that disables the new DDL by default. We can remove this once the rest of the DML changes get committed, but in the meantime I don't think it makes sense to make the DDL visible to users. So to recap, I'm proposing the following:

          • Add a configuration property named "hive.internal.ddl.list.bucketing.enable" and set the default value to false.
          • Add a comment in HiveConf explaining that this will be removed once the rest of the DML changes are committed.
          • Do not add this property to hive-default.xml.template since we don't want users messing with it.
          • Throw an error if the user tries to use the DDL with hive.internal.ddl.list.bucketing.enable set to false.
          Show
          cwsteinbach Carl Steinbach added a comment - @Namit: I'll make another pass through the patch later today. One thing I'd like to request is that we add an internal configuration property that disables the new DDL by default. We can remove this once the rest of the DML changes get committed, but in the meantime I don't think it makes sense to make the DDL visible to users. So to recap, I'm proposing the following: Add a configuration property named "hive.internal.ddl.list.bucketing.enable" and set the default value to false. Add a comment in HiveConf explaining that this will be removed once the rest of the DML changes are committed. Do not add this property to hive-default.xml.template since we don't want users messing with it. Throw an error if the user tries to use the DDL with hive.internal.ddl.list.bucketing.enable set to false.
          Hide
          gangtimliu Gang Tim Liu added a comment -

          Patch is available on both jara and phabricator.

          Show
          gangtimliu Gang Tim Liu added a comment - Patch is available on both jara and phabricator.
          Hide
          cwsteinbach Carl Steinbach added a comment -

          @Tim: Can you please add the configuration property to disable this DDL? Thanks.

          Show
          cwsteinbach Carl Steinbach added a comment - @Tim: Can you please add the configuration property to disable this DDL? Thanks.
          Hide
          gangtimliu Gang Tim Liu added a comment -

          @Carl, yes , will code after dinner. Thanks a lot

          Show
          gangtimliu Gang Tim Liu added a comment - @Carl, yes , will code after dinner. Thanks a lot
          Hide
          gangtimliu Gang Tim Liu added a comment -

          Patch is available.

          Show
          gangtimliu Gang Tim Liu added a comment - Patch is available.
          Hide
          namit Namit Jain added a comment -

          Committed. Thanks Tim

          Show
          namit Namit Jain added a comment - Committed. Thanks Tim
          Hide
          hudson Hudson added a comment -

          Integrated in Hive-trunk-h0.21 #1627 (See https://builds.apache.org/job/Hive-trunk-h0.21/1627/)
          HIVE-3072 Hive List Bucketing - DDL support
          (Gang Tim Liu via namit) (Revision 1377187)

          Result = FAILURE
          namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1377187
          Files :

          • /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
          • /hive/trunk/metastore/if/hive_metastore.thrift
          • /hive/trunk/metastore/scripts/upgrade/derby/010-HIVE-3072.derby.sql
          • /hive/trunk/metastore/scripts/upgrade/derby/hive-schema-0.10.0.derby.sql
          • /hive/trunk/metastore/scripts/upgrade/derby/upgrade-0.9.0-to-0.10.0.derby.sql
          • /hive/trunk/metastore/scripts/upgrade/mysql/010-HIVE-3072.mysql.sql
          • /hive/trunk/metastore/scripts/upgrade/mysql/hive-schema-0.10.0.mysql.sql
          • /hive/trunk/metastore/scripts/upgrade/mysql/upgrade-0.9.0-to-0.10.0.mysql.sql
          • /hive/trunk/metastore/scripts/upgrade/oracle/010-HIVE-3072.oracle.sql
          • /hive/trunk/metastore/scripts/upgrade/oracle/hive-schema-0.10.0.oracle.sql
          • /hive/trunk/metastore/scripts/upgrade/postgres/010-HIVE-3072.postgres.sql
          • /hive/trunk/metastore/scripts/upgrade/postgres/hive-schema-0.10.0.postgres.sql
          • /hive/trunk/metastore/scripts/upgrade/postgres/upgrade-0.9.0-to-0.10.0.postgres.sql
          • /hive/trunk/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.cpp
          • /hive/trunk/metastore/src/gen/thrift/gen-cpp/hive_metastore_types.cpp
          • /hive/trunk/metastore/src/gen/thrift/gen-cpp/hive_metastore_types.h
          • /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/EnvironmentContext.java
          • /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Index.java
          • /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Partition.java
          • /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Schema.java
          • /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SkewedInfo.java
          • /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/StorageDescriptor.java
          • /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Table.java
          • /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java
          • /hive/trunk/metastore/src/gen/thrift/gen-php/hive_metastore/ThriftHiveMetastore.php
          • /hive/trunk/metastore/src/gen/thrift/gen-php/hive_metastore/hive_metastore_types.php
          • /hive/trunk/metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py
          • /hive/trunk/metastore/src/gen/thrift/gen-py/hive_metastore/ttypes.py
          • /hive/trunk/metastore/src/gen/thrift/gen-rb/hive_metastore_types.rb
          • /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
          • /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java
          • /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
          • /hive/trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MStorageDescriptor.java
          • /hive/trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MStringList.java
          • /hive/trunk/metastore/src/model/package.jdo
          • /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnInfo.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/MetaDataFormatUtils.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ImportSemanticAnalyzer.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ParseContext.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ParseUtils.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/QB.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/CreateTableDesc.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeColumnDesc.java
          • /hive/trunk/ql/src/test/queries/clientnegative/column_rename5.q
          • /hive/trunk/ql/src/test/queries/clientnegative/create_skewed_table_col_name_value_no_mismatch.q
          • /hive/trunk/ql/src/test/queries/clientnegative/create_skewed_table_dup_col_name.q
          • /hive/trunk/ql/src/test/queries/clientnegative/create_skewed_table_failure_invalid_col_name.q
          • /hive/trunk/ql/src/test/queries/clientpositive/create_skewed_table.q
          • /hive/trunk/ql/src/test/results/clientnegative/column_rename5.q.out
          • /hive/trunk/ql/src/test/results/clientnegative/create_skewed_table_col_name_value_no_mismatch.q.out
          • /hive/trunk/ql/src/test/results/clientnegative/create_skewed_table_dup_col_name.q.out
          • /hive/trunk/ql/src/test/results/clientnegative/create_skewed_table_failure_invalid_col_name.q.out
          • /hive/trunk/ql/src/test/results/clientpositive/create_skewed_table.q.out
          Show
          hudson Hudson added a comment - Integrated in Hive-trunk-h0.21 #1627 (See https://builds.apache.org/job/Hive-trunk-h0.21/1627/ ) HIVE-3072 Hive List Bucketing - DDL support (Gang Tim Liu via namit) (Revision 1377187) Result = FAILURE namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1377187 Files : /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java /hive/trunk/metastore/if/hive_metastore.thrift /hive/trunk/metastore/scripts/upgrade/derby/010- HIVE-3072 .derby.sql /hive/trunk/metastore/scripts/upgrade/derby/hive-schema-0.10.0.derby.sql /hive/trunk/metastore/scripts/upgrade/derby/upgrade-0.9.0-to-0.10.0.derby.sql /hive/trunk/metastore/scripts/upgrade/mysql/010- HIVE-3072 .mysql.sql /hive/trunk/metastore/scripts/upgrade/mysql/hive-schema-0.10.0.mysql.sql /hive/trunk/metastore/scripts/upgrade/mysql/upgrade-0.9.0-to-0.10.0.mysql.sql /hive/trunk/metastore/scripts/upgrade/oracle/010- HIVE-3072 .oracle.sql /hive/trunk/metastore/scripts/upgrade/oracle/hive-schema-0.10.0.oracle.sql /hive/trunk/metastore/scripts/upgrade/postgres/010- HIVE-3072 .postgres.sql /hive/trunk/metastore/scripts/upgrade/postgres/hive-schema-0.10.0.postgres.sql /hive/trunk/metastore/scripts/upgrade/postgres/upgrade-0.9.0-to-0.10.0.postgres.sql /hive/trunk/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.cpp /hive/trunk/metastore/src/gen/thrift/gen-cpp/hive_metastore_types.cpp /hive/trunk/metastore/src/gen/thrift/gen-cpp/hive_metastore_types.h /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/EnvironmentContext.java /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Index.java /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Partition.java /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Schema.java /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SkewedInfo.java /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/StorageDescriptor.java /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Table.java /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java /hive/trunk/metastore/src/gen/thrift/gen-php/hive_metastore/ThriftHiveMetastore.php /hive/trunk/metastore/src/gen/thrift/gen-php/hive_metastore/hive_metastore_types.php /hive/trunk/metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py /hive/trunk/metastore/src/gen/thrift/gen-py/hive_metastore/ttypes.py /hive/trunk/metastore/src/gen/thrift/gen-rb/hive_metastore_types.rb /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java /hive/trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MStorageDescriptor.java /hive/trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MStringList.java /hive/trunk/metastore/src/model/package.jdo /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnInfo.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/MetaDataFormatUtils.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ImportSemanticAnalyzer.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ParseContext.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ParseUtils.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/QB.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/CreateTableDesc.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeColumnDesc.java /hive/trunk/ql/src/test/queries/clientnegative/column_rename5.q /hive/trunk/ql/src/test/queries/clientnegative/create_skewed_table_col_name_value_no_mismatch.q /hive/trunk/ql/src/test/queries/clientnegative/create_skewed_table_dup_col_name.q /hive/trunk/ql/src/test/queries/clientnegative/create_skewed_table_failure_invalid_col_name.q /hive/trunk/ql/src/test/queries/clientpositive/create_skewed_table.q /hive/trunk/ql/src/test/results/clientnegative/column_rename5.q.out /hive/trunk/ql/src/test/results/clientnegative/create_skewed_table_col_name_value_no_mismatch.q.out /hive/trunk/ql/src/test/results/clientnegative/create_skewed_table_dup_col_name.q.out /hive/trunk/ql/src/test/results/clientnegative/create_skewed_table_failure_invalid_col_name.q.out /hive/trunk/ql/src/test/results/clientpositive/create_skewed_table.q.out
          Hide
          hudson Hudson added a comment -

          Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/)
          HIVE-3072 Hive List Bucketing - DDL support
          (Gang Tim Liu via namit) (Revision 1377187)

          Result = ABORTED
          namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1377187
          Files :

          • /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
          • /hive/trunk/metastore/if/hive_metastore.thrift
          • /hive/trunk/metastore/scripts/upgrade/derby/010-HIVE-3072.derby.sql
          • /hive/trunk/metastore/scripts/upgrade/derby/hive-schema-0.10.0.derby.sql
          • /hive/trunk/metastore/scripts/upgrade/derby/upgrade-0.9.0-to-0.10.0.derby.sql
          • /hive/trunk/metastore/scripts/upgrade/mysql/010-HIVE-3072.mysql.sql
          • /hive/trunk/metastore/scripts/upgrade/mysql/hive-schema-0.10.0.mysql.sql
          • /hive/trunk/metastore/scripts/upgrade/mysql/upgrade-0.9.0-to-0.10.0.mysql.sql
          • /hive/trunk/metastore/scripts/upgrade/oracle/010-HIVE-3072.oracle.sql
          • /hive/trunk/metastore/scripts/upgrade/oracle/hive-schema-0.10.0.oracle.sql
          • /hive/trunk/metastore/scripts/upgrade/postgres/010-HIVE-3072.postgres.sql
          • /hive/trunk/metastore/scripts/upgrade/postgres/hive-schema-0.10.0.postgres.sql
          • /hive/trunk/metastore/scripts/upgrade/postgres/upgrade-0.9.0-to-0.10.0.postgres.sql
          • /hive/trunk/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.cpp
          • /hive/trunk/metastore/src/gen/thrift/gen-cpp/hive_metastore_types.cpp
          • /hive/trunk/metastore/src/gen/thrift/gen-cpp/hive_metastore_types.h
          • /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/EnvironmentContext.java
          • /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Index.java
          • /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Partition.java
          • /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Schema.java
          • /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SkewedInfo.java
          • /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/StorageDescriptor.java
          • /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Table.java
          • /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java
          • /hive/trunk/metastore/src/gen/thrift/gen-php/hive_metastore/ThriftHiveMetastore.php
          • /hive/trunk/metastore/src/gen/thrift/gen-php/hive_metastore/hive_metastore_types.php
          • /hive/trunk/metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py
          • /hive/trunk/metastore/src/gen/thrift/gen-py/hive_metastore/ttypes.py
          • /hive/trunk/metastore/src/gen/thrift/gen-rb/hive_metastore_types.rb
          • /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
          • /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java
          • /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
          • /hive/trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MStorageDescriptor.java
          • /hive/trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MStringList.java
          • /hive/trunk/metastore/src/model/package.jdo
          • /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnInfo.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/MetaDataFormatUtils.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ImportSemanticAnalyzer.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ParseContext.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ParseUtils.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/QB.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/CreateTableDesc.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeColumnDesc.java
          • /hive/trunk/ql/src/test/queries/clientnegative/column_rename5.q
          • /hive/trunk/ql/src/test/queries/clientnegative/create_skewed_table_col_name_value_no_mismatch.q
          • /hive/trunk/ql/src/test/queries/clientnegative/create_skewed_table_dup_col_name.q
          • /hive/trunk/ql/src/test/queries/clientnegative/create_skewed_table_failure_invalid_col_name.q
          • /hive/trunk/ql/src/test/queries/clientpositive/create_skewed_table.q
          • /hive/trunk/ql/src/test/results/clientnegative/column_rename5.q.out
          • /hive/trunk/ql/src/test/results/clientnegative/create_skewed_table_col_name_value_no_mismatch.q.out
          • /hive/trunk/ql/src/test/results/clientnegative/create_skewed_table_dup_col_name.q.out
          • /hive/trunk/ql/src/test/results/clientnegative/create_skewed_table_failure_invalid_col_name.q.out
          • /hive/trunk/ql/src/test/results/clientpositive/create_skewed_table.q.out
          Show
          hudson Hudson added a comment - Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/ ) HIVE-3072 Hive List Bucketing - DDL support (Gang Tim Liu via namit) (Revision 1377187) Result = ABORTED namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1377187 Files : /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java /hive/trunk/metastore/if/hive_metastore.thrift /hive/trunk/metastore/scripts/upgrade/derby/010- HIVE-3072 .derby.sql /hive/trunk/metastore/scripts/upgrade/derby/hive-schema-0.10.0.derby.sql /hive/trunk/metastore/scripts/upgrade/derby/upgrade-0.9.0-to-0.10.0.derby.sql /hive/trunk/metastore/scripts/upgrade/mysql/010- HIVE-3072 .mysql.sql /hive/trunk/metastore/scripts/upgrade/mysql/hive-schema-0.10.0.mysql.sql /hive/trunk/metastore/scripts/upgrade/mysql/upgrade-0.9.0-to-0.10.0.mysql.sql /hive/trunk/metastore/scripts/upgrade/oracle/010- HIVE-3072 .oracle.sql /hive/trunk/metastore/scripts/upgrade/oracle/hive-schema-0.10.0.oracle.sql /hive/trunk/metastore/scripts/upgrade/postgres/010- HIVE-3072 .postgres.sql /hive/trunk/metastore/scripts/upgrade/postgres/hive-schema-0.10.0.postgres.sql /hive/trunk/metastore/scripts/upgrade/postgres/upgrade-0.9.0-to-0.10.0.postgres.sql /hive/trunk/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.cpp /hive/trunk/metastore/src/gen/thrift/gen-cpp/hive_metastore_types.cpp /hive/trunk/metastore/src/gen/thrift/gen-cpp/hive_metastore_types.h /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/EnvironmentContext.java /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Index.java /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Partition.java /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Schema.java /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SkewedInfo.java /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/StorageDescriptor.java /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Table.java /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java /hive/trunk/metastore/src/gen/thrift/gen-php/hive_metastore/ThriftHiveMetastore.php /hive/trunk/metastore/src/gen/thrift/gen-php/hive_metastore/hive_metastore_types.php /hive/trunk/metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py /hive/trunk/metastore/src/gen/thrift/gen-py/hive_metastore/ttypes.py /hive/trunk/metastore/src/gen/thrift/gen-rb/hive_metastore_types.rb /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java /hive/trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MStorageDescriptor.java /hive/trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MStringList.java /hive/trunk/metastore/src/model/package.jdo /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnInfo.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/MetaDataFormatUtils.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ImportSemanticAnalyzer.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ParseContext.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ParseUtils.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/QB.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/CreateTableDesc.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeColumnDesc.java /hive/trunk/ql/src/test/queries/clientnegative/column_rename5.q /hive/trunk/ql/src/test/queries/clientnegative/create_skewed_table_col_name_value_no_mismatch.q /hive/trunk/ql/src/test/queries/clientnegative/create_skewed_table_dup_col_name.q /hive/trunk/ql/src/test/queries/clientnegative/create_skewed_table_failure_invalid_col_name.q /hive/trunk/ql/src/test/queries/clientpositive/create_skewed_table.q /hive/trunk/ql/src/test/results/clientnegative/column_rename5.q.out /hive/trunk/ql/src/test/results/clientnegative/create_skewed_table_col_name_value_no_mismatch.q.out /hive/trunk/ql/src/test/results/clientnegative/create_skewed_table_dup_col_name.q.out /hive/trunk/ql/src/test/results/clientnegative/create_skewed_table_failure_invalid_col_name.q.out /hive/trunk/ql/src/test/results/clientpositive/create_skewed_table.q.out
          Hide
          ashutoshc Ashutosh Chauhan added a comment -

          This issue is fixed and released as part of 0.10.0 release. If you find an issue which seems to be related to this one, please create a new jira and link this one with new jira.

          Show
          ashutoshc Ashutosh Chauhan added a comment - This issue is fixed and released as part of 0.10.0 release. If you find an issue which seems to be related to this one, please create a new jira and link this one with new jira.
          Hide
          leftylev Lefty Leverenz added a comment -

          This is documented in the wiki here:

          Show
          leftylev Lefty Leverenz added a comment - This is documented in the wiki here: Language Manual – DDL – Skewed Tables

            People

            • Assignee:
              gangtimliu Gang Tim Liu
              Reporter:
              gangtimliu Gang Tim Liu
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development