Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.10.0
    • Component/s: Statistics
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    1. HIVE-1362-gen_thrift.6.patch.txt
      2.36 MB
      Shreepadma Venugopalan
    2. HIVE-1362-gen_thrift.5.patch.txt
      2.35 MB
      Shreepadma Venugopalan
    3. HIVE-1362-gen_thrift.4.patch.txt
      2.34 MB
      Shreepadma Venugopalan
    4. HIVE-1362-gen_thrift.3.patch.txt
      2.34 MB
      Shreepadma Venugopalan
    5. HIVE-1362-gen_thrift.2.patch.txt
      2.34 MB
      Shreepadma Venugopalan
    6. HIVE-1362-gen_thrift.1.patch.txt
      1.61 MB
      Shreepadma Venugopalan
    7. HIVE-1362.D6339.1.patch
      269 kB
      Phabricator
    8. HIVE-1362.9.patch.txt
      317 kB
      Shreepadma Venugopalan
    9. HIVE-1362.8.patch.txt
      280 kB
      Shreepadma Venugopalan
    10. HIVE-1362.7.patch.txt
      275 kB
      Shreepadma Venugopalan
    11. HIVE-1362.6.patch.txt
      280 kB
      Shreepadma Venugopalan
    12. HIVE-1362.5.patch.txt
      266 kB
      Shreepadma Venugopalan
    13. HIVE-1362.4.patch.txt
      261 kB
      Shreepadma Venugopalan
    14. HIVE-1362.3.patch.txt
      261 kB
      Shreepadma Venugopalan
    15. HIVE-1362.2.patch.txt
      264 kB
      Shreepadma Venugopalan
    16. HIVE-1362.11.patch.txt
      3 kB
      Shreepadma Venugopalan
    17. HIVE-1362.10.patch.txt
      320 kB
      Shreepadma Venugopalan
    18. HIVE-1362.1.patch.txt
      243 kB
      Shreepadma Venugopalan
    19. HIVE-1362_gen-thrift.9.patch.txt
      2.40 MB
      Shreepadma Venugopalan
    20. HIVE-1362_gen-thrift.8.patch.txt
      2.36 MB
      Shreepadma Venugopalan
    21. HIVE-1362_gen-thrift.7.patch.txt
      2.36 MB
      Shreepadma Venugopalan
    22. HIVE-1362_gen-thrift.10.patch.txt
      2.38 MB
      Shreepadma Venugopalan

      Issue Links

        Activity

        Ning Zhang created issue -
        Hide
        Ning Zhang added a comment -

        This is the 2nd subtask of HIVE-33 (stats in Hive tables).

        We will gather column level stats based on users' request. It also depends on HIVE-1361 in that the metastore API should suport storing and retrieving stats.

        The major milestone for this subtasks are:
        1) add a new HiveQL command to gather column level stats. Please see HIVE-33 for the syntax.
        2) add new UDFs/UDAFs to compute these statistics.

        The proposed statistics are:

        • number of distinct values
        • number of NULL values
        • min/max k values where k could be given by user
        • histogram: frequency and height balanced
        • average size of the column
        • avg/sum of all values in the column if their type is numerical
        • percentiles of the value
        Show
        Ning Zhang added a comment - This is the 2nd subtask of HIVE-33 (stats in Hive tables). We will gather column level stats based on users' request. It also depends on HIVE-1361 in that the metastore API should suport storing and retrieving stats. The major milestone for this subtasks are: 1) add a new HiveQL command to gather column level stats. Please see HIVE-33 for the syntax. 2) add new UDFs/UDAFs to compute these statistics. The proposed statistics are: number of distinct values number of NULL values min/max k values where k could be given by user histogram: frequency and height balanced average size of the column avg/sum of all values in the column if their type is numerical percentiles of the value
        Jeff Hammerbacher made changes -
        Field Original Value New Value
        Link This issue is part of HIVE-33 [ HIVE-33 ]
        Hide
        Ning Zhang added a comment -

        Good catch Carl. Yeah, it won't work for HiveServer.

        I'm uploading a new patch taking your 2 comments.

        Show
        Ning Zhang added a comment - Good catch Carl. Yeah, it won't work for HiveServer. I'm uploading a new patch taking your 2 comments.
        Ning Zhang made changes -
        Attachment HIVE-1937.2.patch [ 12469890 ]
        Hide
        Ning Zhang added a comment -

        Sorry, please ignore my previous comment. It should goes to HIVE-1937.

        Show
        Ning Zhang added a comment - Sorry, please ignore my previous comment. It should goes to HIVE-1937 .
        Ning Zhang made changes -
        Attachment HIVE-1937.2.patch [ 12469890 ]
        bharath v made changes -
        Link This issue is blocked by HIVE-1938 [ HIVE-1938 ]
        bharath v made changes -
        Link This issue is blocked by HIVE-1938 [ HIVE-1938 ]
        bharath v made changes -
        Link This issue blocks HIVE-1938 [ HIVE-1938 ]
        Mark Thomas made changes -
        Assignee Ahmed M Aly [ ahmedaly ]
        Hide
        Namit Jain added a comment -

        Is anyone working on this ?

        Show
        Namit Jain added a comment - Is anyone working on this ?
        Carl Steinbach made changes -
        Component/s Statistics [ 12314312 ]
        Namit Jain made changes -
        Labels gsoc gsoc2012
        Hide
        Ashutosh Chauhan added a comment -

        @Namit,
        Bharath plans to work on this and has a gsoc proposal : http://www.google-melange.com/gsoc/proposal/review/google/gsoc2012/bharathv/18002

        Show
        Ashutosh Chauhan added a comment - @Namit, Bharath plans to work on this and has a gsoc proposal : http://www.google-melange.com/gsoc/proposal/review/google/gsoc2012/bharathv/18002
        Carl Steinbach made changes -
        Link This issue is duplicated by HIVE-1940 [ HIVE-1940 ]
        Carl Steinbach made changes -
        Labels gsoc gsoc2012
        Assignee Shreepadma Venugopalan [ shreepadma ]
        Hide
        Feng Lu added a comment -

        A patch for computing column level top k values is attached. Review is available at https://reviews.facebook.net/D5073 .

        Show
        Feng Lu added a comment - A patch for computing column level top k values is attached. Review is available at https://reviews.facebook.net/D5073 .
        Feng Lu made changes -
        Attachment HIVE-1362.patch.txt [ 12543309 ]
        Feng Lu made changes -
        Attachment HIVE-1362.patch.txt [ 12543313 ]
        Feng Lu made changes -
        Attachment HIVE-1362.patch.txt [ 12543313 ]
        Hide
        Carl Steinbach added a comment -

        @Feng: Please open a new JIRA for your patch and post it there. This ticket covers the task of implementing a general framework for computing and persisting column-level statistics. It doesn't look like your patch does that. Thanks.

        Show
        Carl Steinbach added a comment - @Feng: Please open a new JIRA for your patch and post it there. This ticket covers the task of implementing a general framework for computing and persisting column-level statistics. It doesn't look like your patch does that. Thanks.
        Feng Lu made changes -
        Attachment HIVE-1362.patch.txt [ 12543309 ]
        Hide
        Feng Lu added a comment -

        @Carl, created a new JIRA at https://issues.apache.org/jira/browse/HIVE-3421 .

        Show
        Feng Lu added a comment - @Carl, created a new JIRA at https://issues.apache.org/jira/browse/HIVE-3421 .
        Shreepadma Venugopalan made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Shreepadma Venugopalan made changes -
        Attachment HIVE-1362.1.patch.txt [ 12543349 ]
        Shreepadma Venugopalan made changes -
        Attachment HIVE-1362-gen_thrift.1.patch.txt [ 12543350 ]
        Hide
        Shreepadma Venugopalan added a comment -

        This patch implements version 1 of the column statistics project in Hive. It adds support for computing and persisting statistical summary of column values in Hive Tables and Partitions. In order to support column statistics in Hive, this patch does the following,

        • Adds a new compute stats UDAF to compute scalar statistics for all primitive Hive data types. In version 1 of the project, we support the following scalar statistics on primitive types - estimate of number of distinct values, number of null values, number of trues/falses for boolean typed columsn, max and avg length for string and binary typed columns, max and min value for long and double typed columns. Note that version 1 of the column stats project includes support for column statistics both at the table and partition level.
        • Adds Metastore schema tables to persist the newly added statistics both at table and partition level.
        • Adds Metastore Thrift API to persist, retrieve and delete column statistics at both table and partition level.
          Please refer to the following wiki link for the details of the schema and the Thrift API changes - https://cwiki.apache.org/confluence/display/Hive/Column+Statistics+in+Hive
        • Extends the analyze table compute statistics statement to trigger statistics computation and persistence for one or more columns. Please note that statistics for multiple columns is computed through a single scan of the table data. Please refer to the following wiki link for the syntax changes - https://cwiki.apache.org/confluence/display/Hive/Column+Statistics+in+Hive

        One thing missing from the patch at this point is the metastore upgrade scrips for MySQL/Derby/Postgres/Oracle. I'm waiting for the review to finalize the metastore schema changes before I go ahead and add the upgrade scripts.

        In a follow on patch, as part of version 2 of the column statistics project, we will add support for computing, persisting and retrieving histograms on long and double typed column values.

        Show
        Shreepadma Venugopalan added a comment - This patch implements version 1 of the column statistics project in Hive. It adds support for computing and persisting statistical summary of column values in Hive Tables and Partitions. In order to support column statistics in Hive, this patch does the following, Adds a new compute stats UDAF to compute scalar statistics for all primitive Hive data types. In version 1 of the project, we support the following scalar statistics on primitive types - estimate of number of distinct values, number of null values, number of trues/falses for boolean typed columsn, max and avg length for string and binary typed columns, max and min value for long and double typed columns. Note that version 1 of the column stats project includes support for column statistics both at the table and partition level. Adds Metastore schema tables to persist the newly added statistics both at table and partition level. Adds Metastore Thrift API to persist, retrieve and delete column statistics at both table and partition level. Please refer to the following wiki link for the details of the schema and the Thrift API changes - https://cwiki.apache.org/confluence/display/Hive/Column+Statistics+in+Hive Extends the analyze table compute statistics statement to trigger statistics computation and persistence for one or more columns. Please note that statistics for multiple columns is computed through a single scan of the table data. Please refer to the following wiki link for the syntax changes - https://cwiki.apache.org/confluence/display/Hive/Column+Statistics+in+Hive One thing missing from the patch at this point is the metastore upgrade scrips for MySQL/Derby/Postgres/Oracle. I'm waiting for the review to finalize the metastore schema changes before I go ahead and add the upgrade scripts. In a follow on patch, as part of version 2 of the column statistics project, we will add support for computing, persisting and retrieving histograms on long and double typed column values.
        Hide
        Shreepadma Venugopalan added a comment -

        Review is available at : https://reviews.apache.org/r/6878/

        Show
        Shreepadma Venugopalan added a comment - Review is available at : https://reviews.apache.org/r/6878/
        Hide
        Carl Steinbach added a comment -

        @Shreepadma: please see my comments on reviewboard.

        Show
        Carl Steinbach added a comment - @Shreepadma: please see my comments on reviewboard.
        Carl Steinbach made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Shreepadma Venugopalan made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Shreepadma Venugopalan made changes -
        Attachment HIVE-1362-gen_thrift.2.patch.txt [ 12547483 ]
        Hide
        Shreepadma Venugopalan added a comment -

        I've provide a review board link shortly.

        Show
        Shreepadma Venugopalan added a comment - I've provide a review board link shortly.
        Hide
        Shreepadma Venugopalan added a comment -

        Meant to say I'll provide a review board link shortly.

        Show
        Shreepadma Venugopalan added a comment - Meant to say I'll provide a review board link shortly.
        Shreepadma Venugopalan made changes -
        Attachment HIVE-1362.2.patch.txt [ 12547488 ]
        Hide
        Shreepadma Venugopalan added a comment -

        Patch is available for review at : https://reviews.apache.org/r/6878/

        Show
        Shreepadma Venugopalan added a comment - Patch is available for review at :  https://reviews.apache.org/r/6878/
        Hide
        Shreepadma Venugopalan added a comment -
        Show
        Shreepadma Venugopalan added a comment - Latest revision is here: https://reviews.apache.org/r/6878/diff/#index_header
        Hide
        Namit Jain added a comment -

        Are the stats collected while the table is being scanned, or is it part of analyze only ?

        Show
        Namit Jain added a comment - Are the stats collected while the table is being scanned, or is it part of analyze only ?
        Hide
        Namit Jain added a comment -

        Shreepadma, some comments on reviewboard

        Show
        Namit Jain added a comment - Shreepadma, some comments on reviewboard
        Namit Jain made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Hide
        Shreepadma Venugopalan added a comment -

        @Namit: The stats is collected as part of analyze. We will look into implicit stats collection i.e., when the table is scanned/loaded in next version of this project.

        Show
        Shreepadma Venugopalan added a comment - @Namit: The stats is collected as part of analyze. We will look into implicit stats collection i.e., when the table is scanned/loaded in next version of this project.
        Shreepadma Venugopalan made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Shreepadma Venugopalan made changes -
        Attachment HIVE-1362.3.patch.txt [ 12547580 ]
        Hide
        Shreepadma Venugopalan added a comment -

        Latest revision which addresses Namit's comments is on review board.

        Show
        Shreepadma Venugopalan added a comment - Latest revision which addresses Namit's comments is on review board.
        Shreepadma Venugopalan made changes -
        Attachment HIVE-1362-gen_thrift.3.patch.txt [ 12547582 ]
        Hide
        shrikanth shankar added a comment -

        I had a couple of high level comments on the patch that seem to fit better here rather than on the review board. Apologies if this violates protocol
        (1) The count_stats aggregation operator 'repeats' many existing aggregates that Hive already supports (count of nulls, count true's, max, min etc). It might make a lot more sense to just add an aggregate to return the approximate number of distinct values for a column. Any reason why stats collection cant just generate more expressions in the SQL?
        (2) There might even be value in adding a different UDAF which just returns a serialized numDV estimator. Storing this (instead of the count) could be useful in other places e.g. combining numDV estimates across partitions (A second UDAF would be needed to support aggregating these but that seems easy)

        Show
        shrikanth shankar added a comment - I had a couple of high level comments on the patch that seem to fit better here rather than on the review board. Apologies if this violates protocol (1) The count_stats aggregation operator 'repeats' many existing aggregates that Hive already supports (count of nulls, count true's, max, min etc). It might make a lot more sense to just add an aggregate to return the approximate number of distinct values for a column. Any reason why stats collection cant just generate more expressions in the SQL? (2) There might even be value in adding a different UDAF which just returns a serialized numDV estimator. Storing this (instead of the count) could be useful in other places e.g. combining numDV estimates across partitions (A second UDAF would be needed to support aggregating these but that seems easy)
        Hide
        Namit Jain added a comment -

        Questions on the jira ?

        Show
        Namit Jain added a comment - Questions on the jira ?
        Namit Jain made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Shreepadma Venugopalan made changes -
        Attachment HIVE-1362.4.patch.txt [ 12547683 ]
        Shreepadma Venugopalan made changes -
        Attachment HIVE-1362-gen_thrift.4.patch.txt [ 12547684 ]
        Hide
        Shreepadma Venugopalan added a comment -

        @Shrikanth: Thank you for your comments. We can certainly add a new UDAF with the Flajolet-Martin sketch that returns a serialized numDV estimator. I've already filed a new JIRA (HIVE-3516) for the incremental stats computation work. I'll add the UDAF as part of that JIRA.

        A couple of reasons why we decided to create a new compute_stats aggregation operator instead of generating more expressions in the SQL,

        1. We felt its a lot cleaner to encapsulate the stats for a column within a single UDAF. The compute_stats UDAF returns a struct with the relevant stats depending on the data type of the column and as a result makes the parsing as well as the SQL we generate simple.

        2. Adding a new compute_stats UDAF allows the gathering of statistical summaries of the underlying data even outside of the column stats framework. One use I can think of is, it can be used to model the statistical properties of data which in turn can be used to generate data whose statistical properties mimic that of the underlying data.

        Even though max, min, total count exist as UDAFs today, we need these to be part of the histogram UDAF. Estimating quantiles for equi-height histogram is a lot more efficient if we know the range of values the column can take. We need to know the total_count to generate the histogram bins. Given that we need these stats for generating histograms, I think its a good idea to encapsulate all of these stats within the compute_stats UDAF. Thanks.

        Show
        Shreepadma Venugopalan added a comment - @Shrikanth: Thank you for your comments. We can certainly add a new UDAF with the Flajolet-Martin sketch that returns a serialized numDV estimator. I've already filed a new JIRA ( HIVE-3516 ) for the incremental stats computation work. I'll add the UDAF as part of that JIRA. A couple of reasons why we decided to create a new compute_stats aggregation operator instead of generating more expressions in the SQL, 1. We felt its a lot cleaner to encapsulate the stats for a column within a single UDAF. The compute_stats UDAF returns a struct with the relevant stats depending on the data type of the column and as a result makes the parsing as well as the SQL we generate simple. 2. Adding a new compute_stats UDAF allows the gathering of statistical summaries of the underlying data even outside of the column stats framework. One use I can think of is, it can be used to model the statistical properties of data which in turn can be used to generate data whose statistical properties mimic that of the underlying data. Even though max, min, total count exist as UDAFs today, we need these to be part of the histogram UDAF. Estimating quantiles for equi-height histogram is a lot more efficient if we know the range of values the column can take. We need to know the total_count to generate the histogram bins. Given that we need these stats for generating histograms, I think its a good idea to encapsulate all of these stats within the compute_stats UDAF. Thanks.
        Shreepadma Venugopalan made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Shreepadma Venugopalan added a comment -

        @Namit: I've addressed your review comments in the latest revision to the patch. Its available on review board.

        Show
        Shreepadma Venugopalan added a comment - @Namit: I've addressed your review comments in the latest revision to the patch. Its available on review board.
        Carl Steinbach made changes -
        Link This issue relates HIVE-3516 [ HIVE-3516 ]
        Hide
        Namit Jain added a comment -

        Would be easier to comment here:

        >> It does not interact with hive.stats.reliable.

        Can you fix that ? I mean, if you want to do it in a follow-up that's fine. But, please throw an error in that case if hive.stats.reliable
        is set to true for now.

        >>> Why does it make sense to add a thrift API for updating statistics ? There doesn't exist a interface for updating
        row level statistics. How is the user supposed to compute these other than analyze, which anyway updates the stats.

        Show
        Namit Jain added a comment - Would be easier to comment here: >> It does not interact with hive.stats.reliable. Can you fix that ? I mean, if you want to do it in a follow-up that's fine. But, please throw an error in that case if hive.stats.reliable is set to true for now. >>> Why does it make sense to add a thrift API for updating statistics ? There doesn't exist a interface for updating row level statistics. How is the user supposed to compute these other than analyze, which anyway updates the stats.
        Hide
        Shreepadma Venugopalan added a comment -

        @Namit: I don't understand hive.stats.reliable very clearly. Can you please explain how hive.stats.reliable works? What are the semantics of hive.stats.reliable? Why do we need hive.stats.reliable? Thanks.

        Show
        Shreepadma Venugopalan added a comment - @Namit: I don't understand hive.stats.reliable very clearly. Can you please explain how hive.stats.reliable works? What are the semantics of hive.stats.reliable? Why do we need hive.stats.reliable? Thanks.
        Hide
        Shreepadma Venugopalan added a comment -

        I assume when you say row level statistics you are referring to table statistics. Today, table statistics is stored as part of the table_params. table_params table gets mapped to the TTable object in memory and it looks like the existing APIs sufficed. We want to have a dedicated Thrift API for column stats for the following reasons,

        1. Column statistics is a property of the column and not the table and hence doesn't belong with the table_params. Furthermore, we have seen customers with tables that are 100s-1000s of columns wide. Storing this information as a table_param is going to bloat, and it will also make the output of DESCRIBE EXTENDED unreadable.

        2. We want column statistics to be a first class metadata. In order to do so, we have to provide dedicated Thrift APIs to query and update it. We want the Thrift API to be self-documenting, i.e. if someone tells you that metastore supports column stats, you should be able to look at the Thrift IDL and figure out which method you need to use to store/retrieve column stats. Right now a lot of the API doesn't satisfy that goal since many methods are overloaded, and other features are implemented by adding new key/value properties to different catalog objects that aren't easy to document via the thrift API

        3. Additionally storing column statistics as a key/value pair in the table_params table is not space efficient. We need to repeat the keys for each one of the columns in the table for which statistics is gathered. Furthermore, by storing column stats in the table_params table we would de-normalize the schema completely and incur a performance penalty performing self-joins, though not necessarily in the metasote db, to retrieve the statistics associated with a column.

        Show
        Shreepadma Venugopalan added a comment - I assume when you say row level statistics you are referring to table statistics. Today, table statistics is stored as part of the table_params. table_params table gets mapped to the TTable object in memory and it looks like the existing APIs sufficed. We want to have a dedicated Thrift API for column stats for the following reasons, 1. Column statistics is a property of the column and not the table and hence doesn't belong with the table_params. Furthermore, we have seen customers with tables that are 100s-1000s of columns wide. Storing this information as a table_param is going to bloat, and it will also make the output of DESCRIBE EXTENDED unreadable. 2. We want column statistics to be a first class metadata. In order to do so, we have to provide dedicated Thrift APIs to query and update it. We want the Thrift API to be self-documenting, i.e. if someone tells you that metastore supports column stats, you should be able to look at the Thrift IDL and figure out which method you need to use to store/retrieve column stats. Right now a lot of the API doesn't satisfy that goal since many methods are overloaded, and other features are implemented by adding new key/value properties to different catalog objects that aren't easy to document via the thrift API 3. Additionally storing column statistics as a key/value pair in the table_params table is not space efficient. We need to repeat the keys for each one of the columns in the table for which statistics is gathered. Furthermore, by storing column stats in the table_params table we would de-normalize the schema completely and incur a performance penalty performing self-joins, though not necessarily in the metasote db, to retrieve the statistics associated with a column.
        Hide
        Shreepadma Venugopalan added a comment -

        @Namit: In the first version of the project, which this patch implements, the only way to trigger stats gathering is through an explicit ANALYZE command.

        Show
        Shreepadma Venugopalan added a comment - @Namit: In the first version of the project, which this patch implements, the only way to trigger stats gathering is through an explicit ANALYZE command.
        Hide
        Carl Steinbach added a comment -

        @Shreepadma: More comments on RB. Thanks.

        Show
        Carl Steinbach added a comment - @Shreepadma: More comments on RB. Thanks.
        Carl Steinbach made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Shreepadma Venugopalan made changes -
        Link This issue is depended upon by HIVE-3602 [ HIVE-3602 ]
        Shreepadma Venugopalan made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Shreepadma Venugopalan made changes -
        Attachment HIVE-1362.5.patch.txt [ 12550248 ]
        Shreepadma Venugopalan made changes -
        Attachment HIVE-1362-gen_thrift.5.patch.txt [ 12550249 ]
        Hide
        Shreepadma Venugopalan added a comment -

        Updated patch available on both JIRA and review board.

        Show
        Shreepadma Venugopalan added a comment - Updated patch available on both JIRA and review board.
        Hide
        Carl Steinbach added a comment -

        +1. Will commit if tests pass.

        @Namit: Let me know if you need more time to review this. If so I'll hold off until you're ready. Thanks.

        Show
        Carl Steinbach added a comment - +1. Will commit if tests pass. @Namit: Let me know if you need more time to review this. If so I'll hold off until you're ready. Thanks.
        Shreepadma Venugopalan made changes -
        Attachment HIVE-1362.6.patch.txt [ 12551296 ]
        Shreepadma Venugopalan made changes -
        Attachment HIVE-1362-gen_thrift.6.patch.txt [ 12551297 ]
        Hide
        Shreepadma Venugopalan added a comment -

        Attached new patch rebased off the tip of current trunk.

        Show
        Shreepadma Venugopalan added a comment - Attached new patch rebased off the tip of current trunk.
        Hide
        Namit Jain added a comment -

        @Carl, Can you hold off for just 1 day ? I will definitely take a look today.

        Show
        Namit Jain added a comment - @Carl, Can you hold off for just 1 day ? I will definitely take a look today.
        Hide
        Namit Jain added a comment - - edited

        https://reviews.facebook.net/D6321

        Created the phabricator entry for easy review from HIVE-1362.6.patch

        Show
        Namit Jain added a comment - - edited https://reviews.facebook.net/D6321 Created the phabricator entry for easy review from HIVE-1362 .6.patch
        Hide
        Namit Jain added a comment -

        Looks like this patch file is old.
        I couldn't see any reference to StatsSemanticAnalyzer.java.
        Can you load the latest patch file ?

        If you dont mind, can you create a phabricator entry ? I know, there is no official policy, but phabricator is much more usable.

        Show
        Namit Jain added a comment - Looks like this patch file is old. I couldn't see any reference to StatsSemanticAnalyzer.java. Can you load the latest patch file ? If you dont mind, can you create a phabricator entry ? I know, there is no official policy, but phabricator is much more usable.
        Hide
        Shreepadma Venugopalan added a comment -

        @Namit: Thanks for creating the phabricator entry. HIVE-1362.6.patch is the latest patch. StatsSemanticAnalyzer is referenced in SemanticAnalyzerFactory.java. I'll try to create a phabricator entry in the future. In the past when I've tried to use it, I've had problems with it.

        Show
        Shreepadma Venugopalan added a comment - @Namit: Thanks for creating the phabricator entry. HIVE-1362 .6.patch is the latest patch. StatsSemanticAnalyzer is referenced in SemanticAnalyzerFactory.java. I'll try to create a phabricator entry in the future. In the past when I've tried to use it, I've had problems with it.
        Hide
        Namit Jain added a comment -

        There is something wrong in my patch. Can you create the final patch ?
        There are a bunch of lint warnings, and we need to atleast resolve them.
        Can you create a new phabricator entry ? I will definitely take a look.
        Thanks a lot.

        Show
        Namit Jain added a comment - There is something wrong in my patch. Can you create the final patch ? There are a bunch of lint warnings, and we need to atleast resolve them. Can you create a new phabricator entry ? I will definitely take a look. Thanks a lot.
        Hide
        Namit Jain added a comment -

        Anyway, you need to create it, how else will you refresh it ?

        Show
        Namit Jain added a comment - Anyway, you need to create it, how else will you refresh it ?
        Shreepadma Venugopalan made changes -
        Attachment HIVE-1362.7.patch.txt [ 12551403 ]
        Shreepadma Venugopalan made changes -
        Attachment HIVE-1362_gen-thrift.7.patch.txt [ 12551405 ]
        Hide
        Shreepadma Venugopalan added a comment -

        @Namit: I've uploaded the final patch. I've fixed all the lint problems from the previous revision. Please note that I was just following the convention in those files and didn't realize that it would result in lint problems.

        I tried to create a phabricator review but ran into errors. Can you please use review board this time? I'll try to use Phabricator for future reviews. Thanks.

        Show
        Shreepadma Venugopalan added a comment - @Namit: I've uploaded the final patch. I've fixed all the lint problems from the previous revision. Please note that I was just following the convention in those files and didn't realize that it would result in lint problems. I tried to create a phabricator review but ran into errors. Can you please use review board this time? I'll try to use Phabricator for future reviews. Thanks.
        Hide
        Phabricator added a comment -

        cwsteinbach requested code review of "HIVE-1362 [jira] column level statistics".
        Reviewers: JIRA

        https://issues.apache.org/jira/secure/attachment/12551403/HIVE-1362.7.patch.txt

        TEST PLAN
        EMPTY

        REVISION DETAIL
        https://reviews.facebook.net/D6339

        AFFECTED FILES
        common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
        conf/hive-default.xml.template
        data/files/UserVisits.dat
        data/files/binary.txt
        data/files/bool.txt
        data/files/double.txt
        data/files/employee.dat
        data/files/employee2.dat
        data/files/int.txt
        metastore/if/hive_metastore.thrift
        metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
        metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
        metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
        metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
        metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java
        metastore/src/java/org/apache/hadoop/hive/metastore/Warehouse.java
        metastore/src/model/org/apache/hadoop/hive/metastore/model/MPartitionColumnStatistics.java
        metastore/src/model/org/apache/hadoop/hive/metastore/model/MTableColumnStatistics.java
        metastore/src/model/package.jdo
        metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
        metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java
        ql/build.xml
        ql/if/queryplan.thrift
        ql/ivy.xml
        ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsTask.java
        ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java
        ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java
        ql/src/java/org/apache/hadoop/hive/ql/exec/TaskFactory.java
        ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
        ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteParseContextGenerator.java
        ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java
        ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g
        ql/src/java/org/apache/hadoop/hive/ql/parse/QB.java
        ql/src/java/org/apache/hadoop/hive/ql/parse/QBParseInfo.java
        ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
        ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java
        ql/src/java/org/apache/hadoop/hive/ql/parse/StatsSemanticAnalyzer.java
        ql/src/java/org/apache/hadoop/hive/ql/plan/ColumnStatsDesc.java
        ql/src/java/org/apache/hadoop/hive/ql/plan/ColumnStatsWork.java
        ql/src/java/org/apache/hadoop/hive/ql/plan/HiveOperation.java
        ql/src/java/org/apache/hadoop/hive/ql/udf/generic/DoubleNumDistinctValueEstimator.java
        ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFComputeStats.java
        ql/src/java/org/apache/hadoop/hive/ql/udf/generic/LongNumDistinctValueEstimator.java
        ql/src/java/org/apache/hadoop/hive/ql/udf/generic/NumDistinctValueEstimator.java
        ql/src/java/org/apache/hadoop/hive/ql/udf/generic/StringNumDistinctValueEstimator.java
        ql/src/test/queries/clientnegative/columnstats_partlvl.q
        ql/src/test/queries/clientpositive/columnstats_partlvl.q
        ql/src/test/queries/clientpositive/columnstats_tbllvl.q
        ql/src/test/queries/clientpositive/compute_stats_binary.q
        ql/src/test/queries/clientpositive/compute_stats_boolean.q
        ql/src/test/queries/clientpositive/compute_stats_double.q
        ql/src/test/queries/clientpositive/compute_stats_long.q
        ql/src/test/queries/clientpositive/compute_stats_string.q
        ql/src/test/results/clientnegative/columnstats_partlvl.q.out
        ql/src/test/results/clientpositive/columnstats_partlvl.q.out
        ql/src/test/results/clientpositive/columnstats_tbllvl.q.out
        ql/src/test/results/clientpositive/compute_stats_binary.q.out
        ql/src/test/results/clientpositive/compute_stats_boolean.q.out
        ql/src/test/results/clientpositive/compute_stats_double.q.out
        ql/src/test/results/clientpositive/compute_stats_long.q.out
        ql/src/test/results/clientpositive/compute_stats_string.q.out
        ql/src/test/results/clientpositive/show_functions.q.out
        serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorUtils.java

        MANAGE HERALD DIFFERENTIAL RULES
        https://reviews.facebook.net/herald/view/differential/

        WHY DID I GET THIS EMAIL?
        https://reviews.facebook.net/herald/transcript/15051/

        To: JIRA, cwsteinbach

        Show
        Phabricator added a comment - cwsteinbach requested code review of " HIVE-1362 [jira] column level statistics". Reviewers: JIRA https://issues.apache.org/jira/secure/attachment/12551403/HIVE-1362.7.patch.txt TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D6339 AFFECTED FILES common/src/java/org/apache/hadoop/hive/conf/HiveConf.java conf/hive-default.xml.template data/files/UserVisits.dat data/files/binary.txt data/files/bool.txt data/files/double.txt data/files/employee.dat data/files/employee2.dat data/files/int.txt metastore/if/hive_metastore.thrift metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java metastore/src/java/org/apache/hadoop/hive/metastore/Warehouse.java metastore/src/model/org/apache/hadoop/hive/metastore/model/MPartitionColumnStatistics.java metastore/src/model/org/apache/hadoop/hive/metastore/model/MTableColumnStatistics.java metastore/src/model/package.jdo metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java ql/build.xml ql/if/queryplan.thrift ql/ivy.xml ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsTask.java ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java ql/src/java/org/apache/hadoop/hive/ql/exec/TaskFactory.java ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteParseContextGenerator.java ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g ql/src/java/org/apache/hadoop/hive/ql/parse/QB.java ql/src/java/org/apache/hadoop/hive/ql/parse/QBParseInfo.java ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java ql/src/java/org/apache/hadoop/hive/ql/parse/StatsSemanticAnalyzer.java ql/src/java/org/apache/hadoop/hive/ql/plan/ColumnStatsDesc.java ql/src/java/org/apache/hadoop/hive/ql/plan/ColumnStatsWork.java ql/src/java/org/apache/hadoop/hive/ql/plan/HiveOperation.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/DoubleNumDistinctValueEstimator.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFComputeStats.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/LongNumDistinctValueEstimator.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/NumDistinctValueEstimator.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/StringNumDistinctValueEstimator.java ql/src/test/queries/clientnegative/columnstats_partlvl.q ql/src/test/queries/clientpositive/columnstats_partlvl.q ql/src/test/queries/clientpositive/columnstats_tbllvl.q ql/src/test/queries/clientpositive/compute_stats_binary.q ql/src/test/queries/clientpositive/compute_stats_boolean.q ql/src/test/queries/clientpositive/compute_stats_double.q ql/src/test/queries/clientpositive/compute_stats_long.q ql/src/test/queries/clientpositive/compute_stats_string.q ql/src/test/results/clientnegative/columnstats_partlvl.q.out ql/src/test/results/clientpositive/columnstats_partlvl.q.out ql/src/test/results/clientpositive/columnstats_tbllvl.q.out ql/src/test/results/clientpositive/compute_stats_binary.q.out ql/src/test/results/clientpositive/compute_stats_boolean.q.out ql/src/test/results/clientpositive/compute_stats_double.q.out ql/src/test/results/clientpositive/compute_stats_long.q.out ql/src/test/results/clientpositive/compute_stats_string.q.out ql/src/test/results/clientpositive/show_functions.q.out serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorUtils.java MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/15051/ To: JIRA, cwsteinbach
        Phabricator made changes -
        Attachment HIVE-1362.D6339.1.patch [ 12551446 ]
        Hide
        Carl Steinbach added a comment -

        @Namit: I created a review request on phabricator: https://reviews.facebook.net/D6339

        @Shreepadma: I had to correct one lint error in order to get phabricator to accept this patch (non ASCII character in the GenericUDAF). Phabricator also flagged a bunch of lines that exceed the length limit, and several files that don't end with a newline. Please fix these issues and upload a new patch to JIRA. Thanks.

        Show
        Carl Steinbach added a comment - @Namit: I created a review request on phabricator: https://reviews.facebook.net/D6339 @Shreepadma: I had to correct one lint error in order to get phabricator to accept this patch (non ASCII character in the GenericUDAF). Phabricator also flagged a bunch of lines that exceed the length limit, and several files that don't end with a newline. Please fix these issues and upload a new patch to JIRA. Thanks.
        Hide
        Shreepadma Venugopalan added a comment -

        I've uploaded a new patch to JIRA. It a) fixes the lint problems I saw on phabricator, b) Adds a new line to the files this patch adds.

        Show
        Shreepadma Venugopalan added a comment - I've uploaded a new patch to JIRA. It a) fixes the lint problems I saw on phabricator, b) Adds a new line to the files this patch adds.
        Shreepadma Venugopalan made changes -
        Attachment HIVE-1362_gen-thrift.8.patch.txt [ 12551464 ]
        Shreepadma Venugopalan made changes -
        Attachment HIVE-1362.8.patch.txt [ 12551465 ]
        Hide
        Shreepadma Venugopalan added a comment -

        Forgot to add I removed the invalid ASCII character as well. However, I still have problems with phabricator.

        Show
        Shreepadma Venugopalan added a comment - Forgot to add I removed the invalid ASCII character as well. However, I still have problems with phabricator.
        Hide
        Namit Jain added a comment -

        Thanks a lot for creating a phabricator entry. I will take a look tomorrow.

        Show
        Namit Jain added a comment - Thanks a lot for creating a phabricator entry. I will take a look tomorrow.
        Hide
        Phabricator added a comment -

        njain has commented on the revision "HIVE-1362 [jira] column level statistics".

        I wont go over cosmetic changes – newline at the end of the file etc.
        Please fix those.

        One higher level question: (SA == SemanticAnalyzer)

        1. StatsSemanticAnalyzer - looks like a wrong name - can you rename it to AnalyzeColumnStatsSema.. given that we already have a StatsTask etc. for something totally different.
        2. The code looks more complicated - I mean, calling SemanticAnalyzer from StatsSemanticAnalyzer, specially for non-rewritten queries. Won't it be simpler to:
        a. Either StatsSA should extend SA
        b. Move the logic for analyze (old) in StatsSA

        INLINE COMMENTS
        ql/src/test/queries/clientnegative/columnstats_partlvl.q:13 Looks fishy - wont the test error out at the first error.
        Subsequent errors would just be ignored

        Can you add more negative tests - wrong column name.

        REVISION DETAIL
        https://reviews.facebook.net/D6339

        To: JIRA, njain, cwsteinbach

        Show
        Phabricator added a comment - njain has commented on the revision " HIVE-1362 [jira] column level statistics". I wont go over cosmetic changes – newline at the end of the file etc. Please fix those. One higher level question: (SA == SemanticAnalyzer) 1. StatsSemanticAnalyzer - looks like a wrong name - can you rename it to AnalyzeColumnStatsSema.. given that we already have a StatsTask etc. for something totally different. 2. The code looks more complicated - I mean, calling SemanticAnalyzer from StatsSemanticAnalyzer, specially for non-rewritten queries. Won't it be simpler to: a. Either StatsSA should extend SA b. Move the logic for analyze (old) in StatsSA INLINE COMMENTS ql/src/test/queries/clientnegative/columnstats_partlvl.q:13 Looks fishy - wont the test error out at the first error. Subsequent errors would just be ignored Can you add more negative tests - wrong column name. REVISION DETAIL https://reviews.facebook.net/D6339 To: JIRA, njain, cwsteinbach
        Hide
        Phabricator added a comment -

        njain has commented on the revision "HIVE-1362 [jira] column level statistics".

        you have a columnstatsdesc/columnstatswork etc.
        Rename the file as columnstatssemanticanalyzer

        I did not look into too much detail in the UDAF - I am assuming Carl has already reviewed that part

        INLINE COMMENTS
        ql/src/java/org/apache/hadoop/hive/ql/parse/StatsSemanticAnalyzer.java:216 create a entry in ErrorMsg - similarly to all other exceptions

        REVISION DETAIL
        https://reviews.facebook.net/D6339

        To: JIRA, njain, cwsteinbach

        Show
        Phabricator added a comment - njain has commented on the revision " HIVE-1362 [jira] column level statistics". you have a columnstatsdesc/columnstatswork etc. Rename the file as columnstatssemanticanalyzer I did not look into too much detail in the UDAF - I am assuming Carl has already reviewed that part INLINE COMMENTS ql/src/java/org/apache/hadoop/hive/ql/parse/StatsSemanticAnalyzer.java:216 create a entry in ErrorMsg - similarly to all other exceptions REVISION DETAIL https://reviews.facebook.net/D6339 To: JIRA, njain, cwsteinbach
        Hide
        Phabricator added a comment -

        njain has commented on the revision "HIVE-1362 [jira] column level statistics".

        if u haven't done already - file a jira for upgrade scripts for other dbs.

        INLINE COMMENTS
        ql/src/test/queries/clientpositive/columnstats_partlvl.q:10 add some tests with explain extended.
        ql/src/java/org/apache/hadoop/hive/ql/plan/ColumnStatsWork.java:33 More stuff should be dumped in explain(extended)
        ql/src/java/org/apache/hadoop/hive/ql/plan/ColumnStatsDesc.java:27 explain should output this

        REVISION DETAIL
        https://reviews.facebook.net/D6339

        To: JIRA, njain, cwsteinbach

        Show
        Phabricator added a comment - njain has commented on the revision " HIVE-1362 [jira] column level statistics". if u haven't done already - file a jira for upgrade scripts for other dbs. INLINE COMMENTS ql/src/test/queries/clientpositive/columnstats_partlvl.q:10 add some tests with explain extended. ql/src/java/org/apache/hadoop/hive/ql/plan/ColumnStatsWork.java:33 More stuff should be dumped in explain(extended) ql/src/java/org/apache/hadoop/hive/ql/plan/ColumnStatsDesc.java:27 explain should output this REVISION DETAIL https://reviews.facebook.net/D6339 To: JIRA, njain, cwsteinbach
        Hide
        Namit Jain added a comment -

        minor comments on phabricator

        Show
        Namit Jain added a comment - minor comments on phabricator
        Namit Jain made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Hide
        Phabricator added a comment -

        shreepadma has commented on the revision "HIVE-1362 [jira] column level statistics".

        INLINE COMMENTS
        ql/src/java/org/apache/hadoop/hive/ql/plan/ColumnStatsWork.java:33 Can you please let me know what you would like to see dumped in the explain extended?
        ql/src/test/queries/clientnegative/columnstats_partlvl.q:13 Will add negative test for wrong column name and move each one of the query to a separate q file.
        ql/src/java/org/apache/hadoop/hive/ql/plan/ColumnStatsDesc.java:27 Can you please let me know what you would like to include in the explain output?
        ql/src/java/org/apache/hadoop/hive/ql/parse/StatsSemanticAnalyzer.java:216 Can you please explain the purpose of creating an error message entry in addition to raising an exception? Throwing an exception should present all the error information that is needed.
        ql/src/test/queries/clientpositive/columnstats_partlvl.q:10 Will add tests wil explain extended. However, I'm not sure what needs to be printed as part of explain extended output for columnstatswork. Please refer to my earlier comment on it.

        REVISION DETAIL
        https://reviews.facebook.net/D6339

        To: JIRA, njain, cwsteinbach
        Cc: shreepadma

        Show
        Phabricator added a comment - shreepadma has commented on the revision " HIVE-1362 [jira] column level statistics". INLINE COMMENTS ql/src/java/org/apache/hadoop/hive/ql/plan/ColumnStatsWork.java:33 Can you please let me know what you would like to see dumped in the explain extended? ql/src/test/queries/clientnegative/columnstats_partlvl.q:13 Will add negative test for wrong column name and move each one of the query to a separate q file. ql/src/java/org/apache/hadoop/hive/ql/plan/ColumnStatsDesc.java:27 Can you please let me know what you would like to include in the explain output? ql/src/java/org/apache/hadoop/hive/ql/parse/StatsSemanticAnalyzer.java:216 Can you please explain the purpose of creating an error message entry in addition to raising an exception? Throwing an exception should present all the error information that is needed. ql/src/test/queries/clientpositive/columnstats_partlvl.q:10 Will add tests wil explain extended. However, I'm not sure what needs to be printed as part of explain extended output for columnstatswork. Please refer to my earlier comment on it. REVISION DETAIL https://reviews.facebook.net/D6339 To: JIRA, njain, cwsteinbach Cc: shreepadma
        Hide
        Shreepadma Venugopalan added a comment -

        @Namit: Thanks for your comments! I responded to most of your comments on phabricator. Thanks!

        Show
        Shreepadma Venugopalan added a comment - @Namit: Thanks for your comments! I responded to most of your comments on phabricator. Thanks!
        Shreepadma Venugopalan made changes -
        Status Open [ 1 ] In Progress [ 3 ]
        Shreepadma Venugopalan made changes -
        Attachment HIVE-1362_gen-thrift.9.patch.txt [ 12552056 ]
        Shreepadma Venugopalan made changes -
        Attachment HIVE-1362.9.patch.txt [ 12552057 ]
        Hide
        Shreepadma Venugopalan added a comment -

        @Namit: New patch that addresses your comments is available on both JIRA and phabricator. Phabricator link: https://reviews.facebook.net/D6459. I had to create a new review request since Carl owns the old one and I can't update it. Thanks!

        Show
        Shreepadma Venugopalan added a comment - @Namit: New patch that addresses your comments is available on both JIRA and phabricator. Phabricator link: https://reviews.facebook.net/D6459 . I had to create a new review request since Carl owns the old one and I can't update it. Thanks!
        Shreepadma Venugopalan made changes -
        Status In Progress [ 3 ] Patch Available [ 10002 ]
        Namit Jain made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Hide
        Namit Jain added a comment -

        comments on phabricator

        Show
        Namit Jain added a comment - comments on phabricator
        Hide
        Shreepadma Venugopalan added a comment -

        @Namit: Updated patch is available on phabricator. Thanks.

        Show
        Shreepadma Venugopalan added a comment - @Namit: Updated patch is available on phabricator. Thanks.
        Shreepadma Venugopalan made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Shreepadma Venugopalan added a comment -

        @Namit: I responded to your comment about Exceptions on phabricator. Thanks.

        Show
        Shreepadma Venugopalan added a comment - @Namit: I responded to your comment about Exceptions on phabricator. Thanks.
        Hide
        Namit Jain added a comment -

        I think you should wrap the error even if is being thrown by a function being called.

        Show
        Namit Jain added a comment - I think you should wrap the error even if is being thrown by a function being called.
        Hide
        Namit Jain added a comment -

        comments

        Show
        Namit Jain added a comment - comments
        Namit Jain made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Shreepadma Venugopalan made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Shreepadma Venugopalan made changes -
        Attachment HIVE-1362_gen-thrift.10.patch.txt [ 12552228 ]
        Shreepadma Venugopalan made changes -
        Attachment HIVE-1362.10.patch.txt [ 12552229 ]
        Hide
        Shreepadma Venugopalan added a comment -

        @Namit: The latest patch addressed your comments. Its available both on JIRA and phabricator. Thanks.

        Show
        Shreepadma Venugopalan added a comment - @Namit: The latest patch addressed your comments. Its available both on JIRA and phabricator. Thanks.
        Hide
        Namit Jain added a comment -

        Fine with me.

        @Carl, do you have any additional comments ?
        If not, can you take care of committing it ?

        Show
        Namit Jain added a comment - Fine with me. @Carl, do you have any additional comments ? If not, can you take care of committing it ?
        Hide
        Carl Steinbach added a comment -

        +1. Will commit if tests pass.

        Show
        Carl Steinbach added a comment - +1. Will commit if tests pass.
        Hide
        Shreepadma Venugopalan added a comment -

        @Carl: Please take the latest patch from JIRA. If you have trouble applying it, let me know. Thanks.

        Show
        Shreepadma Venugopalan added a comment - @Carl: Please take the latest patch from JIRA. If you have trouble applying it, let me know. Thanks.
        Hide
        Shreepadma Venugopalan added a comment -

        @Carl: You will see 6 failures in testParse (groupby1.q .. groupby6.q) when you run the tests. It is in the proces of being fixed by HIVE-3674.

        Show
        Shreepadma Venugopalan added a comment - @Carl: You will see 6 failures in testParse (groupby1.q .. groupby6.q) when you run the tests. It is in the proces of being fixed by HIVE-3674 .
        Carl Steinbach made changes -
        Link This issue relates to HIVE-3526 [ HIVE-3526 ]
        Carl Steinbach made changes -
        Summary column level statistics Column level scalar valued statistics
        Shreepadma Venugopalan made changes -
        Summary Column level scalar valued statistics Column level scalar valued statistics on Tables and Partitions
        Hide
        Carl Steinbach added a comment -

        Committed to trunk. Thanks Shreepadma!

        Show
        Carl Steinbach added a comment - Committed to trunk. Thanks Shreepadma!
        Carl Steinbach made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags Reviewed [ 10343 ]
        Fix Version/s 0.10.0 [ 12320745 ]
        Resolution Fixed [ 1 ]
        Carl Steinbach made changes -
        Link This issue requires HIVE-3678 [ HIVE-3678 ]
        Hide
        Namit Jain added a comment -

        I am getting some compile errors. Can you take a look ?

        Show
        Namit Jain added a comment - I am getting some compile errors. Can you take a look ?
        Hide
        Hudson added a comment -

        Integrated in Hive-trunk-h0.21 #1778 (See https://builds.apache.org/job/Hive-trunk-h0.21/1778/)
        HIVE-1362. Column level scalar valued statistics on Tables and Partitions (Shreepadma Venugopalan via cws) (Revision 1406465)

        Result = FAILURE
        cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1406465
        Files :

        • /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
        • /hive/trunk/conf/hive-default.xml.template
        • /hive/trunk/data/files/UserVisits.dat
        • /hive/trunk/data/files/binary.txt
        • /hive/trunk/data/files/bool.txt
        • /hive/trunk/data/files/double.txt
        • /hive/trunk/data/files/employee.dat
        • /hive/trunk/data/files/employee2.dat
        • /hive/trunk/data/files/employee_part.txt
        • /hive/trunk/data/files/int.txt
        • /hive/trunk/metastore/if/hive_metastore.thrift
        • /hive/trunk/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.cpp
        • /hive/trunk/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.h
        • /hive/trunk/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore_server.skeleton.cpp
        • /hive/trunk/metastore/src/gen/thrift/gen-cpp/hive_metastore_types.cpp
        • /hive/trunk/metastore/src/gen/thrift/gen-cpp/hive_metastore_types.h
        • /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/BinaryColumnStatsData.java
        • /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/BooleanColumnStatsData.java
        • /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ColumnStatistics.java
        • /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ColumnStatisticsData.java
        • /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ColumnStatisticsDesc.java
        • /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ColumnStatisticsObj.java
        • /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/DoubleColumnStatsData.java
        • /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/EnvironmentContext.java
        • /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/InvalidInputException.java
        • /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/LongColumnStatsData.java
        • /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Schema.java
        • /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/StringColumnStatsData.java
        • /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java
        • /hive/trunk/metastore/src/gen/thrift/gen-php/hive_metastore/ThriftHiveMetastore.php
        • /hive/trunk/metastore/src/gen/thrift/gen-php/hive_metastore/hive_metastore_types.php
        • /hive/trunk/metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote
        • /hive/trunk/metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py
        • /hive/trunk/metastore/src/gen/thrift/gen-py/hive_metastore/ttypes.py
        • /hive/trunk/metastore/src/gen/thrift/gen-rb/hive_metastore_types.rb
        • /hive/trunk/metastore/src/gen/thrift/gen-rb/thrift_hive_metastore.rb
        • /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
        • /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
        • /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
        • /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
        • /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java
        • /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/Warehouse.java
        • /hive/trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MPartitionColumnStatistics.java
        • /hive/trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MTableColumnStatistics.java
        • /hive/trunk/metastore/src/model/package.jdo
        • /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
        • /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java
        • /hive/trunk/ql/build.xml
        • /hive/trunk/ql/if/queryplan.thrift
        • /hive/trunk/ql/ivy.xml
        • /hive/trunk/ql/src/gen/thrift/gen-cpp/queryplan_types.cpp
        • /hive/trunk/ql/src/gen/thrift/gen-cpp/queryplan_types.h
        • /hive/trunk/ql/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/ql/plan/api/StageType.java
        • /hive/trunk/ql/src/gen/thrift/gen-php/queryplan/queryplan_types.php
        • /hive/trunk/ql/src/gen/thrift/gen-py/queryplan/ttypes.py
        • /hive/trunk/ql/src/gen/thrift/gen-rb/queryplan_types.rb
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsTask.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/TaskFactory.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteParseContextGenerator.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ExportSemanticAnalyzer.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/FunctionSemanticAnalyzer.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ImportSemanticAnalyzer.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/QB.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/QBParseInfo.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ColumnStatsDesc.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ColumnStatsWork.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/HiveOperation.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/DoubleNumDistinctValueEstimator.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFComputeStats.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/LongNumDistinctValueEstimator.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/NumDistinctValueEstimator.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/StringNumDistinctValueEstimator.java
        • /hive/trunk/ql/src/test/queries/clientnegative/columnstats_partlvl_dp.q
        • /hive/trunk/ql/src/test/queries/clientnegative/columnstats_partlvl_incorrect_num_keys.q
        • /hive/trunk/ql/src/test/queries/clientnegative/columnstats_partlvl_invalid_values.q
        • /hive/trunk/ql/src/test/queries/clientnegative/columnstats_partlvl_multiple_part_clause.q
        • /hive/trunk/ql/src/test/queries/clientnegative/columnstats_tbllvl.q
        • /hive/trunk/ql/src/test/queries/clientnegative/columnstats_tbllvl_complex_type.q
        • /hive/trunk/ql/src/test/queries/clientnegative/columnstats_tbllvl_incorrect_column.q
        • /hive/trunk/ql/src/test/queries/clientpositive/columnstats_partlvl.q
        • /hive/trunk/ql/src/test/queries/clientpositive/columnstats_tbllvl.q
        • /hive/trunk/ql/src/test/queries/clientpositive/compute_stats_binary.q
        • /hive/trunk/ql/src/test/queries/clientpositive/compute_stats_boolean.q
        • /hive/trunk/ql/src/test/queries/clientpositive/compute_stats_double.q
        • /hive/trunk/ql/src/test/queries/clientpositive/compute_stats_long.q
        • /hive/trunk/ql/src/test/queries/clientpositive/compute_stats_string.q
        • /hive/trunk/ql/src/test/results/clientnegative/columnstats_partlvl_dp.q.out
        • /hive/trunk/ql/src/test/results/clientnegative/columnstats_partlvl_incorrect_num_keys.q.out
        • /hive/trunk/ql/src/test/results/clientnegative/columnstats_partlvl_invalid_values.q.out
        • /hive/trunk/ql/src/test/results/clientnegative/columnstats_partlvl_multiple_part_clause.q.out
        • /hive/trunk/ql/src/test/results/clientnegative/columnstats_tbllvl.q.out
        • /hive/trunk/ql/src/test/results/clientnegative/columnstats_tbllvl_complex_type.q.out
        • /hive/trunk/ql/src/test/results/clientnegative/columnstats_tbllvl_incorrect_column.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/columnstats_partlvl.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/columnstats_tbllvl.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/compute_stats_binary.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/compute_stats_boolean.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/compute_stats_double.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/compute_stats_long.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/compute_stats_string.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/show_functions.q.out
        • /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorUtils.java
        Show
        Hudson added a comment - Integrated in Hive-trunk-h0.21 #1778 (See https://builds.apache.org/job/Hive-trunk-h0.21/1778/ ) HIVE-1362 . Column level scalar valued statistics on Tables and Partitions (Shreepadma Venugopalan via cws) (Revision 1406465) Result = FAILURE cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1406465 Files : /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java /hive/trunk/conf/hive-default.xml.template /hive/trunk/data/files/UserVisits.dat /hive/trunk/data/files/binary.txt /hive/trunk/data/files/bool.txt /hive/trunk/data/files/double.txt /hive/trunk/data/files/employee.dat /hive/trunk/data/files/employee2.dat /hive/trunk/data/files/employee_part.txt /hive/trunk/data/files/int.txt /hive/trunk/metastore/if/hive_metastore.thrift /hive/trunk/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.cpp /hive/trunk/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.h /hive/trunk/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore_server.skeleton.cpp /hive/trunk/metastore/src/gen/thrift/gen-cpp/hive_metastore_types.cpp /hive/trunk/metastore/src/gen/thrift/gen-cpp/hive_metastore_types.h /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/BinaryColumnStatsData.java /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/BooleanColumnStatsData.java /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ColumnStatistics.java /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ColumnStatisticsData.java /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ColumnStatisticsDesc.java /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ColumnStatisticsObj.java /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/DoubleColumnStatsData.java /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/EnvironmentContext.java /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/InvalidInputException.java /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/LongColumnStatsData.java /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Schema.java /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/StringColumnStatsData.java /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java /hive/trunk/metastore/src/gen/thrift/gen-php/hive_metastore/ThriftHiveMetastore.php /hive/trunk/metastore/src/gen/thrift/gen-php/hive_metastore/hive_metastore_types.php /hive/trunk/metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote /hive/trunk/metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py /hive/trunk/metastore/src/gen/thrift/gen-py/hive_metastore/ttypes.py /hive/trunk/metastore/src/gen/thrift/gen-rb/hive_metastore_types.rb /hive/trunk/metastore/src/gen/thrift/gen-rb/thrift_hive_metastore.rb /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/Warehouse.java /hive/trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MPartitionColumnStatistics.java /hive/trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MTableColumnStatistics.java /hive/trunk/metastore/src/model/package.jdo /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java /hive/trunk/ql/build.xml /hive/trunk/ql/if/queryplan.thrift /hive/trunk/ql/ivy.xml /hive/trunk/ql/src/gen/thrift/gen-cpp/queryplan_types.cpp /hive/trunk/ql/src/gen/thrift/gen-cpp/queryplan_types.h /hive/trunk/ql/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/ql/plan/api/StageType.java /hive/trunk/ql/src/gen/thrift/gen-php/queryplan/queryplan_types.php /hive/trunk/ql/src/gen/thrift/gen-py/queryplan/ttypes.py /hive/trunk/ql/src/gen/thrift/gen-rb/queryplan_types.rb /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsTask.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/TaskFactory.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteParseContextGenerator.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ExportSemanticAnalyzer.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/FunctionSemanticAnalyzer.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ImportSemanticAnalyzer.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/QB.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/QBParseInfo.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ColumnStatsDesc.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ColumnStatsWork.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/HiveOperation.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/DoubleNumDistinctValueEstimator.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFComputeStats.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/LongNumDistinctValueEstimator.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/NumDistinctValueEstimator.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/StringNumDistinctValueEstimator.java /hive/trunk/ql/src/test/queries/clientnegative/columnstats_partlvl_dp.q /hive/trunk/ql/src/test/queries/clientnegative/columnstats_partlvl_incorrect_num_keys.q /hive/trunk/ql/src/test/queries/clientnegative/columnstats_partlvl_invalid_values.q /hive/trunk/ql/src/test/queries/clientnegative/columnstats_partlvl_multiple_part_clause.q /hive/trunk/ql/src/test/queries/clientnegative/columnstats_tbllvl.q /hive/trunk/ql/src/test/queries/clientnegative/columnstats_tbllvl_complex_type.q /hive/trunk/ql/src/test/queries/clientnegative/columnstats_tbllvl_incorrect_column.q /hive/trunk/ql/src/test/queries/clientpositive/columnstats_partlvl.q /hive/trunk/ql/src/test/queries/clientpositive/columnstats_tbllvl.q /hive/trunk/ql/src/test/queries/clientpositive/compute_stats_binary.q /hive/trunk/ql/src/test/queries/clientpositive/compute_stats_boolean.q /hive/trunk/ql/src/test/queries/clientpositive/compute_stats_double.q /hive/trunk/ql/src/test/queries/clientpositive/compute_stats_long.q /hive/trunk/ql/src/test/queries/clientpositive/compute_stats_string.q /hive/trunk/ql/src/test/results/clientnegative/columnstats_partlvl_dp.q.out /hive/trunk/ql/src/test/results/clientnegative/columnstats_partlvl_incorrect_num_keys.q.out /hive/trunk/ql/src/test/results/clientnegative/columnstats_partlvl_invalid_values.q.out /hive/trunk/ql/src/test/results/clientnegative/columnstats_partlvl_multiple_part_clause.q.out /hive/trunk/ql/src/test/results/clientnegative/columnstats_tbllvl.q.out /hive/trunk/ql/src/test/results/clientnegative/columnstats_tbllvl_complex_type.q.out /hive/trunk/ql/src/test/results/clientnegative/columnstats_tbllvl_incorrect_column.q.out /hive/trunk/ql/src/test/results/clientpositive/columnstats_partlvl.q.out /hive/trunk/ql/src/test/results/clientpositive/columnstats_tbllvl.q.out /hive/trunk/ql/src/test/results/clientpositive/compute_stats_binary.q.out /hive/trunk/ql/src/test/results/clientpositive/compute_stats_boolean.q.out /hive/trunk/ql/src/test/results/clientpositive/compute_stats_double.q.out /hive/trunk/ql/src/test/results/clientpositive/compute_stats_long.q.out /hive/trunk/ql/src/test/results/clientpositive/compute_stats_string.q.out /hive/trunk/ql/src/test/results/clientpositive/show_functions.q.out /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorUtils.java
        Hide
        Shreepadma Venugopalan added a comment -

        HIVE-3524 changed the signature of endFunction in HiveMetaStore.java. HIVE-3524 was committed hours before this patch. The compile errors are due to the signature change. I'm working on a fix. Thanks.

        Show
        Shreepadma Venugopalan added a comment - HIVE-3524 changed the signature of endFunction in HiveMetaStore.java. HIVE-3524 was committed hours before this patch. The compile errors are due to the signature change. I'm working on a fix. Thanks.
        Hide
        Shreepadma Venugopalan added a comment -

        @Namit: Not sure what the protocol is but I've attach the new patch to this JIRA. Thanks.

        Show
        Shreepadma Venugopalan added a comment - @Namit: Not sure what the protocol is but I've attach the new patch to this JIRA. Thanks.
        Shreepadma Venugopalan made changes -
        Attachment HIVE-1362.11.patch.txt [ 12552493 ]
        Hide
        Shreepadma Venugopalan added a comment -

        Please look at HIVE-1362.11.patch.txt to fix the compile errors introduced earlier.

        Show
        Shreepadma Venugopalan added a comment - Please look at HIVE-1362 .11.patch.txt to fix the compile errors introduced earlier.
        Hide
        Shreepadma Venugopalan added a comment -

        Filed a new JIRA - HIVE-3686 to fix the compile errors.

        Show
        Shreepadma Venugopalan added a comment - Filed a new JIRA - HIVE-3686 to fix the compile errors.
        Carl Steinbach made changes -
        Link This issue breaks HIVE-3686 [ HIVE-3686 ]
        Shreepadma Venugopalan made changes -
        Link This issue is broken by HIVE-3524 [ HIVE-3524 ]
        Hide
        Shreepadma Venugopalan added a comment -

        HIVE-3524 introduced a signature change to endFunction in HiveMetaStore.java. HIVE-3524 was committed a few hours before HIVE-1362. HIVE-1362 adds new functions to HiveMetaStore.java that call endFunction with the old signature.

        Show
        Shreepadma Venugopalan added a comment - HIVE-3524 introduced a signature change to endFunction in HiveMetaStore.java. HIVE-3524 was committed a few hours before HIVE-1362 . HIVE-1362 adds new functions to HiveMetaStore.java that call endFunction with the old signature.
        Shreepadma Venugopalan made changes -
        Link This issue incorporates HIVE-3686 [ HIVE-3686 ]
        Shreepadma Venugopalan made changes -
        Link This issue breaks HIVE-3686 [ HIVE-3686 ]
        Hide
        Hudson added a comment -

        Integrated in Hive-trunk-h0.21 #1780 (See https://builds.apache.org/job/Hive-trunk-h0.21/1780/)
        HIVE-3686. Fix compile errors introduced by the interaction of HIVE-1362 and HIVE-3524. (Shreepadma Venugopalan via kevinwilfong) (Revision 1406783)

        Result = FAILURE
        kevinwilfong : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1406783
        Files :

        • /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
        Show
        Hudson added a comment - Integrated in Hive-trunk-h0.21 #1780 (See https://builds.apache.org/job/Hive-trunk-h0.21/1780/ ) HIVE-3686 . Fix compile errors introduced by the interaction of HIVE-1362 and HIVE-3524 . (Shreepadma Venugopalan via kevinwilfong) (Revision 1406783) Result = FAILURE kevinwilfong : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1406783 Files : /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
        Hide
        Hudson added a comment -

        Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/)
        HIVE-3686. Fix compile errors introduced by the interaction of HIVE-1362 and HIVE-3524. (Shreepadma Venugopalan via kevinwilfong) (Revision 1406783)
        HIVE-1362. Column level scalar valued statistics on Tables and Partitions (Shreepadma Venugopalan via cws) (Revision 1406465)

        Result = ABORTED
        kevinwilfong : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1406783
        Files :

        • /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java

        cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1406465
        Files :

        • /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
        • /hive/trunk/conf/hive-default.xml.template
        • /hive/trunk/data/files/UserVisits.dat
        • /hive/trunk/data/files/binary.txt
        • /hive/trunk/data/files/bool.txt
        • /hive/trunk/data/files/double.txt
        • /hive/trunk/data/files/employee.dat
        • /hive/trunk/data/files/employee2.dat
        • /hive/trunk/data/files/employee_part.txt
        • /hive/trunk/data/files/int.txt
        • /hive/trunk/metastore/if/hive_metastore.thrift
        • /hive/trunk/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.cpp
        • /hive/trunk/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.h
        • /hive/trunk/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore_server.skeleton.cpp
        • /hive/trunk/metastore/src/gen/thrift/gen-cpp/hive_metastore_types.cpp
        • /hive/trunk/metastore/src/gen/thrift/gen-cpp/hive_metastore_types.h
        • /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/BinaryColumnStatsData.java
        • /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/BooleanColumnStatsData.java
        • /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ColumnStatistics.java
        • /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ColumnStatisticsData.java
        • /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ColumnStatisticsDesc.java
        • /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ColumnStatisticsObj.java
        • /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/DoubleColumnStatsData.java
        • /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/EnvironmentContext.java
        • /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/InvalidInputException.java
        • /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/LongColumnStatsData.java
        • /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Schema.java
        • /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/StringColumnStatsData.java
        • /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java
        • /hive/trunk/metastore/src/gen/thrift/gen-php/hive_metastore/ThriftHiveMetastore.php
        • /hive/trunk/metastore/src/gen/thrift/gen-php/hive_metastore/hive_metastore_types.php
        • /hive/trunk/metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote
        • /hive/trunk/metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py
        • /hive/trunk/metastore/src/gen/thrift/gen-py/hive_metastore/ttypes.py
        • /hive/trunk/metastore/src/gen/thrift/gen-rb/hive_metastore_types.rb
        • /hive/trunk/metastore/src/gen/thrift/gen-rb/thrift_hive_metastore.rb
        • /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
        • /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
        • /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
        • /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
        • /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java
        • /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/Warehouse.java
        • /hive/trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MPartitionColumnStatistics.java
        • /hive/trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MTableColumnStatistics.java
        • /hive/trunk/metastore/src/model/package.jdo
        • /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
        • /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java
        • /hive/trunk/ql/build.xml
        • /hive/trunk/ql/if/queryplan.thrift
        • /hive/trunk/ql/ivy.xml
        • /hive/trunk/ql/src/gen/thrift/gen-cpp/queryplan_types.cpp
        • /hive/trunk/ql/src/gen/thrift/gen-cpp/queryplan_types.h
        • /hive/trunk/ql/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/ql/plan/api/StageType.java
        • /hive/trunk/ql/src/gen/thrift/gen-php/queryplan/queryplan_types.php
        • /hive/trunk/ql/src/gen/thrift/gen-py/queryplan/ttypes.py
        • /hive/trunk/ql/src/gen/thrift/gen-rb/queryplan_types.rb
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsTask.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/TaskFactory.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteParseContextGenerator.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ExportSemanticAnalyzer.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/FunctionSemanticAnalyzer.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ImportSemanticAnalyzer.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/QB.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/QBParseInfo.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ColumnStatsDesc.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ColumnStatsWork.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/HiveOperation.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/DoubleNumDistinctValueEstimator.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFComputeStats.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/LongNumDistinctValueEstimator.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/NumDistinctValueEstimator.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/StringNumDistinctValueEstimator.java
        • /hive/trunk/ql/src/test/queries/clientnegative/columnstats_partlvl_dp.q
        • /hive/trunk/ql/src/test/queries/clientnegative/columnstats_partlvl_incorrect_num_keys.q
        • /hive/trunk/ql/src/test/queries/clientnegative/columnstats_partlvl_invalid_values.q
        • /hive/trunk/ql/src/test/queries/clientnegative/columnstats_partlvl_multiple_part_clause.q
        • /hive/trunk/ql/src/test/queries/clientnegative/columnstats_tbllvl.q
        • /hive/trunk/ql/src/test/queries/clientnegative/columnstats_tbllvl_complex_type.q
        • /hive/trunk/ql/src/test/queries/clientnegative/columnstats_tbllvl_incorrect_column.q
        • /hive/trunk/ql/src/test/queries/clientpositive/columnstats_partlvl.q
        • /hive/trunk/ql/src/test/queries/clientpositive/columnstats_tbllvl.q
        • /hive/trunk/ql/src/test/queries/clientpositive/compute_stats_binary.q
        • /hive/trunk/ql/src/test/queries/clientpositive/compute_stats_boolean.q
        • /hive/trunk/ql/src/test/queries/clientpositive/compute_stats_double.q
        • /hive/trunk/ql/src/test/queries/clientpositive/compute_stats_long.q
        • /hive/trunk/ql/src/test/queries/clientpositive/compute_stats_string.q
        • /hive/trunk/ql/src/test/results/clientnegative/columnstats_partlvl_dp.q.out
        • /hive/trunk/ql/src/test/results/clientnegative/columnstats_partlvl_incorrect_num_keys.q.out
        • /hive/trunk/ql/src/test/results/clientnegative/columnstats_partlvl_invalid_values.q.out
        • /hive/trunk/ql/src/test/results/clientnegative/columnstats_partlvl_multiple_part_clause.q.out
        • /hive/trunk/ql/src/test/results/clientnegative/columnstats_tbllvl.q.out
        • /hive/trunk/ql/src/test/results/clientnegative/columnstats_tbllvl_complex_type.q.out
        • /hive/trunk/ql/src/test/results/clientnegative/columnstats_tbllvl_incorrect_column.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/columnstats_partlvl.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/columnstats_tbllvl.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/compute_stats_binary.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/compute_stats_boolean.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/compute_stats_double.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/compute_stats_long.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/compute_stats_string.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/show_functions.q.out
        • /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorUtils.java
        Show
        Hudson added a comment - Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/ ) HIVE-3686 . Fix compile errors introduced by the interaction of HIVE-1362 and HIVE-3524 . (Shreepadma Venugopalan via kevinwilfong) (Revision 1406783) HIVE-1362 . Column level scalar valued statistics on Tables and Partitions (Shreepadma Venugopalan via cws) (Revision 1406465) Result = ABORTED kevinwilfong : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1406783 Files : /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1406465 Files : /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java /hive/trunk/conf/hive-default.xml.template /hive/trunk/data/files/UserVisits.dat /hive/trunk/data/files/binary.txt /hive/trunk/data/files/bool.txt /hive/trunk/data/files/double.txt /hive/trunk/data/files/employee.dat /hive/trunk/data/files/employee2.dat /hive/trunk/data/files/employee_part.txt /hive/trunk/data/files/int.txt /hive/trunk/metastore/if/hive_metastore.thrift /hive/trunk/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.cpp /hive/trunk/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.h /hive/trunk/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore_server.skeleton.cpp /hive/trunk/metastore/src/gen/thrift/gen-cpp/hive_metastore_types.cpp /hive/trunk/metastore/src/gen/thrift/gen-cpp/hive_metastore_types.h /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/BinaryColumnStatsData.java /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/BooleanColumnStatsData.java /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ColumnStatistics.java /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ColumnStatisticsData.java /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ColumnStatisticsDesc.java /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ColumnStatisticsObj.java /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/DoubleColumnStatsData.java /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/EnvironmentContext.java /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/InvalidInputException.java /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/LongColumnStatsData.java /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Schema.java /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/StringColumnStatsData.java /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java /hive/trunk/metastore/src/gen/thrift/gen-php/hive_metastore/ThriftHiveMetastore.php /hive/trunk/metastore/src/gen/thrift/gen-php/hive_metastore/hive_metastore_types.php /hive/trunk/metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote /hive/trunk/metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py /hive/trunk/metastore/src/gen/thrift/gen-py/hive_metastore/ttypes.py /hive/trunk/metastore/src/gen/thrift/gen-rb/hive_metastore_types.rb /hive/trunk/metastore/src/gen/thrift/gen-rb/thrift_hive_metastore.rb /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/Warehouse.java /hive/trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MPartitionColumnStatistics.java /hive/trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MTableColumnStatistics.java /hive/trunk/metastore/src/model/package.jdo /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java /hive/trunk/ql/build.xml /hive/trunk/ql/if/queryplan.thrift /hive/trunk/ql/ivy.xml /hive/trunk/ql/src/gen/thrift/gen-cpp/queryplan_types.cpp /hive/trunk/ql/src/gen/thrift/gen-cpp/queryplan_types.h /hive/trunk/ql/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/ql/plan/api/StageType.java /hive/trunk/ql/src/gen/thrift/gen-php/queryplan/queryplan_types.php /hive/trunk/ql/src/gen/thrift/gen-py/queryplan/ttypes.py /hive/trunk/ql/src/gen/thrift/gen-rb/queryplan_types.rb /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsTask.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/TaskFactory.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteParseContextGenerator.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ExportSemanticAnalyzer.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/FunctionSemanticAnalyzer.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ImportSemanticAnalyzer.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/QB.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/QBParseInfo.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ColumnStatsDesc.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ColumnStatsWork.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/HiveOperation.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/DoubleNumDistinctValueEstimator.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFComputeStats.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/LongNumDistinctValueEstimator.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/NumDistinctValueEstimator.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/StringNumDistinctValueEstimator.java /hive/trunk/ql/src/test/queries/clientnegative/columnstats_partlvl_dp.q /hive/trunk/ql/src/test/queries/clientnegative/columnstats_partlvl_incorrect_num_keys.q /hive/trunk/ql/src/test/queries/clientnegative/columnstats_partlvl_invalid_values.q /hive/trunk/ql/src/test/queries/clientnegative/columnstats_partlvl_multiple_part_clause.q /hive/trunk/ql/src/test/queries/clientnegative/columnstats_tbllvl.q /hive/trunk/ql/src/test/queries/clientnegative/columnstats_tbllvl_complex_type.q /hive/trunk/ql/src/test/queries/clientnegative/columnstats_tbllvl_incorrect_column.q /hive/trunk/ql/src/test/queries/clientpositive/columnstats_partlvl.q /hive/trunk/ql/src/test/queries/clientpositive/columnstats_tbllvl.q /hive/trunk/ql/src/test/queries/clientpositive/compute_stats_binary.q /hive/trunk/ql/src/test/queries/clientpositive/compute_stats_boolean.q /hive/trunk/ql/src/test/queries/clientpositive/compute_stats_double.q /hive/trunk/ql/src/test/queries/clientpositive/compute_stats_long.q /hive/trunk/ql/src/test/queries/clientpositive/compute_stats_string.q /hive/trunk/ql/src/test/results/clientnegative/columnstats_partlvl_dp.q.out /hive/trunk/ql/src/test/results/clientnegative/columnstats_partlvl_incorrect_num_keys.q.out /hive/trunk/ql/src/test/results/clientnegative/columnstats_partlvl_invalid_values.q.out /hive/trunk/ql/src/test/results/clientnegative/columnstats_partlvl_multiple_part_clause.q.out /hive/trunk/ql/src/test/results/clientnegative/columnstats_tbllvl.q.out /hive/trunk/ql/src/test/results/clientnegative/columnstats_tbllvl_complex_type.q.out /hive/trunk/ql/src/test/results/clientnegative/columnstats_tbllvl_incorrect_column.q.out /hive/trunk/ql/src/test/results/clientpositive/columnstats_partlvl.q.out /hive/trunk/ql/src/test/results/clientpositive/columnstats_tbllvl.q.out /hive/trunk/ql/src/test/results/clientpositive/compute_stats_binary.q.out /hive/trunk/ql/src/test/results/clientpositive/compute_stats_boolean.q.out /hive/trunk/ql/src/test/results/clientpositive/compute_stats_double.q.out /hive/trunk/ql/src/test/results/clientpositive/compute_stats_long.q.out /hive/trunk/ql/src/test/results/clientpositive/compute_stats_string.q.out /hive/trunk/ql/src/test/results/clientpositive/show_functions.q.out /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorUtils.java
        Hide
        Ashutosh Chauhan added a comment -

        This issue is fixed and released as part of 0.10.0 release. If you find an issue which seems to be related to this one, please create a new jira and link this one with new jira.

        Show
        Ashutosh Chauhan added a comment - This issue is fixed and released as part of 0.10.0 release. If you find an issue which seems to be related to this one, please create a new jira and link this one with new jira.
        Ashutosh Chauhan made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Shreepadma Venugopalan made changes -
        Summary Column level scalar valued statistics on Tables and Partitions Optimizer statistics on columns in tables and partitions
        Gavin made changes -
        Link This issue relates to HIVE-3516 [ HIVE-3516 ]
        Gavin made changes -
        Link This issue relates to HIVE-3516 [ HIVE-3516 ]
        caofangkun made changes -
        Link This issue relates to HIVE-4561 [ HIVE-4561 ]
        Hide
        Sergey Shelukhin added a comment -

        This jira adds but doesn't use decimal fields in the schema... I am going to "reuse" them for HIVE-6701. We probably cannot use decimal due to derby limitations (31 precision max, Hive is 38), so between string and binary there might not be a difference that matters

        Show
        Sergey Shelukhin added a comment - This jira adds but doesn't use decimal fields in the schema... I am going to "reuse" them for HIVE-6701 . We probably cannot use decimal due to derby limitations (31 precision max, Hive is 38), so between string and binary there might not be a difference that matters
        Hide
        Lefty Leverenz added a comment -

        For the record: patch 10 adds the hive.stats.ndv.error configuration parameter to HiveConf.java and hive-default.xml.template.

        Show
        Lefty Leverenz added a comment - For the record: patch 10 adds the hive.stats.ndv.error configuration parameter to HiveConf.java and hive-default.xml.template.

          People

          • Assignee:
            Shreepadma Venugopalan
            Reporter:
            Ning Zhang
          • Votes:
            0 Vote for this issue
            Watchers:
            30 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development