Hive
  1. Hive
  2. HIVE-4115

Introduce cube abstraction in hive

    Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Won't Fix
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      We would like to define a cube abstraction so that user can query at cube layer and do not know anything about storage and rollups.

      Will describe the model more in following comments.

      1. cube-design.docx
        59 kB
        Amareshwari Sriramadasu
      2. cube-design-2.pdf
        75 kB
        Amareshwari Sriramadasu
      3. HIVE-4115.D10689.1.patch
        243 kB
        Phabricator
      4. HIVE-4115.D10689.2.patch
        278 kB
        Phabricator
      5. HIVE-4115.D10689.3.patch
        298 kB
        Phabricator
      6. HIVE-4115.D10689.4.patch
        306 kB
        Phabricator

        Activity

        Amareshwari Sriramadasu created issue -
        Hide
        Amareshwari Sriramadasu added a comment -

        Logical model :
        -----------------
        Cube :

        • A cube is a set of dimensions and measures in a particular subject.
        • A measure is a quantity that you are interested in measuring.
        • A dimension is an attribute, or set of attributes, by which you can divide measures into sub-categories.

        Fact Tables :

        • Cube will have fact tables associated with it.
        • A fact table would have subset of measures and dimensions.
        • Fact tables can be rolled at any dimension and time.

        Dimensions :

        • The cube dimension can refer to a dimension table
        • The cube dimension can have hierarchy of elements.

        Dimension tables :

        • A table with list of columns.
        • The table can have references to other dimension tables.
        • The dimension tables can be shared across cubes.

        Storage:

        • Fact or dimension table can have storages associated with it.

        Storage Model :
        ---------------------
        A physical table will be created in hive metastore for each fact, per storage per rollup.

        Show
        Amareshwari Sriramadasu added a comment - Logical model : ----------------- Cube : A cube is a set of dimensions and measures in a particular subject. A measure is a quantity that you are interested in measuring. A dimension is an attribute, or set of attributes, by which you can divide measures into sub-categories. Fact Tables : Cube will have fact tables associated with it. A fact table would have subset of measures and dimensions. Fact tables can be rolled at any dimension and time. Dimensions : The cube dimension can refer to a dimension table The cube dimension can have hierarchy of elements. Dimension tables : A table with list of columns. The table can have references to other dimension tables. The dimension tables can be shared across cubes. Storage : Fact or dimension table can have storages associated with it. Storage Model : --------------------- A physical table will be created in hive metastore for each fact, per storage per rollup.
        Hide
        Amareshwari Sriramadasu added a comment -

        Illustrating above model with an example :

        • Define a SALES_CUBE cube with measures : "Sales, Discount" and Dimensions: "CustomerID, Location, Transaction-time"
        • Dimensions:
          • CustomerID is a simple dimension which refers to the customer table on column ID. CustomerTable is having the schema : "ID, Age, Gender"
          • Location is hierarchical dimension with the hierarchy : "Zipcode, CityID, StateID, CountryID, RegionID"
            • Zipcode refers to ZipTable on column code. ZipTable schema : "code, street-name, cityID, stateID"
            • CityID refers to cityTable on column ID. CityTable schema : "ID, name, stateID"
            • stateID refers to stateTable on column ID. StateTable schema : "ID, name, capital, countryID"
            • countryID refers to counteryTable on column ID. CounterTable : "ID, name, capital, Region"
            • Region is an inline dimension with values "'APAC', 'EMEA', 'USA'"
          • Transaction-time is simple dimension with timestamp field.
        • Facts :Sales_cube can have the following fact tables :
          1. RawFact with columns "Sales, Discount, CustomerId, ZipCode, Transaction-time"
          2. CountryFact with columns "Sales, Discount, CountryID"

        Physical storage tables :
        ------------------------------------
        In the example described above say that RawFact is rolled hourly in Cluster c1, is rolled daily and monthly on Cluster C2; CountryFact is rolled daily, monthly, quarterly and yearly on Cluster C2; Also, Customer table is available in HBase cluster H1; All the location tables are available in HDFS cluster C2.

        The physical tables would be :

        • C1_Rawfact_hourly - schema : "Sales, Discount, CustomerId, ZipCode, Transaction-time" Partitioned by dt and state.
        • C2_Rawfact_daily - schema : "Sales, Discount, CustomerId, ZipCode, Transaction-time" Partitioned by dt and state.
        • C2_Rawfact_monthly - schema : "Sales, Discount, CustomerId, ZipCode, Transaction-time" Partitioned by dt and state.
        • C2_CountryFact_daily - Schema : "Sales, Discount, CountryID" Partitioned by dt
        • C2_CountryFact_monthly - Schema : "Sales, Discount, CountryID" Partitioned by dt
        • C2_CountryFact_quarterly - Schema : "Sales, Discount, CountryID" Partitioned by dt
        • C2_CountryFact_yearly - Schema : "Sales, Discount, CountryID" Partitioned by dt
        • H1_CustomerTable - schema : "ID, Age, Gender"
        • C2_ZipTable - schema : "code, street-name, cityID, stateID"
        • C2_CityTable - schema : "ID, name, stateID"
        • C2_StateTable -schema : "ID, name, capital, countryID"
        • C2_CountryTable -schema : "ID, name, capital, Region"

        If User queries the data on cube with a query like the following :

        • Select sales from SALES_CUBE where region = 'APAC' and time_range_in(09/01/2012, 12/31/2012) // Q4 -2012.

        Cube Abstraction provided would be smart enough to figure out which table to go and give the result . In this case the query translates to :

        • Select sales from C2_CountryFact_quarterly join C2_countryTable on C2_CountryFact_quarterly.CountryID = C2_countryTable.ID where dt = "Q4-2012" and C2_countryTable.region = 'APAC';
        Show
        Amareshwari Sriramadasu added a comment - Illustrating above model with an example : Define a SALES_CUBE cube with measures : "Sales, Discount" and Dimensions: "CustomerID, Location, Transaction-time" Dimensions: CustomerID is a simple dimension which refers to the customer table on column ID. CustomerTable is having the schema : "ID, Age, Gender" Location is hierarchical dimension with the hierarchy : "Zipcode, CityID, StateID, CountryID, RegionID" Zipcode refers to ZipTable on column code. ZipTable schema : "code, street-name, cityID, stateID" CityID refers to cityTable on column ID. CityTable schema : "ID, name, stateID" stateID refers to stateTable on column ID. StateTable schema : "ID, name, capital, countryID" countryID refers to counteryTable on column ID. CounterTable : "ID, name, capital, Region" Region is an inline dimension with values "'APAC', 'EMEA', 'USA'" Transaction-time is simple dimension with timestamp field. Facts :Sales_cube can have the following fact tables : RawFact with columns "Sales, Discount, CustomerId, ZipCode, Transaction-time" CountryFact with columns "Sales, Discount, CountryID" Physical storage tables : ------------------------------------ In the example described above say that RawFact is rolled hourly in Cluster c1, is rolled daily and monthly on Cluster C2; CountryFact is rolled daily, monthly, quarterly and yearly on Cluster C2; Also, Customer table is available in HBase cluster H1; All the location tables are available in HDFS cluster C2. The physical tables would be : C1_Rawfact_hourly - schema : "Sales, Discount, CustomerId, ZipCode, Transaction-time" Partitioned by dt and state. C2_Rawfact_daily - schema : "Sales, Discount, CustomerId, ZipCode, Transaction-time" Partitioned by dt and state. C2_Rawfact_monthly - schema : "Sales, Discount, CustomerId, ZipCode, Transaction-time" Partitioned by dt and state. C2_CountryFact_daily - Schema : "Sales, Discount, CountryID" Partitioned by dt C2_CountryFact_monthly - Schema : "Sales, Discount, CountryID" Partitioned by dt C2_CountryFact_quarterly - Schema : "Sales, Discount, CountryID" Partitioned by dt C2_CountryFact_yearly - Schema : "Sales, Discount, CountryID" Partitioned by dt H1_CustomerTable - schema : "ID, Age, Gender" C2_ZipTable - schema : "code, street-name, cityID, stateID" C2_CityTable - schema : "ID, name, stateID" C2_StateTable -schema : "ID, name, capital, countryID" C2_CountryTable -schema : "ID, name, capital, Region" If User queries the data on cube with a query like the following : Select sales from SALES_CUBE where region = 'APAC' and time_range_in(09/01/2012, 12/31/2012) // Q4 -2012. Cube Abstraction provided would be smart enough to figure out which table to go and give the result . In this case the query translates to : Select sales from C2_CountryFact_quarterly join C2_countryTable on C2_CountryFact_quarterly.CountryID = C2_countryTable.ID where dt = "Q4-2012" and C2_countryTable.region = 'APAC';
        Hide
        Amareshwari Sriramadasu added a comment -

        In the example described above say that RawFact is rolled hourly in Cluster c1, is rolled daily and monthly on Cluster C2; CountryFact is rolled daily, monthly, quarterly and yearly on Cluster C2; Also, Customer table is available in HBase cluster H1; All the location tables are available in HDFS cluster C2.

        Forgot to mention that, along with timely rolling RawFact is rolled at dimension state also.

        Show
        Amareshwari Sriramadasu added a comment - In the example described above say that RawFact is rolled hourly in Cluster c1, is rolled daily and monthly on Cluster C2; CountryFact is rolled daily, monthly, quarterly and yearly on Cluster C2; Also, Customer table is available in HBase cluster H1; All the location tables are available in HDFS cluster C2. Forgot to mention that, along with timely rolling RawFact is rolled at dimension state also.
        Hide
        Amareshwari Sriramadasu added a comment -

        Attaching the first cut design document for adding cube abstraction in hive.

        Pushed the code (being developed) to the branch HIVE-4115. Will be developing on the branch going forward.

        Show
        Amareshwari Sriramadasu added a comment - Attaching the first cut design document for adding cube abstraction in hive. Pushed the code (being developed) to the branch HIVE-4115 . Will be developing on the branch going forward.
        Amareshwari Sriramadasu made changes -
        Field Original Value New Value
        Attachment cube-design.docx [ 12573712 ]
        Hide
        Amareshwari Sriramadasu added a comment -

        Attaching the updated design doc

        Show
        Amareshwari Sriramadasu added a comment - Attaching the updated design doc
        Amareshwari Sriramadasu made changes -
        Attachment cube-design-2.pdf [ 12582270 ]
        Hide
        Phabricator added a comment -

        Amareshwari requested code review of "HIVE-4115 [jira] Introduce cube abstraction in hive".

        Reviewers: JIRA

        HIVE-4115. Cube Abstraction in Hive

        We would like to define a cube abstraction so that user can query at cube layer and do not know anything about storage and rollups.

        Will describe the model more in following comments.

        TEST PLAN
        EMPTY

        REVISION DETAIL
        https://reviews.facebook.net/D10689

        AFFECTED FILES
        common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
        ql/src/java/org/apache/hadoop/hive/ql/Driver.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/AbstractCubeTable.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/BaseDimension.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/ColumnMeasure.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/Cube.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/CubeDimension.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/CubeDimensionTable.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/CubeFactTable.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/CubeMeasure.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/CubeMetastoreClient.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/CubeTableType.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/ExprMeasure.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/HDFSStorage.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/HierarchicalDimension.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/InlineDimension.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/MetastoreConstants.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/MetastoreUtil.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/Named.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/ReferencedDimension.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/Storage.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/StorageConstants.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/TableReference.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/UpdatePeriod.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/parse/AggregateResolver.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/parse/AliasReplacer.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/parse/CheckColumnMapping.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/parse/CheckDateRange.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/parse/CheckTableNames.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/parse/ContextRewriter.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/parse/CubeQueryContext.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/parse/CubeQueryExpr.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/parse/CubeQueryRewriter.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/parse/CubeSemanticAnalyzer.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/parse/DateUtil.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/parse/GroupbyResolver.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/parse/HQLParser.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/parse/JoinResolver.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/parse/LeastDimensionResolver.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/parse/LeastPartitionResolver.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/parse/PartitionResolver.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/parse/StorageTableResolver.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/parse/ValidationRule.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/processors/CubeDriver.java
        ql/src/java/org/apache/hadoop/hive/ql/processors/CommandProcessorFactory.java
        ql/src/test/org/apache/hadoop/hive/ql/cube/metadata/TestCubeMetastoreClient.java
        ql/src/test/org/apache/hadoop/hive/ql/cube/parse/CubeTestSetup.java
        ql/src/test/org/apache/hadoop/hive/ql/cube/parse/TestCubeSemanticAnalyzer.java
        ql/src/test/org/apache/hadoop/hive/ql/cube/parse/TestDateUtil.java
        ql/src/test/org/apache/hadoop/hive/ql/cube/parse/TestMaxUpdateInterval.java
        ql/src/test/org/apache/hadoop/hive/ql/cube/processors/TestCubeDriver.java

        MANAGE HERALD RULES
        https://reviews.facebook.net/herald/view/differential/

        WHY DID I GET THIS EMAIL?
        https://reviews.facebook.net/herald/transcript/25647/

        To: JIRA, Amareshwari

        Show
        Phabricator added a comment - Amareshwari requested code review of " HIVE-4115 [jira] Introduce cube abstraction in hive". Reviewers: JIRA HIVE-4115 . Cube Abstraction in Hive We would like to define a cube abstraction so that user can query at cube layer and do not know anything about storage and rollups. Will describe the model more in following comments. TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D10689 AFFECTED FILES common/src/java/org/apache/hadoop/hive/conf/HiveConf.java ql/src/java/org/apache/hadoop/hive/ql/Driver.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/AbstractCubeTable.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/BaseDimension.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/ColumnMeasure.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/Cube.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/CubeDimension.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/CubeDimensionTable.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/CubeFactTable.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/CubeMeasure.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/CubeMetastoreClient.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/CubeTableType.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/ExprMeasure.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/HDFSStorage.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/HierarchicalDimension.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/InlineDimension.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/MetastoreConstants.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/MetastoreUtil.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/Named.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/ReferencedDimension.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/Storage.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/StorageConstants.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/TableReference.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/UpdatePeriod.java ql/src/java/org/apache/hadoop/hive/ql/cube/parse/AggregateResolver.java ql/src/java/org/apache/hadoop/hive/ql/cube/parse/AliasReplacer.java ql/src/java/org/apache/hadoop/hive/ql/cube/parse/CheckColumnMapping.java ql/src/java/org/apache/hadoop/hive/ql/cube/parse/CheckDateRange.java ql/src/java/org/apache/hadoop/hive/ql/cube/parse/CheckTableNames.java ql/src/java/org/apache/hadoop/hive/ql/cube/parse/ContextRewriter.java ql/src/java/org/apache/hadoop/hive/ql/cube/parse/CubeQueryContext.java ql/src/java/org/apache/hadoop/hive/ql/cube/parse/CubeQueryExpr.java ql/src/java/org/apache/hadoop/hive/ql/cube/parse/CubeQueryRewriter.java ql/src/java/org/apache/hadoop/hive/ql/cube/parse/CubeSemanticAnalyzer.java ql/src/java/org/apache/hadoop/hive/ql/cube/parse/DateUtil.java ql/src/java/org/apache/hadoop/hive/ql/cube/parse/GroupbyResolver.java ql/src/java/org/apache/hadoop/hive/ql/cube/parse/HQLParser.java ql/src/java/org/apache/hadoop/hive/ql/cube/parse/JoinResolver.java ql/src/java/org/apache/hadoop/hive/ql/cube/parse/LeastDimensionResolver.java ql/src/java/org/apache/hadoop/hive/ql/cube/parse/LeastPartitionResolver.java ql/src/java/org/apache/hadoop/hive/ql/cube/parse/PartitionResolver.java ql/src/java/org/apache/hadoop/hive/ql/cube/parse/StorageTableResolver.java ql/src/java/org/apache/hadoop/hive/ql/cube/parse/ValidationRule.java ql/src/java/org/apache/hadoop/hive/ql/cube/processors/CubeDriver.java ql/src/java/org/apache/hadoop/hive/ql/processors/CommandProcessorFactory.java ql/src/test/org/apache/hadoop/hive/ql/cube/metadata/TestCubeMetastoreClient.java ql/src/test/org/apache/hadoop/hive/ql/cube/parse/CubeTestSetup.java ql/src/test/org/apache/hadoop/hive/ql/cube/parse/TestCubeSemanticAnalyzer.java ql/src/test/org/apache/hadoop/hive/ql/cube/parse/TestDateUtil.java ql/src/test/org/apache/hadoop/hive/ql/cube/parse/TestMaxUpdateInterval.java ql/src/test/org/apache/hadoop/hive/ql/cube/processors/TestCubeDriver.java MANAGE HERALD RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/25647/ To: JIRA, Amareshwari
        Phabricator made changes -
        Attachment HIVE-4115.D10689.1.patch [ 12582433 ]
        Hide
        Amareshwari Sriramadasu added a comment -

        The branch HIVE-4115 is ready for review. Also, created the phabricator entry.

        Changes include :

        • ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/ has classes for Cube Metastore adn CubeMetastoreClient.java has the api to create cube, fact and dimension tables.
        • ql/src/java/org/apache/hadoop/hive/ql/cube/parse/ has code for validating the cube ql and converting the cube ql to HQL involving final storage tables
        • ql/src/java/org/apache/hadoop/hive/ql/cube/processors/CubeDriver.java is the entry point for the cube query. If query start with 'cube', it will be processed by CubeDriver.

        Will add Cube DDL in a followup jira.

        Show
        Amareshwari Sriramadasu added a comment - The branch HIVE-4115 is ready for review. Also, created the phabricator entry. Changes include : ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/ has classes for Cube Metastore adn CubeMetastoreClient.java has the api to create cube, fact and dimension tables. ql/src/java/org/apache/hadoop/hive/ql/cube/parse/ has code for validating the cube ql and converting the cube ql to HQL involving final storage tables ql/src/java/org/apache/hadoop/hive/ql/cube/processors/CubeDriver.java is the entry point for the cube query. If query start with 'cube', it will be processed by CubeDriver. Will add Cube DDL in a followup jira.
        Hide
        Phabricator added a comment -

        Amareshwari updated the revision "HIVE-4115 [jira] Introduce cube abstraction in hive".

        • Add weights to cube tables
        • Add configurations to specify valid fact or storage tables
        • Add configuration to specify valid columns in a fact
        • Add support for explain query
        • Add check for partition existence while resolving partitions to be queried

        Reviewers: JIRA, njain, alanfgates, omalley, cwsteinbach, ashutoshc

        REVISION DETAIL
        https://reviews.facebook.net/D10689

        CHANGE SINCE LAST DIFF
        https://reviews.facebook.net/D10689?vs=33441&id=34029#toc

        AFFECTED FILES
        ql/src/java/org/apache/hadoop/hive/ql/Driver.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/AbstractCubeTable.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/BaseDimension.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/ColumnMeasure.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/Cube.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/CubeDimension.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/CubeDimensionTable.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/CubeFactTable.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/CubeMeasure.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/CubeMetastoreClient.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/CubeTableType.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/ExprMeasure.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/HDFSStorage.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/HierarchicalDimension.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/InlineDimension.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/MetastoreConstants.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/MetastoreUtil.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/Named.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/ReferencedDimension.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/Storage.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/StorageConstants.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/TableReference.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/UpdatePeriod.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/parse/AggregateResolver.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/parse/AliasReplacer.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/parse/CheckColumnMapping.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/parse/CheckDateRange.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/parse/CheckTableNames.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/parse/ContextRewriter.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/parse/CubeQueryConstants.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/parse/CubeQueryContext.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/parse/CubeQueryExpr.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/parse/CubeQueryRewriter.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/parse/CubeSemanticAnalyzer.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/parse/DateUtil.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/parse/GroupbyResolver.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/parse/HQLParser.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/parse/JoinResolver.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/parse/LeastDimensionResolver.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/parse/LeastPartitionResolver.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/parse/LightestFactResolver.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/parse/PartitionResolver.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/parse/StorageTableResolver.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/parse/ValidationRule.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/processors/CubeDriver.java
        ql/src/java/org/apache/hadoop/hive/ql/processors/CommandProcessorFactory.java
        ql/src/test/org/apache/hadoop/hive/ql/cube/metadata/TestCubeMetastoreClient.java
        ql/src/test/org/apache/hadoop/hive/ql/cube/parse/CubeTestSetup.java
        ql/src/test/org/apache/hadoop/hive/ql/cube/parse/TestCubeSemanticAnalyzer.java
        ql/src/test/org/apache/hadoop/hive/ql/cube/parse/TestDateUtil.java
        ql/src/test/org/apache/hadoop/hive/ql/cube/parse/TestMaxUpdateInterval.java
        ql/src/test/org/apache/hadoop/hive/ql/cube/processors/TestCubeDriver.java

        To: JIRA, njain, alanfgates, omalley, cwsteinbach, ashutoshc, Amareshwari

        Show
        Phabricator added a comment - Amareshwari updated the revision " HIVE-4115 [jira] Introduce cube abstraction in hive". Add weights to cube tables Add configurations to specify valid fact or storage tables Add configuration to specify valid columns in a fact Add support for explain query Add check for partition existence while resolving partitions to be queried Reviewers: JIRA, njain, alanfgates, omalley, cwsteinbach, ashutoshc REVISION DETAIL https://reviews.facebook.net/D10689 CHANGE SINCE LAST DIFF https://reviews.facebook.net/D10689?vs=33441&id=34029#toc AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/Driver.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/AbstractCubeTable.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/BaseDimension.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/ColumnMeasure.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/Cube.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/CubeDimension.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/CubeDimensionTable.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/CubeFactTable.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/CubeMeasure.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/CubeMetastoreClient.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/CubeTableType.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/ExprMeasure.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/HDFSStorage.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/HierarchicalDimension.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/InlineDimension.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/MetastoreConstants.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/MetastoreUtil.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/Named.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/ReferencedDimension.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/Storage.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/StorageConstants.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/TableReference.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/UpdatePeriod.java ql/src/java/org/apache/hadoop/hive/ql/cube/parse/AggregateResolver.java ql/src/java/org/apache/hadoop/hive/ql/cube/parse/AliasReplacer.java ql/src/java/org/apache/hadoop/hive/ql/cube/parse/CheckColumnMapping.java ql/src/java/org/apache/hadoop/hive/ql/cube/parse/CheckDateRange.java ql/src/java/org/apache/hadoop/hive/ql/cube/parse/CheckTableNames.java ql/src/java/org/apache/hadoop/hive/ql/cube/parse/ContextRewriter.java ql/src/java/org/apache/hadoop/hive/ql/cube/parse/CubeQueryConstants.java ql/src/java/org/apache/hadoop/hive/ql/cube/parse/CubeQueryContext.java ql/src/java/org/apache/hadoop/hive/ql/cube/parse/CubeQueryExpr.java ql/src/java/org/apache/hadoop/hive/ql/cube/parse/CubeQueryRewriter.java ql/src/java/org/apache/hadoop/hive/ql/cube/parse/CubeSemanticAnalyzer.java ql/src/java/org/apache/hadoop/hive/ql/cube/parse/DateUtil.java ql/src/java/org/apache/hadoop/hive/ql/cube/parse/GroupbyResolver.java ql/src/java/org/apache/hadoop/hive/ql/cube/parse/HQLParser.java ql/src/java/org/apache/hadoop/hive/ql/cube/parse/JoinResolver.java ql/src/java/org/apache/hadoop/hive/ql/cube/parse/LeastDimensionResolver.java ql/src/java/org/apache/hadoop/hive/ql/cube/parse/LeastPartitionResolver.java ql/src/java/org/apache/hadoop/hive/ql/cube/parse/LightestFactResolver.java ql/src/java/org/apache/hadoop/hive/ql/cube/parse/PartitionResolver.java ql/src/java/org/apache/hadoop/hive/ql/cube/parse/StorageTableResolver.java ql/src/java/org/apache/hadoop/hive/ql/cube/parse/ValidationRule.java ql/src/java/org/apache/hadoop/hive/ql/cube/processors/CubeDriver.java ql/src/java/org/apache/hadoop/hive/ql/processors/CommandProcessorFactory.java ql/src/test/org/apache/hadoop/hive/ql/cube/metadata/TestCubeMetastoreClient.java ql/src/test/org/apache/hadoop/hive/ql/cube/parse/CubeTestSetup.java ql/src/test/org/apache/hadoop/hive/ql/cube/parse/TestCubeSemanticAnalyzer.java ql/src/test/org/apache/hadoop/hive/ql/cube/parse/TestDateUtil.java ql/src/test/org/apache/hadoop/hive/ql/cube/parse/TestMaxUpdateInterval.java ql/src/test/org/apache/hadoop/hive/ql/cube/processors/TestCubeDriver.java To: JIRA, njain, alanfgates, omalley, cwsteinbach, ashutoshc, Amareshwari
        Phabricator made changes -
        Attachment HIVE-4115.D10689.2.patch [ 12584902 ]
        Hide
        Phabricator added a comment -

        Amareshwari updated the revision "HIVE-4115 [jira] Introduce cube abstraction in hive".

        • Fix AliasReplacer
        • Queries with starting of the month as start period should be considered for MONTHLY update period
        • Add validations for all the tests in TestCubeDriver

        Reviewers: JIRA, njain, alanfgates, omalley, cwsteinbach, ashutoshc

        REVISION DETAIL
        https://reviews.facebook.net/D10689

        CHANGE SINCE LAST DIFF
        https://reviews.facebook.net/D10689?vs=34029&id=34299#toc

        AFFECTED FILES
        ql/src/java/org/apache/hadoop/hive/ql/Driver.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/AbstractCubeTable.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/BaseDimension.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/ColumnMeasure.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/Cube.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/CubeDimension.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/CubeDimensionTable.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/CubeFactTable.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/CubeMeasure.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/CubeMetastoreClient.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/CubeTableType.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/ExprMeasure.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/HDFSStorage.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/HierarchicalDimension.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/InlineDimension.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/MetastoreConstants.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/MetastoreUtil.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/Named.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/ReferencedDimension.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/Storage.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/StorageConstants.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/TableReference.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/UpdatePeriod.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/parse/AggregateResolver.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/parse/AliasReplacer.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/parse/CheckColumnMapping.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/parse/CheckDateRange.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/parse/CheckTableNames.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/parse/ContextRewriter.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/parse/CubeQueryConstants.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/parse/CubeQueryContext.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/parse/CubeQueryExpr.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/parse/CubeQueryRewriter.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/parse/CubeSemanticAnalyzer.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/parse/DateUtil.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/parse/GroupbyResolver.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/parse/HQLParser.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/parse/JoinResolver.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/parse/LeastDimensionResolver.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/parse/LeastPartitionResolver.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/parse/LightestFactResolver.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/parse/PartitionResolver.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/parse/StorageTableResolver.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/parse/ValidationRule.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/processors/CubeDriver.java
        ql/src/java/org/apache/hadoop/hive/ql/processors/CommandProcessorFactory.java
        ql/src/test/org/apache/hadoop/hive/ql/cube/metadata/TestCubeMetastoreClient.java
        ql/src/test/org/apache/hadoop/hive/ql/cube/parse/CubeTestSetup.java
        ql/src/test/org/apache/hadoop/hive/ql/cube/parse/TestCubeSemanticAnalyzer.java
        ql/src/test/org/apache/hadoop/hive/ql/cube/parse/TestDateUtil.java
        ql/src/test/org/apache/hadoop/hive/ql/cube/parse/TestMaxUpdateInterval.java
        ql/src/test/org/apache/hadoop/hive/ql/cube/processors/TestCubeDriver.java

        To: JIRA, njain, alanfgates, omalley, cwsteinbach, ashutoshc, Amareshwari

        Show
        Phabricator added a comment - Amareshwari updated the revision " HIVE-4115 [jira] Introduce cube abstraction in hive". Fix AliasReplacer Queries with starting of the month as start period should be considered for MONTHLY update period Add validations for all the tests in TestCubeDriver Reviewers: JIRA, njain, alanfgates, omalley, cwsteinbach, ashutoshc REVISION DETAIL https://reviews.facebook.net/D10689 CHANGE SINCE LAST DIFF https://reviews.facebook.net/D10689?vs=34029&id=34299#toc AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/Driver.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/AbstractCubeTable.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/BaseDimension.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/ColumnMeasure.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/Cube.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/CubeDimension.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/CubeDimensionTable.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/CubeFactTable.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/CubeMeasure.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/CubeMetastoreClient.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/CubeTableType.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/ExprMeasure.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/HDFSStorage.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/HierarchicalDimension.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/InlineDimension.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/MetastoreConstants.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/MetastoreUtil.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/Named.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/ReferencedDimension.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/Storage.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/StorageConstants.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/TableReference.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/UpdatePeriod.java ql/src/java/org/apache/hadoop/hive/ql/cube/parse/AggregateResolver.java ql/src/java/org/apache/hadoop/hive/ql/cube/parse/AliasReplacer.java ql/src/java/org/apache/hadoop/hive/ql/cube/parse/CheckColumnMapping.java ql/src/java/org/apache/hadoop/hive/ql/cube/parse/CheckDateRange.java ql/src/java/org/apache/hadoop/hive/ql/cube/parse/CheckTableNames.java ql/src/java/org/apache/hadoop/hive/ql/cube/parse/ContextRewriter.java ql/src/java/org/apache/hadoop/hive/ql/cube/parse/CubeQueryConstants.java ql/src/java/org/apache/hadoop/hive/ql/cube/parse/CubeQueryContext.java ql/src/java/org/apache/hadoop/hive/ql/cube/parse/CubeQueryExpr.java ql/src/java/org/apache/hadoop/hive/ql/cube/parse/CubeQueryRewriter.java ql/src/java/org/apache/hadoop/hive/ql/cube/parse/CubeSemanticAnalyzer.java ql/src/java/org/apache/hadoop/hive/ql/cube/parse/DateUtil.java ql/src/java/org/apache/hadoop/hive/ql/cube/parse/GroupbyResolver.java ql/src/java/org/apache/hadoop/hive/ql/cube/parse/HQLParser.java ql/src/java/org/apache/hadoop/hive/ql/cube/parse/JoinResolver.java ql/src/java/org/apache/hadoop/hive/ql/cube/parse/LeastDimensionResolver.java ql/src/java/org/apache/hadoop/hive/ql/cube/parse/LeastPartitionResolver.java ql/src/java/org/apache/hadoop/hive/ql/cube/parse/LightestFactResolver.java ql/src/java/org/apache/hadoop/hive/ql/cube/parse/PartitionResolver.java ql/src/java/org/apache/hadoop/hive/ql/cube/parse/StorageTableResolver.java ql/src/java/org/apache/hadoop/hive/ql/cube/parse/ValidationRule.java ql/src/java/org/apache/hadoop/hive/ql/cube/processors/CubeDriver.java ql/src/java/org/apache/hadoop/hive/ql/processors/CommandProcessorFactory.java ql/src/test/org/apache/hadoop/hive/ql/cube/metadata/TestCubeMetastoreClient.java ql/src/test/org/apache/hadoop/hive/ql/cube/parse/CubeTestSetup.java ql/src/test/org/apache/hadoop/hive/ql/cube/parse/TestCubeSemanticAnalyzer.java ql/src/test/org/apache/hadoop/hive/ql/cube/parse/TestDateUtil.java ql/src/test/org/apache/hadoop/hive/ql/cube/parse/TestMaxUpdateInterval.java ql/src/test/org/apache/hadoop/hive/ql/cube/processors/TestCubeDriver.java To: JIRA, njain, alanfgates, omalley, cwsteinbach, ashutoshc, Amareshwari
        Phabricator made changes -
        Attachment HIVE-4115.D10689.3.patch [ 12586289 ]
        Hide
        Phabricator added a comment -

        Amareshwari updated the revision "HIVE-4115 [jira] Introduce cube abstraction in hive".

        • Make union query as subquery
        • Add javadoc to cube metastore api

        Reviewers: JIRA, njain, alanfgates, omalley, cwsteinbach, ashutoshc

        REVISION DETAIL
        https://reviews.facebook.net/D10689

        CHANGE SINCE LAST DIFF
        https://reviews.facebook.net/D10689?vs=34299&id=34395#toc

        AFFECTED FILES
        ql/src/java/org/apache/hadoop/hive/ql/Driver.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/AbstractCubeTable.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/BaseDimension.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/ColumnMeasure.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/Cube.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/CubeDimension.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/CubeDimensionTable.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/CubeFactTable.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/CubeMeasure.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/CubeMetastoreClient.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/CubeTableType.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/ExprMeasure.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/HDFSStorage.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/HierarchicalDimension.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/InlineDimension.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/MetastoreConstants.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/MetastoreUtil.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/Named.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/ReferencedDimension.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/Storage.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/StorageConstants.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/TableReference.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/UpdatePeriod.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/parse/AggregateResolver.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/parse/AliasReplacer.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/parse/CheckColumnMapping.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/parse/CheckDateRange.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/parse/CheckTableNames.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/parse/ContextRewriter.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/parse/CubeQueryConstants.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/parse/CubeQueryContext.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/parse/CubeQueryExpr.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/parse/CubeQueryRewriter.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/parse/CubeSemanticAnalyzer.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/parse/DateUtil.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/parse/GroupbyResolver.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/parse/HQLParser.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/parse/JoinResolver.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/parse/LeastDimensionResolver.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/parse/LeastPartitionResolver.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/parse/LightestFactResolver.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/parse/PartitionResolver.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/parse/StorageTableResolver.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/parse/ValidationRule.java
        ql/src/java/org/apache/hadoop/hive/ql/cube/processors/CubeDriver.java
        ql/src/java/org/apache/hadoop/hive/ql/processors/CommandProcessorFactory.java
        ql/src/test/org/apache/hadoop/hive/ql/cube/metadata/TestCubeMetastoreClient.java
        ql/src/test/org/apache/hadoop/hive/ql/cube/parse/CubeTestSetup.java
        ql/src/test/org/apache/hadoop/hive/ql/cube/parse/TestCubeSemanticAnalyzer.java
        ql/src/test/org/apache/hadoop/hive/ql/cube/parse/TestDateUtil.java
        ql/src/test/org/apache/hadoop/hive/ql/cube/parse/TestMaxUpdateInterval.java
        ql/src/test/org/apache/hadoop/hive/ql/cube/processors/TestCubeDriver.java

        To: JIRA, njain, alanfgates, omalley, cwsteinbach, ashutoshc, Amareshwari

        Show
        Phabricator added a comment - Amareshwari updated the revision " HIVE-4115 [jira] Introduce cube abstraction in hive". Make union query as subquery Add javadoc to cube metastore api Reviewers: JIRA, njain, alanfgates, omalley, cwsteinbach, ashutoshc REVISION DETAIL https://reviews.facebook.net/D10689 CHANGE SINCE LAST DIFF https://reviews.facebook.net/D10689?vs=34299&id=34395#toc AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/Driver.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/AbstractCubeTable.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/BaseDimension.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/ColumnMeasure.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/Cube.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/CubeDimension.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/CubeDimensionTable.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/CubeFactTable.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/CubeMeasure.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/CubeMetastoreClient.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/CubeTableType.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/ExprMeasure.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/HDFSStorage.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/HierarchicalDimension.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/InlineDimension.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/MetastoreConstants.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/MetastoreUtil.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/Named.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/ReferencedDimension.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/Storage.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/StorageConstants.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/TableReference.java ql/src/java/org/apache/hadoop/hive/ql/cube/metadata/UpdatePeriod.java ql/src/java/org/apache/hadoop/hive/ql/cube/parse/AggregateResolver.java ql/src/java/org/apache/hadoop/hive/ql/cube/parse/AliasReplacer.java ql/src/java/org/apache/hadoop/hive/ql/cube/parse/CheckColumnMapping.java ql/src/java/org/apache/hadoop/hive/ql/cube/parse/CheckDateRange.java ql/src/java/org/apache/hadoop/hive/ql/cube/parse/CheckTableNames.java ql/src/java/org/apache/hadoop/hive/ql/cube/parse/ContextRewriter.java ql/src/java/org/apache/hadoop/hive/ql/cube/parse/CubeQueryConstants.java ql/src/java/org/apache/hadoop/hive/ql/cube/parse/CubeQueryContext.java ql/src/java/org/apache/hadoop/hive/ql/cube/parse/CubeQueryExpr.java ql/src/java/org/apache/hadoop/hive/ql/cube/parse/CubeQueryRewriter.java ql/src/java/org/apache/hadoop/hive/ql/cube/parse/CubeSemanticAnalyzer.java ql/src/java/org/apache/hadoop/hive/ql/cube/parse/DateUtil.java ql/src/java/org/apache/hadoop/hive/ql/cube/parse/GroupbyResolver.java ql/src/java/org/apache/hadoop/hive/ql/cube/parse/HQLParser.java ql/src/java/org/apache/hadoop/hive/ql/cube/parse/JoinResolver.java ql/src/java/org/apache/hadoop/hive/ql/cube/parse/LeastDimensionResolver.java ql/src/java/org/apache/hadoop/hive/ql/cube/parse/LeastPartitionResolver.java ql/src/java/org/apache/hadoop/hive/ql/cube/parse/LightestFactResolver.java ql/src/java/org/apache/hadoop/hive/ql/cube/parse/PartitionResolver.java ql/src/java/org/apache/hadoop/hive/ql/cube/parse/StorageTableResolver.java ql/src/java/org/apache/hadoop/hive/ql/cube/parse/ValidationRule.java ql/src/java/org/apache/hadoop/hive/ql/cube/processors/CubeDriver.java ql/src/java/org/apache/hadoop/hive/ql/processors/CommandProcessorFactory.java ql/src/test/org/apache/hadoop/hive/ql/cube/metadata/TestCubeMetastoreClient.java ql/src/test/org/apache/hadoop/hive/ql/cube/parse/CubeTestSetup.java ql/src/test/org/apache/hadoop/hive/ql/cube/parse/TestCubeSemanticAnalyzer.java ql/src/test/org/apache/hadoop/hive/ql/cube/parse/TestDateUtil.java ql/src/test/org/apache/hadoop/hive/ql/cube/parse/TestMaxUpdateInterval.java ql/src/test/org/apache/hadoop/hive/ql/cube/processors/TestCubeDriver.java To: JIRA, njain, alanfgates, omalley, cwsteinbach, ashutoshc, Amareshwari
        Phabricator made changes -
        Attachment HIVE-4115.D10689.4.patch [ 12586709 ]
        Hide
        Amareshwari Sriramadasu added a comment -

        Code is ready for review and checkin. Changing the status

        Show
        Amareshwari Sriramadasu added a comment - Code is ready for review and checkin. Changing the status
        Amareshwari Sriramadasu made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Damien Carol added a comment -

        Amareshwari Sriramadasu I think using MDX would be a must hvae.

        MDX is a query language that every OLAP BI tool use. Plus, OSS like Pentaho+Mondrian may query HIVE with this query language.

        A good syntax could be :

        MDX [MDX Query]

        If you can deve MDX part, I can help you to integrate it in Pentaho BI tool.

        Show
        Damien Carol added a comment - Amareshwari Sriramadasu I think using MDX would be a must hvae. MDX is a query language that every OLAP BI tool use. Plus, OSS like Pentaho+Mondrian may query HIVE with this query language. A good syntax could be : MDX [MDX Query] If you can deve MDX part, I can help you to integrate it in Pentaho BI tool.
        Hide
        Hive QA added a comment -

        Overall: -1 no tests executed

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12586709/HIVE-4115.D10689.4.patch

        Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1286/testReport
        Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1286/console

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Tests exited with: NonZeroExitCodeException
        Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ [[ -n '' ]]
        + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
        + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
        + export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128'
        + M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128'
        + cd /data/hive-ptest/working/
        + tee /data/hive-ptest/logs/PreCommit-HIVE-Build-1286/source-prep.txt
        + [[ false == \t\r\u\e ]]
        + mkdir -p maven ivy
        + [[ svn = \s\v\n ]]
        + [[ -n '' ]]
        + [[ -d apache-svn-trunk-source ]]
        + [[ ! -d apache-svn-trunk-source/.svn ]]
        + [[ ! -d apache-svn-trunk-source ]]
        + cd apache-svn-trunk-source
        + svn revert -R .
        Reverted 'itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java'
        Reverted 'common/src/java/org/apache/hadoop/hive/conf/HiveConf.java'
        Reverted 'ql/src/test/results/clientnegative/authorization_role_cycles1.q.out'
        Reverted 'ql/src/test/results/clientnegative/authorization_role_cycles2.q.out'
        Reverted 'ql/src/test/results/clientpositive/authorization_1_sql_std.q.out'
        Reverted 'ql/src/test/results/clientpositive/authorization_set_show_current_role.q.out'
        Reverted 'ql/src/test/results/clientpositive/authorization_role_grant1.q.out'
        Reverted 'ql/src/test/queries/clientnegative/authorization_role_cycles1.q'
        Reverted 'ql/src/test/queries/clientnegative/authorization_role_cycles2.q'
        Reverted 'ql/src/test/queries/clientpositive/authorization_set_show_current_role.q'
        Reverted 'ql/src/test/queries/clientpositive/authorization_role_grant1.q'
        Reverted 'ql/src/test/queries/clientpositive/authorization_1_sql_std.q'
        Reverted 'ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/HiveAuthorizer.java'
        Reverted 'ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/HiveAccessController.java'
        Reverted 'ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/HiveAuthorizerImpl.java'
        Reverted 'ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/sqlstd/SQLStdHiveAccessController.java'
        Reverted 'ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/HiveAccessControlException.java'
        ++ egrep -v '^X|^Performing status on external'
        ++ awk '{print $2}'
        ++ svn status --no-ignore
        + rm -rf target datanucleus.log ant/target shims/target shims/0.20/target shims/0.20S/target shims/0.23/target shims/aggregator/target shims/common/target shims/common-secure/target packaging/target hbase-handler/target testutils/target jdbc/target metastore/target itests/target itests/hcatalog-unit/target itests/test-serde/target itests/qtest/target itests/hive-unit/target itests/custom-serde/target itests/util/target hcatalog/target hcatalog/storage-handlers/hbase/target hcatalog/server-extensions/target hcatalog/core/target hcatalog/webhcat/svr/target hcatalog/webhcat/java-client/target hcatalog/hcatalog-pig-adapter/target hwi/target common/target common/src/gen service/target contrib/target serde/target beeline/target odbc/target cli/target ql/dependency-reduced-pom.xml ql/target ql/src/test/results/clientnegative/authorization_show_roles_no_admin.q.out ql/src/test/results/clientnegative/authorization_create_role_no_admin.q.out ql/src/test/results/clientnegative/authorization_drop_role_no_admin.q.out ql/src/test/results/clientpositive/authorization_set_show_current_role.q.out.orig ql/src/test/queries/clientnegative/authorization_show_roles_no_admin.q ql/src/test/queries/clientnegative/authorization_create_role_no_admin.q ql/src/test/queries/clientnegative/authorization_drop_role_no_admin.q
        + svn update
        
        Fetching external item into 'hcatalog/src/test/e2e/harness'
        External at revision 1567657.
        
        At revision 1567657.
        + patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
        + patchFilePath=/data/hive-ptest/working/scratch/build.patch
        + [[ -f /data/hive-ptest/working/scratch/build.patch ]]
        + chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
        + /data/hive-ptest/working/scratch/smart-apply-patch.sh /data/hive-ptest/working/scratch/build.patch
        The patch does not appear to apply with p0, p1, or p2
        + exit 1
        '
        

        This message is automatically generated.

        ATTACHMENT ID: 12586709

        Show
        Hive QA added a comment - Overall : -1 no tests executed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12586709/HIVE-4115.D10689.4.patch Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1286/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1286/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ [[ -n '' ]] + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + cd /data/hive-ptest/working/ + tee /data/hive-ptest/logs/PreCommit-HIVE-Build-1286/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ svn = \s\v\n ]] + [[ -n '' ]] + [[ -d apache-svn-trunk-source ]] + [[ ! -d apache-svn-trunk-source/.svn ]] + [[ ! -d apache-svn-trunk-source ]] + cd apache-svn-trunk-source + svn revert -R . Reverted 'itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java' Reverted 'common/src/java/org/apache/hadoop/hive/conf/HiveConf.java' Reverted 'ql/src/test/results/clientnegative/authorization_role_cycles1.q.out' Reverted 'ql/src/test/results/clientnegative/authorization_role_cycles2.q.out' Reverted 'ql/src/test/results/clientpositive/authorization_1_sql_std.q.out' Reverted 'ql/src/test/results/clientpositive/authorization_set_show_current_role.q.out' Reverted 'ql/src/test/results/clientpositive/authorization_role_grant1.q.out' Reverted 'ql/src/test/queries/clientnegative/authorization_role_cycles1.q' Reverted 'ql/src/test/queries/clientnegative/authorization_role_cycles2.q' Reverted 'ql/src/test/queries/clientpositive/authorization_set_show_current_role.q' Reverted 'ql/src/test/queries/clientpositive/authorization_role_grant1.q' Reverted 'ql/src/test/queries/clientpositive/authorization_1_sql_std.q' Reverted 'ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/HiveAuthorizer.java' Reverted 'ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/HiveAccessController.java' Reverted 'ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/HiveAuthorizerImpl.java' Reverted 'ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/sqlstd/SQLStdHiveAccessController.java' Reverted 'ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/HiveAccessControlException.java' ++ egrep -v '^X|^Performing status on external' ++ awk '{print $2}' ++ svn status --no-ignore + rm -rf target datanucleus.log ant/target shims/target shims/0.20/target shims/0.20S/target shims/0.23/target shims/aggregator/target shims/common/target shims/common-secure/target packaging/target hbase-handler/target testutils/target jdbc/target metastore/target itests/target itests/hcatalog-unit/target itests/test-serde/target itests/qtest/target itests/hive-unit/target itests/custom-serde/target itests/util/target hcatalog/target hcatalog/storage-handlers/hbase/target hcatalog/server-extensions/target hcatalog/core/target hcatalog/webhcat/svr/target hcatalog/webhcat/java-client/target hcatalog/hcatalog-pig-adapter/target hwi/target common/target common/src/gen service/target contrib/target serde/target beeline/target odbc/target cli/target ql/dependency-reduced-pom.xml ql/target ql/src/test/results/clientnegative/authorization_show_roles_no_admin.q.out ql/src/test/results/clientnegative/authorization_create_role_no_admin.q.out ql/src/test/results/clientnegative/authorization_drop_role_no_admin.q.out ql/src/test/results/clientpositive/authorization_set_show_current_role.q.out.orig ql/src/test/queries/clientnegative/authorization_show_roles_no_admin.q ql/src/test/queries/clientnegative/authorization_create_role_no_admin.q ql/src/test/queries/clientnegative/authorization_drop_role_no_admin.q + svn update Fetching external item into 'hcatalog/src/test/e2e/harness' External at revision 1567657. At revision 1567657. + patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hive-ptest/working/scratch/build.patch + [[ -f /data/hive-ptest/working/scratch/build.patch ]] + chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh + /data/hive-ptest/working/scratch/smart-apply-patch.sh /data/hive-ptest/working/scratch/build.patch The patch does not appear to apply with p0, p1, or p2 + exit 1 ' This message is automatically generated. ATTACHMENT ID: 12586709
        Hide
        Amareshwari Sriramadasu added a comment -

        This effort has been incubated into apache here - http://incubator.apache.org/projects/lens.html

        Show
        Amareshwari Sriramadasu added a comment - This effort has been incubated into apache here - http://incubator.apache.org/projects/lens.html
        Amareshwari Sriramadasu made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Won't Fix [ 2 ]
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Patch Available Patch Available
        97d 1h 4m 1 Amareshwari Sriramadasu 10/Jun/13 12:23
        Patch Available Patch Available Resolved Resolved
        522d 27m 1 Amareshwari Sriramadasu 14/Nov/14 11:50

          People

          • Assignee:
            Amareshwari Sriramadasu
            Reporter:
            Amareshwari Sriramadasu
          • Votes:
            0 Vote for this issue
            Watchers:
            26 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development