Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • v0.7.2
    • v1.1, v1.4.0
    • None
    • None
    • Sprint 47

    Description

      While trying to save a cube with a derived column I get the following error:

      Caused by: java.lang.IllegalArgumentException: Cannot find rowkey column PART_DT in cube CubeDesc [name=test]
              at org.apache.kylin.cube.model.RowKeyDesc.buildRowKey(RowKeyDesc.java:193)
              at org.apache.kylin.cube.model.RowKeyDesc.init(RowKeyDesc.java:172)
              at org.apache.kylin.cube.model.CubeDesc.init(CubeDesc.java:448)
              at org.apache.kylin.cube.CubeDescManager.createCubeDesc(CubeDescManager.java:150)
              at org.apache.kylin.rest.service.CubeService.createCubeAndDesc(CubeService.java:156)
      

      cubeDescData:

      {"name":"test","description":"","dimensions":[{"name":"DEFAULT.KYLIN_SALES.PART_DT","table":"DEFAULT.KYLIN_SALES","hierarchy":false,"derived":null,"column":["PART_DT"],"id":1},{"name":"DEFAULT.KYLIN_SALES.SELLER_ID","table":"DEFAULT.KYLIN_SALES","hierarchy":false,"derived":null,"column":["SELLER_ID"],"id":2},{"name":"DEFAULT.KYLIN_CAL_DT_derived","table":"DEFAULT.KYLIN_CAL_DT","hierarchy":false,"derived":["CAL_DT"],"column":null,"id":3}],"measures":[{"id":1,"name":"_COUNT_","function":{"expression":"COUNT","returntype":"bigint","parameter":{"type":"constant","value":"1"}}}],"rowkey":{"rowkey_columns":[{"column":"PART_DT","length":0,"dictionary":"true","mandatory":false},{"column":"SELLER_ID","length":0,"dictionary":"true","mandatory":false}],"aggregation_groups":[["PART_DT","SELLER_ID"]]},"notify_list":[],"capacity":"","hbase_mapping":{"column_family":[{"name":"f1","columns":[{"qualifier":"m","measure_refs":["_COUNT_"]}]}]},"project":"learn_kylin","model_name":"test"}
      

      modelDescData:

      {"name":"test","fact_table":"DEFAULT.KYLIN_SALES","lookups":[{"table":"DEFAULT.KYLIN_CAL_DT","join":{"type":"left","primary_key":["CAL_DT"],"foreign_key":["PART_DT"]}}],"filter_condition":"","capacity":"MEDIUM","partition_desc":{"partition_date_column":"","partition_date_start":0,"partition_type":"APPEND"},"last_modified":0}
      

      The issue comes from PART_DT being in CubeDesc:derivedToHostMap therefore it's excluded from list of dimension columns in CubeDesc:listDimensionColumnsExcludingDerived(). So colNameAbbr doesn't have PART_DT in RowKeyDesc:170 which causes the exception.

      Attachments

        1. Cube Dimensions.png
          36 kB
          Vadim Semenov
        2. Cube Data Model.png
          17 kB
          Vadim Semenov
        3. Comparison.png
          849 kB
          Vadim Semenov

        Issue Links

        Activity

          shaofengshi Shao Feng Shi added a comment -

          Resolved in release 1.1-incubating (2015-10-25)

          shaofengshi Shao Feng Shi added a comment - Resolved in release 1.1-incubating (2015-10-25)
          liyang.gmt8@gmail.com liyang added a comment -

          Debug through and turns out this is because of PK declared as derived dimension explicitly.

          FK automatically derives PK. However user unaware of this can declare PK as derived dimension explicitly and cause this error. Have it fixed.

          liyang.gmt8@gmail.com liyang added a comment - Debug through and turns out this is because of PK declared as derived dimension explicitly. FK automatically derives PK. However user unaware of this can declare PK as derived dimension explicitly and cause this error. Have it fixed.
          vadim.semenov Vadim Semenov added a comment -

          The issue is still present

          vadim.semenov Vadim Semenov added a comment - The issue is still present
          buryat Vadim Semenov added a comment -

          Tested branch 0.7-staging (latest commit 572bcbc1bf9fce7332124063fb119b8f9ac90ae6), the issue is still present.

          Loaded sample_cube data, and tried to create a cube with the following definition (The same as I reported initially):
          fact table = KYLIN_SALES
          KYLIN_SALES.PART_DT = KYLIN_CAL_DT.CAL_DT

          normal dimensions: KYLIN_SALES.PART_DT, KYLIN_SALES.SELLER_ID
          derived dimension: KYLIN_CAL_DT.CAL_DT

          Got the same exception.

          buryat Vadim Semenov added a comment - Tested branch 0.7-staging (latest commit 572bcbc1bf9fce7332124063fb119b8f9ac90ae6), the issue is still present. Loaded sample_cube data, and tried to create a cube with the following definition (The same as I reported initially): fact table = KYLIN_SALES KYLIN_SALES.PART_DT = KYLIN_CAL_DT.CAL_DT normal dimensions: KYLIN_SALES.PART_DT, KYLIN_SALES.SELLER_ID derived dimension: KYLIN_CAL_DT.CAL_DT Got the same exception.
          liyang.gmt8@gmail.com liyang added a comment -

          Hi Vadim, there were commits and rollbacks about how PK-FK derive each other. I do believe finally it's resolve on current 0.7 branch. Please help verify again. Many thanks!

          liyang.gmt8@gmail.com liyang added a comment - Hi Vadim, there were commits and rollbacks about how PK-FK derive each other. I do believe finally it's resolve on current 0.7 branch. Please help verify again. Many thanks!
          buryat Vadim Semenov added a comment -

          liyang At the time I was testing after this commit, and actually this commit introduced the bug, I had to revert `initDimensionColRef` on my deployment.
          I'll test once again with the latest 0.7-staging

          buryat Vadim Semenov added a comment - liyang At the time I was testing after this commit, and actually this commit introduced the bug, I had to revert `initDimensionColRef` on my deployment. I'll test once again with the latest 0.7-staging
          liyang.gmt8@gmail.com liyang added a comment -

          I believe this has been fixed by https://github.com/apache/incubator-kylin/commit/30f4c6ab31dfb20e4dc7f5e78c5772e30e04d64b

          I've verified 0.7 branch with

          • fact.CAL_DT=lookup.PART_DT
          • derive WEEK_BEGIN_DT from lookup
          • hierarchy of YEAR_BEGIN_DT, MONTH_BEGIN_DT, PART_DT from lookup

          Cube save successful in all cases.

          liyang.gmt8@gmail.com liyang added a comment - I believe this has been fixed by https://github.com/apache/incubator-kylin/commit/30f4c6ab31dfb20e4dc7f5e78c5772e30e04d64b I've verified 0.7 branch with fact.CAL_DT=lookup.PART_DT derive WEEK_BEGIN_DT from lookup hierarchy of YEAR_BEGIN_DT, MONTH_BEGIN_DT, PART_DT from lookup Cube save successful in all cases.

          People

            liyang.gmt8@gmail.com liyang
            vadim.semenov Vadim Semenov
            Votes:
            0 Vote for this issue
            Watchers:
            Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Agile

                Completed Sprint:
                Sprint 47 ended 25/Aug/15
                View on Board

                Slack

                  Error getting Slack data from Jira

                  Issue deployment

                    Client must be authenticated to access this resource.