Uploaded image for project: 'Kylin'
  1. Kylin
  2. KYLIN-913

Cannot find rowkey column XXX in cube CubeDesc

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • v0.7.2
    • v1.1, v1.4.0
    • None
    • None
    • Sprint 47

    Description

      While trying to save a cube with a derived column I get the following error:

      Caused by: java.lang.IllegalArgumentException: Cannot find rowkey column PART_DT in cube CubeDesc [name=test]
              at org.apache.kylin.cube.model.RowKeyDesc.buildRowKey(RowKeyDesc.java:193)
              at org.apache.kylin.cube.model.RowKeyDesc.init(RowKeyDesc.java:172)
              at org.apache.kylin.cube.model.CubeDesc.init(CubeDesc.java:448)
              at org.apache.kylin.cube.CubeDescManager.createCubeDesc(CubeDescManager.java:150)
              at org.apache.kylin.rest.service.CubeService.createCubeAndDesc(CubeService.java:156)
      

      cubeDescData:

      {"name":"test","description":"","dimensions":[{"name":"DEFAULT.KYLIN_SALES.PART_DT","table":"DEFAULT.KYLIN_SALES","hierarchy":false,"derived":null,"column":["PART_DT"],"id":1},{"name":"DEFAULT.KYLIN_SALES.SELLER_ID","table":"DEFAULT.KYLIN_SALES","hierarchy":false,"derived":null,"column":["SELLER_ID"],"id":2},{"name":"DEFAULT.KYLIN_CAL_DT_derived","table":"DEFAULT.KYLIN_CAL_DT","hierarchy":false,"derived":["CAL_DT"],"column":null,"id":3}],"measures":[{"id":1,"name":"_COUNT_","function":{"expression":"COUNT","returntype":"bigint","parameter":{"type":"constant","value":"1"}}}],"rowkey":{"rowkey_columns":[{"column":"PART_DT","length":0,"dictionary":"true","mandatory":false},{"column":"SELLER_ID","length":0,"dictionary":"true","mandatory":false}],"aggregation_groups":[["PART_DT","SELLER_ID"]]},"notify_list":[],"capacity":"","hbase_mapping":{"column_family":[{"name":"f1","columns":[{"qualifier":"m","measure_refs":["_COUNT_"]}]}]},"project":"learn_kylin","model_name":"test"}
      

      modelDescData:

      {"name":"test","fact_table":"DEFAULT.KYLIN_SALES","lookups":[{"table":"DEFAULT.KYLIN_CAL_DT","join":{"type":"left","primary_key":["CAL_DT"],"foreign_key":["PART_DT"]}}],"filter_condition":"","capacity":"MEDIUM","partition_desc":{"partition_date_column":"","partition_date_start":0,"partition_type":"APPEND"},"last_modified":0}
      

      The issue comes from PART_DT being in CubeDesc:derivedToHostMap therefore it's excluded from list of dimension columns in CubeDesc:listDimensionColumnsExcludingDerived(). So colNameAbbr doesn't have PART_DT in RowKeyDesc:170 which causes the exception.

      Attachments

        1. Comparison.png
          849 kB
          Vadim Semenov
        2. Cube Data Model.png
          17 kB
          Vadim Semenov
        3. Cube Dimensions.png
          36 kB
          Vadim Semenov

        Issue Links

          Activity

            liyang.gmt8@gmail.com liyang added a comment -

            I believe this has been fixed by https://github.com/apache/incubator-kylin/commit/30f4c6ab31dfb20e4dc7f5e78c5772e30e04d64b

            I've verified 0.7 branch with

            • fact.CAL_DT=lookup.PART_DT
            • derive WEEK_BEGIN_DT from lookup
            • hierarchy of YEAR_BEGIN_DT, MONTH_BEGIN_DT, PART_DT from lookup

            Cube save successful in all cases.

            liyang.gmt8@gmail.com liyang added a comment - I believe this has been fixed by https://github.com/apache/incubator-kylin/commit/30f4c6ab31dfb20e4dc7f5e78c5772e30e04d64b I've verified 0.7 branch with fact.CAL_DT=lookup.PART_DT derive WEEK_BEGIN_DT from lookup hierarchy of YEAR_BEGIN_DT, MONTH_BEGIN_DT, PART_DT from lookup Cube save successful in all cases.
            buryat Vadim Semenov added a comment -

            liyang.gmt8@gmail.com At the time I was testing after this commit, and actually this commit introduced the bug, I had to revert `initDimensionColRef` on my deployment.
            I'll test once again with the latest 0.7-staging

            buryat Vadim Semenov added a comment - liyang.gmt8@gmail.com At the time I was testing after this commit, and actually this commit introduced the bug, I had to revert `initDimensionColRef` on my deployment. I'll test once again with the latest 0.7-staging
            liyang.gmt8@gmail.com liyang added a comment -

            Hi Vadim, there were commits and rollbacks about how PK-FK derive each other. I do believe finally it's resolve on current 0.7 branch. Please help verify again. Many thanks!

            liyang.gmt8@gmail.com liyang added a comment - Hi Vadim, there were commits and rollbacks about how PK-FK derive each other. I do believe finally it's resolve on current 0.7 branch. Please help verify again. Many thanks!
            buryat Vadim Semenov added a comment -

            Tested branch 0.7-staging (latest commit 572bcbc1bf9fce7332124063fb119b8f9ac90ae6), the issue is still present.

            Loaded sample_cube data, and tried to create a cube with the following definition (The same as I reported initially):
            fact table = KYLIN_SALES
            KYLIN_SALES.PART_DT = KYLIN_CAL_DT.CAL_DT

            normal dimensions: KYLIN_SALES.PART_DT, KYLIN_SALES.SELLER_ID
            derived dimension: KYLIN_CAL_DT.CAL_DT

            Got the same exception.

            buryat Vadim Semenov added a comment - Tested branch 0.7-staging (latest commit 572bcbc1bf9fce7332124063fb119b8f9ac90ae6), the issue is still present. Loaded sample_cube data, and tried to create a cube with the following definition (The same as I reported initially): fact table = KYLIN_SALES KYLIN_SALES.PART_DT = KYLIN_CAL_DT.CAL_DT normal dimensions: KYLIN_SALES.PART_DT, KYLIN_SALES.SELLER_ID derived dimension: KYLIN_CAL_DT.CAL_DT Got the same exception.
            vadim.semenov Vadim Semenov added a comment -

            The issue is still present

            vadim.semenov Vadim Semenov added a comment - The issue is still present
            liyang.gmt8@gmail.com liyang added a comment -

            Debug through and turns out this is because of PK declared as derived dimension explicitly.

            FK automatically derives PK. However user unaware of this can declare PK as derived dimension explicitly and cause this error. Have it fixed.

            liyang.gmt8@gmail.com liyang added a comment - Debug through and turns out this is because of PK declared as derived dimension explicitly. FK automatically derives PK. However user unaware of this can declare PK as derived dimension explicitly and cause this error. Have it fixed.
            shaofengshi Shao Feng Shi added a comment -

            Resolved in release 1.1-incubating (2015-10-25)

            shaofengshi Shao Feng Shi added a comment - Resolved in release 1.1-incubating (2015-10-25)

            People

              liyang.gmt8@gmail.com liyang
              vadim.semenov Vadim Semenov
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: