Uploaded image for project: 'Kylin'
  1. Kylin
  2. KYLIN-1427

Enable partition date column to support date and hour as separate columns for increment cube build

    Details

    • Type: Task
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: v1.5.0, v1.4.0
    • Fix Version/s: v1.5.0
    • Component/s: Metadata, REST Service, Web
    • Labels:
      None

      Description

      There are some data table, which save date and hour at 2 columns. For this kind of source data, kylin currently doesn't support incrementally build the cube. Open this JIRA to track this feature.

      {
       "partition_desc": {
          "partition_date_column": "DEFAULT.TEST_KYLIN_FACT.CAL_DT",
          "partition_time_column": "DEFAULT.TEST_KYLIN_FACT.CAL_HOUR",
          "partition_date_format": "yyyy-MM-dd",
          "partition_time_format": "HH:mm",
          "partition_date_start": 0,
          "partition_type": "APPEND",
          "partition_condition_builder": "org.apache.kylin.metadata.model.PartitionDesc$DefaultPartitionConditionBuilder"
        }
      }
      

        Issue Links

          Activity

          Hide
          liyang.gmt8@gmail.com liyang added a comment -

          More info copied from discussion with customer.

          What We Want

          • Hourly incremental build a cube

          What We Need

          • A timestamp column on FACT_TABLE that Kylin can use to tell old data from the new
          • Overwriting the hive table every hour is not required
          • Partitioning the hive table is not required
          • The above can work but maybe slow when reading hive if hive partition is not enabled on the timestamp column

          Alternative

          • Kylin to support incremental load from a date column + a time (hour) column (KYLIN-1427), letting Hive partition be comfortably enabled on the date column
          Show
          liyang.gmt8@gmail.com liyang added a comment - More info copied from discussion with customer. What We Want Hourly incremental build a cube What We Need A timestamp column on FACT_TABLE that Kylin can use to tell old data from the new Overwriting the hive table every hour is not required Partitioning the hive table is not required The above can work but maybe slow when reading hive if hive partition is not enabled on the timestamp column Alternative Kylin to support incremental load from a date column + a time (hour) column ( KYLIN-1427 ), letting Hive partition be comfortably enabled on the date column
          Hide
          haoch Hao Chen added a comment - - edited
          • "partition_time_column": "DEFAULT.TEST_KYLIN_FACT.CAL_HOUR"
          • "partition_time_format": "HH:mm"
          {
           "partition_desc": {
              "partition_date_column": "DEFAULT.TEST_KYLIN_FACT.CAL_DT",
              "partition_time_column": "DEFAULT.TEST_KYLIN_FACT.CAL_HOUR",
              "partition_date_format": "yyyy-MM-dd",
              "partition_time_format": "HH",
              "partition_date_start": 0,
              "partition_type": "APPEND",
              "partition_condition_builder": "org.apache.kylin.metadata.model.PartitionDesc$DefaultPartitionConditionBuilder"
            }
          }
          
          Show
          haoch Hao Chen added a comment - - edited "partition_time_column": "DEFAULT.TEST_KYLIN_FACT.CAL_HOUR" "partition_time_format": "HH:mm" { "partition_desc" : { "partition_date_column" : "DEFAULT.TEST_KYLIN_FACT.CAL_DT" , "partition_time_column" : "DEFAULT.TEST_KYLIN_FACT.CAL_HOUR" , "partition_date_format" : "yyyy-MM-dd" , "partition_time_format" : "HH" , "partition_date_start" : 0, "partition_type" : "APPEND" , "partition_condition_builder" : "org.apache.kylin.metadata.model.PartitionDesc$DefaultPartitionConditionBuilder" } }
          Hide
          haoch Hao Chen added a comment -

          Support partition_time_column and partition_time_format

          Show
          haoch Hao Chen added a comment - Support partition_time_column and partition_time_format
          Hide
          haoch Hao Chen added a comment - - edited
          Show
          haoch Hao Chen added a comment - - edited liyang Please help review https://github.com/apache/kylin/pull/27/files
          Hide
          liyang.gmt8@gmail.com liyang added a comment -

          Reviewed and merged. Slightly revised the coding style and used `org.junit.Assert` instead of the deprecated `junit.framework.Assert`

          Show
          liyang.gmt8@gmail.com liyang added a comment - Reviewed and merged. Slightly revised the coding style and used `org.junit.Assert` instead of the deprecated `junit.framework.Assert`
          Hide
          haoch Hao Chen added a comment -

          Got it, thanks.

          Show
          haoch Hao Chen added a comment - Got it, thanks.
          Hide
          lukehan Luke Han added a comment -

          Could we have "fixed version" to be correct one

          Thanks

          Show
          lukehan Luke Han added a comment - Could we have "fixed version" to be correct one Thanks
          Hide
          yonzhang2012 Edward Zhang added a comment -

          big issue today is that I do see some important are in 1.x but not in 2.x, for example query related.

          Show
          yonzhang2012 Edward Zhang added a comment - big issue today is that I do see some important are in 1.x but not in 2.x, for example query related.
          Hide
          haoch Hao Chen added a comment -

          May I know more detail about the issue on 1.x ?

          Show
          haoch Hao Chen added a comment - May I know more detail about the issue on 1.x ?
          Hide
          haoch Hao Chen added a comment -

          Is it ok to change the fixed version as `v2.1`?

          Show
          haoch Hao Chen added a comment - Is it ok to change the fixed version as `v2.1`?
          Hide
          zhongjian Zhong,Jason added a comment -

          For this jira, UI is not ready in the 1.5 release, the related ticket is https://issues.apache.org/jira/browse/KYLIN-1441

          I'm working on this. please be informed.

          Show
          zhongjian Zhong,Jason added a comment - For this jira, UI is not ready in the 1.5 release, the related ticket is https://issues.apache.org/jira/browse/KYLIN-1441 I'm working on this. please be informed.
          Hide
          haoch Hao Chen added a comment - - edited

          I think Dipesh is working on the front-end part. qianqiaoneng please confirm with Dipesh.

          Show
          haoch Hao Chen added a comment - - edited I think Dipesh is working on the front-end part. qianqiaoneng please confirm with Dipesh .
          Hide
          dipeshkhakhkhar Dipesh added a comment -

          Hi Jason - I have committed a patch for this issue. Kindly review it.
          -Thanks.
          Dipesh

          Show
          dipeshkhakhkhar Dipesh added a comment - Hi Jason - I have committed a patch for this issue. Kindly review it. -Thanks. Dipesh
          Hide
          liyang.gmt8@gmail.com liyang added a comment -

          Resolved in release v1.5.0 (2016-03-12).

          Note v1.4.0 is never actually released so v1.4.0 issues are closed with v.1.5.0 release.

          Show
          liyang.gmt8@gmail.com liyang added a comment - Resolved in release v1.5.0 (2016-03-12). Note v1.4.0 is never actually released so v1.4.0 issues are closed with v.1.5.0 release.

            People

            • Assignee:
              haoch Hao Chen
              Reporter:
              qianqiaoneng qianqiaoneng
              Request participants:
              None
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: