Uploaded image for project: 'Kylin'
  1. Kylin
  2. KYLIN-2004

Make the creating intermediate hive table steps configurable (two options)

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: v1.5.3
    • Fix Version/s: v1.5.4
    • Component/s: Job Engine
    • Labels:
      None

      Description

      In 1.5.3 for KYLIN-1677, Kylin changed the steps when pull data from hive as: 1) count the source table 2) create intermediate flat table with redistribution (reducer number is determined by step1)

      This works good for many cases, but may not benefit for case like a view (which joins several tables) as the fact table.

      So it is better to keep the other option: 1) create intermediate flat table 2) count the intermediate table 3) redistribute it based on the output of step 2)

      Plan to make this configurable (system level & cube level).

        Attachments

          Activity

            People

            • Assignee:
              shaofengshi Shao Feng Shi
              Reporter:
              shaofengshi Shao Feng Shi
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: