Uploaded image for project: 'CarbonData'
  1. CarbonData
  2. CARBONDATA-3994

Skip Order by for map task if it is sort column and use limit pushdown for array_contains filter

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.1.0
    • Component/s: None
    • Labels:
      None

      Description

      When the order by column is in sort column, every map task output will be already sorted. No need to sort the data again.
      Hence skipping the order at map task by changing plan node from TakeOrderedAndProject --> CarbonTakeOrderedAndProjectExec
      Also in this scenario collecting the limit at map task and Array_contains() will use this limit value for row scan filtering to break scan once limit value is reached.
      Also added a carbon property to control this .
      carbon.mapOrderPushDown.<db_name>_<table_name>.column

      Note: later we can improve this for other filters also to use the limit value.

        Attachments

          Activity

            People

            • Assignee:
              Ajantha_Bhat Ajantha Bhat
              Reporter:
              Ajantha_Bhat Ajantha Bhat
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 6h
                6h