Uploaded image for project: 'CarbonData'
  1. CarbonData
  2. CARBONDATA-107

Remove unnecessary ConverToSafe in spark planner

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 0.1.0-incubating
    • 0.1.0-incubating
    • spark-integration
    • None

    Description

      Query:
      select ch, sum(c) from (select ch,count(1) as c from t2 group by ch) temp where c > 1 group by ch

      Output plan looks like:

      == Physical Plan ==
      Limit 21
      ConvertToSafe
      CarbonDictionaryDecoder CarbonDecoderRelation(Map(word#39 -> word#39, ch#40 -> ch#40, value#41 -> value#41),CarbonDatasourceRelation(`default`.`t1`,None)), ExcludeProfile(ArrayBuffer(#103)), CarbonAliasDecoderRelation()
      ConvertToSafe
      TungstenAggregate(key=ch#40, functions=(sum(c#101L),mode=Final,isDistinct=false), output=ch#40,_c1#102L)
      TungstenAggregate(key=ch#40, functions=(sum(c#101L),mode=Partial,isDistinct=false), output=ch#40,currentSum#122L)
      Filter (c#101L > FakeCarbonCast(1 as bigint))
      CarbonDictionaryDecoder CarbonDecoderRelation(Map(word#39 -> word#39, ch#40 -> ch#40, value#41 -> value#41),CarbonDatasourceRelation(`default`.`t1`,None)), IncludeProfile(ArrayBuffer(#103)), CarbonAliasDecoderRelation()
      ConvertToSafe
      TungstenAggregate(key=ch#40, functions=[(count(1),mode=Final,isDistinct=false)], output=ch#40,c#101L)
      TungstenExchange hashpartitioning(ch#40)
      TungstenAggregate(key=ch#40, functions=[(count(1),mode=Partial,isDistinct=false)], output=ch#40,currentCount#126L)
      Project ch#40
      ConvertToSafe
      CarbonScan ch#40, (CarbonRelation default, t1, CarbonMetaData(ArrayBuffer(word, ch),ArrayBuffer(value),org.carbondata.core.carbon.metadata.schema.table.CarbonTable@52d54ca2,DictionaryMap(Map(word -> true, ch -> true))), TableMeta(default_t1,/Users/jackylk/code/incubator-carbondata/target/store,org.carbondata.core.carbon.metadata.schema.table.CarbonTable@52d54ca2,Partitioner(org.carbondata.spark.partition.api.impl.SampleDataPartitionerImpl,[Ljava.lang.String;@62a877f4,1,[Ljava.lang.String;@3c180da5)), None), true

      There are unnecessary ConvertToSafe before CarbonDictionaryDecoder.

      Attachments

        Issue Links

          Activity

            People

              jackylk Jacky Li
              jackylk Jacky Li
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: