Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.13.0
    • Component/s: Query Processor
    • Labels:

      Description

      For map join, all data in small aliases are hashed and stored into temporary file in MapRedLocalTask. But for some aliases without filter or projection, it seemed not necessary to do that. For example.

      select a.* from src a join src b on a.key=b.key;
      

      makes plan like this.

      STAGE PLANS:
        Stage: Stage-4
          Map Reduce Local Work
            Alias -> Map Local Tables:
              a 
                Fetch Operator
                  limit: -1
            Alias -> Map Local Operator Tree:
              a 
                TableScan
                  alias: a
                  HashTable Sink Operator
                    condition expressions:
                      0 {key} {value}
                      1 
                    handleSkewJoin: false
                    keys:
                      0 [Column[key]]
                      1 [Column[key]]
                    Position of Big Table: 1
      
        Stage: Stage-3
          Map Reduce
            Alias -> Map Operator Tree:
              b 
                TableScan
                  alias: b
                  Map Join Operator
                    condition map:
                         Inner Join 0 to 1
                    condition expressions:
                      0 {key} {value}
                      1 
                    handleSkewJoin: false
                    keys:
                      0 [Column[key]]
                      1 [Column[key]]
                    outputColumnNames: _col0, _col1
                    Position of Big Table: 1
                    Select Operator
                      File Output Operator
            Local Work:
              Map Reduce Local Work
        Stage: Stage-0
          Fetch Operator
      

      table src(a) is fetched and stored as-is in MRLocalTask. With this patch, plan can be like below.

        Stage: Stage-3
          Map Reduce
            Alias -> Map Operator Tree:
              b 
                TableScan
                  alias: b
                  Map Join Operator
                    condition map:
                         Inner Join 0 to 1
                    condition expressions:
                      0 {key} {value}
                      1 
                    handleSkewJoin: false
                    keys:
                      0 [Column[key]]
                      1 [Column[key]]
                    outputColumnNames: _col0, _col1
                    Position of Big Table: 1
                    Select Operator
                        File Output Operator
            Local Work:
              Map Reduce Local Work
                Alias -> Map Local Tables:
                  a 
                    Fetch Operator
                      limit: -1
                Alias -> Map Local Operator Tree:
                  a 
                    TableScan
                      alias: a
                Has Any Stage Alias: false
        Stage: Stage-0
          Fetch Operator
      

        Attachments

        1. HIVE-6144.1.patch.txt
          65 kB
          Navis
        2. HIVE-6144.2.patch.txt
          66 kB
          Navis
        3. HIVE-6144.3.patch.txt
          67 kB
          Navis
        4. HIVE-6144.4.patch.txt
          1.14 MB
          Navis
        5. HIVE-6144.5.patch.txt
          1.14 MB
          Navis
        6. HIVE-6144.6.patch.txt
          1.04 MB
          Navis
        7. HIVE-6144.7.patch.txt
          1013 kB
          Navis
        8. HIVE-6144.8.patch.txt
          1011 kB
          Navis
        9. HIVE-6144.9.patch.txt
          1010 kB
          Navis

          Issue Links

            Activity

              People

              • Assignee:
                navis Navis
                Reporter:
                navis Navis
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: