Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-4078

Delay the serialize-deserialize pair in CommonJoinTaskDispatcher

    XMLWordPrintableJSON

    Details

    • Release Note:
      Create a copy of the MapredWork only if a conditional task is involved and avoid the copy if the task is non-conditional.

      Description

      CommonJoinProcessor tries to clone a MapredWork while attempting a conversion to a map-join

        // deep copy a new mapred work from xml
        InputStream in = new ByteArrayInputStream(xml.getBytes("UTF-8"));
        MapredWork newWork = Utilities.deserializeMapRedWork(in, physicalContext.getConf());
      

      which is a very heavy operation memory wise & cpu-wise.

      It would be better to do this only if a conditional task is required, resulting in a copy of the task.

        Attachments

        1. HIVE-4078-20130305.2.patch
          3 kB
          Gopal Vijayaraghavan
        2. HIVE-4078-20130305.patch
          3 kB
          Gopal Vijayaraghavan
        3. HIVE-4078-20130406.patch
          4 kB
          Gopal Vijayaraghavan
        4. HIVE-4078-trunk-rebase.patch
          4 kB
          Gopal Vijayaraghavan

          Issue Links

            Activity

              People

              • Assignee:
                gopalv Gopal Vijayaraghavan
                Reporter:
                gopalv Gopal Vijayaraghavan
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: