Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-4078

Delay the serialize-deserialize pair in CommonJoinTaskDispatcher

    XMLWordPrintableJSON

Details

    • Create a copy of the MapredWork only if a conditional task is involved and avoid the copy if the task is non-conditional.

    Description

      CommonJoinProcessor tries to clone a MapredWork while attempting a conversion to a map-join

        // deep copy a new mapred work from xml
        InputStream in = new ByteArrayInputStream(xml.getBytes("UTF-8"));
        MapredWork newWork = Utilities.deserializeMapRedWork(in, physicalContext.getConf());
      

      which is a very heavy operation memory wise & cpu-wise.

      It would be better to do this only if a conditional task is required, resulting in a copy of the task.

      Attachments

        1. HIVE-4078-20130305.2.patch
          3 kB
          Gopal Vijayaraghavan
        2. HIVE-4078-20130305.patch
          3 kB
          Gopal Vijayaraghavan
        3. HIVE-4078-20130406.patch
          4 kB
          Gopal Vijayaraghavan
        4. HIVE-4078-trunk-rebase.patch
          4 kB
          Gopal Vijayaraghavan

        Issue Links

          Activity

            People

              gopalv Gopal Vijayaraghavan
              gopalv Gopal Vijayaraghavan
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: