Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-3251

Hive doesn't remove scrach directories while killing running MR job

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.10.0
    • Component/s: Server Infrastructure
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      While killing running MR job, hive doesn't clean up scratch directory (mapred.cache.files). So that, afterwards, scratch directory is left there in hdfs. HDFS name node doesn't know it and try to do lease recovery. while such instances happen more, it will eventually crash namenode.

      The fix is to leverage hdfs clean up functionality. While creating scratch dirs, hive registers it to hdfs cleanup hook. While killing happens, hdfs will clean them up.

        Attachments

        1. HIVE-3151.patch
          3 kB
          Gang Tim Liu

          Activity

            People

            • Assignee:
              gangtimliu Gang Tim Liu
              Reporter:
              gangtimliu Gang Tim Liu
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: