Uploaded image for project: 'Nutch'
  1. Nutch
  2. NUTCH-2161

Interrupted failed and/or killed tasks fail to clean up temp directories in HDFS

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Duplicate
    • 1.11
    • 1.15
    • None
    • None

    Description

      If for example one kills an inject or generate job, Nutch does not clean up 'temporary' directories and I have witnessed them remain within HDFS. This is far from ideal if we have a large team of users all hammering away on Yarn and persisting data into HDFS.
      We should investigate how to clean up these directories such that a cluster admin is not left with all of the dross at the end of the long day

      Attachments

        Issue Links

          is superceded by

          Bug - A problem which impairs or prevents the functions of the product. NUTCH-2518 Must check return value of job.waitForCompletion()

          • Blocker - Blocks development and/or testing work, production could not run
          • Closed

          Activity

            People

              Unassigned Unassigned
              lewismc Lewis John McGibbney
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: