Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-7764

DirectoryScanner shouldn't abort the scan if one directory had an error

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.7.0
    • 2.9.0, 3.0.0-alpha1, 2.8.2
    • datanode
    • None

    Description

      If there is an exception while preparing the ScanInfo for the blocks in the directory, DirectoryScanner is immediately throwing exception and coming out of the current scan cycle. The idea of this jira is to discuss & improve the exception handling mechanism.

      DirectoryScanner.java

          for (Entry<Integer, Future<ScanInfoPerBlockPool>> report :
              compilersInProgress.entrySet()) {
            try {
              dirReports[report.getKey()] = report.getValue().get();
            } catch (Exception ex) {
              LOG.error("Error compiling report", ex);
              // Propagate ex to DataBlockScanner to deal with
              throw new RuntimeException(ex);
            }
      

      Attachments

        1. HDFS-7764-04.patch
          10 kB
          Rakesh Radhakrishnan
        2. HDFS-7764-03.patch
          9 kB
          Rakesh Radhakrishnan
        3. HDFS-7764-02.patch
          9 kB
          Rakesh Radhakrishnan
        4. HDFS-7764-01.patch
          7 kB
          Rakesh Radhakrishnan
        5. HDFS-7764.patch
          2 kB
          Rakesh Radhakrishnan

        Activity

          People

            rakeshr Rakesh Radhakrishnan
            rakeshr Rakesh Radhakrishnan
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: