Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1900

MapReduce daemons should close FileSystems that are not needed anymore

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.22.0
    • Component/s: jobtracker, tasktracker
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Related to HADOOP-6843, this jira is to make MapReduce behave better with respect to closing FileSystems when they are not needed anymore.

      1. fs-close-delta.patch
        11 kB
        Devaraj Das
      2. m1900-04.patch
        13 kB
        Kan Zhang
      3. mapred-fs-close.patch
        6 kB
        Devaraj Das

        Issue Links

          Activity

          Hide
          Devaraj Das added a comment -

          Patch for Y20S. The patch basically closes the filesystems, created during the job execution, at the end of the job.

          Show
          Devaraj Das added a comment - Patch for Y20S. The patch basically closes the filesystems, created during the job execution, at the end of the job.
          Hide
          Hong Tang added a comment -

          //sigh. Had we implemented ref-counted fs cache (HADOOP-4665) and call close() every time a local context finishes using an fs instance, we are unlikely hit this problem.

          Show
          Hong Tang added a comment - //sigh. Had we implemented ref-counted fs cache ( HADOOP-4665 ) and call close() every time a local context finishes using an fs instance, we are unlikely hit this problem.
          Hide
          Hong Tang added a comment -

          I mean to say HADOOP-4655 instead of HADOOP-4665.

          Show
          Hong Tang added a comment - I mean to say HADOOP-4655 instead of HADOOP-4665 .
          Hide
          Devaraj Das added a comment -

          Missed one case. This patch fixes it.

          Show
          Devaraj Das added a comment - Missed one case. This patch fixes it.
          Hide
          Kan Zhang added a comment -

          a port for trunk.

          Show
          Kan Zhang added a comment - a port for trunk.
          Hide
          Kan Zhang added a comment -

          Here is the test-patch result. Also ran ant test and test-contrib. TestSimulatorDeterministicReplay failed, but it also fails without the patch. Other tests are passing.

          [exec] -1 overall.
          [exec]
          [exec] +1 @author. The patch does not contain any @author tags.
          [exec]
          [exec] -1 tests included. The patch doesn't appear to include any new or modified tests.
          [exec] Please justify why no new tests are needed for this patch.
          [exec] Also please list what manual steps were performed to verify this patch.
          [exec]
          [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
          [exec]
          [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
          [exec]
          [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
          [exec]
          [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.

          Show
          Kan Zhang added a comment - Here is the test-patch result. Also ran ant test and test-contrib. TestSimulatorDeterministicReplay failed, but it also fails without the patch. Other tests are passing. [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] -1 tests included. The patch doesn't appear to include any new or modified tests. [exec] Please justify why no new tests are needed for this patch. [exec] Also please list what manual steps were performed to verify this patch. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          +1 patch looks good.
          "Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch."

          Show
          Tsz Wo Nicholas Sze added a comment - +1 patch looks good. "Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch."
          Hide
          Devaraj Das added a comment -

          The patch was tested manually on Yahoo clusters. I just committed this. Thanks, Kan for the trunk patch.

          Show
          Devaraj Das added a comment - The patch was tested manually on Yahoo clusters. I just committed this. Thanks, Kan for the trunk patch.
          Hide
          Dan Adkins added a comment -

          This patch breaks hadoop for me. Can we reconsider it?

          It is not safe to call

          FileSystem.closeAllForUGI(UserGroupInformation.getCurrentUser());

          in order to close one filesystem connection.

          See MAPREDUCE-2060

          It causes the parent JobTracker's FileSystem object to be closed as well, making it impossible (for me) to successfully submit a job.

          Show
          Dan Adkins added a comment - This patch breaks hadoop for me. Can we reconsider it? It is not safe to call FileSystem.closeAllForUGI(UserGroupInformation.getCurrentUser()); in order to close one filesystem connection. See MAPREDUCE-2060 It causes the parent JobTracker's FileSystem object to be closed as well, making it impossible (for me) to successfully submit a job.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #523 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/523/)

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #523 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/523/ )

            People

            • Assignee:
              Kan Zhang
              Reporter:
              Devaraj Das
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development