Hadoop Common
  1. Hadoop Common
  2. HADOOP-4997

workaround for tmp file handling on DataNodes in 0.18 (HADOOP-4663)

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.18.0
    • Fix Version/s: 0.18.3
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change
    • Release Note:
      Revert tmp files handling on DataNodes back to 0.17. sync() introduced in 0.18 has less gaurantees.

      Description

      This is a temporary work around issues discussed in HADOOP-4663.

      The proposal is to remove all the files under tmp directory, thus bringing the behavior back to 0.17. The main cost is that sync() will not be supported. This is incompatible with 0.18.x, but not with 0.17 because of this reason.

      1. HADOOP-4997-branch-18.patch
        4 kB
        Raghu Angadi
      2. deleteTmp-0.18.patch
        0.9 kB
        dhruba borthakur

        Issue Links

          Activity

          Hide
          dhruba borthakur added a comment -

          when the datanode restarts, remove all blocks from the tmp directory of the datanode. However, fsync does not throw an exception.

          Show
          dhruba borthakur added a comment - when the datanode restarts, remove all blocks from the tmp directory of the datanode. However, fsync does not throw an exception.
          Hide
          Hairong Kuang added a comment -

          Dhruba, we should also remove the unit tests that test the tmp directory does not get removed after restarting the cluster.

          Show
          Hairong Kuang added a comment - Dhruba, we should also remove the unit tests that test the tmp directory does not get removed after restarting the cluster.
          Hide
          Hairong Kuang added a comment -

          The two junit tests that would fail with this patch are TestFileCreationDelete and TestRenameWhileOpen.

          Show
          Hairong Kuang added a comment - The two junit tests that would fail with this patch are TestFileCreationDelete and TestRenameWhileOpen.
          Hide
          Raghu Angadi added a comment -

          Thanks Hairong.

          The updated patch disables the test Hairong mentioned with a very explicit comment that it is a temporary change and it has the original patch Dhruba attached.

          Show
          Raghu Angadi added a comment - Thanks Hairong. The updated patch disables the test Hairong mentioned with a very explicit comment that it is a temporary change and it has the original patch Dhruba attached.
          Hide
          dhruba borthakur added a comment -

          Hi Hairong, I saw ur comment just now about the unit tests. Thanks for fixing it.

          Show
          dhruba borthakur added a comment - Hi Hairong, I saw ur comment just now about the unit tests. Thanks for fixing it.
          Hide
          Konstantin Shvachko added a comment -

          +1 on Raghu's patch. It is critical for 0.18.

          Show
          Konstantin Shvachko added a comment - +1 on Raghu's patch. It is critical for 0.18.
          Hide
          Raghu Angadi added a comment -

          test-patch :
          ------------

          [exec] -1 overall.

          [exec] +1 @author. The patch does not contain any @author tags.

          [exec] +1 tests included. The patch appears to include 6 new or modified tests.

          [exec] -1 javadoc. The javadoc tool appears to have generated 1 warning messages.

          [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.

          ----------

          JavaDoc failures are unrelated to this patch (they are for "CyclicIterator.java").

          Show
          Raghu Angadi added a comment - test-patch : ------------ [exec] -1 overall. [exec] +1 @author. The patch does not contain any @author tags. [exec] +1 tests included. The patch appears to include 6 new or modified tests. [exec] -1 javadoc. The javadoc tool appears to have generated 1 warning messages. [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. ---------- JavaDoc failures are unrelated to this patch (they are for "CyclicIterator.java").
          Hide
          Raghu Angadi added a comment -

          I just committed this to 0.18 only.

          Show
          Raghu Angadi added a comment - I just committed this to 0.18 only.
          Hide
          Konstantin Shvachko added a comment -

          Could you please file a bug against CyclicIterator.java

          Show
          Konstantin Shvachko added a comment - Could you please file a bug against CyclicIterator.java
          Hide
          Raghu Angadi added a comment -

          > Could you please file a bug against CyclicIterator.java
          done : HADOOP-5077.

          Show
          Raghu Angadi added a comment - > Could you please file a bug against CyclicIterator.java done : HADOOP-5077 .

            People

            • Assignee:
              Raghu Angadi
              Reporter:
              Raghu Angadi
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development