Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-12678

Handle empty rename pending metadata file during atomic rename in redo path

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: fs/azure
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Handle empty rename pending metadata file during atomic rename in redo path
      During atomic rename we create metadata file for rename(-renamePending.json). We create that in 2 steps
      1. We create an empty blob corresponding to the .json file in its real location
      2. We create a scratch file to which we write the contents of the rename pending which is then copied over into the blob described in 1
      If process crash occurs after step 1 and before step 2 is complete - we will be left with a zero size blob corresponding to the pending rename metadata file.
      This kind of scenario can happen in the /hbase/.tmp folder because it is considered a candidate folder for atomic rename. Now when HMaster starts up it executes listStatus on the .tmp folder to clean up pending data. At this stage due to the lazy pending rename complete process we look for these json files. On seeing an empty file the process simply throws a fatal exception assuming something went wrong.

        Attachments

        1. HADOOP-12678.001.patch
          12 kB
          madhumita chakraborty
        2. HADOOP-12678.002.patch
          6 kB
          madhumita chakraborty
        3. HADOOP-12678.003.patch
          6 kB
          madhumita chakraborty
        4. HADOOP-12678.004.patch
          6 kB
          madhumita chakraborty
        5. HADOOP-12678.005.patch
          6 kB
          madhumita chakraborty
        6. HADOOP-12678.006.patch
          6 kB
          madhumita chakraborty

          Activity

            People

            • Assignee:
              madhuch-ms madhumita chakraborty
              Reporter:
              madhuch-ms madhumita chakraborty
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: