Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-6981

Fix DN upgrade with layout version change

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-alpha1
    • Fix Version/s: 2.6.0
    • Component/s: datanode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Post HDFS-6800, we can encounter the following scenario:

      1. We start with DN software version -55 and initiate a rolling upgrade to version -56
      2. We delete some blocks, and they are moved to trash
      3. We roll back to DN software version -55 using the -rollback flag – since we are running the old code (prior to this patch), we will restore the previous directory but will not delete the trash
      4. We append to some of the blocks that were deleted in step 2
      5. We then restart a DN that contains blocks that were appended to – since the trash still exists, it will be restored at this point, the appended-to blocks will be overwritten, and we will lose the appended data

      So I think we need to avoid writing anything to the trash directory if we have a previous directory.

      Thanks to James Thomas for reporting this.

        Attachments

        1. HDFS-6981.08.patch
          35 kB
          Arpit Agarwal
        2. HDFS-6981.07.patch
          34 kB
          Arpit Agarwal
        3. HDFS-6981.06.patch
          34 kB
          Arpit Agarwal
        4. HDFS-6981.05.patch
          35 kB
          Arpit Agarwal
        5. HDFS-6981.04.patch
          27 kB
          Arpit Agarwal
        6. HDFS-6981.03.patch
          21 kB
          Arpit Agarwal
        7. HDFS-6981.02.patch
          44 kB
          Arpit Agarwal
        8. HDFS-6981.01.patch
          44 kB
          Arpit Agarwal

          Issue Links

            Activity

              People

              • Assignee:
                arp Arpit Agarwal
                Reporter:
                james.thomas James Thomas
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: