Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-6981

Fix DN upgrade with layout version change

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 3.0.0-alpha1
    • 2.6.0
    • datanode
    • None
    • Reviewed

    Description

      Post HDFS-6800, we can encounter the following scenario:

      1. We start with DN software version -55 and initiate a rolling upgrade to version -56
      2. We delete some blocks, and they are moved to trash
      3. We roll back to DN software version -55 using the -rollback flag – since we are running the old code (prior to this patch), we will restore the previous directory but will not delete the trash
      4. We append to some of the blocks that were deleted in step 2
      5. We then restart a DN that contains blocks that were appended to – since the trash still exists, it will be restored at this point, the appended-to blocks will be overwritten, and we will lose the appended data

      So I think we need to avoid writing anything to the trash directory if we have a previous directory.

      Thanks to james.thomas for reporting this.

      Attachments

        1. HDFS-6981.01.patch
          44 kB
          Arpit Agarwal
        2. HDFS-6981.02.patch
          44 kB
          Arpit Agarwal
        3. HDFS-6981.03.patch
          21 kB
          Arpit Agarwal
        4. HDFS-6981.04.patch
          27 kB
          Arpit Agarwal
        5. HDFS-6981.05.patch
          35 kB
          Arpit Agarwal
        6. HDFS-6981.06.patch
          34 kB
          Arpit Agarwal
        7. HDFS-6981.07.patch
          34 kB
          Arpit Agarwal
        8. HDFS-6981.08.patch
          35 kB
          Arpit Agarwal

        Issue Links

          Activity

            People

              arp Arpit Agarwal
              james.thomas James Thomas
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: