Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-5535 Umbrella jira for improved HDFS rolling upgrades
  3. HDFS-5945

Add rolling upgrade information to fsimage

        Details

        • Hadoop Flags:
          Reviewed

          Description

          When rolling upgrade is in progress, the standby namenode may create checkpoint. The rolling upgrade information should be added to fsimage in order to support namenode restart and continue rolling upgrade.

          1. Text File
            HDFS-5945.protobuf.patch
            16 kB
            Jing Zhao
          2. Text File
            h5945_20140216.patch
            13 kB
            Tsz Wo Nicholas Sze
          3. Text File
            h5945_20140214.patch
            12 kB
            Tsz Wo Nicholas Sze
          4. Text File
            h5945_20140213.patch
            8 kB
            Tsz Wo Nicholas Sze

            Issue Links

            incorporates

            Sub-task - The sub-task of the issue HDFS-5905 Upgrade and rolling upgrade should not be allowed simultaneously

            • Major - Major loss of function.
            • Resolved

              Activity

              Descending order - Click to sort in ascending order
              Hide
              Permalink
              szetszwo Tsz Wo Nicholas Sze added a comment -

              I have committed this.

              Show
              szetszwo Tsz Wo Nicholas Sze added a comment - I have committed this.
              Hide
              Permalink
              szetszwo Tsz Wo Nicholas Sze added a comment -

              +1 the protobuf change looks good.

              Show
              szetszwo Tsz Wo Nicholas Sze added a comment - +1 the protobuf change looks good.
              Hide
              Permalink
              jingzhao Jing Zhao added a comment -

              Make a small change to the latest patch to use protobuf based fsimage.

              Show
              jingzhao Jing Zhao added a comment - Make a small change to the latest patch to use protobuf based fsimage.
              Hide
              Permalink
              szetszwo Tsz Wo Nicholas Sze added a comment -

              h5945_20140216.patch: fix a bug.

              Show
              szetszwo Tsz Wo Nicholas Sze added a comment - h5945_20140216.patch: fix a bug.
              Hide
              Permalink
              szetszwo Tsz Wo Nicholas Sze added a comment -

              h5945_20140214.patch: allows saveNamespace since checkpoint is allowed; adds new tests.

              Show
              szetszwo Tsz Wo Nicholas Sze added a comment - h5945_20140214.patch: allows saveNamespace since checkpoint is allowed; adds new tests.
              Hide
              Permalink
              szetszwo Tsz Wo Nicholas Sze added a comment -

              > Since HDFS-5698 is already merged to trunk, why not merge those changes first and then make changes related to this jira using protobuf itself?

              There are some conflicts with the branch. I plan to resolve them later on since I have started to work on this.

              > Instead of extracting checking part to checkUpgrade() can't we use nn.getFSImage().isUpgradeFinalized() directly? This will avoid checking disks on RPC calls holding writeLock.

              You brought up a good point about the write lock. However, isUpgradeFinalized() is different from checkUpgrade(), which is to check if upgrade is started. Since starting rolling upgrade is an admin command, we can safely assume that it won't be abused.

              Show
              szetszwo Tsz Wo Nicholas Sze added a comment - > Since HDFS-5698 is already merged to trunk, why not merge those changes first and then make changes related to this jira using protobuf itself? There are some conflicts with the branch. I plan to resolve them later on since I have started to work on this. > Instead of extracting checking part to checkUpgrade() can't we use nn.getFSImage().isUpgradeFinalized() directly? This will avoid checking disks on RPC calls holding writeLock. You brought up a good point about the write lock. However, isUpgradeFinalized() is different from checkUpgrade(), which is to check if upgrade is started. Since starting rolling upgrade is an admin command, we can safely assume that it won't be abused.
              Hide
              Permalink
              vinayrpet Vinayakumar B added a comment -

              Instead of extracting checking part to checkUpgrade() can't we use nn.getFSImage().isUpgradeFinalized() directly? This will avoid checking disks on RPC calls holding writeLock.

              Show
              vinayrpet Vinayakumar B added a comment - Instead of extracting checking part to checkUpgrade() can't we use nn.getFSImage().isUpgradeFinalized() directly? This will avoid checking disks on RPC calls holding writeLock.
              Hide
              Permalink
              vinayrpet Vinayakumar B added a comment -

              Hi Nicholas,
              Since HDFS-5698 is already merged to trunk, why not merge those changes first and then make changes related to this jira using protobuf itself?

              Show
              vinayrpet Vinayakumar B added a comment - Hi Nicholas, Since HDFS-5698 is already merged to trunk, why not merge those changes first and then make changes related to this jira using protobuf itself?
              Hide
              Permalink
              szetszwo Tsz Wo Nicholas Sze added a comment -

              h5945_20140213.patch: adds upgrade info to fsimage and incorporates HDFS-5905.

              Show
              szetszwo Tsz Wo Nicholas Sze added a comment - h5945_20140213.patch: adds upgrade info to fsimage and incorporates HDFS-5905 .

                People

                • Assignee:
                  szetszwo Tsz Wo Nicholas Sze
                  Reporter:
                  szetszwo Tsz Wo Nicholas Sze
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  3 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved:

                    Development