Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-7185

The active NameNode will not accept an fsimage sent from the standby during rolling upgrade

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 2.4.0
    • 2.6.0
    • namenode
    • None
    • Reviewed

    Description

      The active NameNode will not accept an fsimage sent from the standby during rolling upgrade. The active fails with the exception:

      18:25:07,620  WARN ImageServlet:198 - Received an invalid request file transfer request from a secondary with storage info -59:65195028:0:CID-385de4d7-64e4-4dde-9f5d-0a6e431987f6
      18:25:07,620  WARN log:76 - Committed before 410 PutImage failed. java.io.IOException: This namenode has storage info -55:65195028:0:CID-385de4d7-64e4-4dde-9f5d-0a6e431987f6 but the secondary expected -59:65195028:0:CID-385de4d7-64e4-4dde-9f5d-
      0a6e431987f6
              at org.apache.hadoop.hdfs.server.namenode.ImageServlet.validateRequest(ImageServlet.java:200)
              at org.apache.hadoop.hdfs.server.namenode.ImageServlet.doPut(ImageServlet.java:443)
              at javax.servlet.http.HttpServlet.service(HttpServlet.java:730)
      

      On the standby, the exception is:

      java.io.IOException: Exception during image upload: org.apache.hadoop.hdfs.server.namenode.TransferFsImage$HttpPutFailedException: This namenode has storage info -55:65195028:0:CID-385de4d7-64e4-4dde-9f5d-0a6e431987f6 but the secondary expected
       -59:65195028:0:CID-385de4d7-64e4-4dde-9f5d-0a6e431987f6
              at org.apache.hadoop.hdfs.server.namenode.ha.StandbyCheckpointer.doCheckpoint(StandbyCheckpointer.java:218)
              at org.apache.hadoop.hdfs.server.namenode.ha.StandbyCheckpointer.access$1400(StandbyCheckpointer.java:62)
      

      This seems to be a consequence of the fact that the VERSION file still is at -55 (the old version) even after the rolling upgrade has started. When the rolling upgrade is finalized with hdfs dfsadmin -rollingUpgrade finalize, both VERSION files get set to the new version, and the problem goes away.

      Attachments

        1. HDFS-7185.000.patch
          1.0 kB
          Jing Zhao
        2. HDFS-7185.001.patch
          4 kB
          Jing Zhao
        3. HDFS-7185.002.patch
          10 kB
          Jing Zhao
        4. HDFS-7185.003.patch
          10 kB
          Jing Zhao
        5. HDFS-7185.004.patch
          17 kB
          Jing Zhao

        Activity

          People

            jingzhao Jing Zhao
            cmccabe Colin McCabe
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: