Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Invalid
    • Affects Version/s: 2.4.0
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      I've a hadoop-2.3 running non-securable on the cluster. then I built a trunk instance, also non securable.

      NN1 - active
      NN2 - standby
      DN1 - datanode
      DN2 - datanode
      JN1,JN2,JN3 - Journal and ZK

      then on the NN2:

      hadoop-dameon.sh stop namenode
      hadoop-dameon.sh stop zkfc
      

      then:
      change the environment variables to the new hadoop.(trunk version)

      then:

      hadoop-dameon.sh start namenode
      

      NN2 throws exception:

      org.apache.hadoop.hdfs.qjournal.client.QuorumException: Could not journal CTime for one more JournalNodes. 1 exceptions thrown:
      10.100.91.33:8485: Failed on local exception: java.io.EOFException; Host Details : local host is: "10-204-8-136/10.204.8.136"; destination host is: "jn33.com":8485;
              at org.apache.hadoop.hdfs.qjournal.client.QuorumException.create(QuorumException.java:81)
              at org.apache.hadoop.hdfs.qjournal.client.QuorumCall.rethrowException(QuorumCall.java:223)
              at org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.getJournalCTime(QuorumJournalManager.java:631)
              at org.apache.hadoop.hdfs.server.namenode.FSEditLog.getSharedLogCTime(FSEditLog.java:1383)
              at org.apache.hadoop.hdfs.server.namenode.FSImage.initEditLog(FSImage.java:738)
              at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:600)
              at org.apache.hadoop.hdfs.server.namenode.FSImage.doUpgrade(FSImage.java:360)
              at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:258)
              at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:894)
              at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:653)
              at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:444)
              at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:500)
              at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:656)
              at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:641)
              at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1294)
              at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1360)
      

      JN throws Exception:

      2014-03-18 12:19:01,960 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 8485: readAndProcess threw exception java.io.IOException: Unable to read authentication method from client 10.204.8.136. Count of bytes read: 0
      java.io.IOException: Unable to read authentication method
      	at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:1344)
      	at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:761)
      	at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:560)
      	at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:535)
      2014-03-18 12:19:01,960 DEBUG org.apache.hadoop.ipc.Server: IPC Server listener on 8485: disconnecting client 10.204.8.136:39063. Number of active connections: 1
      

        Activity

        Hide
        Jing Zhao added a comment -

        Based on Kihwal and Nicholas's comments, let's close this jira first. Fengdong, thanks for the testing, and please feel free to open new jiras if you think there are other issues.

        Show
        Jing Zhao added a comment - Based on Kihwal and Nicholas's comments, let's close this jira first. Fengdong, thanks for the testing, and please feel free to open new jiras if you think there are other issues.
        Hide
        Tsz Wo Nicholas Sze added a comment -

        > ... then we'd better add some addtional text in the HDFS-5778, ...

        For the user guide, it is only available in the new version so that I think there should be very little confusion – users running an old version software should not expect a new feature described in the new version software documentation working in the old version.

        Show
        Tsz Wo Nicholas Sze added a comment - > ... then we'd better add some addtional text in the HDFS-5778 , ... For the user guide, it is only available in the new version so that I think there should be very little confusion – users running an old version software should not expect a new feature described in the new version software documentation working in the old version.
        Hide
        Kihwal Lee added a comment -

        Live upgrades from 2.x to trunk is not supported. This has nothing to do with the rolling upgrade feature. Whenever there is incompatible changes, live upgrades may not work.

        The rolling upgrade feature provides a safe mechanism to upgrade or downgrade and also minimizes impact on service and data availability while doing live upgrades. In order to take advantage of it, both new & old version of software must have this feature. Also, the two version should be compatible. For more details on the compatibility between versions, please refer to the design documentation of HDFS-5535.

        Show
        Kihwal Lee added a comment - Live upgrades from 2.x to trunk is not supported. This has nothing to do with the rolling upgrade feature. Whenever there is incompatible changes, live upgrades may not work. The rolling upgrade feature provides a safe mechanism to upgrade or downgrade and also minimizes impact on service and data availability while doing live upgrades. In order to take advantage of it, both new & old version of software must have this feature. Also, the two version should be compatible. For more details on the compatibility between versions, please refer to the design documentation of HDFS-5535 .
        Hide
        Fengdong Yu added a comment -

        continue above, if your answer is yes, then we'd better add some addtional text in the HDFS-5778, Tsz Wo Nicholas Sze , do you think so?

        Show
        Fengdong Yu added a comment - continue above, if your answer is yes, then we'd better add some addtional text in the HDFS-5778 , Tsz Wo Nicholas Sze , do you think so?
        Hide
        Fengdong Yu added a comment -

        Jing Zhao, I know RU is start from 2.4, but HDFS-5535 has been merged with the trunk.
        my target version is built from trunk, but, do you mean, both the new and the old version should all support rolling upgrade?

        Show
        Fengdong Yu added a comment - Jing Zhao , I know RU is start from 2.4, but HDFS-5535 has been merged with the trunk. my target version is built from trunk, but, do you mean, both the new and the old version should all support rolling upgrade?
        Hide
        Jing Zhao added a comment - - edited

        Hi Fengdong, thanks for testing. But hadoop 2.3 does not support rolling upgrade... And HA upgrade support only starts from 2.4. Also, please check the document for rolling upgrade detailed steps.

        Show
        Jing Zhao added a comment - - edited Hi Fengdong, thanks for testing. But hadoop 2.3 does not support rolling upgrade... And HA upgrade support only starts from 2.4. Also, please check the document for rolling upgrade detailed steps.

          People

          • Assignee:
            Unassigned
            Reporter:
            Fengdong Yu
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development