Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-10540

Datanode upgrade in Windows fails with hardlink error.

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.4.0
    • Fix Version/s: 2.5.0
    • Component/s: tools
    • Labels:
      None
    • Environment:

      Windows + JDK7. The issue was hit while upgrading from 1.x to 2.4.

    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      I try to upgrade Hadoop from 1.x and 2.4, but DataNode failed to start due to hard link exception.
      Repro steps:
      *Installed Hadoop 1.x
      *hadoop dfsadmin -safemode enter
      *hadoop dfsadmin -saveNamespace
      *hadoop namenode -finalize
      *Stop all services
      *Uninstall Hadoop 1.x
      *Install Hadoop 2.4
      *Start namenode with -upgrade option
      *Try to start datanode, begin to see Hardlink exception in datanode service log.

      2014-04-10 22:47:11,655 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 8010: starting
      2014-04-10 22:47:11,656 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
      2014-04-10 22:47:11,999 INFO org.apache.hadoop.hdfs.server.common.Storage: Data-node version: -55 and name-node layout version: -56
      2014-04-10 22:47:12,008 INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on d:\hadoop\data\hdfs\dn\in_use.lock acquired by nodename 7268@myhost
      2014-04-10 22:47:12,011 INFO org.apache.hadoop.hdfs.server.common.Storage: Recovering storage directory D:\hadoop\data\hdfs\dn from previous upgrade
      2014-04-10 22:47:12,017 INFO org.apache.hadoop.hdfs.server.common.Storage: Upgrading storage directory d:\hadoop\data\hdfs\dn.
         old LV = -44; old CTime = 0.
         new LV = -55; new CTime = 1397168400373
      2014-04-10 22:47:12,021 INFO org.apache.hadoop.hdfs.server.common.Storage: Formatting block pool BP-39008719-10.0.0.1-1397168400092 directory d:\hadoop\data\hdfs\dn\current\BP-39008719-10.0.0.1-1397168400092\current
      2014-04-10 22:47:12,254 FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for block pool Block pool <registering> (Datanode Uuid unassigned) service to myhost/10.0.0.1:8020
      java.io.IOException: Usage: hardlink create [LINKNAME] [FILENAME] |Incorrect command line arguments.
      	at org.apache.hadoop.fs.HardLink.createHardLinkMult(HardLink.java:479)
      	at org.apache.hadoop.fs.HardLink.createHardLinkMult(HardLink.java:416)
      	at org.apache.hadoop.hdfs.server.datanode.DataStorage.linkBlocks(DataStorage.java:816)
      	at org.apache.hadoop.hdfs.server.datanode.DataStorage.linkAllBlocks(DataStorage.java:759)
      	at org.apache.hadoop.hdfs.server.datanode.DataStorage.doUpgrade(DataStorage.java:566)
      	at org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:486)
      	at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:225)
      	at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:249)
      	at org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:929)
      	at org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:900)
      	at org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:274)
      	at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:220)
      	at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:815)
      	at java.lang.Thread.run(Thread.java:722)
      2014-04-10 22:47:12,258 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Ending block pool service for: Block pool <registering> (Datanode Uuid unassigned) service to myhost/10.0.0.1:8020
      2014-04-10 22:47:12,359 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool ID needed, but service not yet registered with NN
      java.lang.Exception: trace
      	at org.apache.hadoop.hdfs.server.datanode.BPOfferService.getBlockPoolId(BPOfferService.java:143)
      	at org.apache.hadoop.hdfs.server.datanode.BlockPoolManager.remove(BlockPoolManager.java:91)
      	at org.apache.hadoop.hdfs.server.datanode.DataNode.shutdownBlockPool(DataNode.java:859)
      	at org.apache.hadoop.hdfs.server.datanode.BPOfferService.shutdownActor(BPOfferService.java:350)
      	at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.cleanUp(BPServiceActor.java:619)
      	at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:837)
      	at java.lang.Thread.run(Thread.java:722)
      2014-04-10 22:47:12,359 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Removed Block pool <registering> (Datanode Uuid unassigned)
      2014-04-10 22:47:12,360 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool ID needed, but service not yet registered with NN
      java.lang.Exception: trace
      	at org.apache.hadoop.hdfs.server.datanode.BPOfferService.getBlockPoolId(BPOfferService.java:143)
      	at org.apache.hadoop.hdfs.server.datanode.DataNode.shutdownBlockPool(DataNode.java:861)
      	at org.apache.hadoop.hdfs.server.datanode.BPOfferService.shutdownActor(BPOfferService.java:350)
      	at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.cleanUp(BPServiceActor.java:619)
      	at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:837)
      	at java.lang.Thread.run(Thread.java:722)
      2014-04-10 22:47:14,360 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Exiting Datanode
      2014-04-10 22:47:14,361 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 0
      2014-04-10 22:47:14,362 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG: 
      /************************************************************
      SHUTDOWN_MSG: Shutting down DataNode at myhost/10.0.0.1
      ************************************************************/
      

        Attachments

        1. HDFS-6233.01.patch
          2 kB
          Arpit Agarwal
        2. HDFS-6233.02.patch
          7 kB
          Chris Nauroth
        3. HDFS-6233.03.patch
          8 kB
          Arpit Agarwal

          Activity

            People

            • Assignee:
              arp Arpit Agarwal
              Reporter:
              hhuang2014 Huan Huang
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: