Hadoop Common
  1. Hadoop Common
  2. HADOOP-10540

Datanode upgrade in Windows fails with hardlink error.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.4.0
    • Fix Version/s: 2.5.0
    • Component/s: tools
    • Labels:
      None
    • Environment:

      Windows + JDK7. The issue was hit while upgrading from 1.x to 2.4.

    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      I try to upgrade Hadoop from 1.x and 2.4, but DataNode failed to start due to hard link exception.
      Repro steps:
      *Installed Hadoop 1.x
      *hadoop dfsadmin -safemode enter
      *hadoop dfsadmin -saveNamespace
      *hadoop namenode -finalize
      *Stop all services
      *Uninstall Hadoop 1.x
      *Install Hadoop 2.4
      *Start namenode with -upgrade option
      *Try to start datanode, begin to see Hardlink exception in datanode service log.

      
      2014-04-10 22:47:11,655 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 8010: starting
      2014-04-10 22:47:11,656 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
      2014-04-10 22:47:11,999 INFO org.apache.hadoop.hdfs.server.common.Storage: Data-node version: -55 and name-node layout version: -56
      2014-04-10 22:47:12,008 INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on d:\hadoop\data\hdfs\dn\in_use.lock acquired by nodename 7268@myhost
      2014-04-10 22:47:12,011 INFO org.apache.hadoop.hdfs.server.common.Storage: Recovering storage directory D:\hadoop\data\hdfs\dn from previous upgrade
      2014-04-10 22:47:12,017 INFO org.apache.hadoop.hdfs.server.common.Storage: Upgrading storage directory d:\hadoop\data\hdfs\dn.
         old LV = -44; old CTime = 0.
         new LV = -55; new CTime = 1397168400373
      2014-04-10 22:47:12,021 INFO org.apache.hadoop.hdfs.server.common.Storage: Formatting block pool BP-39008719-10.0.0.1-1397168400092 directory d:\hadoop\data\hdfs\dn\current\BP-39008719-10.0.0.1-1397168400092\current
      2014-04-10 22:47:12,254 FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for block pool Block pool <registering> (Datanode Uuid unassigned) service to myhost/10.0.0.1:8020
      java.io.IOException: Usage: hardlink create [LINKNAME] [FILENAME] |Incorrect command line arguments.
      	at org.apache.hadoop.fs.HardLink.createHardLinkMult(HardLink.java:479)
      	at org.apache.hadoop.fs.HardLink.createHardLinkMult(HardLink.java:416)
      	at org.apache.hadoop.hdfs.server.datanode.DataStorage.linkBlocks(DataStorage.java:816)
      	at org.apache.hadoop.hdfs.server.datanode.DataStorage.linkAllBlocks(DataStorage.java:759)
      	at org.apache.hadoop.hdfs.server.datanode.DataStorage.doUpgrade(DataStorage.java:566)
      	at org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:486)
      	at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:225)
      	at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:249)
      	at org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:929)
      	at org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:900)
      	at org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:274)
      	at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:220)
      	at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:815)
      	at java.lang.Thread.run(Thread.java:722)
      2014-04-10 22:47:12,258 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Ending block pool service for: Block pool <registering> (Datanode Uuid unassigned) service to myhost/10.0.0.1:8020
      2014-04-10 22:47:12,359 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool ID needed, but service not yet registered with NN
      java.lang.Exception: trace
      	at org.apache.hadoop.hdfs.server.datanode.BPOfferService.getBlockPoolId(BPOfferService.java:143)
      	at org.apache.hadoop.hdfs.server.datanode.BlockPoolManager.remove(BlockPoolManager.java:91)
      	at org.apache.hadoop.hdfs.server.datanode.DataNode.shutdownBlockPool(DataNode.java:859)
      	at org.apache.hadoop.hdfs.server.datanode.BPOfferService.shutdownActor(BPOfferService.java:350)
      	at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.cleanUp(BPServiceActor.java:619)
      	at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:837)
      	at java.lang.Thread.run(Thread.java:722)
      2014-04-10 22:47:12,359 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Removed Block pool <registering> (Datanode Uuid unassigned)
      2014-04-10 22:47:12,360 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool ID needed, but service not yet registered with NN
      java.lang.Exception: trace
      	at org.apache.hadoop.hdfs.server.datanode.BPOfferService.getBlockPoolId(BPOfferService.java:143)
      	at org.apache.hadoop.hdfs.server.datanode.DataNode.shutdownBlockPool(DataNode.java:861)
      	at org.apache.hadoop.hdfs.server.datanode.BPOfferService.shutdownActor(BPOfferService.java:350)
      	at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.cleanUp(BPServiceActor.java:619)
      	at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:837)
      	at java.lang.Thread.run(Thread.java:722)
      2014-04-10 22:47:14,360 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Exiting Datanode
      2014-04-10 22:47:14,361 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 0
      2014-04-10 22:47:14,362 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG: 
      /************************************************************
      SHUTDOWN_MSG: Shutting down DataNode at myhost/10.0.0.1
      ************************************************************/
      
      1. HDFS-6233.03.patch
        8 kB
        Arpit Agarwal
      2. HDFS-6233.02.patch
        7 kB
        Chris Nauroth
      3. HDFS-6233.01.patch
        2 kB
        Arpit Agarwal

        Activity

          People

          • Assignee:
            Arpit Agarwal
            Reporter:
            Huan Huang
          • Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development