Hadoop Common
  1. Hadoop Common
  2. HADOOP-3426

Datanode does not start up if the local machines DNS isnt working right and dfs.datanode.dns.interface==default

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.21.0
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None
    • Environment:

      Ubuntu 8.04, at home, no reverse DNS

    • Hadoop Flags:
      Reviewed

      Description

      This is the third Java project I've been involved in that doesnt work on my home network, due to implementation issues with java.net.InetAddress.getLocalHost(), issues that only show up on an unamanged network. Fortunately my home network exists to find these problems early.

      In hadoop, if the local hostname doesnt resolve, the datanode does not start up:

      Caused by: java.net.UnknownHostException: k2: k2
      at java.net.InetAddress.getLocalHost(InetAddress.java:1353)
      at org.apache.hadoop.net.DNS.getDefaultHost(DNS.java:185)
      at org.apache.hadoop.dfs.DataNode.startDataNode(DataNode.java:184)
      at org.apache.hadoop.dfs.DataNode.(DataNode.java:162)
      at org.apache.hadoop.dfs.ExtDataNode.(ExtDataNode.java:55)
      at org.smartfrog.services.hadoop.components.datanode.DatanodeImpl.sfStart(DatanodeImpl.java:60)

      While this is a valid option in a production (non-virtual) cluster, if you are playing with VMWare/Xen private networks or on a home network, you can't rely on DNS.

      1. In these situations, its usually better to fall back to using "localhost" or 127.0.0.1 as a hostname if Java can't work it out for itself,
      2. Its often good to cache this if used in lots of parts of the system, otherwise the 30s timeouts can cause problems of their own.

      1. hadoop-3426.patch
        14 kB
        steve_l
      2. hadoop-3426.patch
        14 kB
        steve_l
      3. hadoop-3426.patch
        19 kB
        steve_l
      4. hadoop-3426.patch
        19 kB
        steve_l
      5. hadoop-3426.patch
        12 kB
        steve_l

        Issue Links

          Activity

          Steve Loughran made changes -
          Link This issue incorporates HADOOP-3619 [ HADOOP-3619 ]
          Steve Loughran made changes -
          Link This issue contains HDFS-95 [ HDFS-95 ]
          Tom White made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Gavin made changes -
          Assignee Steve Loughran [ steve_l ] Steve Loughran [ stevel@apache.org ]
          Gavin made changes -
          Reporter Steve Loughran [ steve_l ] Steve Loughran [ stevel@apache.org ]
          Owen O'Malley made changes -
          Component/s dfs [ 12310710 ]
          Chris Douglas made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags [Reviewed]
          Resolution Fixed [ 1 ]
          steve_l made changes -
          Fix Version/s 0.21.0 [ 12313563 ]
          Status Open [ 1 ] Patch Available [ 10002 ]
          Affects Version/s 0.19.0 [ 12313211 ]
          Affects Version/s 0.21.0 [ 12313563 ]
          steve_l made changes -
          Attachment hadoop-3426.patch [ 12401170 ]
          steve_l made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          steve_l made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          steve_l made changes -
          Link This issue incorporates HADOOP-5339 [ HADOOP-5339 ]
          steve_l made changes -
          Attachment hadoop-3426.patch [ 12401055 ]
          Chris Douglas made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          steve_l made changes -
          Attachment hadoop-3426.patch [ 12390380 ]
          steve_l made changes -
          Attachment hadoop-3426.patch [ 12385638 ]
          steve_l made changes -
          Affects Version/s 0.19.0 [ 12313211 ]
          Affects Version/s 0.16.3 [ 12313092 ]
          Status In Progress [ 3 ] Patch Available [ 10002 ]
          Affects Version/s 0.18.0 [ 12312972 ]
          steve_l made changes -
          Attachment dns-fixes.patch [ 12384410 ]
          steve_l made changes -
          Attachment dns-fixes-2.patch [ 12384510 ]
          steve_l made changes -
          Attachment hadoop-3426.patch [ 12385637 ]
          steve_l made changes -
          Attachment TestDNS.java [ 12384511 ]
          steve_l made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          steve_l made changes -
          Assignee Steve Loughran [ steve_l ]
          steve_l made changes -
          Link This issue incorporates HADOOP-3619 [ HADOOP-3619 ]
          steve_l made changes -
          Attachment TestDNS.java [ 12384511 ]
          steve_l made changes -
          Attachment dns-fixes-2.patch [ 12384510 ]
          steve_l made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          steve_l made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          steve_l made changes -
          Attachment dns-fixes.patch [ 12384410 ]
          steve_l made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          steve_l made changes -
          Affects Version/s 0.18.0 [ 12312972 ]
          Status Open [ 1 ] Patch Available [ 10002 ]
          steve_l made changes -
          Link This issue depends on HADOOP-197 [ HADOOP-197 ]
          steve_l made changes -
          Link This issue is related to HADOOP-3613 [ HADOOP-3613 ]
          steve_l made changes -
          Link This issue is related to HADOOP-3612 [ HADOOP-3612 ]
          steve_l made changes -
          Link This issue depends on HADOOP-197 [ HADOOP-197 ]
          steve_l made changes -
          Link This issue blocks HADOOP-197 [ HADOOP-197 ]
          steve_l made changes -
          Link This issue blocks HADOOP-197 [ HADOOP-197 ]
          steve_l made changes -
          Field Original Value New Value
          Link This issue relates to IVY-817 [ IVY-817 ]
          steve_l created issue -

            People

            • Assignee:
              Steve Loughran
              Reporter:
              Steve Loughran
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development