Hadoop Common
  1. Hadoop Common
  2. HADOOP-3564

Sometime after successful hod allocation datanode fails to come up with java.net.BindException for dfs.datanode.ipc.address

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.18.0
    • Fix Version/s: 0.18.0
    • Component/s: contrib/hod
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Modifed HOD to generate the dfs.datanode.ipc.address parameter in the hadoop-site.xml of datanodes that it launches.

      Description

      From Jira: HADOOP-3283 which introduced new conf parameter dfs.datanode.ipc.address which defaults to 50020.
      When static dfs of hadoop version 0.18.0 running and its conf is not having dfs.datanode.ipc.address specified, then datanode start with 50020 port for ipc, w
      When we use hod allocate without using static dfs.datanode on some machine fails to come. On further investigation it has been found sometimes when torque provides list nodes, that list also contain some static dfs node.
      When hodring tries to start datanode on a machine where a static dfs datanode of hadoop 0.18.0 is running, hod's dynamic dfs datanode fails to come with exception -: java.net.BindException: Problem binding to /0.0.0.0:50020 : Address already in use
      beacuse hod provides ports for dfs.datanode.address and dfs.datanode.http.address.

      1. HADOOP-3564.1
        0.7 kB
        Vinod Kumar Vavilapalli

        Activity

        Hide
        Hemanth Yamijala added a comment -

        This scenario will not occur in environments running a static HDFS cluster and using HOD only for Map/Reduce clusters. It will only occur if a static HDFS cluster is present and we use HOD for bringing up our own HDFS and Map/Reduce clusters. The latter is a case only in test environments, that too in a few limited cases.

        Show
        Hemanth Yamijala added a comment - This scenario will not occur in environments running a static HDFS cluster and using HOD only for Map/Reduce clusters. It will only occur if a static HDFS cluster is present and we use HOD for bringing up our own HDFS and Map/Reduce clusters. The latter is a case only in test environments, that too in a few limited cases.
        Hide
        Hemanth Yamijala added a comment -

        I am marking this a blocker for Hadoop 0.18, because test environments are going to suffer once upgraded to Hadoop 0.18.

        Show
        Hemanth Yamijala added a comment - I am marking this a blocker for Hadoop 0.18, because test environments are going to suffer once upgraded to Hadoop 0.18.
        Hide
        Vinod Kumar Vavilapalli added a comment -

        Attaching patch(trivial). Tested with hadoop 0.18 and hadoop 0.17 tarballs, verified that dfs.datanode.ipc.address gets written into Datanode configuration only in the earlier case.

        Show
        Vinod Kumar Vavilapalli added a comment - Attaching patch(trivial). Tested with hadoop 0.18 and hadoop 0.17 tarballs, verified that dfs.datanode.ipc.address gets written into Datanode configuration only in the earlier case.
        Hide
        Hemanth Yamijala added a comment -

        Reviewed patch. Tested with hadoop 0.15, 0.17 and hadoop 0.18, and noted that the new parameter gets written for hadoop 0.18 and above.

        +1

        Show
        Hemanth Yamijala added a comment - Reviewed patch. Tested with hadoop 0.15, 0.17 and hadoop 0.18, and noted that the new parameter gets written for hadoop 0.18 and above. +1
        Hide
        Tsz Wo Nicholas Sze added a comment -

        The ipc port numbers of datanodes can be different since the datanodes will inform the NameNode about the ipc port they are using during datanode registration.

        Show
        Tsz Wo Nicholas Sze added a comment - The ipc port numbers of datanodes can be different since the datanodes will inform the NameNode about the ipc port they are using during datanode registration.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12384118/HADOOP-3564.1
        against trunk revision 668612.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no tests are needed for this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2673/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2673/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2673/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2673/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12384118/HADOOP-3564.1 against trunk revision 668612. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2673/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2673/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2673/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2673/console This message is automatically generated.
        Hide
        Hemanth Yamijala added a comment -

        Writing test cases will require some refactoring of HOD code, which is not currently in the scope. Relied on cross testing to make sure things are working. The fix can be committed.

        Show
        Hemanth Yamijala added a comment - Writing test cases will require some refactoring of HOD code, which is not currently in the scope. Relied on cross testing to make sure things are working. The fix can be committed.
        Hide
        Devaraj Das added a comment -

        I just committed this. Thanks, Vinod!

        Show
        Devaraj Das added a comment - I just committed this. Thanks, Vinod!
        Hide
        Hudson added a comment -
        Show
        Hudson added a comment - Integrated in Hadoop-trunk #522 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/522/ )

          People

          • Assignee:
            Vinod Kumar Vavilapalli
            Reporter:
            Karam Singh
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development