Hadoop Common
  1. Hadoop Common
  2. HADOOP-3564

Sometime after successful hod allocation datanode fails to come up with java.net.BindException for dfs.datanode.ipc.address

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.18.0
    • Fix Version/s: 0.18.0
    • Component/s: contrib/hod
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Modifed HOD to generate the dfs.datanode.ipc.address parameter in the hadoop-site.xml of datanodes that it launches.

      Description

      From Jira: HADOOP-3283 which introduced new conf parameter dfs.datanode.ipc.address which defaults to 50020.
      When static dfs of hadoop version 0.18.0 running and its conf is not having dfs.datanode.ipc.address specified, then datanode start with 50020 port for ipc, w
      When we use hod allocate without using static dfs.datanode on some machine fails to come. On further investigation it has been found sometimes when torque provides list nodes, that list also contain some static dfs node.
      When hodring tries to start datanode on a machine where a static dfs datanode of hadoop 0.18.0 is running, hod's dynamic dfs datanode fails to come with exception -: java.net.BindException: Problem binding to /0.0.0.0:50020 : Address already in use
      beacuse hod provides ports for dfs.datanode.address and dfs.datanode.http.address.

      1. HADOOP-3564.1
        0.7 kB
        Vinod Kumar Vavilapalli

        Activity

        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Patch Available Patch Available
        1d 4h 29m 1 Vinod Kumar Vavilapalli 17/Jun/08 12:26
        Patch Available Patch Available Resolved Resolved
        18h 20m 1 Devaraj Das 18/Jun/08 06:47
        Resolved Resolved Closed Closed
        65d 14h 3m 1 Nigel Daley 22/Aug/08 20:50
        Nigel Daley made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Hide
        Hudson added a comment -
        Show
        Hudson added a comment - Integrated in Hadoop-trunk #522 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/522/ )
        Devaraj Das made changes -
        Resolution Fixed [ 1 ]
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hide
        Devaraj Das added a comment -

        I just committed this. Thanks, Vinod!

        Show
        Devaraj Das added a comment - I just committed this. Thanks, Vinod!
        Hemanth Yamijala made changes -
        Release Note Modifed HOD to generate the dfs.datanode.ipc.address parameter in the hadoop-site.xml of datanodes that it launches.
        Hadoop Flags [Reviewed]
        Hide
        Hemanth Yamijala added a comment -

        Writing test cases will require some refactoring of HOD code, which is not currently in the scope. Relied on cross testing to make sure things are working. The fix can be committed.

        Show
        Hemanth Yamijala added a comment - Writing test cases will require some refactoring of HOD code, which is not currently in the scope. Relied on cross testing to make sure things are working. The fix can be committed.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12384118/HADOOP-3564.1
        against trunk revision 668612.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no tests are needed for this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2673/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2673/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2673/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2673/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12384118/HADOOP-3564.1 against trunk revision 668612. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2673/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2673/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2673/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2673/console This message is automatically generated.
        Hide
        Tsz Wo Nicholas Sze added a comment -

        The ipc port numbers of datanodes can be different since the datanodes will inform the NameNode about the ipc port they are using during datanode registration.

        Show
        Tsz Wo Nicholas Sze added a comment - The ipc port numbers of datanodes can be different since the datanodes will inform the NameNode about the ipc port they are using during datanode registration.
        Hide
        Hemanth Yamijala added a comment -

        Reviewed patch. Tested with hadoop 0.15, 0.17 and hadoop 0.18, and noted that the new parameter gets written for hadoop 0.18 and above.

        +1

        Show
        Hemanth Yamijala added a comment - Reviewed patch. Tested with hadoop 0.15, 0.17 and hadoop 0.18, and noted that the new parameter gets written for hadoop 0.18 and above. +1
        Vinod Kumar Vavilapalli made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Vinod Kumar Vavilapalli made changes -
        Attachment HADOOP-3564.1 [ 12384118 ]
        Hide
        Vinod Kumar Vavilapalli added a comment -

        Attaching patch(trivial). Tested with hadoop 0.18 and hadoop 0.17 tarballs, verified that dfs.datanode.ipc.address gets written into Datanode configuration only in the earlier case.

        Show
        Vinod Kumar Vavilapalli added a comment - Attaching patch(trivial). Tested with hadoop 0.18 and hadoop 0.17 tarballs, verified that dfs.datanode.ipc.address gets written into Datanode configuration only in the earlier case.
        Hemanth Yamijala made changes -
        Field Original Value New Value
        Assignee Vinod Kumar Vavilapalli [ vinodkv ]
        Priority Major [ 3 ] Blocker [ 1 ]
        Fix Version/s 0.18.0 [ 12312972 ]
        Hide
        Hemanth Yamijala added a comment -

        I am marking this a blocker for Hadoop 0.18, because test environments are going to suffer once upgraded to Hadoop 0.18.

        Show
        Hemanth Yamijala added a comment - I am marking this a blocker for Hadoop 0.18, because test environments are going to suffer once upgraded to Hadoop 0.18.
        Hide
        Hemanth Yamijala added a comment -

        This scenario will not occur in environments running a static HDFS cluster and using HOD only for Map/Reduce clusters. It will only occur if a static HDFS cluster is present and we use HOD for bringing up our own HDFS and Map/Reduce clusters. The latter is a case only in test environments, that too in a few limited cases.

        Show
        Hemanth Yamijala added a comment - This scenario will not occur in environments running a static HDFS cluster and using HOD only for Map/Reduce clusters. It will only occur if a static HDFS cluster is present and we use HOD for bringing up our own HDFS and Map/Reduce clusters. The latter is a case only in test environments, that too in a few limited cases.
        Karam Singh created issue -

          People

          • Assignee:
            Vinod Kumar Vavilapalli
            Reporter:
            Karam Singh
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development