Hadoop Common
  1. Hadoop Common
  2. HADOOP-3437

mapred.job.tracker default value/docs appear out of sync with code

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Minor Minor
    • Resolution: Unresolved
    • Affects Version/s: 0.18.0
    • Fix Version/s: None
    • Component/s: conf
    • Labels:
      None

      Description

      This is the settings for mapred.job.tracker in SVN_HEAD. If you dont override these, you get a RuntimeException
      <property>
      <name>mapred.job.tracker</name>
      <value>local</value>
      <description>The host and port that the MapReduce job tracker runs
      at. If "local", then jobs are run in-process as a single map
      and reduce task.
      </description>
      </property>

        Activity

        Hide
        steve_l added a comment -

        stack trace when mapred.job.tracker=local

        Caused by: java.lang.RuntimeException: Not a host:port pair: local
        at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:132)
        at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:119)
        at org.apache.hadoop.mapred.JobTracker.getAddress(JobTracker.java:755)
        at org.apache.hadoop.mapred.JobTracker.(JobTracker.java:646)

        //my code from here
        at org.apache.hadoop.mapred.ExtJobTracker.(ExtJobTracker.java:41)
        at org.smartfrog.services.hadoop.components.tracker.JobTrackerImpl.sfStart(JobTrackerImpl.java:53)
        at org.smartfrog.sfcore.compound.CompoundImpl.sfStartChildren(CompoundImpl.java:659)
        at org.smartfrog.sfcore.compound.CompoundImpl.sfStart(CompoundImpl.java:632)
        at org.smartfrog.services.assertions.TestCompoundImpl.sfStart(TestCompoundImpl.java:249)
        at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305)
        at sun.rmi.transport.Transport$1.run(Transport.java:159)
        at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
        at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:619)

        Show
        steve_l added a comment - stack trace when mapred.job.tracker=local Caused by: java.lang.RuntimeException: Not a host:port pair: local at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:132) at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:119) at org.apache.hadoop.mapred.JobTracker.getAddress(JobTracker.java:755) at org.apache.hadoop.mapred.JobTracker.(JobTracker.java:646) //my code from here at org.apache.hadoop.mapred.ExtJobTracker.(ExtJobTracker.java:41) at org.smartfrog.services.hadoop.components.tracker.JobTrackerImpl.sfStart(JobTrackerImpl.java:53) at org.smartfrog.sfcore.compound.CompoundImpl.sfStartChildren(CompoundImpl.java:659) at org.smartfrog.sfcore.compound.CompoundImpl.sfStart(CompoundImpl.java:632) at org.smartfrog.services.assertions.TestCompoundImpl.sfStart(TestCompoundImpl.java:249) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305) at sun.rmi.transport.Transport$1.run(Transport.java:159) at sun.rmi.transport.Transport.serviceCall(Transport.java:155) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:619)
        Hide
        steve_l added a comment -

        I should add that this is a change since 0.16.3; so "local" used to be valid. A new default value/comment string is probably appropriate

        Show
        steve_l added a comment - I should add that this is a change since 0.16.3; so "local" used to be valid. A new default value/comment string is probably appropriate
        Hide
        Doug Cutting added a comment -

        If mapred.job.tracker=local, then the JobTracker class should not be used, but rather LocalRunner. This logic is in JobClient. There is no corresponding server-side logic, since, when mapred.job.tracker=local, it is assumed that you are not starting any server daemons.

        What do you expect to happen here?

        Show
        Doug Cutting added a comment - If mapred.job.tracker=local, then the JobTracker class should not be used, but rather LocalRunner. This logic is in JobClient. There is no corresponding server-side logic, since, when mapred.job.tracker=local, it is assumed that you are not starting any server daemons. What do you expect to happen here?
        Hide
        Erik Forsberg added a comment - - edited

        Could the problem be IPv6-related? I'm getting this error when trying out the configuration listed at http://hadoop.apache.org/core/docs/current/quickstart.html#PseudoDistributed

        Reading http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Linux_(Single-Node_Cluster)#Disabling_IPv6, I got the idea to explicitly set 127.0.0.1 instead of 0.0.0.0 for values where this was the default.

        I.e., I replaced 0.0.0.0 with 127.0.0.1 for dfs.secondary.http.address, dfs.datanode.address, dfs.datanode.http.address, dfs.datanode.ipc.address, dfs.http.address, dfs.datanode.https.address, dfs.https.address, mapred.job.tracker.http.address, and mapred.task.tracker.http.address.

        That solved the problem.

        Show
        Erik Forsberg added a comment - - edited Could the problem be IPv6-related? I'm getting this error when trying out the configuration listed at http://hadoop.apache.org/core/docs/current/quickstart.html#PseudoDistributed Reading http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Linux_(Single-Node_Cluster)#Disabling_IPv6 , I got the idea to explicitly set 127.0.0.1 instead of 0.0.0.0 for values where this was the default. I.e., I replaced 0.0.0.0 with 127.0.0.1 for dfs.secondary.http.address, dfs.datanode.address, dfs.datanode.http.address, dfs.datanode.ipc.address, dfs.http.address, dfs.datanode.https.address, dfs.https.address, mapred.job.tracker.http.address, and mapred.task.tracker.http.address. That solved the problem.
        Hide
        Erik Forsberg added a comment -

        After some fiddling I found that the solution I proposed in the previous comment does work halfway - you can submit jobs, but the reduce processes fails to connect to hdfs to fetch files.

        I found a better workaround:

        1) Leave the configuration as detailed at http://hadoop.apache.org/core/docs/current/quickstart.html#PseudoDistributed

        2) Disable IPv6 in Java by editing conf/hadoop-env.sh, as follows:

        HADOOP_OPTS=-Djava.net.preferIPv4Stack=true

        Show
        Erik Forsberg added a comment - After some fiddling I found that the solution I proposed in the previous comment does work halfway - you can submit jobs, but the reduce processes fails to connect to hdfs to fetch files. I found a better workaround: 1) Leave the configuration as detailed at http://hadoop.apache.org/core/docs/current/quickstart.html#PseudoDistributed 2) Disable IPv6 in Java by editing conf/hadoop-env.sh, as follows: HADOOP_OPTS=-Djava.net.preferIPv4Stack=true
        Hide
        steve_l added a comment -

        1. JobTracker should clearly recognise that the mapred.job.tracker ref is local, and refuse to play, with a more useful message.

        2. Erik, I think your problem is separate, and perhaps should be its own bugrep.

        We should disable IPv6 anyway; created a bug report for that HADOOP-6056 -as Ubuntu 9.04 doesnt let you turn IPv6 off in the kernel, doing it in Java is the best way to kill it.

        Show
        steve_l added a comment - 1. JobTracker should clearly recognise that the mapred.job.tracker ref is local, and refuse to play, with a more useful message. 2. Erik, I think your problem is separate, and perhaps should be its own bugrep. We should disable IPv6 anyway; created a bug report for that HADOOP-6056 -as Ubuntu 9.04 doesnt let you turn IPv6 off in the kernel, doing it in Java is the best way to kill it.
        Hide
        Derek Morr added a comment -

        What would it take to enable IPv6 support in Hadoop?

        Show
        Derek Morr added a comment - What would it take to enable IPv6 support in Hadoop?

          People

          • Assignee:
            Unassigned
            Reporter:
            Steve Loughran
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:

              Development