Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-959

JobConf::setWorkingDirectory requires that the default FileSystem is reachable

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Minor Minor
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: client, test
    • Labels:
      None

      Description

      If mapred.working.dir is not set, JobConf::setWorkingDirectory will attempt to obtain the default working directory for the default FileSystem. In trunk at least, if the default fs is HDFS and not reachable, set will fail:

      java.net.UnknownHostException: unknown host: notahost
      java.lang.RuntimeException: java.net.UnknownHostException: unknown host: notahost
        at org.apache.hadoop.mapred.JobConf.getWorkingDirectory(JobConf.java:541)
        at org.apache.hadoop.mapred.JobConf.setWorkingDirectory(JobConf.java:522)
        at org.apache.hadoop.conf.TestJobConf.testSetWorkingDir(TestJobConf.java:67)
      Caused by: java.net.UnknownHostException: unknown host: notahost
        at org.apache.hadoop.ipc.Client$Connection.<init>(Client.java:216)
        at org.apache.hadoop.ipc.Client.getConnection(Client.java:876)
        at org.apache.hadoop.ipc.Client.call(Client.java:746)
        at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:223)
        at $Proxy4.getProtocolVersion(Unknown Source)
        at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:366)
        at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:169)
        at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:276)
        at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:235)
        at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:83)
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1430)
        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:69)
        at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:1458)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1446)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:190)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:98)
        at org.apache.hadoop.mapred.JobConf.getWorkingDirectory(JobConf.java:537)
      
      1. M959-test.patch
        2 kB
        Chris Douglas

        Activity

        Chris Douglas created issue -
        Hide
        Chris Douglas added a comment -

        Unit test illustrating the issue.

        Arguably, this is an issue with new instances of DistributedFileSystem and protocol compatibility should be established lazily rather than in the constructor.

        Show
        Chris Douglas added a comment - Unit test illustrating the issue. Arguably, this is an issue with new instances of DistributedFileSystem and protocol compatibility should be established lazily rather than in the constructor.
        Chris Douglas made changes -
        Field Original Value New Value
        Attachment M959-test.patch [ 12418858 ]

          People

          • Assignee:
            Unassigned
            Reporter:
            Chris Douglas
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:

              Development