Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5224

JobTracker should allow the system directory to be in non-default FS

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 1-win
    • Fix Version/s: 1-win
    • Component/s: jobtracker
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      JobTracker today expects the system directory to be in the default file system
      if (fs == null) {
      fs = mrOwner.doAs(new PrivilegedExceptionAction<FileSystem>() {
      public FileSystem run() throws IOException {
      return FileSystem.get(conf);
      }});
      }

      ...

      public String getSystemDir()

      { Path sysDir = new Path(conf.get("mapred.system.dir", "/tmp/hadoop/mapred/system")); return fs.makeQualified(sysDir).toString(); }

      In Cloud like Azure the default file system is set as ASV (Windows Azure Blob Storage), but we would still like the system directory to be in DFS. We should change JobTracker to allow that.

        Attachments

        1. MAPREDUCE-5224.2.patch
          13 kB
          Douma Fang
        2. MAPREDUCE-5224.3.patch
          16 kB
          Douma Fang
        3. MAPREDUCE-5224.4.patch
          12 kB
          Douma Fang
        4. MAPREDUCE-5224.5.patch
          4 kB
          Douma Fang
        5. MAPREDUCE-5224.patch
          3 kB
          Douma Fang

          Issue Links

            Activity

              People

              • Assignee:
                xifang Douma Fang
                Reporter:
                xifang Douma Fang
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: