Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6988

Let JHS support different file systems for intermediate_done and done

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Patch Available
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 2.7.4
    • Fix Version/s: None
    • Component/s: jobhistoryserver
    • Labels:
      None
    • Target Version/s:

      Description

      Currently JHS uses filecontext to move files from intermediate_done to done folder. Since filecontext limits the use to 1 filesystem it makes it harder to use s3 as a storage for jhist files. By moving this to filesystem interface we can set hdfs for intermediate storage and s3 as long term storage therefore reducing the number of puts to s3 and removing the need for all M/R containers to carry a s3 sdk.

        Attachments

        1. MAPREDUCE-6988.000.patch
          6 kB
          Johan Gustavsson
        2. MAPREDUCE-6988.001.patch
          6 kB
          Johan Gustavsson
        3. MAPREDUCE-6988.002.patch
          9 kB
          Johan Gustavsson
        4. MAPREDUCE-6988.003.patch
          9 kB
          Johan Gustavsson

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              johang Johan Gustavsson
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: