Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Incomplete
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Environment:

      High availability enterprise system

      Description

      The Hadoop framework has been designed, in an eort to enhance perfor-
      mances, with a single JobTracker (master node). It's responsibilities varies
      from managing job submission process, compute the input splits, schedule
      the tasks to the slave nodes (TaskTrackers) and monitor their health.
      In some environments, like the IBM and Google's Internet-scale com-
      puting initiative, there is the need for high-availability, and performances
      becomes a secondary issue. In this environments, having a system with
      a Single Point of Failure (such as Hadoop's single JobTracker) is a major
      concern.
      My proposal is to provide a redundant version of Hadoop by adding
      support for multiple replicated JobTrackers. This design can be approached
      in many dierent ways.

      In the document at: http://sites.google.com/site/hadoopthesis/Home/FaultTolerantHadoop.pdf?attredirects=0

      I wrote an overview of the problem and some approaches to solve it.

      I post this to the community to gather feedback on the best way to proceed in my work.

      Thank you!

      1. Enhancing the Hadoop MapReduce framework by adding fault.ppt
        511 kB
        Francesco Salbaroli
      2. FaultTolerantHadoop.pdf
        136 kB
        Francesco Salbaroli
      3. HADOOP-4586-0.1.patch
        35 kB
        Francesco Salbaroli
      4. HADOOP-4586v0.3.patch
        39 kB
        Francesco Salbaroli
      5. jgroups-all.jar
        1.92 MB
        Francesco Salbaroli

        Issue Links

          Activity

          Allen Wittenauer made changes -
          Status In Progress [ 3 ] Resolved [ 5 ]
          Resolution Incomplete [ 4 ]
          Allen Wittenauer made changes -
          Link This issue is duplicated by YARN-149 [ YARN-149 ]
          Gavin made changes -
          Link This issue depends upon HADOOP-1876 [ HADOOP-1876 ]
          Gavin made changes -
          Link This issue depends on HADOOP-1876 [ HADOOP-1876 ]
          Gavin made changes -
          Link This issue depends upon HADOOP-3245 [ HADOOP-3245 ]
          Gavin made changes -
          Link This issue depends on HADOOP-3245 [ HADOOP-3245 ]
          Eli Collins made changes -
          Link This issue is related to MAPREDUCE-2288 [ MAPREDUCE-2288 ]
          Owen O'Malley made changes -
          Project Hadoop Common [ 12310240 ] Hadoop Map/Reduce [ 12310941 ]
          Key HADOOP-4586 MAPREDUCE-225
          Component/s mapred [ 12310690 ]
          Fix Version/s 0.21.0 [ 12313563 ]
          Francesco Salbaroli made changes -
          Attachment HADOOP-4586v0.3.patch [ 12397891 ]
          Francesco Salbaroli made changes -
          Link This issue depends on HADOOP-3245 [ HADOOP-3245 ]
          Francesco Salbaroli made changes -
          Link This issue depends on HADOOP-1876 [ HADOOP-1876 ]
          Francesco Salbaroli made changes -
          Affects Version/s 0.18.1 [ 12313357 ]
          Affects Version/s 0.18.0 [ 12312972 ]
          Affects Version/s 0.18.2 [ 12313424 ]
          Nigel Daley made changes -
          Fix Version/s 0.18.3 [ 12313494 ]
          Fix Version/s 0.21.0 [ 12313563 ]
          Francesco Salbaroli made changes -
          Francesco Salbaroli made changes -
          Remaining Estimate 960h [ 3456000 ]
          Original Estimate 960h [ 3456000 ]
          Francesco Salbaroli made changes -
          Remaining Estimate 2,016h [ 7257600 ] 960h [ 3456000 ]
          Original Estimate 2,016h [ 7257600 ] 960h [ 3456000 ]
          Francesco Salbaroli made changes -
          Affects Version/s 0.18.1 [ 12313357 ]
          Fix Version/s 0.18.3 [ 12313494 ]
          Affects Version/s 0.18.2 [ 12313424 ]
          Francesco Salbaroli made changes -
          Attachment jgroups-all.jar [ 12396380 ]
          Attachment HADOOP-4586-0.1.patch [ 12396379 ]
          Francesco Salbaroli made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          Francesco Salbaroli made changes -
          Assignee Francesco Salbaroli [ fsalbaroli ]
          Francesco Salbaroli made changes -
          Link This issue incorporates HADOOP-4016 [ HADOOP-4016 ]
          Francesco Salbaroli made changes -
          Field Original Value New Value
          Attachment FaultTolerantHadoop.pdf [ 12393292 ]
          Francesco Salbaroli created issue -

            People

            • Assignee:
              Francesco Salbaroli
              Reporter:
              Francesco Salbaroli
            • Votes:
              3 Vote for this issue
              Watchers:
              46 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development