Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-326

Add a lifecycle interface for Hadoop components: namenodes, job clients, etc.

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Won't Fix
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      I'd like to propose we have a standard interface for hadoop components, the things that get started or stopped when you bring up a namenode. currently, some of these classes have a stop() or shutdown() method, with no standard name/interface, but no way of seeing if they are live, checking their health of shutting them down reliably. Indeed, there is a tendency for the spawned threads to not want to die; to require the entire process to be killed to stop the workers.

      Having a standard interface would make it easier for

      • management tools to manage the different things
      • monitoring the state of things
      • subclassing

      The latter is interesting as right now TaskTracker and JobTracker start up threads in their constructor; that's very dangerous as subclasses may have their methods called before they are full initialised. Adding this interface would be the right time to clean up the startup process so that subclassing is less risky.

      1. HDFS-326.patch
        17 kB
        Steve Loughran
      2. hadoop-lifecycle-tomw.sxw
        144 kB
        Steve Loughran
      3. hadoop-lifecycle.sxw
        143 kB
        Steve Loughran
      4. hadoop-lifecycle.pdf
        316 kB
        Steve Loughran
      5. hadoop-lifecycle.pdf
        321 kB
        Steve Loughran
      6. HADOOP-3628-19.patch
        347 kB
        Steve Loughran
      7. HADOOP-3628-18.patch
        126 kB
        Steve Loughran
      8. hadoop-3628.patch
        49 kB
        Steve Loughran
      9. hadoop-3628.patch
        55 kB
        Steve Loughran
      10. hadoop-3628.patch
        70 kB
        Steve Loughran
      11. hadoop-3628.patch
        55 kB
        Steve Loughran
      12. hadoop-3628.patch
        96 kB
        Steve Loughran
      13. hadoop-3628.patch
        121 kB
        Steve Loughran
      14. hadoop-3628.patch
        147 kB
        Steve Loughran
      15. hadoop-3628.patch
        152 kB
        Steve Loughran
      16. hadoop-3628.patch
        139 kB
        Steve Loughran
      17. hadoop-3628.patch
        143 kB
        Steve Loughran
      18. hadoop-3628.patch
        142 kB
        Steve Loughran
      19. hadoop-3628.patch
        154 kB
        Steve Loughran
      20. hadoop-3628.patch
        165 kB
        Steve Loughran
      21. hadoop-3628.patch
        164 kB
        Steve Loughran
      22. hadoop-3628.patch
        164 kB
        Steve Loughran
      23. hadoop-3628.patch
        167 kB
        Steve Loughran
      24. hadoop-3628.patch
        162 kB
        Steve Loughran
      25. AbstractHadoopComponent.java
        3 kB
        Steve Loughran

        Issue Links

          Activity

          Gavin made changes -
          Link This issue depends upon HADOOP-5348 [ HADOOP-5348 ]
          Gavin made changes -
          Link This issue depends on HADOOP-5348 [ HADOOP-5348 ]
          Gavin made changes -
          Link This issue is depended upon by HADOOP-3969 [ HADOOP-3969 ]
          Gavin made changes -
          Link This issue blocks HADOOP-3969 [ HADOOP-3969 ]
          Gavin made changes -
          Link This issue is depended upon by MAPREDUCE-233 [ MAPREDUCE-233 ]
          Gavin made changes -
          Link This issue blocks MAPREDUCE-233 [ MAPREDUCE-233 ]
          Gavin made changes -
          Link This issue depends upon HADOOP-4059 [ HADOOP-4059 ]
          Gavin made changes -
          Link This issue depends on HADOOP-4059 [ HADOOP-4059 ]
          Gavin made changes -
          Link This issue depends upon HADOOP-3642 [ HADOOP-3642 ]
          Gavin made changes -
          Link This issue depends on HADOOP-3642 [ HADOOP-3642 ]
          Gavin made changes -
          Link This issue is depended upon by HDFS-239 [ HDFS-239 ]
          Gavin made changes -
          Link This issue blocks HDFS-239 [ HDFS-239 ]
          Steve Loughran made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Won't Fix [ 2 ]
          Steve Loughran made changes -
          Status In Progress [ 3 ] Open [ 1 ]
          Jeff Hammerbacher made changes -
          Link This issue relates to HADOOP-6915 [ HADOOP-6915 ]
          Gavin made changes -
          Assignee Steve Loughran [ steve_l ] Steve Loughran [ stevel@apache.org ]
          Gavin made changes -
          Reporter Steve Loughran [ steve_l ] Steve Loughran [ stevel@apache.org ]
          Steve Loughran made changes -
          Attachment HDFS-326.patch [ 12443123 ]
          Steve Loughran made changes -
          Attachment hadoop-lifecycle.pdf [ 12419194 ]
          Steve Loughran made changes -
          Attachment HADOOP-3628-19.patch [ 12416603 ]
          Steve Loughran made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          Steve Loughran made changes -
          Link This issue incorporates HADOOP-6194 [ HADOOP-6194 ]
          Chris Douglas made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Owen O'Malley made changes -
          Project Hadoop Common [ 12310240 ] HDFS [ 12310942 ]
          Key HADOOP-3628 HDFS-326
          Affects Version/s 0.21.0 [ 12313563 ]
          Component/s mapred [ 12310690 ]
          Component/s dfs [ 12310710 ]
          Fix Version/s 0.21.0 [ 12313563 ]
          Steve Loughran made changes -
          Status In Progress [ 3 ] Patch Available [ 10002 ]
          Affects Version/s 0.21.0 [ 12313563 ]
          Affects Version/s 0.20.0 [ 12313438 ]
          Steve Loughran made changes -
          Attachment HADOOP-3628-18.patch [ 12410781 ]
          Steve Loughran made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          Amareshwari Sriramadasu made changes -
          Link This issue is related to HADOOP-5525 [ HADOOP-5525 ]
          Amareshwari Sriramadasu made changes -
          Link This issue blocks HADOOP-5525 [ HADOOP-5525 ]
          Amareshwari Sriramadasu made changes -
          Link This issue is related to HADOOP-5525 [ HADOOP-5525 ]
          Steve Loughran made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Steve Loughran made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Steve Loughran made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Eric Yang made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Eric Yang made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Steve Loughran made changes -
          Fix Version/s 0.21.0 [ 12313563 ]
          Status Open [ 1 ] Patch Available [ 10002 ]
          Steve Loughran made changes -
          Attachment hadoop-3628.patch [ 12401531 ]
          Steve Loughran made changes -
          Link This issue depends on HADOOP-5348 [ HADOOP-5348 ]
          Steve Loughran made changes -
          Attachment hadoop-3628.patch [ 12399838 ]
          Steve Loughran made changes -
          Attachment hadoop-3628.patch [ 12398487 ]
          Steve Loughran made changes -
          Attachment hadoop-lifecycle-tomw.sxw [ 12398486 ]
          Steve Loughran made changes -
          Attachment hadoop-3628.patch [ 12397797 ]
          Steve Loughran made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Nigel Daley made changes -
          Fix Version/s 0.20.0 [ 12313438 ]
          Steve Loughran made changes -
          Affects Version/s 0.20.0 [ 12313438 ]
          Fix Version/s 0.20.0 [ 12313438 ]
          Affects Version/s 0.19.0 [ 12313211 ]
          Status Open [ 1 ] Patch Available [ 10002 ]
          Steve Loughran made changes -
          Attachment hadoop-3628.patch [ 12395934 ]
          Steve Loughran made changes -
          Attachment hadoop-3628.patch [ 12391822 ]
          Steve Loughran made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Steve Loughran made changes -
          Link This issue depends on HADOOP-4059 [ HADOOP-4059 ]
          Steve Loughran made changes -
          Attachment hadoop-lifecycle.pdf [ 12389357 ]
          Steve Loughran made changes -
          Attachment hadoop-lifecycle.sxw [ 12389356 ]
          Steve Loughran made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Steve Loughran made changes -
          Attachment hadoop-3628.patch [ 12389295 ]
          Steve Loughran made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Steve Loughran made changes -
          Attachment hadoop-3628.patch [ 12388752 ]
          Steve Loughran made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Steve Loughran made changes -
          Attachment hadoop-3628.patch [ 12388602 ]
          Pete Wyckoff made changes -
          Link This issue blocks HADOOP-3969 [ HADOOP-3969 ]
          Steve Loughran made changes -
          Attachment hadoop-3628.patch [ 12388320 ]
          Steve Loughran made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Steve Loughran made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Steve Loughran made changes -
          Attachment hadoop-3628.patch [ 12387480 ]
          Steve Loughran made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Steve Loughran made changes -
          Status In Progress [ 3 ] Patch Available [ 10002 ]
          Steve Loughran made changes -
          Attachment hadoop-3628.patch [ 12387213 ]
          Steve Loughran made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          Steve Loughran made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Chris Douglas made changes -
          Link This issue blocks HADOOP-3767 [ HADOOP-3767 ]
          Steve Loughran made changes -
          Status In Progress [ 3 ] Patch Available [ 10002 ]
          Steve Loughran made changes -
          Attachment hadoop-3628.patch [ 12387094 ]
          Steve Loughran made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          Steve Loughran made changes -
          Link This issue incorporates HADOOP-3415 [ HADOOP-3415 ]
          Steve Loughran made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Steve Loughran made changes -
          Attachment hadoop-3628.patch [ 12385643 ]
          Steve Loughran made changes -
          Attachment hadoop-3628.patch [ 12385401 ]
          Steve Loughran made changes -
          Link This issue blocks HADOOP-3729 [ HADOOP-3729 ]
          Steve Loughran made changes -
          Status In Progress [ 3 ] Patch Available [ 10002 ]
          Steve Loughran made changes -
          Attachment hadoop-3628.patch [ 12385642 ]
          Steve Loughran made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          Steve Loughran made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Steve Loughran made changes -
          Attachment hadoop-3628.patch [ 12385519 ]
          Steve Loughran made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Steve Loughran made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Steve Loughran made changes -
          Attachment hadoop-3628.patch [ 12385485 ]
          Steve Loughran made changes -
          Status In Progress [ 3 ] Patch Available [ 10002 ]
          Steve Loughran made changes -
          Attachment hadoop-3628.patch [ 12385401 ]
          Steve Loughran made changes -
          Link This issue depends on HADOOP-3642 [ HADOOP-3642 ]
          Steve Loughran made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          Steve Loughran made changes -
          Assignee Steve Loughran [ steve_l ]
          Steve Loughran made changes -
          Field Original Value New Value
          Attachment AbstractHadoopComponent.java [ 12384662 ]
          Steve Loughran created issue -

            People

            • Assignee:
              Steve Loughran
              Reporter:
              Steve Loughran
            • Votes:
              3 Vote for this issue
              Watchers:
              32 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development