Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-5640

Allow ServicePlugins to hook callbacks into key service events

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Won't Fix
    • None
    • None
    • util
    • None

    Description

      HADOOP-5257 added the ability for NameNode and DataNode to start and stop ServicePlugin implementations at NN/DN start/stop. However, this is insufficient integration for some common use cases.

      We should add some functionality for Plugins to subscribe to events generated by the service they're plugging into. Some potential hook points are:

      NameNode:

      • new datanode registered
      • datanode has died
      • exception caught
      • etc?

      DataNode:

      • startup
      • initial registration with NN complete (this is important for HADOOP-4707 to sync up datanode.dnRegistration.name with the NN-side registration)
      • namenode reconnect
      • some block transfer hooks?
      • exception caught

      I see two potential routes for implementation:

      1) We make an enum for the types of hookpoints and have a general function in the ServicePlugin interface. Something like:

      enum HookPoint {
        DN_STARTUP,
        DN_RECEIVED_NEW_BLOCK,
        DN_CAUGHT_EXCEPTION,
       ...
      }
      
      void runHook(HookPoint hp, Object value);
      

      2) We make classes specific to each "pluggable" as was originally suggested in HADDOP-5257. Something like:

      class DataNodePlugin {
        void datanodeStarted() {}
        void receivedNewBlock(block info, etc) {}
        void caughtException(Exception e) {}
        ...
      }
      

      I personally prefer option (2) since we can ensure plugin API compatibility at compile-time, and we avoid an ugly switch statement in a runHook() function.

      Interested to hear what people's thoughts are here.

      Attachments

        1. hadoop-5640.v3.txt
          20 kB
          Todd Lipcon
        2. HADOOP-5640.v2.txt
          20 kB
          Todd Lipcon
        3. hadoop-5640.txt
          19 kB
          Todd Lipcon
        4. hadoop-5640.txt
          11 kB
          Todd Lipcon
        5. 0149-HADOOP-5640-puts-this-in-the-new-test-dir.-It-needs.patch
          8 kB
          Todd Lipcon
        6. 0093-HADOOP-5640-Add-dispatch-mechanism-for-services-to.patch
          21 kB
          Todd Lipcon

        Issue Links

          Activity

            People

              tlipcon Todd Lipcon
              tlipcon Todd Lipcon
              Votes:
              0 Vote for this issue
              Watchers:
              18 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: