Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Data Processors
    • Labels:
      None
    • Release Note:
      Hooks added to trigger actions upon successful completion of a Demux run.

      Description

      Add the ability to trigger an action upon successful completion of a Demux process.

      1. CHUKWA-477.1.patch
        10 kB
        Bill Graham
      2. CHUKWA-477.2.patch
        18 kB
        Bill Graham
      3. CHUKWA-477.3.patch
        18 kB
        Bill Graham
      4. CHUKWA-477.4.patch
        21 kB
        Bill Graham

        Activity

        Hide
        billgraham Bill Graham added a comment -

        Here's a first pass at this functionality (CHUKWA-477.1.patch) for review. Please review and provide feedback re design and naming. A few things to note:

        • I've added the following constant and interface/package:
          package org.apache.hadoop.chukwa.datatrigger;
          
          public interface TriggerAction {
          
            public void load(ChukwaConfiguration conf, FileSystem fs, FileStatus[] src) throws IOException;
          }
          
          public interface CHUKWA_CONSTANT {
            public static final String POST_DEMUX_SUCCESS_ACTION = "chukwa.post.demux.success.action";  
          }
          
        • The idea is that you can implement specific TriggerActions for to occur in different scenarios. They know how to react to some action, which is relayed via the FileStatus[] objects. The method signature of TriggerAction is the same as that of DataLoaderFactory, so the latter could implement the former if we want to standardize and go that route. Or we could create a new signature for TriggerAction that contains more context re the trigger itself.
        • PostProcessorMananger does not know the names of the files that were added to repos/, since those details are never returned by MoveToRepository. I'd like to change MoveToRepository to return a collection of Paths to the action can see what's been added. This could be done in a separate JIRA if that makes more sense.
        Show
        billgraham Bill Graham added a comment - Here's a first pass at this functionality ( CHUKWA-477 .1.patch) for review. Please review and provide feedback re design and naming. A few things to note: I've added the following constant and interface/package: package org.apache.hadoop.chukwa.datatrigger; public interface TriggerAction { public void load(ChukwaConfiguration conf, FileSystem fs, FileStatus[] src) throws IOException; } public interface CHUKWA_CONSTANT { public static final String POST_DEMUX_SUCCESS_ACTION = "chukwa.post.demux.success.action"; } The idea is that you can implement specific TriggerActions for to occur in different scenarios. They know how to react to some action, which is relayed via the FileStatus[] objects. The method signature of TriggerAction is the same as that of DataLoaderFactory, so the latter could implement the former if we want to standardize and go that route. Or we could create a new signature for TriggerAction that contains more context re the trigger itself. PostProcessorMananger does not know the names of the files that were added to repos/, since those details are never returned by MoveToRepository. I'd like to change MoveToRepository to return a collection of Paths to the action can see what's been added. This could be done in a separate JIRA if that makes more sense.
        Hide
        eyang Eric Yang added a comment -

        +1 looks good.

        Show
        eyang Eric Yang added a comment - +1 looks good.
        Hide
        billgraham Bill Graham added a comment -

        Attaching CHUKWA-477.2.patch, which includes the following changes:

        • I've changed the 'load' method to 'execute' in TriggerAction, which makes more sense. It has the same signature though, so we could easily refactor the DataLoaders to implement TriggerAction at some point.
        • I modified MoveToRepository to return the HDFS file names that were created as part of the move operation.
        • I'm passing the moved files collection into the TriggerAction.
        • I added a sample config in chukwa-demux-conf.xml.template to show how to configure a post-demux trigger.
        Show
        billgraham Bill Graham added a comment - Attaching CHUKWA-477 .2.patch, which includes the following changes: I've changed the 'load' method to 'execute' in TriggerAction, which makes more sense. It has the same signature though, so we could easily refactor the DataLoaders to implement TriggerAction at some point. I modified MoveToRepository to return the HDFS file names that were created as part of the move operation. I'm passing the moved files collection into the TriggerAction. I added a sample config in chukwa-demux-conf.xml.template to show how to configure a post-demux trigger.
        Hide
        billgraham Bill Graham added a comment -

        One last minor rev of this patch. The TriggerAction signature should take the more generic Configuration, not ChukwaConfiguration.

        Show
        billgraham Bill Graham added a comment - One last minor rev of this patch. The TriggerAction signature should take the more generic Configuration, not ChukwaConfiguration.
        Hide
        billgraham Bill Graham added a comment -

        Canceling latest patch 3, since I plan to resubmit with a change to the signature of TriggerAction to accept info regarding the event causing the trigger.

        Show
        billgraham Bill Graham added a comment - Canceling latest patch 3, since I plan to resubmit with a change to the signature of TriggerAction to accept info regarding the event causing the trigger.
        Hide
        billgraham Bill Graham added a comment -

        Attached is CHUKWA-477.4.patch, which adds the concept of a TriggerEvent enum, which gets passed to the TriggerAction. Unless there are other comments, I'm code-compete on the trigger work.

        I've been running my HttpTriggerAction (as discussed on the mailing list) with this patch and will submit that as a separate JIRA.

        Show
        billgraham Bill Graham added a comment - Attached is CHUKWA-477 .4.patch, which adds the concept of a TriggerEvent enum, which gets passed to the TriggerAction. Unless there are other comments, I'm code-compete on the trigger work. I've been running my HttpTriggerAction (as discussed on the mailing list) with this patch and will submit that as a separate JIRA.
        Hide
        eyang Eric Yang added a comment -

        I committed this yesterday, thanks Bill.

        Show
        eyang Eric Yang added a comment - I committed this yesterday, thanks Bill.

          People

          • Assignee:
            billgraham Bill Graham
            Reporter:
            billgraham Bill Graham
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development