Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Later
    • Affects Version/s: 0.5.0
    • Fix Version/s: 0.6
    • Labels:
      None

      Description

      Extend Guice AbstractModule to create a simple application builder layer. The idea is to evaluate if we can use Guice to do all the wiring including PEs and Streams and Apps.

      1. s4-fluent.pdf
        128 kB
        Leo Neumeyer
      2. s4-app.png
        1.16 MB
        Leo Neumeyer

        Activity

        Hide
        Leo Neumeyer added a comment -

        From: Aron Sogor
        Date: Tue, Nov 15, 2011 at 6:01 AM
        Subject: S4 piper object creation design
        To: s4-dev@incubator.apache.org

        Hi,

        First of congrats, I really like the stuff you have built, I think it is
        one of the cleanest actor models in java, gives me all the reasons not to
        battle an erlang vm. I also really like that you took up Guice and the
        whole idea of ditching the XML config paradigm is sweet!

        What remains strange is that PE and Stream classes are created by the app
        and not via Guice. Here is my concern:

        • In the case of the Stream it is probably not a big deal as a developer I
          should not be extending or writing my stream classes. (Day #1 I might be
          missing something)
        • In the case of a PE, I would probably like to inject things, from the
          most basic and beloved @Inject Logger, some other custom objects of my own
          to probably even Streams.

        The current model makes this pretty clunky as the app is created by the
        injector but the PE and the Streams are made by the App. I think This could
        overcome with some Tricks with Provider<T> or other helpers. Clearly the is
        a fundamental design choice:

        • Why the App.init() method was created this way? Besides it felt cute, and
          give a sort of DSL to it, is there any other reason?

        I wonder if you up for revisiting this aspect of piper and if you are I
        could help experiment on a github fork, but I would like to understand the
        goals or intentions behind the current idea.

        Aron

        Show
        Leo Neumeyer added a comment - From: Aron Sogor Date: Tue, Nov 15, 2011 at 6:01 AM Subject: S4 piper object creation design To: s4-dev@incubator.apache.org Hi, First of congrats, I really like the stuff you have built, I think it is one of the cleanest actor models in java, gives me all the reasons not to battle an erlang vm. I also really like that you took up Guice and the whole idea of ditching the XML config paradigm is sweet! What remains strange is that PE and Stream classes are created by the app and not via Guice. Here is my concern: In the case of the Stream it is probably not a big deal as a developer I should not be extending or writing my stream classes. (Day #1 I might be missing something) In the case of a PE, I would probably like to inject things, from the most basic and beloved @Inject Logger, some other custom objects of my own to probably even Streams. The current model makes this pretty clunky as the app is created by the injector but the PE and the Streams are made by the App. I think This could overcome with some Tricks with Provider<T> or other helpers. Clearly the is a fundamental design choice: Why the App.init() method was created this way? Besides it felt cute, and give a sort of DSL to it, is there any other reason? I wonder if you up for revisiting this aspect of piper and if you are I could help experiment on a github fork, but I would like to understand the goals or intentions behind the current idea. Aron
        Hide
        Leo Neumeyer added a comment -

        I started to work on the AppMaker package. The goal is to use a builder pattern to build apps and do the bindings with Guice behind the scenes. I started with the end user API. I'm also considering modifying the App class to extend Guice's AbstractModule. Suggestions welcomed.

        This is how a user would create an app:

        Prototyping App Builder
        package org.apache.s4.appbuilder;
        
        public class Main {
        
            /**
             * @param args
             */
            public static void main(String[] args) {
        
                AppMaker am = new AppMaker();
        
                PEMaker pem1, pem2;
                StreamMaker s1;
                StreamMaker s2, s3;
        
                pem1 = am.addPE(PEZ.class);
        
                s1 = am.addStream(EventA.class).withName("My first stream.").withKeyFinder("{gender}").to(pem1);
        
                pem2 = am.addPE(PEY.class).to(s1);
        
                s2 = am.addStream(EventB.class).withName("My second stream.").withKeyFinder("{age}").to(pem2);
        
                s3 = am.addStream(EventB.class).withName("My third stream.").withKeyFinder("{height}").to(pem2);
        
                am.addPE(PEX.class).to(s2).to(s3);
            }
        }
        

        Notes:

        • I am not using generics here, instead I pass the PE and Event types. (I assume we can build what we want with reflection and only need the type.)
        • For now I am doing this as a separate package but this may end up: PEMaker -> ProcessingElement and StreamMaker -> Stream.
        • The KeyFinder needs to be generated from a string. (Can we reuse the code from 0.3?)
        • Need to build the graph using the Guice API.

        The branch name is appbuilder. - GH repo

        Show
        Leo Neumeyer added a comment - I started to work on the AppMaker package. The goal is to use a builder pattern to build apps and do the bindings with Guice behind the scenes. I started with the end user API. I'm also considering modifying the App class to extend Guice's AbstractModule. Suggestions welcomed. This is how a user would create an app: Prototyping App Builder package org.apache.s4.appbuilder; public class Main { /** * @param args */ public static void main( String [] args) { AppMaker am = new AppMaker(); PEMaker pem1, pem2; StreamMaker s1; StreamMaker s2, s3; pem1 = am.addPE(PEZ.class); s1 = am.addStream(EventA.class).withName( "My first stream." ).withKeyFinder( "{gender}" ).to(pem1); pem2 = am.addPE(PEY.class).to(s1); s2 = am.addStream(EventB.class).withName( "My second stream." ).withKeyFinder( "{age}" ).to(pem2); s3 = am.addStream(EventB.class).withName( "My third stream." ).withKeyFinder( "{height}" ).to(pem2); am.addPE(PEX.class).to(s2).to(s3); } } Notes: I am not using generics here, instead I pass the PE and Event types. (I assume we can build what we want with reflection and only need the type.) For now I am doing this as a separate package but this may end up: PEMaker -> ProcessingElement and StreamMaker -> Stream. The KeyFinder needs to be generated from a string. (Can we reuse the code from 0.3?) Need to build the graph using the Guice API. The branch name is appbuilder. - GH repo
        Hide
        kishore gopalakrishna added a comment -

        Hi Leo,

        I really like this approach hiding the Guice details underneath.

        Few thoughts,

        • Feels like there are multiple ways to add PE to streams ( while creating the StreamMaker s1,s2,s3) and then am.addPE.to(s2).to(s3).
        • will it help separating the two one step to add stream and one step to addPE to stream.

        Also it looks like keyFinder is tied to stream 'am.addStream(EventA.class).withName("My first stream.").withKeyFinder("

        {gender}")' but its actually tied not because one can have many keyFinder on the streams.

        Does this make sense

        s1 = am.addStream("My first stream").ofType(EventA.class)

        am.onStream(s1).invoke(PEX.class).usingKeyFinder("{gender}

        ")

        Do we really need StreamMaker and PEMaker

        Show
        kishore gopalakrishna added a comment - Hi Leo, I really like this approach hiding the Guice details underneath. Few thoughts, Feels like there are multiple ways to add PE to streams ( while creating the StreamMaker s1,s2,s3) and then am.addPE.to(s2).to(s3). will it help separating the two one step to add stream and one step to addPE to stream. Also it looks like keyFinder is tied to stream 'am.addStream(EventA.class).withName("My first stream.").withKeyFinder(" {gender}")' but its actually tied not because one can have many keyFinder on the streams. Does this make sense s1 = am.addStream("My first stream").ofType(EventA.class) am.onStream(s1).invoke(PEX.class).usingKeyFinder("{gender} ") Do we really need StreamMaker and PEMaker
        Hide
        Leo Neumeyer added a comment -

        Good ideas, let me think. I used PEMaker/StreamMaker to experiment but you are right, I am now implementing directly in PE/Stream classes and it is pretty neat. In fact, it is even backward compatible so far. I am struggling with a generics problem so I'm a bit stuck. I will try to unstuck myself and update the repo soon. For now I am trying to use Generics instead of types (like SomePE.class). More later.

        Show
        Leo Neumeyer added a comment - Good ideas, let me think. I used PEMaker/StreamMaker to experiment but you are right, I am now implementing directly in PE/Stream classes and it is pretty neat. In fact, it is even backward compatible so far. I am struggling with a generics problem so I'm a bit stuck. I will try to unstuck myself and update the repo soon. For now I am trying to use Generics instead of types (like SomePE.class). More later.
        Hide
        Leo Neumeyer added a comment -

        I pushed the appbuilder branch. Check App Stream ProcessingElement classes and Counter Example.

        https://github.com/leoneu/s4-piper/tree/appbuilder/subprojects/s4-core/src/main/java/org/apache/s4/appbuilder

        I couldn't figure out how to do the builder for PEs without a helper class. The problem is that some methods are in ProcessingElement and some are in the subclass eg. CounterPE. Ideas welcomed, my brain is out.

        All this effort only seems to be a stylistic change (using builder pattern instead of setters). Not sure this makes such a big difference. This is still not being injected by Guice and it is still not clear how to do it.

        Show
        Leo Neumeyer added a comment - I pushed the appbuilder branch. Check App Stream ProcessingElement classes and Counter Example. https://github.com/leoneu/s4-piper/tree/appbuilder/subprojects/s4-core/src/main/java/org/apache/s4/appbuilder I couldn't figure out how to do the builder for PEs without a helper class. The problem is that some methods are in ProcessingElement and some are in the subclass eg. CounterPE. Ideas welcomed, my brain is out. All this effort only seems to be a stylistic change (using builder pattern instead of setters). Not sure this makes such a big difference. This is still not being injected by Guice and it is still not clear how to do it.
        Hide
        Leo Neumeyer added a comment -

        I summarize the problem here:

        Builder Pattern
        class PE {
          PE builderMethod() {
          }
        }
        
        class SomePE extends PE {
          void someSetter() {
          }
        }
        
        class MyApp extends App {
          onInit {
             PE pe = createPE(SomePE.class).with("XXX");
             SomePE somePE = (SomePE) pe;
             somePE.someSetter();
          }
        }
        

        How can we do this without requiring a cast? Asking end user to cast is out of the question of course. I also want to avoid a separate builder class. Ideas?

        Show
        Leo Neumeyer added a comment - I summarize the problem here: Builder Pattern class PE { PE builderMethod() { } } class SomePE extends PE { void someSetter() { } } class MyApp extends App { onInit { PE pe = createPE(SomePE.class).with( "XXX" ); SomePE somePE = (SomePE) pe; somePE.someSetter(); } } How can we do this without requiring a cast? Asking end user to cast is out of the question of course. I also want to avoid a separate builder class. Ideas?
        Hide
        Matthieu Morel added a comment -

        I also like the builder pattern exposed from the S4 API, looks very nice and intuitive.

        You can avoid casting by using generics:

        public static <T> T createPE(Class<T> peClass) throws InstantiationException, IllegalAccessException {
            return peClass.newInstance();
        }
        
        Show
        Matthieu Morel added a comment - I also like the builder pattern exposed from the S4 API, looks very nice and intuitive. You can avoid casting by using generics: public static <T> T createPE( Class <T> peClass) throws InstantiationException, IllegalAccessException { return peClass.newInstance(); }
        Hide
        Leo Neumeyer added a comment -

        That's how createPE is implemented. The problem is with with() which returns PE and and createPE() returns SomePE. In any case after the Skype meeting (Matthieu and I) concluded that we do need helper classes to do the fluent API. It solves this problem and will probably be required to do the Guice wiring behind the scenes so I am leaving the original API as is and will build the fluent one on top. (PEMaker, StreamMaker, AppMaker).

        Show
        Leo Neumeyer added a comment - That's how createPE is implemented. The problem is with with() which returns PE and and createPE() returns SomePE. In any case after the Skype meeting (Matthieu and I) concluded that we do need helper classes to do the fluent API. It solves this problem and will probably be required to do the Guice wiring behind the scenes so I am leaving the original API as is and will build the fluent one on top. (PEMaker, StreamMaker, AppMaker).
        Hide
        Leo Neumeyer added a comment -

        Checked Fluent S4 API to master.
        https://github.com/leoneu/s4-piper/tree/master/subprojects/s4-core/src/test/java/org/apache/s4/appmaker

        For now, this is what an app looks like:

        S4 Fluent API
        package org.apache.s4.appmaker;
        
        public class MyApp extends AppMaker {
        
            @Override
            protected void configure() {
        
                PEMaker pe1, pe2;
                StreamMaker s1;
                StreamMaker s2, s3;
        
                pe1 = addPE(PEZ.class);
        
                s1 = addStream(EventA.class).withName("My first stream.").withKey("{gender}").to(pe1);
        
                pe2 = addPE(PEY.class).to(s1);
        
                s2 = addStream(EventB.class).withName("My second stream.").withKey("{age}").to(pe2);
        
                s3 = addStream(EventB.class).withName("My third stream.").withKey("{height}").to(pe2);
        
                addPE(PEX.class).to(s2).to(s3);
            }
        }
        
        Show
        Leo Neumeyer added a comment - Checked Fluent S4 API to master. https://github.com/leoneu/s4-piper/tree/master/subprojects/s4-core/src/test/java/org/apache/s4/appmaker For now, this is what an app looks like: S4 Fluent API package org.apache.s4.appmaker; public class MyApp extends AppMaker { @Override protected void configure() { PEMaker pe1, pe2; StreamMaker s1; StreamMaker s2, s3; pe1 = addPE(PEZ.class); s1 = addStream(EventA.class).withName( "My first stream." ).withKey( "{gender}" ).to(pe1); pe2 = addPE(PEY.class).to(s1); s2 = addStream(EventB.class).withName( "My second stream." ).withKey( "{age}" ).to(pe2); s3 = addStream(EventB.class).withName( "My third stream." ).withKey( "{height}" ).to(pe2); addPE(PEX.class).to(s2).to(s3); } }
        Hide
        Leo Neumeyer added a comment -

        I had to add a couple of elements.

        1. To define a stream we need the event type and the property name. The property name needs to match exactly the property name used by a ProcessingElement that consumes this stream. I dont' think there is a way around that and of course it is not type safe.
        2. Concrete PEs need configuration properties. To pass them using a generic API, I use the method property(String key, Object value).

        Let ne know what you think. Will this be easy to understand given the required conventions?

        S4 Fluent API
        package org.apache.s4.appmaker;
        
        public class MyApp extends AppMaker {
        
            @Override
            protected void configure() {
        
                PEMaker pez, pey;
                StreamMaker s1;
                StreamMaker s2, s3;
        
                pez = addPE(PEZ.class);
        
                s1 = addStream("stream1", EventA.class).withName("My first stream.").withKey("{gender}").to(pez);
        
                pey = addPE(PEY.class).to(s1).property("duration", 4).property("height", 99);
        
                s2 = addStream("stream2", EventB.class).withName("My second stream.").withKey("{age}").to(pey).to(pez);
        
                s3 = addStream("stream3", EventB.class).withKey("{height}").to(pey);
        
                addPE(PEX.class).to(s2).to(s3).property("keyword", "money");
            }
        }
        
        Show
        Leo Neumeyer added a comment - I had to add a couple of elements. To define a stream we need the event type and the property name. The property name needs to match exactly the property name used by a ProcessingElement that consumes this stream. I dont' think there is a way around that and of course it is not type safe. Concrete PEs need configuration properties. To pass them using a generic API, I use the method property(String key, Object value). Let ne know what you think. Will this be easy to understand given the required conventions? S4 Fluent API package org.apache.s4.appmaker; public class MyApp extends AppMaker { @Override protected void configure() { PEMaker pez, pey; StreamMaker s1; StreamMaker s2, s3; pez = addPE(PEZ.class); s1 = addStream( "stream1" , EventA.class).withName( "My first stream." ).withKey( "{gender}" ).to(pez); pey = addPE(PEY.class).to(s1).property( "duration" , 4).property( "height" , 99); s2 = addStream( "stream2" , EventB.class).withName( "My second stream." ).withKey( "{age}" ).to(pey).to(pez); s3 = addStream( "stream3" , EventB.class).withKey( "{height}" ).to(pey); addPE(PEX.class).to(s2).to(s3).property( "keyword" , "money" ); } }
        Hide
        Leo Neumeyer added a comment -

        Another round.

        • Got rid of StreamMaker. Now it is a graph from PE to PE using an emit() method. I can easily infer the stream from the graph, no need to create a stream element.
        • Added fluent configuration to Trigger/Timer by creating some additional helper classes that the user doesn't have to see.
        • I will try to do the stream wiring in the PEs using the Event type of the streams but I'm not sure if it is possible.
        S4 Fluent API
        package org.apache.s4.fluent;
        
        import java.util.concurrent.TimeUnit;
        
        public class MyApp extends AppMaker {
        
            @Override
            protected void configure() {
        
                PEMaker pez, pey, pex;
        
                pez = addPE(PEZ.class);
                pez.withTrigger().fireOn(EventA.class).ifInterval(5, TimeUnit.SECONDS);
                pez.withCache().ofSize(1000).withDuration(3, TimeUnit.HOURS);
        
                pey = addPE(PEY.class).property("duration", 4).property("height", 99);
                pey.withTimerInterval(50, TimeUnit.MILLISECONDS);
        
                pex = addPE(PEX.class).property("keyword", "money");
        
                pey.emit(EventA.class).withKey("gender").to(pez);
                pex.emit(EventB.class).withKey("height").to(pez);
                pex.emit(EventB.class).withKey("age").to(pey).to(pez);
            }
        }
        
        Show
        Leo Neumeyer added a comment - Another round. Got rid of StreamMaker. Now it is a graph from PE to PE using an emit() method. I can easily infer the stream from the graph, no need to create a stream element. Added fluent configuration to Trigger/Timer by creating some additional helper classes that the user doesn't have to see. I will try to do the stream wiring in the PEs using the Event type of the streams but I'm not sure if it is possible. S4 Fluent API package org.apache.s4.fluent; import java.util.concurrent.TimeUnit; public class MyApp extends AppMaker { @Override protected void configure() { PEMaker pez, pey, pex; pez = addPE(PEZ.class); pez.withTrigger().fireOn(EventA.class).ifInterval(5, TimeUnit.SECONDS); pez.withCache().ofSize(1000).withDuration(3, TimeUnit.HOURS); pey = addPE(PEY.class).property( "duration" , 4).property( "height" , 99); pey.withTimerInterval(50, TimeUnit.MILLISECONDS); pex = addPE(PEX.class).property( "keyword" , "money" ); pey.emit(EventA.class).withKey( "gender" ).to(pez); pex.emit(EventB.class).withKey( "height" ).to(pez); pex.emit(EventB.class).withKey( "age" ).to(pey).to(pez); } }
        Hide
        Leo Neumeyer added a comment - - edited

        I pushed a new version that seems to be working. I created a new Counter example using the Fluent API. The maker class tries to infer all the required info and if it cant, it exits with a message. The main ambiguity happens when a PE has more than one stream field with the same EventType. In that case, the user must provide the field name using the method withField(). Anyway, implementing and debugging was quite a bit of work. Hope this is useful! The Fluent API is completely independent and optional. Should be a good example for others who may want to write a DSL. Doing a DSL in Java is quite cumbersome.

        This is what the example looks like:

        S4 Fluent API
            @Override
            public void configure() {
        
                /* PE that prints counts to console. */
                PEMaker printPE = addPE(PrintPE.class).asSingleton();
        
                /* PEs that count events by user, gender, and age. */
                PEMaker userCountPE = addPE(CounterPE.class);
                userCountPE.addTrigger().fireOn(Event.class).ifInterval(100, TimeUnit.MILLISECONDS);
        
                PEMaker genderCountPE = addPE(CounterPE.class);
                genderCountPE.addTrigger().fireOn(Event.class).ifInterval(100, TimeUnit.MILLISECONDS);
        
                PEMaker ageCountPE = addPE(CounterPE.class);
                ageCountPE.addTrigger().fireOn(Event.class).ifInterval(100, TimeUnit.MILLISECONDS);
        
                /* PE that generates random events. */
                generateUserEventPE = addPE(GenerateUserEventPE.class).asSingleton();
                generateUserEventPE.addTimer().withDuration(1, TimeUnit.MILLISECONDS);
        
                /* Create the application graph. */
                ageCountPE.emit(CountEvent.class).onKey(new CountKeyFinder()).to(printPE);
                genderCountPE.emit(CountEvent.class).onKey(new CountKeyFinder()).to(printPE);
                userCountPE.emit(CountEvent.class).onKey(new CountKeyFinder()).to(printPE);
        
                generateUserEventPE.emit(UserEvent.class).onKey(new AgeKeyFinder()).to(ageCountPE);
                generateUserEventPE.emit(UserEvent.class).onKey(new GenderKeyFinder()).to(genderCountPE);
                generateUserEventPE.emit(UserEvent.class).onKey(new UserIDKeyFinder()).to(userCountPE);
            }
        

        I made a few additional changes:

        • By convention stream fields in PEs must be arrays: Stream<EventType>[]. This convention helps standardize and build tools.
        • Added helper method emit(<? extends Event> event, Stream<? extends Event>[] streams) in ProcessingElement to avoid having to use a for loop in each PE implementation. Simply use emit(someOutEvent, someStreamArray).
        • Removed the SingletonPE class because it was complicating the API. Instead I added a method setSingleton() that creates an eagerly singleton in the app. That is, the PE instance is created and initialized just before the app starts and before events arrive.
        • I improved initialization coordination.
        • Started to use the Guava Precondition class because it makes the code cleaner. Please look at ProcessingElement and use a similar pattern.
        • Fixed some bugs.

        TODO:

        • Expressive language to specify key finder in the DSL.
        • More extensive unit testing.
        Show
        Leo Neumeyer added a comment - - edited I pushed a new version that seems to be working. I created a new Counter example using the Fluent API. The maker class tries to infer all the required info and if it cant, it exits with a message. The main ambiguity happens when a PE has more than one stream field with the same EventType. In that case, the user must provide the field name using the method withField(). Anyway, implementing and debugging was quite a bit of work. Hope this is useful! The Fluent API is completely independent and optional. Should be a good example for others who may want to write a DSL. Doing a DSL in Java is quite cumbersome. This is what the example looks like: S4 Fluent API @Override public void configure() { /* PE that prints counts to console. */ PEMaker printPE = addPE(PrintPE.class).asSingleton(); /* PEs that count events by user, gender, and age. */ PEMaker userCountPE = addPE(CounterPE.class); userCountPE.addTrigger().fireOn(Event.class).ifInterval(100, TimeUnit.MILLISECONDS); PEMaker genderCountPE = addPE(CounterPE.class); genderCountPE.addTrigger().fireOn(Event.class).ifInterval(100, TimeUnit.MILLISECONDS); PEMaker ageCountPE = addPE(CounterPE.class); ageCountPE.addTrigger().fireOn(Event.class).ifInterval(100, TimeUnit.MILLISECONDS); /* PE that generates random events. */ generateUserEventPE = addPE(GenerateUserEventPE.class).asSingleton(); generateUserEventPE.addTimer().withDuration(1, TimeUnit.MILLISECONDS); /* Create the application graph. */ ageCountPE.emit(CountEvent.class).onKey( new CountKeyFinder()).to(printPE); genderCountPE.emit(CountEvent.class).onKey( new CountKeyFinder()).to(printPE); userCountPE.emit(CountEvent.class).onKey( new CountKeyFinder()).to(printPE); generateUserEventPE.emit(UserEvent.class).onKey( new AgeKeyFinder()).to(ageCountPE); generateUserEventPE.emit(UserEvent.class).onKey( new GenderKeyFinder()).to(genderCountPE); generateUserEventPE.emit(UserEvent.class).onKey( new UserIDKeyFinder()).to(userCountPE); } I made a few additional changes: By convention stream fields in PEs must be arrays: Stream<EventType>[]. This convention helps standardize and build tools. Added helper method emit(<? extends Event> event, Stream<? extends Event>[] streams) in ProcessingElement to avoid having to use a for loop in each PE implementation. Simply use emit(someOutEvent, someStreamArray). Removed the SingletonPE class because it was complicating the API. Instead I added a method setSingleton() that creates an eagerly singleton in the app. That is, the PE instance is created and initialized just before the app starts and before events arrive. I improved initialization coordination. Started to use the Guava Precondition class because it makes the code cleaner. Please look at ProcessingElement and use a similar pattern. Fixed some bugs. TODO: Expressive language to specify key finder in the DSL. More extensive unit testing.
        Hide
        kishore gopalakrishna added a comment -

        How are you inferring the stream from PE. by using Event.class ? What if multiple streams use the same EventType. I liked the earlier one with stream, it was clearer.

        I am assuming you will be calling PEMaker.make() some where in the framework and validate the settings

        Show
        kishore gopalakrishna added a comment - How are you inferring the stream from PE. by using Event.class ? What if multiple streams use the same EventType. I liked the earlier one with stream, it was clearer. I am assuming you will be calling PEMaker.make() some where in the framework and validate the settings
        Hide
        Leo Neumeyer added a comment -

        It's a two step process.

        If there is no ambiguity (one stream field per event type) the assignment is done based on type.

        Otherwise, the user provides the PE class field name for the stream using withField() as follows:

        pey.emit(EventA.class).withField("stream3").onKey(new DurationKeyFinder()).to(pez);
        

        If the ambiguity is not resolved, the program exits with an error message.

        The problem is that using a stream entity to define the graph makes it hard to visualize the connections. pe2stream -> stream2pe as opposed to pe2pe. With lots of PEs removing the intermediate should make the code cleaner, I think.

        Anyway, let's play a bit with some variations and talk more on Tue.

        Show
        Leo Neumeyer added a comment - It's a two step process. If there is no ambiguity (one stream field per event type) the assignment is done based on type. Otherwise, the user provides the PE class field name for the stream using withField() as follows: pey.emit(EventA.class).withField( "stream3" ).onKey( new DurationKeyFinder()).to(pez); If the ambiguity is not resolved, the program exits with an error message. The problem is that using a stream entity to define the graph makes it hard to visualize the connections. pe2stream -> stream2pe as opposed to pe2pe. With lots of PEs removing the intermediate should make the code cleaner, I think. Anyway, let's play a bit with some variations and talk more on Tue.
        Hide
        Leo Neumeyer added a comment -

        Implemented GenericKeyFinder to specify event fields of primitive types as strings OR event attribute keys as strings. Supported in the std and fluent APIs. (Event attributes are key-value pairs with typed values set in the base Event class.) First it checks if there is a a field that matches the name and returns the value as a String. Otherwise checks if there is an attribute that matches the key name and returns the value as a String. Otherwise it fails at runtime. (It is still desirable to use the type safe KeyFinder for production work but the string descriptor is handy for rapid prototyping.)

        Example:

        Fluent API using KeyFinder class:

        generateUserEventPE.emit(UserEvent.class).onKey(new GenderKeyFinder()).to(ageCountPE);
        

        Fluent API using string descriptor:

        generateUserEventPE.emit(UserEvent.class).onKey("gender").to(genderCountPE);
        

        Standard API using KeyFinder:

        Stream<UserEvent> genderStream = createStream(UserEvent.class);
        genderStream.setName("Gender Stream");
        genderStream.setKey(new GenderKeyFinder());
        genderStream.setPE(genderCountPE);
        

        Standard API using string descriptor:

        Stream<UserEvent> genderStream = createStream(UserEvent.class);
        genderStream.setName("Gender Stream");
        genderStream.setKey("gender");
        genderStream.setPE(genderCountPE);
        
        Show
        Leo Neumeyer added a comment - Implemented GenericKeyFinder to specify event fields of primitive types as strings OR event attribute keys as strings. Supported in the std and fluent APIs. (Event attributes are key-value pairs with typed values set in the base Event class.) First it checks if there is a a field that matches the name and returns the value as a String. Otherwise checks if there is an attribute that matches the key name and returns the value as a String. Otherwise it fails at runtime. (It is still desirable to use the type safe KeyFinder for production work but the string descriptor is handy for rapid prototyping.) Example: Fluent API using KeyFinder class: generateUserEventPE.emit(UserEvent.class).onKey( new GenderKeyFinder()).to(ageCountPE); Fluent API using string descriptor: generateUserEventPE.emit(UserEvent.class).onKey( "gender" ).to(genderCountPE); Standard API using KeyFinder: Stream<UserEvent> genderStream = createStream(UserEvent.class); genderStream.setName( "Gender Stream" ); genderStream.setKey( new GenderKeyFinder()); genderStream.setPE(genderCountPE); Standard API using string descriptor: Stream<UserEvent> genderStream = createStream(UserEvent.class); genderStream.setName( "Gender Stream" ); genderStream.setKey( "gender" ); genderStream.setPE(genderCountPE);
        Hide
        Leo Neumeyer added a comment -

        After studying how to implement fluent grammars in Java, I came up with a formal definition. Please review.

        Grammar in EBNF format (railroad diagram attached generated with this great tool: http://railroad.my28msec.com/rr/ui )

        s4 ::= 'graph' 'info'? 'create'
        
        graph ::= 'pe' '('STRING')'  'type' '('CLASS_TYPE')' 
        
                       (FIRE_ON )?
        
                       (TIMER)?
        
                       (CACHE)?
        
                       'asSingleton'?
        
                       EMIT
        
        EMIT ::= 'emit' '(' ('event' '(' STRING ')' ('onField''('STRING')')? ('onKey''('(CLASS_TYPE|STRING)')')? 'to' '(' STRING ')' )+
        
        FIRE_ON ::= 'fireOn''('STRING')' ('ifInterval''('NUMBER ',' TIMEUNIT')' )?
        
        TIMER ::= 'timer' 'period''('NUMBER ',' TIMEUNIT')' 
        
        CACHE ::= 'size''('NUMBER')' ('expires''('NUMBER ',' TIMEUNIT')')?
        
        PROPERTY ::= ('p''('STRING',' VALUE')')+
        
        TIMEUNIT ::= 'TimeUnit.'('HOURS'|'MINUTES'|'SECONDS')
        
        STRING ::= '"'([A-Z][a-z][_])+'"'
        
        NUMBER ::= [0-9]+
        
        CLASS_TYPE ::= STRING'.class'
        
        VALUE ::= STRING | NUMBER
        

        Example:

        s4().graph( 
        
                     pe("PEZ").type(PEZ.class)
                        .fireOn("EVC").ifInterval(5, TimeUnit.SECONDS)
                        .cache().size(1000).expires(3, TimeUnit.HOURS)
                        .emit( event("EVB").to("PEX") ),
        
                     pe("PEY").type(PEY.class)
                        .property( p("duration", 4),
                                   p("height", 99)
                                 )
                        .timer().period(2, TimeUnit.MINUTES)
                        .emit( event("EVA").onField("stream3")
                                 .onKey(DurationKeyFinder.class).to("PEZ"),
        
                               event("EVA").onField("heightpez")
                                 .onKey(HeightKeyFinder.class).to("PEZ") ),
        
                     pe("PEX").type(PEX.class)
                        .property( p("query", "money") )
                        .asSingleton()
                        .cache().size(100).expires(1, TimeUnit.MINUTES)
                        .emit( event("EVB").onKey(QueryKeyFinder.class).to("PEY", "PEZ") )
                   ),
        
              .info(
                     organization(name("Algo Tech"), url("http://algotech.com")),
                     app(name("Meaning Extractor"), icon("http://algotech.com/meicon.png")),
                     author("Thomas Edison"),
                   )
        
        ).create();   
        
        Show
        Leo Neumeyer added a comment - After studying how to implement fluent grammars in Java, I came up with a formal definition. Please review. Grammar in EBNF format (railroad diagram attached generated with this great tool: http://railroad.my28msec.com/rr/ui ) s4 ::= 'graph' 'info'? 'create' graph ::= 'pe' '('STRING')' 'type' '('CLASS_TYPE')' (FIRE_ON )? (TIMER)? (CACHE)? 'asSingleton'? EMIT EMIT ::= 'emit' '(' ('event' '(' STRING ')' ('onField''('STRING')')? ('onKey''('(CLASS_TYPE|STRING)')')? 'to' '(' STRING ')' )+ FIRE_ON ::= 'fireOn''('STRING')' ('ifInterval''('NUMBER ',' TIMEUNIT')' )? TIMER ::= 'timer' 'period''('NUMBER ',' TIMEUNIT')' CACHE ::= 'size''('NUMBER')' ('expires''('NUMBER ',' TIMEUNIT')')? PROPERTY ::= ('p''('STRING',' VALUE')')+ TIMEUNIT ::= 'TimeUnit.'('HOURS'|'MINUTES'|'SECONDS') STRING ::= ' "'([A-Z][a-z][_])+'" ' NUMBER ::= [0-9]+ CLASS_TYPE ::= STRING'.class' VALUE ::= STRING | NUMBER Example: s4().graph( pe( "PEZ" ).type(PEZ.class) .fireOn( "EVC" ).ifInterval(5, TimeUnit.SECONDS) .cache().size(1000).expires(3, TimeUnit.HOURS) .emit( event( "EVB" ).to( "PEX" ) ), pe( "PEY" ).type(PEY.class) .property( p( "duration" , 4), p( "height" , 99) ) .timer().period(2, TimeUnit.MINUTES) .emit( event( "EVA" ).onField( "stream3" ) .onKey(DurationKeyFinder.class).to( "PEZ" ), event( "EVA" ).onField( "heightpez" ) .onKey(HeightKeyFinder.class).to( "PEZ" ) ), pe( "PEX" ).type(PEX.class) .property( p( "query" , "money" ) ) .asSingleton() .cache().size(100).expires(1, TimeUnit.MINUTES) .emit( event( "EVB" ).onKey(QueryKeyFinder.class).to( "PEY" , "PEZ" ) ) ), .info( organization(name( "Algo Tech" ), url( "http: //algotech.com" )), app(name( "Meaning Extractor" ), icon( "http: //algotech.com/meicon.png" )), author( "Thomas Edison" ), ) ).create();
        Hide
        Leo Neumeyer added a comment -

        I just committed the first version of the S4 Embedded Domain-Specific Language to the S4-5 branch. I ended up using Diezel to implement it. Eric, the author, was very responsive fixing the few bugs I found. The advantage of using Diezel is that it is almost impossible to code by hand and it makes it easy to modify and extend the grammar in the future. I created a Gradle build script that uses the Diezel Maven plugin to generate the EDSL Java source code. Here is the grammar I committed:

        S4 EDSL Grammar
        (pe , type , prop* , (fireOn , afterInterval? , afterNumEvents?)? , (timer, withPeriod)? , (cache, size , expires? )? , asSingleton? , (emit, onField?, (withKey|withKeyFinder)?, to+ )*  )+ , build
        

        and this is the counter app:

        Counter App using the S4 EDSL
         protected void onInit() {
        
                pe("Print").type(PrintPE.class).asSingleton().
        
                pe("User Count").type(CounterPE.class).fireOn(Event.class).afterInterval(100, TimeUnit.MILLISECONDS)
                        .emit(CountEvent.class).withKeyFinder(new CountKeyFinder()).to("Print").
        
                        pe("Gender Count").type(CounterPE.class).fireOn(Event.class).afterInterval(100, TimeUnit.MILLISECONDS)
                        .emit(CountEvent.class).withKeyFinder(new CountKeyFinder()).to("Print").
        
                        pe("Age Count").type(CounterPE.class).fireOn(Event.class).afterInterval(100, TimeUnit.MILLISECONDS)
                        .emit(CountEvent.class).withKeyFinder(new CountKeyFinder()).to("Print").
        
                        pe("Generate User Event").type(GenerateUserEventPE.class).timer().withPeriod(1, TimeUnit.MILLISECONDS)
                        .asSingleton().
        
                        emit(UserEvent.class).withKeyFinder(new UserIDKeyFinder()).to("User Count").
        
                        emit(UserEvent.class).withKey("gender").to("Gender Count").
        
                        emit(UserEvent.class).withKeyFinder(new AgeKeyFinder()).to("Age Count").
        
                        build();
            }
        

        We can iterate a bit on the names of the tokens.

        For more details look at:

        • The Diezel Project
        • New subproject s4-edsl
        • Test case in s4-edsl
        • The Counter example in the s4-example subproject package org.apache.s4.edsl.counter
        Show
        Leo Neumeyer added a comment - I just committed the first version of the S4 Embedded Domain-Specific Language to the S4-5 branch. I ended up using Diezel to implement it. Eric, the author, was very responsive fixing the few bugs I found. The advantage of using Diezel is that it is almost impossible to code by hand and it makes it easy to modify and extend the grammar in the future. I created a Gradle build script that uses the Diezel Maven plugin to generate the EDSL Java source code. Here is the grammar I committed: S4 EDSL Grammar (pe , type , prop* , (fireOn , afterInterval? , afterNumEvents?)? , (timer, withPeriod)? , (cache, size , expires? )? , asSingleton? , (emit, onField?, (withKey|withKeyFinder)?, to+ )* )+ , build and this is the counter app: Counter App using the S4 EDSL protected void onInit() { pe( "Print" ).type(PrintPE.class).asSingleton(). pe( "User Count" ).type(CounterPE.class).fireOn(Event.class).afterInterval(100, TimeUnit.MILLISECONDS) .emit(CountEvent.class).withKeyFinder( new CountKeyFinder()).to( "Print" ). pe( "Gender Count" ).type(CounterPE.class).fireOn(Event.class).afterInterval(100, TimeUnit.MILLISECONDS) .emit(CountEvent.class).withKeyFinder( new CountKeyFinder()).to( "Print" ). pe( "Age Count" ).type(CounterPE.class).fireOn(Event.class).afterInterval(100, TimeUnit.MILLISECONDS) .emit(CountEvent.class).withKeyFinder( new CountKeyFinder()).to( "Print" ). pe( "Generate User Event" ).type(GenerateUserEventPE.class).timer().withPeriod(1, TimeUnit.MILLISECONDS) .asSingleton(). emit(UserEvent.class).withKeyFinder( new UserIDKeyFinder()).to( "User Count" ). emit(UserEvent.class).withKey( "gender" ).to( "Gender Count" ). emit(UserEvent.class).withKeyFinder( new AgeKeyFinder()).to( "Age Count" ). build(); } We can iterate a bit on the names of the tokens. For more details look at: The Diezel Project New subproject s4-edsl Test case in s4-edsl The Counter example in the s4-example subproject package org.apache.s4.edsl.counter
        Hide
        Leo Neumeyer added a comment -

        Attached the EDSL state diagram generated by Diezel. Happy I didn't have to code that by hand

        Show
        Leo Neumeyer added a comment - Attached the EDSL state diagram generated by Diezel. Happy I didn't have to code that by hand
        Hide
        Matthieu Morel added a comment -

        I ported a simple example to this API and it seems to work nicely.

        Notes:

        • "onField" should rather be named "onStream" no?
        • for dispatching to several types of PEs, you currently need to write .to("1").to("2") right? would it be possible to write .to("1", "2") ?
        • the "stream" component has now disappeared from the API. Shouldn't it be included? It may be nicer to define streams as first class entities and include the key finder in their definition (instead of writing "emit.onField.withKeyFinder")
        • edsl depends on core, that makes it impossible to write app-level tests in core that would use the fluent API. Not sure it's an issue though.
        Show
        Matthieu Morel added a comment - I ported a simple example to this API and it seems to work nicely. Notes: "onField" should rather be named "onStream" no? for dispatching to several types of PEs, you currently need to write .to("1").to("2") right? would it be possible to write .to("1", "2") ? the "stream" component has now disappeared from the API. Shouldn't it be included? It may be nicer to define streams as first class entities and include the key finder in their definition (instead of writing "emit.onField.withKeyFinder") edsl depends on core, that makes it impossible to write app-level tests in core that would use the fluent API. Not sure it's an issue though.
        Hide
        Leo Neumeyer added a comment -

        Great feedback, thanks!

        • "onField" refers to the field name in the PE class that has a reference to a target Stream array. That's why I call it onField. "onStream" seems to imply the name of the stream which would be misleading. "onField" is not required when there is no ambiguity. That is: all stream fields in the PE are parametrized with different Event types. In that case the EDSL will figure out what field to use. This is the best idea I came up with, if anyone has an alternative, we can change. Perhaps we can call it "usePEField"?
        • I will look into using a variable number of args for to()
        • so you would use something like onStream(SomeStream.class, SomeKeyFinder.class).usePEField("someField") [I don't think I can overload the method so the optional "usePEField" method would have to be separate. What do people think?
        • edsl is a completely separate and optional project that depends on core but core doesn't depend on edsl. This pattern will help create alternate UIs. So I don't think we want UI code in core. The unit tests cases should be in the examples or in edsl projects.
        Show
        Leo Neumeyer added a comment - Great feedback, thanks! "onField" refers to the field name in the PE class that has a reference to a target Stream array. That's why I call it onField. "onStream" seems to imply the name of the stream which would be misleading. "onField" is not required when there is no ambiguity. That is: all stream fields in the PE are parametrized with different Event types. In that case the EDSL will figure out what field to use. This is the best idea I came up with, if anyone has an alternative, we can change. Perhaps we can call it "usePEField"? I will look into using a variable number of args for to() so you would use something like onStream(SomeStream.class, SomeKeyFinder.class).usePEField("someField") [I don't think I can overload the method so the optional "usePEField" method would have to be separate. What do people think? edsl is a completely separate and optional project that depends on core but core doesn't depend on edsl. This pattern will help create alternate UIs. So I don't think we want UI code in core. The unit tests cases should be in the examples or in edsl projects.
        Hide
        Leo Neumeyer added a comment -

        Pushed new version:

        • EDSL grammar change: Single required to() method with var args instead of one-or-more.
        • Upgraded Gradle wrapper to ver 1M7.

        NOTES:

        • Because we want to support to ways of specifying the key: KeyFinder and String in generic Event, we cannot use a single method name for both (I dont' think that is supported in Diezel.) In any case, the point of the fluent API is to have named parameters instead of a single method with many unnamed params so I am leaving as is for now.
        • I will add Javadoc entries to the Diezel XML so it gets generated.
        Show
        Leo Neumeyer added a comment - Pushed new version: EDSL grammar change: Single required to() method with var args instead of one-or-more. Upgraded Gradle wrapper to ver 1M7. NOTES: Because we want to support to ways of specifying the key: KeyFinder and String in generic Event, we cannot use a single method name for both (I dont' think that is supported in Diezel.) In any case, the point of the fluent API is to have named parameters instead of a single method with many unnamed params so I am leaving as is for now. I will add Javadoc entries to the Diezel XML so it gets generated.
        Hide
        Leo Neumeyer added a comment -

        Added Javadoc and made some changes. Merged with piper branch and pushed. Please verify that I didn't break anything. Use gradle ver 1M7 (or wrapper gradlew will download it for you.) Leaving this open to see if there any additional comments. Still to do to get better type checking: enforce Generics in Diezel.

        Show
        Leo Neumeyer added a comment - Added Javadoc and made some changes. Merged with piper branch and pushed. Please verify that I didn't break anything. Use gradle ver 1M7 (or wrapper gradlew will download it for you.) Leaving this open to see if there any additional comments. Still to do to get better type checking: enforce Generics in Diezel.
        Hide
        Matthieu Morel added a comment -

        I had a new look at the edsl framework and added, in branch S4-5-alt:

        • a simple word count test ported from core
        • re-enabled the CounterApp from example subproject

        Things work ok but I did the following modification:

        • AppBuilder (and affected PEs): changed the type of the stream field to Stream<Event> instead of Stream<Event>[] . Indeed, I don't really see the need for using arrays of streams. Is there a reason for that? Should we keep this modification?

        For info, here is the code for the word count example with EDSL:

        pe("Classifier").type(WordClassifierPE.class).pe("Counter").type(WordCounterPE.class)
                        .emit(WordCountEvent.class).withKeyFinder(WordCountKeyFinder.class).to("Classifier")
                        .pe("Splitter").type(WordSplitterPE.class).emit(WordSeenEvent.class)
                        .withKeyFinder(WordSeenKeyFinder.class).to("Counter").build();
                Stream<StringEvent> sentenceStream = createStream("sentences stream", new SentenceKeyFinder(),
                        getPE("Splitter"));
        

        (I'm not sure how to create the "sentences stream" as an entry point to the PE graph).

        Here is the same without EDSL:

        WordClassifierPE wordClassifierPrototype = createPE(WordClassifierPE.class);
                Stream<WordCountEvent> wordCountStream = createStream("words counts stream", new WordCountKeyFinder(),
                        wordClassifierPrototype);
                WordCounterPE wordCounterPrototype = createPE(WordCounterPE.class);
                wordCounterPrototype.setWordClassifierStream(wordCountStream);
                Stream<WordSeenEvent> wordSeenStream = createStream("words seen stream", new WordSeenKeyFinder(),
                        wordCounterPrototype);
                WordSplitterPE wordSplitterPrototype = createPE(WordSplitterPE.class);
                wordSplitterPrototype.setWordSeenStream(wordSeenStream);
                Stream<StringEvent> sentenceStream = createStream("sentences stream", new SentenceKeyFinder(),
                        wordSplitterPrototype);
        
        Show
        Matthieu Morel added a comment - I had a new look at the edsl framework and added, in branch S4-5 -alt: a simple word count test ported from core re-enabled the CounterApp from example subproject Things work ok but I did the following modification: AppBuilder (and affected PEs): changed the type of the stream field to Stream<Event> instead of Stream<Event>[] . Indeed, I don't really see the need for using arrays of streams. Is there a reason for that? Should we keep this modification? For info, here is the code for the word count example with EDSL: pe( "Classifier" ).type(WordClassifierPE.class).pe( "Counter" ).type(WordCounterPE.class) .emit(WordCountEvent.class).withKeyFinder(WordCountKeyFinder.class).to( "Classifier" ) .pe( "Splitter" ).type(WordSplitterPE.class).emit(WordSeenEvent.class) .withKeyFinder(WordSeenKeyFinder.class).to( "Counter" ).build(); Stream<StringEvent> sentenceStream = createStream( "sentences stream" , new SentenceKeyFinder(), getPE( "Splitter" )); (I'm not sure how to create the "sentences stream" as an entry point to the PE graph). Here is the same without EDSL: WordClassifierPE wordClassifierPrototype = createPE(WordClassifierPE.class); Stream<WordCountEvent> wordCountStream = createStream( "words counts stream" , new WordCountKeyFinder(), wordClassifierPrototype); WordCounterPE wordCounterPrototype = createPE(WordCounterPE.class); wordCounterPrototype.setWordClassifierStream(wordCountStream); Stream<WordSeenEvent> wordSeenStream = createStream( "words seen stream" , new WordSeenKeyFinder(), wordCounterPrototype); WordSplitterPE wordSplitterPrototype = createPE(WordSplitterPE.class); wordSplitterPrototype.setWordSeenStream(wordSeenStream); Stream<StringEvent> sentenceStream = createStream( "sentences stream" , new SentenceKeyFinder(), wordSplitterPrototype);
        Hide
        Matthieu Morel added a comment -

        Rescheduling to 0.6.

        We still need to iterate on this task in order to:

        • incorporate latest core API updates
        • improve the readability and configurability
        Show
        Matthieu Morel added a comment - Rescheduling to 0.6. We still need to iterate on this task in order to: incorporate latest core API updates improve the readability and configurability
        Hide
        Matthieu Morel added a comment -

        s4-edsl not included in distribution. We currently focus on providing the API in the App class, and this API is fairly simple to use.

        Show
        Matthieu Morel added a comment - s4-edsl not included in distribution. We currently focus on providing the API in the App class, and this API is fairly simple to use.

          People

          • Assignee:
            Unassigned
            Reporter:
            Leo Neumeyer
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development