OFBiz
  1. OFBiz
  2. OFBIZ-1873

Sandbox: Ideas For Mashup Services

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Won't Fix
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: framework
    • Labels:
      None

      Description

      Discuss ideas for potential mashup services and worker classes.

      1. external_services.patch
        13 kB
        Adrian Crum
      2. external_services.patch
        11 kB
        Adrian Crum
      3. external_services.patch
        9 kB
        Adrian Crum

        Activity

        Hide
        Jacques Le Roux added a comment -

        I know what you speak about, I have a good pile too

        Show
        Jacques Le Roux added a comment - I know what you speak about, I have a good pile too
        Hide
        Adrian Crum added a comment -

        Jacques,

        It's on my (long) list of things to finish.

        Show
        Adrian Crum added a comment - Jacques, It's on my (long) list of things to finish.
        Hide
        Jacques Le Roux added a comment -

        Hi Adrian, BJ,

        Did you continue on this direction ? (I found the patch by chance in one of my directories and wondered what it was)

        Show
        Jacques Le Roux added a comment - Hi Adrian, BJ, Did you continue on this direction ? (I found the patch by chance in one of my directories and wondered what it was)
        Hide
        BJ Freeman added a comment -

        for future reference this is the one I used
        http://mozillaparser.sourceforge.net/
        had a mozilla license

        Show
        BJ Freeman added a comment - for future reference this is the one I used http://mozillaparser.sourceforge.net/ had a mozilla license
        Hide
        Jacques Le Roux added a comment -

        This would be great Adrian ! Already commited in too much things to promise any help

        Show
        Jacques Le Roux added a comment - This would be great Adrian ! Already commited in too much things to promise any help
        Hide
        Adrian Crum added a comment -

        BJ,

        That would be great if you could get something implemented using the htmlunit library.

        Right now the patch pulls a page from the Hotwax demo site and puts it in the Example component main page.

        I'd like to use it to link an OFBiz installation to the OFBiz Wiki. Help links in OFBiz could pull down wiki content and display it in the OFBiz screen. I hope to get an entire Help system implemented for Asset Maintenance using this method.

        Show
        Adrian Crum added a comment - BJ, That would be great if you could get something implemented using the htmlunit library. Right now the patch pulls a page from the Hotwax demo site and puts it in the Example component main page. I'd like to use it to link an OFBiz installation to the OFBiz Wiki. Help links in OFBiz could pull down wiki content and display it in the OFBiz screen. I hope to get an entire Help system implemented for Asset Maintenance using this method.
        Hide
        BJ Freeman added a comment -

        will follow work
        I did something in java for yahoo interface.
        it was so could access secure webpages that dealt with order status in the store.
        would like move it over to this, if possible.

        Show
        BJ Freeman added a comment - will follow work I did something in java for yahoo interface. it was so could access secure webpages that dealt with order status in the store. would like move it over to this, if possible.
        Hide
        Adrian Crum added a comment -

        Updated patch - now includes service to get an external website HTML element.

        Show
        Adrian Crum added a comment - Updated patch - now includes service to get an external website HTML element.
        Hide
        Jacques Le Roux added a comment -

        Ha yes : fine :o) Actually I think it was my error, I forgot to put some jar files

        Show
        Jacques Le Roux added a comment - Ha yes : fine :o) Actually I think it was my error, I forgot to put some jar files
        Hide
        Jacques Le Roux added a comment -

        Adrian,

        Ok thks

        Show
        Jacques Le Roux added a comment - Adrian, Ok thks
        Hide
        Adrian Crum added a comment -

        Jacques,

        It's probably due to a jar file version conflict. Sorry, I forgot about that.

        htmlunit comes with a newer version of a common*.jar file than the one in OFBiz. Find the older version jar file and delete it.

        Show
        Adrian Crum added a comment - Jacques, It's probably due to a jar file version conflict. Sorry, I forgot about that. htmlunit comes with a newer version of a common*.jar file than the one in OFBiz. Find the older version jar file and delete it.
        Hide
        Jacques Le Roux added a comment -

        BTW I checked, HWM site is not down.

        Show
        Jacques Le Roux added a comment - BTW I checked, HWM site is not down.
        Hide
        Jacques Le Roux added a comment -

        Hi Adrian,

        I have applied the patch using htmlunit-2.2.jar instead of htmlunit-2.1.jar (I downloaded the last version without care) and opening Example component I get

        2008-07-27 18:45:11,234 (http-0.0.0.0-28443-3) [ ServiceDispatcher.java:497:INFO ] [[Sync service failed...- total:0.0,since last(Begin):0.0]] - 'example / logIntoWebsite'
        2008-07-27 18:45:11,234 (http-0.0.0.0-28443-3) [ ServiceDispatcher.java:500:ERROR]
        ---- exception report ----------------------------------------------------------
        Service [logIntoWebsite] threw an unexpected exception/error
        Exception: org.ofbiz.service.GenericServiceException
        Message: Service [logIntoWebsite] target threw an unexpected exception (org/apache/commons/httpclient/auth/CredentialsProvider)
        ---- stack trace ---------------------------------------------------------------
        org.ofbiz.service.GenericServiceException: Service [logIntoWebsite] target threw an unexpected exception (org/apache/commons/httpclient/auth/CredentialsProvider)
        org.ofbiz.service.engine.StandardJavaEngine.serviceInvoker(StandardJavaEngine.java:108)
        org.ofbiz.service.engine.StandardJavaEngine.runSync(StandardJavaEngine.java:54)
        org.ofbiz.service.ServiceDispatcher.runSync(ServiceDispatcher.java:384)
        org.ofbiz.service.ServiceDispatcher.runSync(ServiceDispatcher.java:213)
        org.ofbiz.service.GenericDispatcher.runSync(GenericDispatcher.java:149)
        org.ofbiz.widget.screen.ModelScreenAction$Service.runAction(ModelScreenAction.java:473)
        org.ofbiz.widget.screen.ModelScreenAction.runSubActions(ModelScreenAction.java:122)
        org.ofbiz.widget.screen.ModelScreenWidget$Section.renderWidgetString(ModelScreenWidget.java:221)
        org.ofbiz.widget.screen.ModelScreen.renderScreenString(ModelScreen.java:393)
        org.ofbiz.widget.screen.ScreenRenderer.render(ScreenRenderer.java:129)
        org.ofbiz.widget.screen.ScreenRenderer.render(ScreenRenderer.java:92)
        org.ofbiz.widget.screen.ScreenWidgetViewHandler.render(ScreenWidgetViewHandler.java:96)
        org.ofbiz.webapp.control.RequestHandler.renderView(RequestHandler.java:651)
        org.ofbiz.webapp.control.RequestHandler.doRequest(RequestHandler.java:435)
        org.ofbiz.webapp.control.ControlServlet.doGet(ControlServlet.java:198)
        javax.servlet.http.HttpServlet.service(HttpServlet.java:690)
        javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
        org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
        org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        org.ofbiz.webapp.control.ContextFilter.doFilter(ContextFilter.java:258)
        org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
        org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
        org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
        org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
        org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:568)
        org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)
        org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
        org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
        org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
        java.lang.Thread.run(Thread.java:595)
        java.lang.NoClassDefFoundError: org/apache/commons/httpclient/auth/CredentialsProvider
        org.ofbiz.common.ExternalServices.getHtmlPage(ExternalServices.java:52)
        org.ofbiz.common.ExternalServices.logIntoWebsite(ExternalServices.java:118)
        sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        java.lang.reflect.Method.invoke(Method.java:585)
        org.ofbiz.service.engine.StandardJavaEngine.serviceInvoker(StandardJavaEngine.java:96)
        org.ofbiz.service.engine.StandardJavaEngine.runSync(StandardJavaEngine.java:54)
        org.ofbiz.service.ServiceDispatcher.runSync(ServiceDispatcher.java:384)
        org.ofbiz.service.ServiceDispatcher.runSync(ServiceDispatcher.java:213)
        org.ofbiz.service.GenericDispatcher.runSync(GenericDispatcher.java:149)
        org.ofbiz.widget.screen.ModelScreenAction$Service.runAction(ModelScreenAction.java:473)
        org.ofbiz.widget.screen.ModelScreenAction.runSubActions(ModelScreenAction.java:122)
        org.ofbiz.widget.screen.ModelScreenWidget$Section.renderWidgetString(ModelScreenWidget.java:221)
        org.ofbiz.widget.screen.ModelScreen.renderScreenString(ModelScreen.java:393)
        org.ofbiz.widget.screen.ScreenRenderer.render(ScreenRenderer.java:129)
        org.ofbiz.widget.screen.ScreenRenderer.render(ScreenRenderer.java:92)
        org.ofbiz.widget.screen.ScreenWidgetViewHandler.render(ScreenWidgetViewHandler.java:96)
        org.ofbiz.webapp.control.RequestHandler.renderView(RequestHandler.java:651)
        org.ofbiz.webapp.control.RequestHandler.doRequest(RequestHandler.java:435)
        org.ofbiz.webapp.control.ControlServlet.doGet(ControlServlet.java:198)
        javax.servlet.http.HttpServlet.service(HttpServlet.java:690)
        javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
        org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
        org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        org.ofbiz.webapp.control.ContextFilter.doFilter(ContextFilter.java:258)
        org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
        org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
        org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
        org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
        org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:568)
        org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)
        org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
        org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
        org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
        java.lang.Thread.run(Thread.java:595)

        Did not look further, any ideas of what I could have done wrong ?

        Thanks

        Show
        Jacques Le Roux added a comment - Hi Adrian, I have applied the patch using htmlunit-2.2.jar instead of htmlunit-2.1.jar (I downloaded the last version without care) and opening Example component I get 2008-07-27 18:45:11,234 (http-0.0.0.0-28443-3) [ ServiceDispatcher.java:497:INFO ] [ [Sync service failed...- total:0.0,since last(Begin):0.0] ] - 'example / logIntoWebsite' 2008-07-27 18:45:11,234 (http-0.0.0.0-28443-3) [ ServiceDispatcher.java:500:ERROR] ---- exception report ---------------------------------------------------------- Service [logIntoWebsite] threw an unexpected exception/error Exception: org.ofbiz.service.GenericServiceException Message: Service [logIntoWebsite] target threw an unexpected exception (org/apache/commons/httpclient/auth/CredentialsProvider) ---- stack trace --------------------------------------------------------------- org.ofbiz.service.GenericServiceException: Service [logIntoWebsite] target threw an unexpected exception (org/apache/commons/httpclient/auth/CredentialsProvider) org.ofbiz.service.engine.StandardJavaEngine.serviceInvoker(StandardJavaEngine.java:108) org.ofbiz.service.engine.StandardJavaEngine.runSync(StandardJavaEngine.java:54) org.ofbiz.service.ServiceDispatcher.runSync(ServiceDispatcher.java:384) org.ofbiz.service.ServiceDispatcher.runSync(ServiceDispatcher.java:213) org.ofbiz.service.GenericDispatcher.runSync(GenericDispatcher.java:149) org.ofbiz.widget.screen.ModelScreenAction$Service.runAction(ModelScreenAction.java:473) org.ofbiz.widget.screen.ModelScreenAction.runSubActions(ModelScreenAction.java:122) org.ofbiz.widget.screen.ModelScreenWidget$Section.renderWidgetString(ModelScreenWidget.java:221) org.ofbiz.widget.screen.ModelScreen.renderScreenString(ModelScreen.java:393) org.ofbiz.widget.screen.ScreenRenderer.render(ScreenRenderer.java:129) org.ofbiz.widget.screen.ScreenRenderer.render(ScreenRenderer.java:92) org.ofbiz.widget.screen.ScreenWidgetViewHandler.render(ScreenWidgetViewHandler.java:96) org.ofbiz.webapp.control.RequestHandler.renderView(RequestHandler.java:651) org.ofbiz.webapp.control.RequestHandler.doRequest(RequestHandler.java:435) org.ofbiz.webapp.control.ControlServlet.doGet(ControlServlet.java:198) javax.servlet.http.HttpServlet.service(HttpServlet.java:690) javax.servlet.http.HttpServlet.service(HttpServlet.java:803) org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) org.ofbiz.webapp.control.ContextFilter.doFilter(ContextFilter.java:258) org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175) org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128) org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:568) org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286) org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844) org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583) org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447) java.lang.Thread.run(Thread.java:595) java.lang.NoClassDefFoundError: org/apache/commons/httpclient/auth/CredentialsProvider org.ofbiz.common.ExternalServices.getHtmlPage(ExternalServices.java:52) org.ofbiz.common.ExternalServices.logIntoWebsite(ExternalServices.java:118) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) java.lang.reflect.Method.invoke(Method.java:585) org.ofbiz.service.engine.StandardJavaEngine.serviceInvoker(StandardJavaEngine.java:96) org.ofbiz.service.engine.StandardJavaEngine.runSync(StandardJavaEngine.java:54) org.ofbiz.service.ServiceDispatcher.runSync(ServiceDispatcher.java:384) org.ofbiz.service.ServiceDispatcher.runSync(ServiceDispatcher.java:213) org.ofbiz.service.GenericDispatcher.runSync(GenericDispatcher.java:149) org.ofbiz.widget.screen.ModelScreenAction$Service.runAction(ModelScreenAction.java:473) org.ofbiz.widget.screen.ModelScreenAction.runSubActions(ModelScreenAction.java:122) org.ofbiz.widget.screen.ModelScreenWidget$Section.renderWidgetString(ModelScreenWidget.java:221) org.ofbiz.widget.screen.ModelScreen.renderScreenString(ModelScreen.java:393) org.ofbiz.widget.screen.ScreenRenderer.render(ScreenRenderer.java:129) org.ofbiz.widget.screen.ScreenRenderer.render(ScreenRenderer.java:92) org.ofbiz.widget.screen.ScreenWidgetViewHandler.render(ScreenWidgetViewHandler.java:96) org.ofbiz.webapp.control.RequestHandler.renderView(RequestHandler.java:651) org.ofbiz.webapp.control.RequestHandler.doRequest(RequestHandler.java:435) org.ofbiz.webapp.control.ControlServlet.doGet(ControlServlet.java:198) javax.servlet.http.HttpServlet.service(HttpServlet.java:690) javax.servlet.http.HttpServlet.service(HttpServlet.java:803) org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) org.ofbiz.webapp.control.ContextFilter.doFilter(ContextFilter.java:258) org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175) org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128) org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:568) org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286) org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844) org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583) org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447) java.lang.Thread.run(Thread.java:595) Did not look further, any ideas of what I could have done wrong ? Thanks
        Hide
        Adrian Crum added a comment -

        Improved patch - now uses the htmlunit library. I was able to log into some public websites with this version. MySpace caused the htmlunit code to choke though.

        There is an example use in the Example component - the main page logs into the hotwax demo server.

        Anyone wanting to try this out will need to do some extra work. The htmlunit library must be downloaded and unzipped, then put the jar files in the framework/base/lib folder.

        The htmlunit library executes slowly. I'll contact them about that.

        Show
        Adrian Crum added a comment - Improved patch - now uses the htmlunit library. I was able to log into some public websites with this version. MySpace caused the htmlunit code to choke though. There is an example use in the Example component - the main page logs into the hotwax demo server. Anyone wanting to try this out will need to do some extra work. The htmlunit library must be downloaded and unzipped, then put the jar files in the framework/base/lib folder. The htmlunit library executes slowly. I'll contact them about that.
        Hide
        Adrian Crum added a comment -

        I did some more research and found a library that would be a good choice for an HTML scraper: HtmlUnit - http://htmlunit.sourceforge.net. It is intended to be used for unit testing of HTML pages - which makes it ideal for interacting with forms and extracting results. It handles javascript, redirects, cookies, and more.

        It uses the Apache 2.0 license so we can include it in the project.

        Show
        Adrian Crum added a comment - I did some more research and found a library that would be a good choice for an HTML scraper: HtmlUnit - http://htmlunit.sourceforge.net . It is intended to be used for unit testing of HTML pages - which makes it ideal for interacting with forms and extracting results. It handles javascript, redirects, cookies, and more. It uses the Apache 2.0 license so we can include it in the project.
        Hide
        Adrian Crum added a comment - - edited

        external_services.patch is a proof of concept implementation. It still needs more work before being committed.

        I created a basic service to retrieve external web pages and I also created a sample scraper - a form extraction service. It all seems to work well.

        I implemented these as services so they can be used inside screen widget <actions> elements and in minilang.

        The sendExternalRequest service takes the following arguments:

        extRequestUrl - the external URL to invoke
        extRequestType - get, post, or empty (for REST type URLs)
        extRequestParams - request parameters Map

        It returns extResponse - the web page (or XML document) as a String.

        The getFormParametersFromExtResponse service takes the following arguments:

        extResponse - The response String
        formId - The id attribute to search for
        formName - The form name to search for

        It returns:

        extResponse - The original response String
        formAction - The form's action attribute
        formMethod - The form's method attribute
        formFields - The form's input fields Map

        The idea being you could modify the contents of the formFields Map and then use the returned values in another call to sendExternalRequest.

        I picture the Mashup services working like a chain - you invoke the initial request, then take the response and pass it through various scrapers and/or XML element extractors.

        Comments are welcome.

        Show
        Adrian Crum added a comment - - edited external_services.patch is a proof of concept implementation. It still needs more work before being committed. I created a basic service to retrieve external web pages and I also created a sample scraper - a form extraction service. It all seems to work well. I implemented these as services so they can be used inside screen widget <actions> elements and in minilang. The sendExternalRequest service takes the following arguments: extRequestUrl - the external URL to invoke extRequestType - get, post, or empty (for REST type URLs) extRequestParams - request parameters Map It returns extResponse - the web page (or XML document) as a String. The getFormParametersFromExtResponse service takes the following arguments: extResponse - The response String formId - The id attribute to search for formName - The form name to search for It returns: extResponse - The original response String formAction - The form's action attribute formMethod - The form's method attribute formFields - The form's input fields Map The idea being you could modify the contents of the formFields Map and then use the returned values in another call to sendExternalRequest. I picture the Mashup services working like a chain - you invoke the initial request, then take the response and pass it through various scrapers and/or XML element extractors. Comments are welcome.

          People

          • Assignee:
            Unassigned
            Reporter:
            Adrian Crum
          • Votes:
            1 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development