Bug 52128 - Add JDBC pre- and post-processor
Add JDBC pre- and post-processor
Status: RESOLVED FIXED
Product: JMeter
Classification: Unclassified
Component: Main
Nightly (Please specify date)
All All
: P3 enhancement (vote)
: ---
Assigned To: JMeter issues mailing list
:
Depends on:
Blocks:
  Show dependency tree
 
Reported: 2011-11-03 14:21 UTC by Roman Bargezi
Modified: 2011-12-02 21:36 UTC (History)
1 user (show)



Attachments
Directional Patch proposal for Jdbc Pre And Post Processor against build 1196526 (63.76 KB, patch)
2011-11-03 14:25 UTC, Roman Bargezi
Details | Diff
Patch proposal for Jdbc Pre And Post Processor against build 1196526 (80.72 KB, patch)
2011-11-04 15:05 UTC, Roman Bargezi
Details | Diff
Patch proposal for Jdbc Pre And Post Processor against build 1196526 (80.57 KB, patch)
2011-11-04 15:14 UTC, Roman Bargezi
Details | Diff
Patch proposal for Jdbc Pre And Post Processor against build 1196526 (61.68 KB, patch)
2011-11-04 18:07 UTC, Roman Bargezi
Details | Diff
Example using JDBC pre and post processors (24.74 KB, application/xml)
2011-11-22 14:49 UTC, Roman Bargezi
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Roman Bargezi 2011-11-03 14:21:27 UTC
Add support for a JDBC PreProcessor and PostProcessor.

I often need to setup things before the actual measuring. Like creating a table (pre-process), then sample filling in some data and finally dropping the table (post-process). Or I need to clear the caches on the DB (pre-process/post-process).

I could do all of this using the JDBC Request sampler, but it polutes my listeners with results I don't want in the final output. Thus this request.
Comment 1 Roman Bargezi 2011-11-03 14:25:09 UTC
Created attachment 27889 [details]
Directional Patch proposal for Jdbc Pre And Post Processor against build 1196526

As this is a major refactoring, please let me know if this is going in the right direction for you to accept. I will then clean up, add docs, etc. and submit a final proposal.
Comment 2 Sebb 2011-11-03 14:52:33 UTC
Sounds like a job for the setUp and tearDown thread groups.

Have you tried those?
Comment 3 Roman Bargezi 2011-11-04 12:30:03 UTC
(In reply to comment #2)
> Sounds like a job for the setUp and tearDown thread groups.
> 
> Have you tried those?

I can't quite see how they can solve my case. Two things I would like to note here:
1. If I use the setup- and tear-down threads, I still get the results of the samples in those threads collected by my listeners. So even in this case I get garbage.

2. I have a testcase where one thread group runs more than one loop (in my case 3). Before each loop starts sampling, I have to reset a field in a DB table. After I have sampled my JDBC test requests, I need to cleanup the data that was created by the sample. All I want to collect is the timings of the JDMC test request. If I use the setup and tear-down threads, the call is only made once before the 3 loops are executed and after all 3 loops have ended. Thus having a JDBC pre- and post-processor would be nice. So far I have used the BSF porcessors with some groovy, but this is not convenient if you have pure SQL testers.
Comment 4 Sebb 2011-11-04 13:00:27 UTC
(In reply to comment #3)
> (In reply to comment #2)
> > Sounds like a job for the setUp and tearDown thread groups.
> > 
> > Have you tried those?
> 
> I can't quite see how they can solve my case. Two things I would like to note
> here:
> 1. If I use the setup- and tear-down threads, I still get the results of the
> samples in those threads collected by my listeners. So even in this case I get
> garbage.

That depends where you put the Listeners.

> 2. I have a testcase where one thread group runs more than one loop (in my case
> 3). Before each loop starts sampling, I have to reset a field in a DB table.
> After I have sampled my JDBC test requests, I need to cleanup the data that was
> created by the sample. All I want to collect is the timings of the JDMC test
> request. If I use the setup and tear-down threads, the call is only made once
> before the 3 loops are executed and after all 3 loops have ended. Thus having a
> JDBC pre- and post-processor would be nice. So far I have used the BSF
> porcessors with some groovy, but this is not convenient if you have pure SQL
> testers.

Again, you can do this by careful placement of the Listeners.

Also, Pre- and Post-Processors are intended for use before/after *each sample*; the use-case here seems rather different.

Although it is possible to adjust the positioning of the listeners, that can become tedious, so another possibility is to add an option to the JDBC Sampler so it returns a null sample (the Test Action Sampler does this). It should probably only nullify the result if successful.
Comment 5 Roman Bargezi 2011-11-04 13:52:35 UTC
(In reply to comment #4)
> (In reply to comment #3)
> > (In reply to comment #2)
> > > Sounds like a job for the setUp and tearDown thread groups.
> > > 
> > > Have you tried those?
> > 
> > I can't quite see how they can solve my case. Two things I would like to note
> > here:
> > 1. If I use the setup- and tear-down threads, I still get the results of the
> > samples in those threads collected by my listeners. So even in this case I get
> > garbage.
> 
> That depends where you put the Listeners.
> 
> > 2. I have a testcase where one thread group runs more than one loop (in my case
> > 3). Before each loop starts sampling, I have to reset a field in a DB table.
> > After I have sampled my JDBC test requests, I need to cleanup the data that was
> > created by the sample. All I want to collect is the timings of the JDMC test
> > request. If I use the setup and tear-down threads, the call is only made once
> > before the 3 loops are executed and after all 3 loops have ended. Thus having a
> > JDBC pre- and post-processor would be nice. So far I have used the BSF
> > porcessors with some groovy, but this is not convenient if you have pure SQL
> > testers.
> 
> Again, you can do this by careful placement of the Listeners.
> 
> Also, Pre- and Post-Processors are intended for use before/after *each sample*;
> the use-case here seems rather different.
> 
> Although it is possible to adjust the positioning of the listeners, that can
> become tedious, so another possibility is to add an option to the JDBC Sampler
> so it returns a null sample (the Test Action Sampler does this). It should
> probably only nullify the result if successful.

Maybe I was not clear enough with my usecase description. My usecase is exactly that I need to pre-/post-process an individual sample. Thus I have so far placed the BSF Pre/Post processors with some SQL code before and after my JDBC Sampler to achieve my goal. In other cases I even have more than one JDBC sample I take in a single thread and need to pre-process and post-process individual samples.

I did a quick test with returning a null sample result from the JDBC sample and it works. I just don't like the solution approach as it is more of a workaround and does not read properly (the user does not really see that things arepre/post-processed). I thus still think adding the JDBC pre/post-processors is the more elegant solution to the problem.
Comment 6 Sebb 2011-11-04 14:38:04 UTC
OK, I understand now - that makes more sense.

I notice that the patch includes the following import:

import ch.infonic.jmeter.jdbc.JDBCSamplerDiagnosticListener;

I assume this is not relevant to this issue?

Otherwise the patch looks reasonable.
Comment 7 Roman Bargezi 2011-11-04 15:05:11 UTC
Created attachment 27898 [details]
Patch proposal for Jdbc Pre And Post Processor against build 1196526

Added final patch proposal w/o the diagnostic listener mentioned by Sebb
Comment 8 Roman Bargezi 2011-11-04 15:06:41 UTC
Sebb, assuming you may now consider the enhancement I have added my final proposal. I am happy to update the Docs/Manual as well. As I don't quite know how, please let me know where I have to edit things and where you would want an upload.
Comment 9 Roman Bargezi 2011-11-04 15:14:49 UTC
Created attachment 27899 [details]
Patch proposal for Jdbc Pre And Post Processor against build 1196526
Comment 10 Roman Bargezi 2011-11-04 18:07:05 UTC
Created attachment 27901 [details]
Patch proposal for Jdbc Pre And Post Processor against build 1196526

Fixed some issues with resources.

SEBB YOU MAY WANT TO CHECK THIS:
- In trying not to duplicate resources, I added a base resource for the AbstractJDBCTestElement
- I then only added the 'displayName' to the additional resource files (JDBCPostProcessorResources.properties and JDBCPreProcessorResources.properties)
- The property labels worked fine but THE GROUP LABELS WERE NOT THERE!
- So I added the group labels to the Processor resources and all was good
- Then I did the same for JDBCSamplerResources.properties
- NOW THE PROPERTY LABELS WERE GONE FOR THIS BEAN!

This really drives me nuts! Maybe you have an answer to this inexplicable behavior. Or is it just a bug?
Comment 11 Sebb 2011-11-04 23:34:17 UTC
(In reply to comment #10)
> Created attachment 27901 [details]
> Patch proposal for Jdbc Pre And Post Processor against build 1196526
> 
> Fixed some issues with resources.
> 
> SEBB YOU MAY WANT TO CHECK THIS:
> - In trying not to duplicate resources, I added a base resource for the
> AbstractJDBCTestElement
> - I then only added the 'displayName' to the additional resource files
> (JDBCPostProcessorResources.properties and
> JDBCPreProcessorResources.properties)
> - The property labels worked fine but THE GROUP LABELS WERE NOT THERE!
> - So I added the group labels to the Processor resources and all was good
> - Then I did the same for JDBCSamplerResources.properties
> - NOW THE PROPERTY LABELS WERE GONE FOR THIS BEAN!
> 
> This really drives me nuts! Maybe you have an answer to this inexplicable
> behavior. Or is it just a bug?

I noticed some odd behaviour with the TestBean property handling some while ago. Never got to the bottom of it - not sure if it is a JMeter bug or a JavaBeans limitation. 

The only existing BeanInfoSupport class hierarchies are the BeanShell and BSF/JSR223 trees, which have a common parent class and don't add any customisation in the subclasses - maybe that is why they work.

Hopefully we can sort out the behaviour once the main code is incorporated.
Comment 12 Sebb 2011-11-05 00:43:21 UTC
Thanks, applied.

Inheritance does not seem to work for TestBean properties files, so I just copied the common fields to the Pre and Post processor properties, and dropped the common parent.

Seems to work.

URL: http://svn.apache.org/viewvc?rev=1197848&view=rev
Log:
Bug 52128 - Add JDBC pre- and post-processor

Added:
   jmeter/trunk/src/protocol/jdbc/org/apache/jmeter/protocol/jdbc/AbstractJDBCTestElement.java   (with props)
   jmeter/trunk/src/protocol/jdbc/org/apache/jmeter/protocol/jdbc/JDBCTestElementBeanInfoSupport.java   (with props)
   jmeter/trunk/src/protocol/jdbc/org/apache/jmeter/protocol/jdbc/processor/
   jmeter/trunk/src/protocol/jdbc/org/apache/jmeter/protocol/jdbc/processor/AbstractJDBCProcessor.java   (with props)
   jmeter/trunk/src/protocol/jdbc/org/apache/jmeter/protocol/jdbc/processor/JDBCPostProcessor.java   (with props)
   jmeter/trunk/src/protocol/jdbc/org/apache/jmeter/protocol/jdbc/processor/JDBCPostProcessorBeanInfo.java   (with props)
   jmeter/trunk/src/protocol/jdbc/org/apache/jmeter/protocol/jdbc/processor/JDBCPostProcessorResources.properties   (with props)
   jmeter/trunk/src/protocol/jdbc/org/apache/jmeter/protocol/jdbc/processor/JDBCPreProcessor.java   (with props)
   jmeter/trunk/src/protocol/jdbc/org/apache/jmeter/protocol/jdbc/processor/JDBCPreProcessorBeanInfo.java   (with props)
   jmeter/trunk/src/protocol/jdbc/org/apache/jmeter/protocol/jdbc/processor/JDBCPreProcessorResources.properties   (with props)
Modified:
   jmeter/trunk/src/protocol/jdbc/org/apache/jmeter/protocol/jdbc/sampler/JDBCSampler.java
   jmeter/trunk/src/protocol/jdbc/org/apache/jmeter/protocol/jdbc/sampler/JDBCSamplerBeanInfo.java
   jmeter/trunk/xdocs/changes.xml
Comment 13 Sebb 2011-11-05 00:51:02 UTC
(In reply to comment #8)
> Sebb, assuming you may now consider the enhancement I have added my final
> proposal. I am happy to update the Docs/Manual as well. As I don't quite know
> how, please let me know where I have to edit things and where you would want an
> upload.

The documentation is in the file

xdocs/usermanual/component_reference.xml

You just need to add the appropriate <component> paragraphs to the end of the relevant sections.

Patch can be provided as for code patches.

Screenshots can be generated using ^G if you want to do those; if not I can add them later. 
If you have some useful (publishable) examples of SQL code that can be used in the pre/post processors that would be very good - either provide screenshots, or a test plan with the fields filled in, and we can the do the screenshots.
Comment 14 Sebb 2011-11-18 18:15:27 UTC
Could you provide some examples for the documentation please?
Comment 15 Roman Bargezi 2011-11-22 14:49:17 UTC
Created attachment 27975 [details]
Example using JDBC pre and post processors

Sebb,

I thought hard about some fictitious sample, but could not come up with something that makes sense. So I have attached a sample from our real world. What it does is it measures the execution for parallel runs of a multi-portfolio valuation.

What needs to be done in preparation is:
- setting up the list of portfolios to value as an XML
- creating a container record (Price cut-off / PCO) for the results
- retrieving the ID of the container

Then we run the actual calculation and sample it

The post-processor then does the cleanup by deleting the results and the PCO

Hope this helps. I'll be off-line for the next 2 weeks, but please let me know if you need anything more. I'll follow up once I am online again.
Comment 16 Philippe Mouawad 2011-12-02 21:36:24 UTC
Date: Fri Dec  2 21:34:56 2011
New Revision: 1209701

URL: http://svn.apache.org/viewvc?rev=1209701&view=rev
Log:
Bug 52128 - Add JDBC pre- and post-processor

Modified:
   jmeter/trunk/xdocs/usermanual/component_reference.xml