OFBiz
  1. OFBiz
  2. OFBIZ-3855

Service engine JobManager polling method has a transaction time-out when the number of jobs is large. Need a way to limit the number of jobs processed during each polling interval.

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: Trunk
    • Fix Version/s: Trunk
    • Component/s: ALL COMPONENTS
    • Labels:
      None

      Description

      Here is an example of when we see the transaction timeout for the polling operation in the JobManager.

      We are running into a problem with the service engine when trying to run an archiving batch service. Here is how the service runs.

      1. We run a service that collects a bunch of records for data warehousing.

      2. The data warehouse service calls an async service call to process each individual record.

      3. The service engine them starts working on each individual record.

      Here is the problem:

      There are over 10K records that are submitted as async service calls in step 2 above. I'm seeing a transaction time out when the service engine tries to process all of these records as it calls an "UPDATE JOB_SANDBOX for job_id=XXX". I believe it is trying to update all 10k of these jobs individually in a single transaction and times out. The jobs are then left in a pending status and never changed but the service engine keeps timing out.

      I was able to fix this problem by adding a new poll-transaction-time attribute to the "thread-pool" definition in the serviceengine.xml. I then looked up this value in the JobManager.poll method and set the transaction timeout at the beginning of the transaction.

      I'll submit a patch for this when I have made it for the latest revision in the trunk.

      Brett

        Activity

        Hide
        Adrian Crum added a comment -

        I think it will be helpful to have a job-queue-size attribute in the <thread-pool> element that will limit to queue size and the number of records processed at one time.

        Show
        Adrian Crum added a comment - I think it will be helpful to have a job-queue-size attribute in the <thread-pool> element that will limit to queue size and the number of records processed at one time.
        Hide
        Jacques Le Roux added a comment -

        Hi Brett,

        Please close if Adrian's recent implementation (r1370566) has correctly addressed the problem. Else I'd be interested to see your patch...

        Show
        Jacques Le Roux added a comment - Hi Brett, Please close if Adrian's recent implementation (r1370566) has correctly addressed the problem. Else I'd be interested to see your patch...
        Hide
        Jacques Le Roux added a comment -

        Of course I understand, it will be outdated...

        Show
        Jacques Le Roux added a comment - Of course I understand, it will be outdated...
        Hide
        Adrian Crum added a comment -

        The Job Scheduler has been overhauled. I believe the issues mentioned here are fixed as of rev 1371140. Thanks Brett for the report and additional info on the dev mailing list.

        Show
        Adrian Crum added a comment - The Job Scheduler has been overhauled. I believe the issues mentioned here are fixed as of rev 1371140. Thanks Brett for the report and additional info on the dev mailing list.

          People

          • Assignee:
            Adrian Crum
            Reporter:
            Brett G. Palmer
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development