OFBiz
  1. OFBiz
  2. OFBIZ-3558

Out of memory when importing large files /framework/datafile

    Details

      Description

      importing a tab delimited file 8megs(discontinued products) in size

      to me this is poor design since most file are even bigger when importing inventory from suppliers.

      javax.servlet.ServletException: Servlet execution threw an exception
      org.ofbiz.webapp.control.ContextFilter.doFilter(ContextFilter.java:259)

      root cause

      java.lang.OutOfMemoryError: Java heap space
      java.util.Arrays.copyOfRange(Unknown Source)
      java.lang.String.<init>(Unknown Source)
      java.lang.StringBuffer.toString(Unknown Source)
      java.io.StringWriter.toString(Unknown Source)
      org.apache.log4j.spi.LocationInfo.<init>(LocationInfo.java:114)
      org.apache.log4j.spi.LoggingEvent.getLocationInformation(LoggingEvent.java:247)
      org.apache.log4j.AsyncAppender.append(AsyncAppender.java:160)
      org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
      org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
      org.apache.log4j.Category.callAppenders(Category.java:206)
      org.apache.log4j.Category.forcedLog(Category.java:391)
      org.apache.log4j.Category.log(Category.java:856)
      org.ofbiz.base.util.Debug.log(Debug.java:166)
      org.ofbiz.base.util.Debug.log(Debug.java:149)
      org.ofbiz.base.util.Debug.logInfo(Debug.java:258)
      org.ofbiz.datafile.RecordIterator.getNextLine(RecordIterator.java:117)
      org.ofbiz.datafile.RecordIterator.next(RecordIterator.java:163)
      org.ofbiz.datafile.DataFile.readDataFile(DataFile.java:126)
      org.ofbiz.datafile.DataFile.readFile(DataFile.java:63)
      sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
      sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
      java.lang.reflect.Method.invoke(Unknown Source)
      org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:86)
      groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:230)
      groovy.lang.MetaClassImpl.invokeStaticMethod(MetaClassImpl.java:1105)
      org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:749)
      org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodN(ScriptBytecodeAdapter.java:170)
      viewdatafile.run(viewdatafile.groovy:65)
      org.ofbiz.base.util.GroovyUtil.runScriptAtLocation(GroovyUtil.java:117)
      org.ofbiz.widget.screen.ModelScreenAction$Script.runAction(ModelScreenAction.java:408)
      org.ofbiz.widget.screen.ModelScreenAction.runSubActions(ModelScreenAction.java:118)

      this is a typical record

      2010-03-15 16:56:15,421 (http-0.0.0.0-8443-1) [ RecordIterator.java:117:INFO ] br.readLine()="569 0.00 Britney 5473236 23320:302273:P "14K Yellow Gold Kansas City Chief's 
      Enamel Helment Pendant" "Reflecting the flawless elegance from a long-gone age of gentility, this luminescent pendant blends the sublime majestic aesthetics of the past with the 
      impeccable advanced metallurgical technology of the modern day. This 14k yellow gold pendant makes a great addition to any jewelry collection and can be worn on many 
      occasions. The total metal weight of this pendant is 2.30 grams. This pendant is part of our sports pendants collection of jewelry. This pendant has a length of 21 mm, a width of 
      21.25 mm." new Britney Britney 0 "2009-11-06 16:41:38" 5910653 23320:302273:P 23320:302273:P "14K Yellow Gold Kansas City Chief's Enamel Helment Pendant" 0 0 0.00 
      181.94 176.72 599.98 10 in-stock 0 "2009-11-06 15:02:07" "Catalog||Apparel, shoes & jewelry||Jewelry & watches||Pendants" http://images.doba.com/products
      /569/images_inventory_23320.jpg 250 250"
      

        Activity

        Hide
        Adam Heath added a comment -

        Are there lots of columns, and few lines, or lots of lines, and few columns? this problem might be related to the number of lines, as it is probably trying to do it all in a single transaction, and the parsed strings end up sharing the raw character array data in some places, but duplicating it in others.

        Show
        Adam Heath added a comment - Are there lots of columns, and few lines, or lots of lines, and few columns? this problem might be related to the number of lines, as it is probably trying to do it all in a single transaction, and the parsed strings end up sharing the raw character array data in some places, but duplicating it in others.
        Hide
        BJ Freeman added a comment -

        this is the datafile definitions.
        since the data is a large file I will not put in the Jira.
        I don't know if it is legal to put a url to the 8meg file.

        Show
        BJ Freeman added a comment - this is the datafile definitions. since the data is a large file I will not put in the Jira. I don't know if it is legal to put a url to the 8meg file.
        Hide
        BJ Freeman added a comment -

        The import does not go directly into the database.
        it creates xml files that are then imported via code or webtools.

        http://cwiki.apache.org/confluence/display/OFBENDUSER/OFBiz%27s+Data+File+Tools

        Show
        BJ Freeman added a comment - The import does not go directly into the database. it creates xml files that are then imported via code or webtools. http://cwiki.apache.org/confluence/display/OFBENDUSER/OFBiz%27s+Data+File+Tools
        Hide
        Jacques Le Roux added a comment -

        I don't think anything prevent to have an URL to a 8meg file.

        Show
        Jacques Le Roux added a comment - I don't think anything prevent to have an URL to a 8meg file.
        Show
        BJ Freeman added a comment - http://www.businessesnetwork.com/customerSupport/ofbiz/download/outofstock.txt is the datafile.
        Hide
        BJ Freeman added a comment -

        still researching.
        I have used creating temp table Idea in my Database applications.
        Currently I am looking at storing temporarily the list of records in persistence layer.
        i have seen code where an entities is created in java code.
        what I don't see is how to implement the entity(create table) in database on the fly.
        I am sure that can be done.
        next is to find a way to remove the entity, from the db at the end of the process.

        Show
        BJ Freeman added a comment - still researching. I have used creating temp table Idea in my Database applications. Currently I am looking at storing temporarily the list of records in persistence layer. i have seen code where an entities is created in java code. what I don't see is how to implement the entity(create table) in database on the fly. I am sure that can be done. next is to find a way to remove the entity, from the db at the end of the process.

          People

          • Assignee:
            Unassigned
            Reporter:
            BJ Freeman
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:

              Development