Texen
  1. Texen
  2. TEXEN-1

No upper limit on cached file handles causes random ResourceLoader exceptions

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.0
    • Fix Version/s: 1.1
    • Labels:
      None
    • Environment:
      Operating System: other
      Platform: Other

      Description

      The Generator class caches filehandles while generating files. There is no
      upper limit on the number of filehandles it will cache, so when generating many
      files with the same generator, it eventually hits the per-process limits on the
      number of open files. This then causes failures in the resource loader because
      it cannot open any more files. I've seen this problem on OS X and Linux, but it
      should be a pretty universal problem.

      You can work around this problem by increasing the number of filehandles
      available to the ant task, but that isn't always easy to do when running it as a
      subtask from an IDE, and it should be fairly simple to implement a basic LRU scheme.

      An alternative fix would be to more accurately report the error - the current
      implementation just throws a ResourceNotFound exception, making it very hard to
      track down the root cause of the error.

      1. Generator.java.patch
        4 kB
        Daniel Martin
      2. breaktexen.xml
        0.8 kB
        Daniel Martin

        Activity

        Hide
        Ian Ragsdale added a comment -

        Greetings,

        This email account is no longer active. For any Skylist or Datran issues, please contact Joel Lucas <jlucas@datranmedia.com>. To contact me, please use my new email address - <ian@bside.com>

        Thank you,
        -Ian

        Show
        Ian Ragsdale added a comment - Greetings, This email account is no longer active. For any Skylist or Datran issues, please contact Joel Lucas <jlucas@datranmedia.com>. To contact me, please use my new email address - <ian@bside.com> Thank you, -Ian
        Hide
        Ian Ragsdale added a comment -

        Greetings,

        This email account is no longer active. For any Skylist or Datran issues, please contact Joel Lucas <jlucas@datranmedia.com>. To contact me, please use my new email address - <ian@bside.com>

        Thank you,
        -Ian

        Show
        Ian Ragsdale added a comment - Greetings, This email account is no longer active. For any Skylist or Datran issues, please contact Joel Lucas <jlucas@datranmedia.com>. To contact me, please use my new email address - <ian@bside.com> Thank you, -Ian
        Hide
        Ian Ragsdale added a comment -

        Greetings,

        This email account is no longer active. For any Skylist or Datran issues, please contact Joel Lucas <jlucas@datranmedia.com>. To contact me, please use my new email address - <ian@bside.com>

        Thank you,
        -Ian

        Show
        Ian Ragsdale added a comment - Greetings, This email account is no longer active. For any Skylist or Datran issues, please contact Joel Lucas <jlucas@datranmedia.com>. To contact me, please use my new email address - <ian@bside.com> Thank you, -Ian
        Hide
        Ian Ragsdale added a comment -

        Greetings,

        This email account is no longer active. For any Skylist or Datran issues, please contact Joel Lucas <jlucas@datranmedia.com>. To contact me, please use my new email address - <ian@bside.com>

        Thank you,
        -Ian

        Show
        Ian Ragsdale added a comment - Greetings, This email account is no longer active. For any Skylist or Datran issues, please contact Joel Lucas <jlucas@datranmedia.com>. To contact me, please use my new email address - <ian@bside.com> Thank you, -Ian
        Hide
        Will Glass-Husain added a comment -

        resolved as noted before. (really, didn't need to reopen).

        Show
        Will Glass-Husain added a comment - resolved as noted before. (really, didn't need to reopen).
        Hide
        Will Glass-Husain added a comment -

        reopening so I can mark the fix version

        Show
        Will Glass-Husain added a comment - reopening so I can mark the fix version
        Hide
        Will Glass-Husain added a comment -

        Added patch. (though I changed the use of Hashtable to Set). Thanks so much!

        As an aside, In the future, try doing "svn diff" when contributing patches. It's easy to do and it puts it in standard unified format which is easy to patch.

        Show
        Will Glass-Husain added a comment - Added patch. (though I changed the use of Hashtable to Set). Thanks so much! As an aside, In the future, try doing "svn diff" when contributing patches. It's easy to do and it puts it in standard unified format which is easy to patch.
        Hide
        Daniel Martin added a comment -

        This patch to Generator.java, when applied, makes the test ant xml file succeed.

        The way it works is that it stores all the writers in a subclass of org.apache.commons.collections.map.LRUMap limited to 100 entries that closes files as they age out of the cache. At the same time, it does remember each pathname written to in case that pathname needs to be written to again - if a pathname is written to a second time, it opens the file in append mode.

        Note that other components of velocity already use org.apache.commons.collections.map.LRUMap so this shouldn't introduce any new dependencies.

        Show
        Daniel Martin added a comment - This patch to Generator.java, when applied, makes the test ant xml file succeed. The way it works is that it stores all the writers in a subclass of org.apache.commons.collections.map.LRUMap limited to 100 entries that closes files as they age out of the cache. At the same time, it does remember each pathname written to in case that pathname needs to be written to again - if a pathname is written to a second time, it opens the file in append mode. Note that other components of velocity already use org.apache.commons.collections.map.LRUMap so this shouldn't introduce any new dependencies.
        Hide
        Daniel Martin added a comment -

        The breaktexen.xml ant build script will, on Linux or Solaris, fail with this error message:
        [texen] SEVERE: ResourceManager : unable to find resource 'NumFacts.vm' in any resource loader.
        This despite the fact that the file exists at that point.

        The problem is with a template that holds onto too many open files.

        In regular production code where we hit this, we were generating ~ 1500 files through several nested foreach statements.

        Show
        Daniel Martin added a comment - The breaktexen.xml ant build script will, on Linux or Solaris, fail with this error message: [texen] SEVERE: ResourceManager : unable to find resource 'NumFacts.vm' in any resource loader. This despite the fact that the file exists at that point. The problem is with a template that holds onto too many open files. In regular production code where we hit this, we were generating ~ 1500 files through several nested foreach statements.
        Hide
        Will Glass-Husain added a comment -

        Responding to this old bug. If you or someone else wants to write a patch, I'll commit it. Even a self-contained test case would be helpful. Thanks.

        Show
        Will Glass-Husain added a comment - Responding to this old bug. If you or someone else wants to write a patch, I'll commit it. Even a self-contained test case would be helpful. Thanks.

          People

          • Assignee:
            Unassigned
            Reporter:
            Ian Ragsdale
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development