Apache Jena
  1. Apache Jena
  2. JENA-217

tdbloader : Unsafe memory access operation

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Cannot Reproduce
    • Affects Version/s: Fuseki 0.2.1
    • Fix Version/s: None
    • Component/s: TDB
    • Labels:
      None
    • Environment:

      Ubuntu 10.04

      Description

      First of all, I'm not sure if this is necessarily a bug or whether my system encountered some other issue. Here it is:

      I'm trying to import world-development-indicators.nt which is 68 GB into a TDB store.

      I've performed the following operation:

      java tdb.tdbloader --loc /usr/lib/fuseki/DB/DB.worldbank2 world-development-indicators --graph=http://worldbank.270a.info/graph/world-bank-indicators /data/worldbank.270a.info/indicators/en/indicator/world-bank-indicators.nt

      ..
      360,393,432 triples loaded in 15,956.80 seconds [Rate: 22,585.57 per second]
      ..

        • Index GSPO->GPOS: 360,202,756 slots indexed in 18,194.61 seconds [Rate: 19,797.23 per second]
          ..
        • Index GSPO->GOSP: 360,202,756 slots indexed in 4,410.64 seconds [Rate: 81,666.85 per second]
          ..
        • Index GSPO->POSG: 360,202,756 slots indexed in 4,354.58 seconds [Rate: 82,718.05 per second]
          ..
        • Index GSPO->OSPG: 360,202,756 slots indexed in 4,084.50 seconds [Rate: 88,187.73 per second]
          ..
          Index GSPO->SPOG: 113,900,000 slots (Batch: 213,675 slots/s / Avg: 113,695 slots/s)
          Exception in thread "main" java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled Java code
          at com.hp.hpl.jena.tdb.lib.TupleLib.tuple(TupleLib.java:208)
          at com.hp.hpl.jena.tdb.index.TupleIndexRecord$1.convert(TupleIndexRecord.java:195)
          at com.hp.hpl.jena.tdb.index.TupleIndexRecord$1.convert(TupleIndexRecord.java:191)
          at org.openjena.atlas.iterator.Iter$4.next(Iter.java:293)
          at com.hp.hpl.jena.tdb.store.bulkloader.LoaderNodeTupleTable.copyIndex(LoaderNodeTupleTable.java:209)
          at com.hp.hpl.jena.tdb.store.bulkloader.BuilderSecondaryIndexesSequential.createSecondaryIndexes(BuilderSecondaryIndexesSequential.java:43)
          at com.hp.hpl.jena.tdb.store.bulkloader.LoaderNodeTupleTable.createSecondaryIndexes(LoaderNodeTupleTable.java:192)
          at com.hp.hpl.jena.tdb.store.bulkloader.LoaderNodeTupleTable.loadSecondaryIndexes(LoaderNodeTupleTable.java:91)
          at com.hp.hpl.jena.tdb.store.bulkloader.LoaderNodeTupleTable.loadIndexStart(LoaderNodeTupleTable.java:144)
          at com.hp.hpl.jena.tdb.store.bulkloader.BulkLoader$1.finish(BulkLoader.java:232)
          at com.hp.hpl.jena.tdb.store.bulkloader.BulkLoader.loadTriples$(BulkLoader.java:141)
          at com.hp.hpl.jena.tdb.store.bulkloader.BulkLoader.loadNamedGraph(BulkLoader.java:107)
          at com.hp.hpl.jena.tdb.TDBLoader.loadNamedGraph$(TDBLoader.java:271)
          at com.hp.hpl.jena.tdb.TDBLoader.loadGraph$(TDBLoader.java:246)
          at com.hp.hpl.jena.tdb.TDBLoader.loadGraph(TDBLoader.java:177)
          at com.hp.hpl.jena.tdb.TDBLoader.load(TDBLoader.java:112)
          at tdb.tdbloader.loadNamedGraph(tdbloader.java:157)
          at tdb.tdbloader.exec(tdbloader.java:142)
          at arq.cmdline.CmdMain.mainMethod(CmdMain.java:97)
          at arq.cmdline.CmdMain.mainRun(CmdMain.java:59)
          at arq.cmdline.CmdMain.mainRun(CmdMain.java:46)
          at tdb.tdbloader.main(tdbloader.java:53)

        Activity

        Hide
        Andy Seaborne added a comment -

        Exception in thread "main" java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled Java code

        java.lang.InternalError is a JVM-internal error. This looks rather like a problem with your system, either by bad configuration or by system bug. It should not be possible to cause from java if things are working correctly. The only system configuration factor I can think of is the maximum process size.

        Maybe tdbloader2 will work - it works very differently.

        I wish I could be more helpful - if the data is available then I can download and try it on my machine but it's not trivially small and the error is something likely to happen only at scale.

        A hacked up version of tdbloader could start from the end of the data phase (the problem is in index generation).

        What is the system setup?
        Size of RAM?
        java version?

        Stackoverflow says:
        http://stackoverflow.com/questions/2949371/java-map-nio-nfs-issue-causing-a-vm-fault-a-fault-occurred-in-a-recent-uns

        I am seeing this occur with memory mapped files on local ext4 and tmpfs file systems with Java 7u1. – Peter Lawrey Dec 15 '11 at 14:01
        ... it happens at the point I run out of physical memory (viewed in top), even though I have 10 GB of swap free. – Peter Lawrey Dec 15 '11 at 14:15

        Also:
        http://tech.groups.yahoo.com/group/jena-dev/message/47100

        Show
        Andy Seaborne added a comment - Exception in thread "main" java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled Java code java.lang.InternalError is a JVM-internal error. This looks rather like a problem with your system, either by bad configuration or by system bug. It should not be possible to cause from java if things are working correctly. The only system configuration factor I can think of is the maximum process size. Maybe tdbloader2 will work - it works very differently. I wish I could be more helpful - if the data is available then I can download and try it on my machine but it's not trivially small and the error is something likely to happen only at scale. A hacked up version of tdbloader could start from the end of the data phase (the problem is in index generation). What is the system setup? Size of RAM? java version? Stackoverflow says: http://stackoverflow.com/questions/2949371/java-map-nio-nfs-issue-causing-a-vm-fault-a-fault-occurred-in-a-recent-uns I am seeing this occur with memory mapped files on local ext4 and tmpfs file systems with Java 7u1. – Peter Lawrey Dec 15 '11 at 14:01 ... it happens at the point I run out of physical memory (viewed in top), even though I have 10 GB of swap free. – Peter Lawrey Dec 15 '11 at 14:15 Also: http://tech.groups.yahoo.com/group/jena-dev/message/47100
        Hide
        Sarven Capadisli added a comment -

        Thanks Andy.

        Just to confirm that the filesystem is ext4.

        16 GB of RAM, however I'd say that the available free is from 4-6GB. The swap is 16GB.

        $ java -version
        java version "1.6.0_20"
        OpenJDK Runtime Environment (IcedTea6 1.9.13) (6b20-1.9.13-0ubuntu1~10.04.1)
        OpenJDK 64-Bit Server VM (build 19.0-b09, mixed mode)

        I'll give tdbloader one more go because this time around it was going at a pretty decent speed (in comparison to my previous imports - I don't have a reason for this either). It took about 13 hours before breaking. I think there was another hour to go before finishing up.

        The only issue I have with switching over to tdbloader2 or 3 is that I have to add the context URI to my N-Triples for N-Quads.

        Show
        Sarven Capadisli added a comment - Thanks Andy. Just to confirm that the filesystem is ext4. 16 GB of RAM, however I'd say that the available free is from 4-6GB. The swap is 16GB. $ java -version java version "1.6.0_20" OpenJDK Runtime Environment (IcedTea6 1.9.13) (6b20-1.9.13-0ubuntu1~10.04.1) OpenJDK 64-Bit Server VM (build 19.0-b09, mixed mode) I'll give tdbloader one more go because this time around it was going at a pretty decent speed (in comparison to my previous imports - I don't have a reason for this either). It took about 13 hours before breaking. I think there was another hour to go before finishing up. The only issue I have with switching over to tdbloader2 or 3 is that I have to add the context URI to my N-Triples for N-Quads.
        Hide
        Sarven Capadisli added a comment -

        Second try:

        360,393,432 triples loaded in 18,865.78 seconds [Rate: 19,103.02 per second]

          • Index GSPO->GPOS: 360,202,756 slots indexed in 13,691.17 seconds [Rate: 26,309.13 per second]
          • Index GSPO->GOSP: 360,202,756 slots indexed in 3,234.70 seconds [Rate: 111,355.88 per second]
          • Index GSPO->POSG: 360,202,756 slots indexed in 3,379.80 seconds [Rate: 106,575.05 per second]
          • Index GSPO->OSPG: 360,202,756 slots indexed in 3,134.57 seconds [Rate: 114,912.98 per second]
          • Index GSPO->SPOG: 360,202,756 slots indexed in 1,530.70 seconds [Rate: 235,319.28 per second]
        Show
        Sarven Capadisli added a comment - Second try: 360,393,432 triples loaded in 18,865.78 seconds [Rate: 19,103.02 per second] Index GSPO->GPOS: 360,202,756 slots indexed in 13,691.17 seconds [Rate: 26,309.13 per second] Index GSPO->GOSP: 360,202,756 slots indexed in 3,234.70 seconds [Rate: 111,355.88 per second] Index GSPO->POSG: 360,202,756 slots indexed in 3,379.80 seconds [Rate: 106,575.05 per second] Index GSPO->OSPG: 360,202,756 slots indexed in 3,134.57 seconds [Rate: 114,912.98 per second] Index GSPO->SPOG: 360,202,756 slots indexed in 1,530.70 seconds [Rate: 235,319.28 per second]
        Hide
        Andy Seaborne added a comment -

        This seems to be a JVM, OS or hardware problem.

        Show
        Andy Seaborne added a comment - This seems to be a JVM, OS or hardware problem.

          People

          • Assignee:
            Andy Seaborne
            Reporter:
            Sarven Capadisli
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development