Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 10.5.3.0
    • Fix Version/s: None
    • Component/s: Miscellaneous
    • Labels:
      None
    • Environment:
      dx 1.2 (from the Android SDK 2.0) running on Java 1.6

      Description

      derby.jar cannot be used on mobile devices running the Android OS. Android uses Dalvik as runtime environment, and the classfiles from derby.jar cannot be translated to Dalvik byte code.

      Steps to repeat:
      1. get the Android SDK from http://developer.android.com
      2. navigate to [android-sdk]/platforms/android-1.5/tools
      3. execute ./dx --dex path/to/derby.jar

      The dx tool should convert the JRE byte code to Dalvik byte code. Instead, it spits out this warning several time:
      warning: Ignoring InnerClasses attribute for an anonymous inner class that doesn't come with an associated EnclosingMethod attribute. (This class was probably produced by a broken compiler.)

      Then it runs out of memory:
      java.lang.OutOfMemoryError: Java heap space
      at com.android.dx.rop.code.ThrowingInsn.withNewRegisters(ThrowingInsn.java:116)
      at com.android.dx.ssa.NormalSsaInsn.toRopInsn(NormalSsaInsn.java:122)
      at com.android.dx.ssa.back.SsaToRop.convertInsns(SsaToRop.java:386)
      at com.android.dx.ssa.back.SsaToRop.convertBasicBlock(SsaToRop.java:365)
      at com.android.dx.ssa.back.SsaToRop.convertBasicBlocks(SsaToRop.java:300)
      at com.android.dx.ssa.back.SsaToRop.convertToRop(SsaToRop.java:277)
      at com.android.dx.ssa.back.SsaToRop.convert(SsaToRop.java:118)
      at com.android.dx.ssa.back.SsaToRop.convertToRopMethod(SsaToRop.java:71)
      at com.android.dx.ssa.Optimizer.optimize(Optimizer.java:103)
      at com.android.dx.ssa.Optimizer.optimize(Optimizer.java:74)
      at com.android.dx.dex.cf.CfTranslator.processMethods(CfTranslator.java:269)
      at com.android.dx.dex.cf.CfTranslator.translate0(CfTranslator.java:131)
      at com.android.dx.dex.cf.CfTranslator.translate(CfTranslator.java:85)
      at com.android.dx.command.dexer.Main.processClass(Main.java:297)
      at com.android.dx.command.dexer.Main.processFileBytes(Main.java:276)
      at com.android.dx.command.dexer.Main.access$100(Main.java:56)
      at com.android.dx.command.dexer.Main$1.processFileBytes(Main.java:228)
      at com.android.dx.cf.direct.ClassPathOpener.processArchive(ClassPathOpener.java:245)
      at com.android.dx.cf.direct.ClassPathOpener.processOne(ClassPathOpener.java:130)
      at com.android.dx.cf.direct.ClassPathOpener.process(ClassPathOpener.java:108)
      at com.android.dx.command.dexer.Main.processOne(Main.java:245)
      at com.android.dx.command.dexer.Main.processAllFiles(Main.java:183)
      at com.android.dx.command.dexer.Main.run(Main.java:139)
      at com.android.dx.command.dexer.Main.main(Main.java:120)
      at com.android.dx.command.Main.main(Main.java:87)

        Activity

        Hide
        Rick Hillegas added a comment -

        Note that as of this time, Android only supports Java 6 (not Java 7) language features: http://developer.android.com/sdk/index.html

        Show
        Rick Hillegas added a comment - Note that as of this time, Android only supports Java 6 (not Java 7) language features: http://developer.android.com/sdk/index.html
        Hide
        Kristian Waagan added a comment -

        As for generating .dex files there's now http://code.google.com/p/dexmaker/ . Other potential libs are smali and asmdex, but no idea how well suited they are.
        Anyone up for making the byte-code generation in Derby pluggable?

        On a similar note, but unrelated to Android, would it be better if we used ASM to generate the byte-code?

        Show
        Kristian Waagan added a comment - As for generating .dex files there's now http://code.google.com/p/dexmaker/ . Other potential libs are smali and asmdex, but no idea how well suited they are. Anyone up for making the byte-code generation in Derby pluggable? On a similar note, but unrelated to Android, would it be better if we used ASM to generate the byte-code?
        Hide
        Dag H. Wanvik added a comment -

        The -source 1.4 is there to make sure that developer's don't accidentally slip Java code in that requires a higher level than 1.4. I believe the code will compile just fine with a newer compiler modulo the JDBC interface: Java 6 comes with JDBC 4, so implementations of JDBC interfaces need special care, cf. the versioned source code in the Derby client, e.g. ClientPooledConnection40 which extends ClientPooledConnection etc. The former (*40.java) is already compiled with a Java 6 compiler. The -target 1.4 is to ensure the resulting byte code will run on Java 1.4. We did not want to release separate jars for each Java level.

        Show
        Dag H. Wanvik added a comment - The -source 1.4 is there to make sure that developer's don't accidentally slip Java code in that requires a higher level than 1.4. I believe the code will compile just fine with a newer compiler modulo the JDBC interface: Java 6 comes with JDBC 4, so implementations of JDBC interfaces need special care, cf. the versioned source code in the Derby client, e.g. ClientPooledConnection40 which extends ClientPooledConnection etc. The former (*40.java) is already compiled with a Java 6 compiler. The -target 1.4 is to ensure the resulting byte code will run on Java 1.4. We did not want to release separate jars for each Java level.
        Hide
        Chetan Narsude added a comment -

        Is that all that is there to it? I wonder then why the target is enforced in the build.xml files by mentioning target="1.4" to javac? Wouldn't it be sufficient to leave target out from the source distribution and release derby binaries with annotation of the target for which the binary is released?

        Seeing 1.4 target made me wonder that there is version specific code that would not compile/run without that version. I am trying to use the derbyclient.jar recompiled by removing "1.4" references from build.xml, and using 1.6 compiler on Android phone. Before I go too deep into it, I just want to make sure that I am not overlooking something which may haunt me later.

        – Thanks.

        Show
        Chetan Narsude added a comment - Is that all that is there to it? I wonder then why the target is enforced in the build.xml files by mentioning target="1.4" to javac? Wouldn't it be sufficient to leave target out from the source distribution and release derby binaries with annotation of the target for which the binary is released? Seeing 1.4 target made me wonder that there is version specific code that would not compile/run without that version. I am trying to use the derbyclient.jar recompiled by removing "1.4" references from build.xml, and using 1.6 compiler on Android phone. Before I go too deep into it, I just want to make sure that I am not overlooking something which may haunt me later. – Thanks.
        Hide
        Kristian Waagan added a comment -

        Not sure if I understand what you mean, but I think the only thing that would happen is that you cannot run the Derby client code with a VM at level 1.4 - assuming you want to produce Java 1.5 - 1.7 class files.
        Maybe you get a better answer if you explain which value you would use instead of 1.4, and which Dalvik/Android related issue you are trying to address.

        Thanks.

        Show
        Kristian Waagan added a comment - Not sure if I understand what you mean, but I think the only thing that would happen is that you cannot run the Derby client code with a VM at level 1.4 - assuming you want to produce Java 1.5 - 1.7 class files. Maybe you get a better answer if you explain which value you would use instead of 1.4, and which Dalvik/Android related issue you are trying to address. Thanks.
        Hide
        Chetan Narsude added a comment - - edited

        If I recompile the jars without using target="1.4", what kind of issues I would be facing?
        Update: Especially if I am going to use just the derbyclient.jar. – Thanks.

        Show
        Chetan Narsude added a comment - - edited If I recompile the jars without using target="1.4", what kind of issues I would be facing? Update: Especially if I am going to use just the derbyclient.jar. – Thanks.
        Hide
        Myrna van Lunteren added a comment -

        I am lowering the priority of this one.
        I understand this 'blocks' people from using derby in this combination, but 'blocker' would mean we cannot release a release without this being fixed, and I do not think this is the case.

        Show
        Myrna van Lunteren added a comment - I am lowering the priority of this one. I understand this 'blocks' people from using derby in this combination, but 'blocker' would mean we cannot release a release without this being fixed, and I do not think this is the case.
        Hide
        Bryan Pendleton added a comment -

        I tried doing something similar, but different, which was to use the derbyclient.jar library in an Android application.

        I made some progress, but am having trouble because Derby's ClientBaseDataSource class implements
        javax.naming.Referenceable from the JNDI API, and Android apparently does not support javax.naming.*

        The error I get is:

        I/dalvikvm( 348): Failed resolving Lorg/apache/derby/jdbc/ClientBaseDataSource; interface 212 'Ljavax/naming/Referenceable;'

        Sure enough, javax.naming is not in the list of supported packages at
        http://developer.android.com/reference/packages.html

        Show
        Bryan Pendleton added a comment - I tried doing something similar, but different, which was to use the derbyclient.jar library in an Android application. I made some progress, but am having trouble because Derby's ClientBaseDataSource class implements javax.naming.Referenceable from the JNDI API, and Android apparently does not support javax.naming.* The error I get is: I/dalvikvm( 348): Failed resolving Lorg/apache/derby/jdbc/ClientBaseDataSource; interface 212 'Ljavax/naming/Referenceable;' Sure enough, javax.naming is not in the list of supported packages at http://developer.android.com/reference/packages.html
        Hide
        Dan Bornstein added a comment -

        Hi. I'm the Dalvik engineer referred to above. In fact, I believe the bytecode spec changed between 1.4 and 1.5. dx's warning is basically saying that 1.4-based code that uses inner classes doesn't provide enough information for a 1.5 runtime to faithfully report about them via reflection. Rather than try to halfway reconstruct the reflection info in such cases, dx just gives up and ignores inner classes info entirely. If your code doesn't make reflection calls to look at inner class stuff, then none of this makes any difference.

        Also, if dx is running out of memory, you can tell it to have a larger maximum heap size by using a passthrough option for the underlying vm that runs dx, e.g. "dx -JXmx2048m ...".

        Also also, while the Dalvik vm doesn't read .class files directly, you can in fact synthesize .dex files and use them, in a similar manner. I have seen a proof of concept where someone generated .class files, ran them through dx (treated as a library), and then loaded the result. It works, but it is admittedly not pretty, especially since the dx code isn't particularly geared for use in a memory-constrained environment (see above). In the long run, you'd be better off generating .dex files more directly, but it might at least unblock you.

        Cheers!

        -dan

        Show
        Dan Bornstein added a comment - Hi. I'm the Dalvik engineer referred to above. In fact, I believe the bytecode spec changed between 1.4 and 1.5. dx's warning is basically saying that 1.4-based code that uses inner classes doesn't provide enough information for a 1.5 runtime to faithfully report about them via reflection. Rather than try to halfway reconstruct the reflection info in such cases, dx just gives up and ignores inner classes info entirely. If your code doesn't make reflection calls to look at inner class stuff, then none of this makes any difference. Also, if dx is running out of memory, you can tell it to have a larger maximum heap size by using a passthrough option for the underlying vm that runs dx, e.g. "dx -JXmx2048m ...". Also also, while the Dalvik vm doesn't read .class files directly, you can in fact synthesize .dex files and use them, in a similar manner. I have seen a proof of concept where someone generated .class files, ran them through dx (treated as a library), and then loaded the result. It works, but it is admittedly not pretty, especially since the dx code isn't particularly geared for use in a memory-constrained environment (see above). In the long run, you'd be better off generating .dex files more directly, but it might at least unblock you. Cheers! -dan
        Hide
        Francois Orsini added a comment -

        > Seems like if there is an improvement needed, it is in the android 'dx' tool, isn't it?

        You would think so but according to the Dalvik lead engineer, it is not:
        http://www.mail-archive.com/android-developers@googlegroups.com/msg29593.html

        This warning (not an error) is also reported by 'dx' translating some other Java libraries, not just derby.jar.
        Unless I'm mistaken, derby.jar GA is built using Java 1.4 which is obviously not a modern compiler? my guess is that this warning would disappear if building with Java 1.5 or 1.6.

        In any case, this is apparently NOT a blocking issue - Thomas was also reporting a Java OutOfMemory error which can be solved in Dalvik by increasing the heap size in dx.bat...

        Of course, even if the translation succeeds, Derby will not run, causing all kinds of errors due to its dynamic generated bytecode not being translated by the Dalvik VM at runtime, which is obviously a separate issue by itself.

        Show
        Francois Orsini added a comment - > Seems like if there is an improvement needed, it is in the android 'dx' tool, isn't it? You would think so but according to the Dalvik lead engineer, it is not: http://www.mail-archive.com/android-developers@googlegroups.com/msg29593.html This warning (not an error) is also reported by 'dx' translating some other Java libraries, not just derby.jar. Unless I'm mistaken, derby.jar GA is built using Java 1.4 which is obviously not a modern compiler? my guess is that this warning would disappear if building with Java 1.5 or 1.6. In any case, this is apparently NOT a blocking issue - Thomas was also reporting a Java OutOfMemory error which can be solved in Dalvik by increasing the heap size in dx.bat... Of course, even if the translation succeeds, Derby will not run, causing all kinds of errors due to its dynamic generated bytecode not being translated by the Dalvik VM at runtime, which is obviously a separate issue by itself.
        Hide
        Bryan Pendleton added a comment -

        What, precisely, is it that we should improve about Derby?

        Seems like if there is an improvement needed, it is in the android 'dx' tool, isn't it?

        Show
        Bryan Pendleton added a comment - What, precisely, is it that we should improve about Derby? Seems like if there is an improvement needed, it is in the android 'dx' tool, isn't it?
        Hide
        Dag H. Wanvik added a comment -

        Francois Orsini wrote on derby-dev Tue, 01 Dec 2009 09:49:00 -0800:

        >This is a separate issue but last I heard about Dalvik is that there is
        >still no support for dynamic bytecode translation at runtime - Derby
        >generates bytecode dynamically for every new SQL statement, so I doubt you
        >might be able to get Derby running on Android until dynamic bytecode
        >translation is available in the Dalvik VM at runtime

        That would be a show-stopper, yes.
        As for why the converter balks at Derby byte code, I have no idea. It is produced by the standard Sun Java compiler. Marking this issue as Improvement.

        Show
        Dag H. Wanvik added a comment - Francois Orsini wrote on derby-dev Tue, 01 Dec 2009 09:49:00 -0800: >This is a separate issue but last I heard about Dalvik is that there is >still no support for dynamic bytecode translation at runtime - Derby >generates bytecode dynamically for every new SQL statement, so I doubt you >might be able to get Derby running on Android until dynamic bytecode >translation is available in the Dalvik VM at runtime That would be a show-stopper, yes. As for why the converter balks at Derby byte code, I have no idea. It is produced by the standard Sun Java compiler. Marking this issue as Improvement.

          People

          • Assignee:
            Unassigned
            Reporter:
            Thomas Görres
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:

              Development