NullPointerException in newBrokeredStatement in app server environment

• Key: DERBY-3513
• Type: Bug
• Status: Closed
• Resolution: Cannot Reproduce
• Priority: Major
• Assignee: Kathey Marsden
• Reporter: Kathey Marsden
• Component/s: JDBC
• Affects Version/s: 10.1.3.2

Environment

Java Version: 1.5.0
Java Vendor: IBM Corporation
OS name: Linux
OS architecture: x86
OS version: 2.6.9-55.ELsmp

Description

User reports in an app server environment an intermittent NullPointerException
with the 10.1 trace:

 R java.lang.NullPointerException
org.apache.derby.iapi.jdbc.BrokeredConnection.newBrokeredStatement(BrokeredConnection.java:448)
   
...org.apache.derby.jdbc.XAStatementControl.<init>(XAStatementControl.java:62)
...org.apache.derby.jdbc.EmbedXAConnection.wrapStatement(EmbedXAConnection.java:827)
org.apache.derby.iapi.jdbc.BrokeredConnection.createStatement(BrokeredConnection.java:296)
 [snip user trace]

The code at line 448 is simply:
return new BrokeredStatement(statementControl, getJDBCLevel());

so not much room for an NPE there. I added println statements to identify the state values and where the NPE is actually occurring but that seemed to make the
problem go away. It may be a JIT issue.

I gave them the fix for DERBY-2142 and that did not correct the problem.

Comments

[Kathey Marsden]

A gave the user a patched derby.jar that just changed the newBrokeredStatement code to
hold the jdbcLevel in a local variable.
With the patch they were not able to reproduce
the error so this small change may serve as a workaround to a probable JIT issue.

Unfortunately because of their complex environment, it is difficult to perform JIT diagnostics and get this resolved from the JVM side, so the
user has requested that we check in the change,
so that they can move forward. I wonder if there are
any objections to checking in this change. I'll plan to check in Thursday if I don't hear back.

The NullPointer failure happened with 30-40% of the builds in a complex build then test environment for an Application Server based system. It could not be reproduced outside of the build-test environment shown below :

org.apache.derby.tools.sysinfo
------------------ Java Information ------------------
Java Version: 1.5.0
Java Vendor: IBM Corporation
Java home: /opt/MyServer/java/jre
Java classpath: /opt/MyServer/derby/lib/derby.jar
OS name: Linux
OS architecture: x86
OS version: 2.6.9-55.ELsmp
Java user name: builduser
Java user home: /home/wsbuild
Java user dir: /opt/MyServer/derby
java.specification.name: Java Platform API Specification
java.specification.version: 1.5
--------- Derby Information --------
JRE - JDBC: J2SE 5.0 - JDBC 3.0

[Kathey Marsden]

Here is the full java version
java version "1.5.0"
Java(TM) 2 Runtime Environment, Standard Edition (build pxi32devifx-20070608 (SR5+IY99712))
IBM J9 VM (build 2.3, J2RE 1.5.0 IBM J9 2.3 Linux x86-32 j9vmxi3223-20070426 (JIT enabled)
J9VM - 20070420_12448_lHdSMR
JIT - 20070419_1806_r8
GC - 200704_19)
JCL - 20070608

[Knut Anders Hatlen]

Wouldn't it be better to remove the jdbcLevel field from BrokeredStatement completely? Then we don't need the call to getJDBCLevel() in those methods at all. The only use of the jdbcLevel field I could find, was this statement in BrokeredStatement.createDuplicateStatement():

if (jdbcLevel == 2)
newStatement = conn.createStatement(resultSetType, resultSetConcurrency);
else
newStatement = conn.createStatement(resultSetType, resultSetConcurrency,
                    resultSetHoldability);

Since BrokeredConnection.getJDBCLevel() never returns 2, it is not needed any more.

[Knut Anders Hatlen]

Ops, sorry, I didn't notice that the affects version said 10.1.3.2. On 10.1 and 10.2, getJDBCLevel() returns 2 if we use Java 1.3, so removing the code wouldn't work. On trunk, I'd prefer that we removed the code. Will open a new issue so that we can fix that later. For now, I think it's OK to check in derby-3513_diff.txt.

[Knut Anders Hatlen]

Created DERBY-3525 to track the clean-up.

[Kathey Marsden]

Thanks Knut, the cleanup (DERBY-3525) sounds like a good solution for 10.3+

[Kathey Marsden]

After running with the test fix for an extendend period of time, the user saw the NullPointerException with the proposed patch, so I won't be committing the change to 10.1.

[Stan Bradbury]

Kathey - Please DO commit this change to the 10.1 codeline afterall. The customer experiencing the failure has continued to test with your patch (in 11 builds to date) and has only observed the NPE failure once (on 3/12/08). The failure rate prior to this change was 35% (a failure approximately every 3 builds). Your patch, if not correcting the problem, has at least greatly reduced the frequency with which it happens. I will post to this issue should another build encounter the NPE problem. If there are no further occurrences I think it is safe to say that the failure reported on 3/12 was an anomaly.

[Kathey Marsden]

OK then, I will go ahead and commit it, but I fear it will only make the problem more elusive and hard to reproduce. When they get to the point that they are able to run JIT diagnostics it might make sense to test with the earlier build where it is easier to reproduce.

[Kathey Marsden]

Closing this issue CNR. I haven't heard back from the user with information on how to reproduce this. If we ever hear back we can reopen.