Derby
  1. Derby
  2. DERBY-3741

SQL LENGTH function materializes CLOB into memory

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 10.3.3.0, 10.4.1.3, 10.5.1.1
    • Fix Version/s: 10.3.3.1, 10.4.2.0, 10.5.1.1
    • Component/s: SQL
    • Labels:
      None
    • Issue & fix info:
      High Value Fix

      Description

      Similar to DERBY-3732, the SQL LENGTH function also materializes CLOB's into memory. See attached repro.

      1. LargeLengthClob.zip
        1 kB
        Kathey Marsden
      2. ClobMemTest.java
        8 kB
        Suran Jayathilaka
      3. derby-3741-1.diff
        11 kB
        Suran Jayathilaka
      4. derby-3741-2.diff
        11 kB
        Suran Jayathilaka
      5. derby-3741-multibyte_test.diff
        7 kB
        Suran Jayathilaka
      6. derby-3741_multibyteclobtest_update_diff.txt
        8 kB
        Kathey Marsden
      7. derby-3741-mergeto-10.4.diff
        142 kB
        Suran Jayathilaka
      8. derby.log
        179 kB
        Suran Jayathilaka
      9. derby-3741-mergeto-10.4-new.diff
        142 kB
        Suran Jayathilaka

        Issue Links

          Activity

          Hide
          Kathey Marsden added a comment -

          I checked this into 10.4 with revision 685028. Resolving it for now. I might backport it to 10.3 later but this will get it in the 10.4 release notes.

          Show
          Kathey Marsden added a comment - I checked this into 10.4 with revision 685028. Resolving it for now. I might backport it to 10.3 later but this will get it in the 10.4 release notes.
          Hide
          Suran Jayathilaka added a comment -

          Resubmitting 10.4 merge patch with references to setAutoCommit() replaced with the old usage of getConnection().setAutoCommit().

          Show
          Suran Jayathilaka added a comment - Resubmitting 10.4 merge patch with references to setAutoCommit() replaced with the old usage of getConnection().setAutoCommit().
          Hide
          Suran Jayathilaka added a comment -

          Attaching derby.log file from the failed tests. This file is from the
          derby-10.4\tests\fail\DerbyNetClient\AuthenticationTest\testChangePasswordAndDatabasePropertiesOnly

          folder.
          Before running the test, I had cleaned the tests folder.

          I used the following commands to run the tests.

          set CLASSPATH=..\classes;..\tools\java\junit.jar;..\tools\java\jakarta-oro-2.0.8.jar
          java -XX:MaxPermSize=128M -Xmx1024M -Xms512M -Dderby.tests.trace=true junit.textui.TestRunner org.apache.derbyTesting.functionTests.tests.jdbcapi.AuthenticationTest

          Show
          Suran Jayathilaka added a comment - Attaching derby.log file from the failed tests. This file is from the derby-10.4\tests\fail\DerbyNetClient\AuthenticationTest\testChangePasswordAndDatabasePropertiesOnly folder. Before running the test, I had cleaned the tests folder. I used the following commands to run the tests. set CLASSPATH=..\classes;..\tools\java\junit.jar;..\tools\java\jakarta-oro-2.0.8.jar java -XX:MaxPermSize=128M -Xmx1024M -Xms512M -Dderby.tests.trace=true junit.textui.TestRunner org.apache.derbyTesting.functionTests.tests.jdbcapi.AuthenticationTest
          Hide
          Kathey Marsden added a comment -

          jdbcapi/AuthenticationTest runs ok for me with your patch. (Actually I ran the merge commands and fixed up the setAutocommit calls.) Can you post the stack trace from your failure and the derby.log so we can try to find out what's wrong with your test.

          Show
          Kathey Marsden added a comment - jdbcapi/AuthenticationTest runs ok for me with your patch. (Actually I ran the merge commands and fixed up the setAutocommit calls.) Can you post the stack trace from your failure and the derby.log so we can try to find out what's wrong with your test.
          Hide
          Suran Jayathilaka added a comment -

          Pls see previous comment.

          Show
          Suran Jayathilaka added a comment - Pls see previous comment.
          Hide
          Suran Jayathilaka added a comment -

          This patch, derby-2741-mergeto-10.4.diff, adds the changes for derby-3741 to the 10.4 code branch.

          It incorporates changes from trunk svn revisions
          674565 : SQLChar file reformat
          680478
          682693.

          The commands used for merging are as follows. (on Windows)
          ---------------------------------------------------------------------------------------------------------------------------
          svn merge -r 674564:674565 https://svn.apache.org/repos/asf/db/derby/code/trunk/

          svn merge -r 680477:680478 https://svn.apache.org/repos/asf/db/derby/code/trunk/

          svn revert java\testing\org\apache\derbyTesting\functionTests\tests\memory\ClobMemTest.java

          svn add java\testing\org\apache\derbyTesting\functionTests\tests\memory\ClobMemTest.java

          svn merge -r 682692:682693 https://svn.apache.org/repos/asf/db/derby/code/trunk/

          svn revert java\testing\org\apache\derbyTesting\functionTests\tests\memory\MultiByteClobTest.java

          svn add java\testing\org\apache\derbyTesting\functionTests\tests\memory\MultiByteClobTest.java
          ----------------------------------------------------------------------------------------------------------------------------------

          When running tests (suites.All), there was a lot of test failures.
          e.g.
          jdbcapi/AuthenticationTest

          Show
          Suran Jayathilaka added a comment - This patch, derby-2741-mergeto-10.4.diff, adds the changes for derby-3741 to the 10.4 code branch. It incorporates changes from trunk svn revisions 674565 : SQLChar file reformat 680478 682693. The commands used for merging are as follows. (on Windows) --------------------------------------------------------------------------------------------------------------------------- svn merge -r 674564:674565 https://svn.apache.org/repos/asf/db/derby/code/trunk/ svn merge -r 680477:680478 https://svn.apache.org/repos/asf/db/derby/code/trunk/ svn revert java\testing\org\apache\derbyTesting\functionTests\tests\memory\ClobMemTest.java svn add java\testing\org\apache\derbyTesting\functionTests\tests\memory\ClobMemTest.java svn merge -r 682692:682693 https://svn.apache.org/repos/asf/db/derby/code/trunk/ svn revert java\testing\org\apache\derbyTesting\functionTests\tests\memory\MultiByteClobTest.java svn add java\testing\org\apache\derbyTesting\functionTests\tests\memory\MultiByteClobTest.java ---------------------------------------------------------------------------------------------------------------------------------- When running tests (suites.All), there was a lot of test failures. e.g. jdbcapi/AuthenticationTest
          Hide
          Kathey Marsden added a comment -

          I am attaching a patch that makes a couple minor modifiications to the test. It changes the pageCacheSize to 100 so the test can be run with low memory configurations and it also changes the test to check the contents of the stream when it reads it back in. I'd go ahead and commit this change, but one thing I noticed was that the client test for the large clob seems really slow. 525000ms vs 9578ms for embedded. So the test will add about 9 minutes to the suite. Two questions:

          1) Is there a client bug logged that would explain the extreme difference?
          2) Is it acceptable to check in this test since it adds so much time to suites.All. One option to shorten the run is to only run for embedded. For this bug we are mostly testing that LENGTH is returning the right value for multibyte characters and not consuming too much memory, so for that an embedded run would have us covered.

          Show
          Kathey Marsden added a comment - I am attaching a patch that makes a couple minor modifiications to the test. It changes the pageCacheSize to 100 so the test can be run with low memory configurations and it also changes the test to check the contents of the stream when it reads it back in. I'd go ahead and commit this change, but one thing I noticed was that the client test for the large clob seems really slow. 525000ms vs 9578ms for embedded. So the test will add about 9 minutes to the suite. Two questions: 1) Is there a client bug logged that would explain the extreme difference? 2) Is it acceptable to check in this test since it adds so much time to suites.All. One option to shorten the run is to only run for embedded. For this bug we are mostly testing that LENGTH is returning the right value for multibyte characters and not consuming too much memory, so for that an embedded run would have us covered.
          Hide
          Suran Jayathilaka added a comment -

          derby-3741-multibyte_test.diff adds a test for small and large clobs with multibyte characters.
          Please review.

          Show
          Suran Jayathilaka added a comment - derby-3741-multibyte_test.diff adds a test for small and large clobs with multibyte characters. Please review.
          Hide
          Kathey Marsden added a comment -

          Thanks Suran for the patch. I committed revision 680478. Before you close the issue though, could you add a test for small and large clobs with multibyte characters.

          Thanks

          Kathey

          Show
          Kathey Marsden added a comment - Thanks Suran for the patch. I committed revision 680478. Before you close the issue though, could you add a test for small and large clobs with multibyte characters. Thanks Kathey
          Hide
          Suran Jayathilaka added a comment -

          Made changes according to Kathey's suggestions. Please review.

          Show
          Suran Jayathilaka added a comment - Made changes according to Kathey's suggestions. Please review.
          Hide
          Kathey Marsden added a comment -

          Hi Suran,

          After the wild goose chase of DERBY-3745 I have some more comments on the patch.

          There is a problem with the patch setting rawLenth to the length without having the corresponding rawData value set. This led to problems like the one I saw with the getProcedures not returning the correct value. The getLength() function should not attempt to set rawLength.
          See https://issues.apache.org/jira/browse/DERBY-3795?focusedCommentId=12616661#action_12616661
          for the getProcedures problem I mentioned.

          Also you have code:
          if (rawLength != -1)
          return rawLength;
          if (stream != null) {
          if (rawLength != -1) {
          return rawLength;
          .....

          There is no need to check rawLength again after checking that the stream is not null, because you would have already returned it.

          Thanks

          Kathey

          Show
          Kathey Marsden added a comment - Hi Suran, After the wild goose chase of DERBY-3745 I have some more comments on the patch. There is a problem with the patch setting rawLenth to the length without having the corresponding rawData value set. This led to problems like the one I saw with the getProcedures not returning the correct value. The getLength() function should not attempt to set rawLength. See https://issues.apache.org/jira/browse/DERBY-3795?focusedCommentId=12616661#action_12616661 for the getProcedures problem I mentioned. Also you have code: if (rawLength != -1) return rawLength; if (stream != null) { if (rawLength != -1) { return rawLength; ..... There is no need to check rawLength again after checking that the stream is not null, because you would have already returned it. Thanks Kathey
          Hide
          Kathey Marsden added a comment -

          On the code I just have one comment. Should we check to see if the stream is instanceof ObjectInput as well as Resettable since we cast it? I realize SQLBinary has the same problem.

          I ran the test with 16M heap and it fails. It looks like we are materializing the lob for client. (I thought we had fixed that #. Perhaps you could disable this part of the test for client and file a bug that the client is materializing the lob into memory. I noticed BlobMemTest with 16M is failing with a similar error (It passed a few weeks ago), so it looks like a regression. Here is the ClobMemTest failure.

          There was 1 error:
          1) testClobLength(org.apache.derbyTesting.functionTests.tests.memory.ClobMemTest)java.sql.SQLException: Attempt to fully
          materialize lob data that is too large for the JVM. The connection has been terminated.
          at org.apache.derby.client.am.SQLExceptionFactory.getSQLException(SQLExceptionFactory.java:46)
          at org.apache.derby.client.am.SqlException.getSQLException(SqlException.java:362)
          at org.apache.derby.client.am.ResultSet.next(ResultSet.java:281)
          at org.apache.derbyTesting.functionTests.tests.memory.ClobMemTest.testClobLength(ClobMemTest.java:116)
          at org.apache.derbyTesting.functionTests.tests.memory.ClobMemTest.testClobLength(ClobMemTest.java:171)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          at org.apache.derbyTesting.junit.BaseTestCase.runBare(BaseTestCase.java:104)
          at junit.extensions.TestDecorator.basicRun(TestDecorator.java:22)
          at junit.extensions.TestSetup$1.protect(TestSetup.java:19)
          at junit.extensions.TestSetup.run(TestSetup.java:23)
          at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57)
          at junit.extensions.TestDecorator.basicRun(TestDecorator.java:22)
          at junit.extensions.TestSetup$1.protect(TestSetup.java:19)
          at junit.extensions.TestSetup.run(TestSetup.java:23)
          at junit.extensions.TestDecorator.basicRun(TestDecorator.java:22)
          at junit.extensions.TestSetup$1.protect(TestSetup.java:19)
          at junit.extensions.TestSetup.run(TestSetup.java:23)
          at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57)
          at junit.extensions.TestDecorator.basicRun(TestDecorator.java:22)
          at junit.extensions.TestSetup$1.protect(TestSetup.java:19)
          at junit.extensions.TestSetup.run(TestSetup.java:23)
          Caused by: org.apache.derby.client.am.DisconnectException: Attempt to fully materialize lob data that is too large for t
          he JVM. The connection has been terminated.
          at org.apache.derby.client.net.NetStatementReply.copyEXTDTA(NetStatementReply.java:1528)
          at org.apache.derby.client.net.NetResultSetReply.parseCNTQRYreply(NetResultSetReply.java:143)
          at org.apache.derby.client.net.NetResultSetReply.readFetch(NetResultSetReply.java:42)
          at org.apache.derby.client.net.ResultSetReply.readFetch(ResultSetReply.java:41)
          at org.apache.derby.client.net.NetResultSet.readFetch_(NetResultSet.java:206)
          at org.apache.derby.client.am.ResultSet.flowFetch(ResultSet.java:4275)
          at org.apache.derby.client.net.NetCursor.getMoreData_(NetCursor.java:1243)
          at org.apache.derby.client.am.Cursor.stepNext(Cursor.java:176)
          at org.apache.derby.client.am.Cursor.next(Cursor.java:195)
          at org.apache.derby.client.am.ResultSet.nextX(ResultSet.java:302)
          at org.apache.derby.client.am.ResultSet.next(ResultSet.java:272)
          ... 38 more
          Caused by: java.lang.OutOfMemoryError
          at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:116)
          at org.apache.derby.client.net.Reply.getData(Reply.java:787)
          at org.apache.derby.client.net.NetStatementReply.copyEXTDTA(NetStatementReply.java:1520)
          ... 48 more

          FAILURES!!!

          Show
          Kathey Marsden added a comment - On the code I just have one comment. Should we check to see if the stream is instanceof ObjectInput as well as Resettable since we cast it? I realize SQLBinary has the same problem. I ran the test with 16M heap and it fails. It looks like we are materializing the lob for client. (I thought we had fixed that # . Perhaps you could disable this part of the test for client and file a bug that the client is materializing the lob into memory. I noticed BlobMemTest with 16M is failing with a similar error (It passed a few weeks ago), so it looks like a regression. Here is the ClobMemTest failure. There was 1 error: 1) testClobLength(org.apache.derbyTesting.functionTests.tests.memory.ClobMemTest)java.sql.SQLException: Attempt to fully materialize lob data that is too large for the JVM. The connection has been terminated. at org.apache.derby.client.am.SQLExceptionFactory.getSQLException(SQLExceptionFactory.java:46) at org.apache.derby.client.am.SqlException.getSQLException(SqlException.java:362) at org.apache.derby.client.am.ResultSet.next(ResultSet.java:281) at org.apache.derbyTesting.functionTests.tests.memory.ClobMemTest.testClobLength(ClobMemTest.java:116) at org.apache.derbyTesting.functionTests.tests.memory.ClobMemTest.testClobLength(ClobMemTest.java:171) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at org.apache.derbyTesting.junit.BaseTestCase.runBare(BaseTestCase.java:104) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:22) at junit.extensions.TestSetup$1.protect(TestSetup.java:19) at junit.extensions.TestSetup.run(TestSetup.java:23) at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:22) at junit.extensions.TestSetup$1.protect(TestSetup.java:19) at junit.extensions.TestSetup.run(TestSetup.java:23) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:22) at junit.extensions.TestSetup$1.protect(TestSetup.java:19) at junit.extensions.TestSetup.run(TestSetup.java:23) at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57) at junit.extensions.TestDecorator.basicRun(TestDecorator.java:22) at junit.extensions.TestSetup$1.protect(TestSetup.java:19) at junit.extensions.TestSetup.run(TestSetup.java:23) Caused by: org.apache.derby.client.am.DisconnectException: Attempt to fully materialize lob data that is too large for t he JVM. The connection has been terminated. at org.apache.derby.client.net.NetStatementReply.copyEXTDTA(NetStatementReply.java:1528) at org.apache.derby.client.net.NetResultSetReply.parseCNTQRYreply(NetResultSetReply.java:143) at org.apache.derby.client.net.NetResultSetReply.readFetch(NetResultSetReply.java:42) at org.apache.derby.client.net.ResultSetReply.readFetch(ResultSetReply.java:41) at org.apache.derby.client.net.NetResultSet.readFetch_(NetResultSet.java:206) at org.apache.derby.client.am.ResultSet.flowFetch(ResultSet.java:4275) at org.apache.derby.client.net.NetCursor.getMoreData_(NetCursor.java:1243) at org.apache.derby.client.am.Cursor.stepNext(Cursor.java:176) at org.apache.derby.client.am.Cursor.next(Cursor.java:195) at org.apache.derby.client.am.ResultSet.nextX(ResultSet.java:302) at org.apache.derby.client.am.ResultSet.next(ResultSet.java:272) ... 38 more Caused by: java.lang.OutOfMemoryError at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:116) at org.apache.derby.client.net.Reply.getData(Reply.java:787) at org.apache.derby.client.net.NetStatementReply.copyEXTDTA(NetStatementReply.java:1520) ... 48 more FAILURES!!!
          Hide
          Suran Jayathilaka added a comment -

          Attaching patch derby-3741-1.diff for initial review only. Test suites not run.

          Show
          Suran Jayathilaka added a comment - Attaching patch derby-3741-1.diff for initial review only. Test suites not run.
          Hide
          Suran Jayathilaka added a comment -

          Attaching ClobMemTest testcase for review. This is based entirely on BlobMemTest. This is NOT intended as a patch.

          Show
          Suran Jayathilaka added a comment - Attaching ClobMemTest testcase for review. This is based entirely on BlobMemTest. This is NOT intended as a patch.
          Hide
          Suran Jayathilaka added a comment -

          In a subsequent discussion, Kathey pointed out that the following path might be the one to take.
          For not so large CLOBs, if the stream is resettable, get the utf8length with readExternal() and return that IF it's not zero.
          If it's zero, (i.e. stream could be longer than 65535) can use the UTF8Util class' skipUntilEOF() method to get how many chars are in the stream,
          but resetting the stream after that.
          If the stream is not Resetable, will have to go with getString().length.

          Show
          Suran Jayathilaka added a comment - In a subsequent discussion, Kathey pointed out that the following path might be the one to take. For not so large CLOBs, if the stream is resettable, get the utf8length with readExternal() and return that IF it's not zero. If it's zero, (i.e. stream could be longer than 65535) can use the UTF8Util class' skipUntilEOF() method to get how many chars are in the stream, but resetting the stream after that. If the stream is not Resetable, will have to go with getString().length.
          Hide
          Kathey Marsden added a comment -

          Suran asked me off-line for some pointers on this issue, but I am posting my response to the issue to keep the discussion on the list and allow room for anyone to correct me if what I say is wrong.

          Probably the first best approach is to write the test. Write a test ClobMemTest which is similar to BlobMemTest and add it to the memory suite. Make sure you have some multibyte characters so we are sure we are getting the character length and not the byte length when we fix this. The test should pass normally but fail when run with the target junit-lowmem.

          As for the code change, I think we are looking to change SQLChar.getLength(). I believe the character length is encoded in the stream in some cases so we want to retrieve that if possible and if not read (skip) the entire stream to get the character length. Either way we will need to reset the stream in the end to make sure we are positioned back at the beginning of the stream. The changes should be the same as those made to SQLBinary.getLength() for BLOBS (DERBY-3732) but not exactly the same because we are dealing with characters.

          Hope this helps. Please let us know as you have more questions.

          Show
          Kathey Marsden added a comment - Suran asked me off-line for some pointers on this issue, but I am posting my response to the issue to keep the discussion on the list and allow room for anyone to correct me if what I say is wrong. Probably the first best approach is to write the test. Write a test ClobMemTest which is similar to BlobMemTest and add it to the memory suite. Make sure you have some multibyte characters so we are sure we are getting the character length and not the byte length when we fix this. The test should pass normally but fail when run with the target junit-lowmem. As for the code change, I think we are looking to change SQLChar.getLength(). I believe the character length is encoded in the stream in some cases so we want to retrieve that if possible and if not read (skip) the entire stream to get the character length. Either way we will need to reset the stream in the end to make sure we are positioned back at the beginning of the stream. The changes should be the same as those made to SQLBinary.getLength() for BLOBS ( DERBY-3732 ) but not exactly the same because we are dealing with characters. Hope this helps. Please let us know as you have more questions.
          Hide
          Kathey Marsden added a comment -

          Reattaching LargeLengthClob.zip, was missing file before.

          Show
          Kathey Marsden added a comment - Reattaching LargeLengthClob.zip, was missing file before.
          Hide
          Kathey Marsden added a comment -

          to reproduce run
          java -Xmx16M LengthLargeClob

          Caused by: java.lang.OutOfMemoryError: Java heap space
          at org.apache.derby.iapi.types.SQLChar.readExternal(SQLChar.java:734)
          at org.apache.derby.iapi.types.SQLChar.getString(SQLChar.java:358)
          at org.apache.derby.iapi.types.SQLChar.getLength(SQLChar.java:315)
          at org.apache.derby.impl.sql.execute.BaseActivation.getDB2Length(BaseActivation.java:1684)
          at org.apache.derby.exe.acf81e0010x011axcc3cx23cbx00000040a4181.e1(Unknown Source)
          at org.apache.derby.impl.services.reflect.DirectCall.invoke(ReflectGeneratedClass.java:141)
          at org.apache.derby.impl.sql.execute.ProjectRestrictResultSet.doProjection(ProjectRestrictResultSet.java:497)
          at org.apache.derby.impl.sql.execute.ProjectRestrictResultSet.getNextRowCore(ProjectRestrictResultSet.java:291)
          at org.apache.derby.impl.sql.execute.BasicNoPutResultSetImpl.getNextRow(BasicNoPutResultSetImpl.java:460)
          at org.apache.derby.impl.jdbc.EmbedResultSet.movePosition(EmbedResultSet.java:423)

          Show
          Kathey Marsden added a comment - to reproduce run java -Xmx16M LengthLargeClob Caused by: java.lang.OutOfMemoryError: Java heap space at org.apache.derby.iapi.types.SQLChar.readExternal(SQLChar.java:734) at org.apache.derby.iapi.types.SQLChar.getString(SQLChar.java:358) at org.apache.derby.iapi.types.SQLChar.getLength(SQLChar.java:315) at org.apache.derby.impl.sql.execute.BaseActivation.getDB2Length(BaseActivation.java:1684) at org.apache.derby.exe.acf81e0010x011axcc3cx23cbx00000040a4181.e1(Unknown Source) at org.apache.derby.impl.services.reflect.DirectCall.invoke(ReflectGeneratedClass.java:141) at org.apache.derby.impl.sql.execute.ProjectRestrictResultSet.doProjection(ProjectRestrictResultSet.java:497) at org.apache.derby.impl.sql.execute.ProjectRestrictResultSet.getNextRowCore(ProjectRestrictResultSet.java:291) at org.apache.derby.impl.sql.execute.BasicNoPutResultSetImpl.getNextRow(BasicNoPutResultSetImpl.java:460) at org.apache.derby.impl.jdbc.EmbedResultSet.movePosition(EmbedResultSet.java:423)

            People

            • Assignee:
              Suran Jayathilaka
              Reporter:
              Kathey Marsden
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development