Hive
  1. Hive
  2. HIVE-2084

Upgrade datanucleus from 2.0.3 to a more recent version (3.?)

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.12.0
    • Component/s: Metastore
    • Labels:

      Description

      It seems the datanucleus 2.2.3 does a better join in caching. The time it takes to get the same set of partition objects takes about 1/4 of the time it took for the first time. While with 2.0.3, it took almost the same amount of time in the second execution. We should retest the test case mentioned in HIVE-1853, HIVE-1862.

      1. HIVE-2084.D5685.1.patch
        6 kB
        Phabricator
      2. ASF.LICENSE.NOT.GRANTED--HIVE-2084.D2397.1.patch
        5 kB
        Phabricator
      3. HIVE-2084.2.patch.txt
        5 kB
        Sushanth Sowmyan
      4. HIVE-2084.1.patch.txt
        3 kB
        Carl Steinbach
      5. HIVE-2084.patch
        6 kB
        Ning Zhang

        Issue Links

          Activity

          Hide
          Ning Zhang added a comment -

          review board: https://reviews.apache.org/r/537/

          Testing using the script provided by HIVE-1862 while adding partitions to the input table. Test has been going on for several hours and so far so good.

          Show
          Ning Zhang added a comment - review board: https://reviews.apache.org/r/537/ Testing using the script provided by HIVE-1862 while adding partitions to the input table. Test has been going on for several hours and so far so good.
          Hide
          Ning Zhang added a comment -

          Devaraj/Mac, could you also test this patch on your production environment if you have other testing scripts? I'm testing the script uploaded by Namit in HIVE-1862 and this patch seems working fine so far.

          Show
          Ning Zhang added a comment - Devaraj/Mac, could you also test this patch on your production environment if you have other testing scripts? I'm testing the script uploaded by Namit in HIVE-1862 and this patch seems working fine so far.
          Hide
          Mac Yang added a comment -

          Ning, I will test this patch with our usual set up and let you know how it goes.

          Show
          Mac Yang added a comment - Ning, I will test this patch with our usual set up and let you know how it goes.
          Hide
          Namit Jain added a comment -

          @Ning/Paul, do you know if data nucleus 2.2.3 has the ability to
          support filter pushdown for predicates for non-equality.

          Show
          Namit Jain added a comment - @Ning/Paul, do you know if data nucleus 2.2.3 has the ability to support filter pushdown for predicates for non-equality.
          Hide
          Namit Jain added a comment -

          +1

          The code changes look good, but I will wait from a confirmation by
          Mac before checking it in.

          Show
          Namit Jain added a comment - +1 The code changes look good, but I will wait from a confirmation by Mac before checking it in.
          Hide
          Ning Zhang added a comment -

          @Namit, yeah, 2.2.3 support filter push down for non-equality. Even the older version of 2.0.3 supposes it too. Mac's patch actually supports range queries, but since range queries could be complicated on multiple partition columns (what if the range is on the column that is not the top partition column), I didn't dig deep into it, but it the push down filtering criteria can certainly be relaxed.

          Having said that, my test results shows that JDO filter pushing down may not be the dominate factor (comparing to the patch in HIVE-2050). In the experiments I've done for HIVE-2050, listing partition names and filtering partitions in the Hive client side may take 10 sec, but retrieving all Partition objects takes about 10 mins in total. The best of pushing down JDO filtering can only reduce the 10 sec to 0, but the 10 mins overhead is still there. We need to find a way to optimize that away.

          Show
          Ning Zhang added a comment - @Namit, yeah, 2.2.3 support filter push down for non-equality. Even the older version of 2.0.3 supposes it too. Mac's patch actually supports range queries, but since range queries could be complicated on multiple partition columns (what if the range is on the column that is not the top partition column), I didn't dig deep into it, but it the push down filtering criteria can certainly be relaxed. Having said that, my test results shows that JDO filter pushing down may not be the dominate factor (comparing to the patch in HIVE-2050 ). In the experiments I've done for HIVE-2050 , listing partition names and filtering partitions in the Hive client side may take 10 sec, but retrieving all Partition objects takes about 10 mins in total. The best of pushing down JDO filtering can only reduce the 10 sec to 0, but the 10 mins overhead is still there. We need to find a way to optimize that away.
          Hide
          Carl Steinbach added a comment -

          @Ning: Why did you modify the class mapping in package.jdo? Does this require a metastore upgrade script?

          Show
          Carl Steinbach added a comment - @Ning: Why did you modify the class mapping in package.jdo? Does this require a metastore upgrade script?
          Hide
          Ning Zhang added a comment -

          @Carl, one change (at line 49) in package.jdo is to fix a bug that was not exposed by the old datanucleus version. Without the change datanucleus will throw an exception in runtime (FCOMMENT is not a column of COLUMNS table). I guess the old version of datanucleus didn't check MFieldSchema mapping in package.jdo, by only retrieving the columns mentioned in the <embedded> elements. The other changes are to make the legacy mappings to confirm to the current relational schema (e.g., MFieldSchema.FNAME should be mapped to COLUMNS.COLUMN_NAME). They currently does not cause any runtime exceptions, but I guess it's better to fix it proactively if we are sure the relational mapping is wrong.

          Show
          Ning Zhang added a comment - @Carl, one change (at line 49) in package.jdo is to fix a bug that was not exposed by the old datanucleus version. Without the change datanucleus will throw an exception in runtime (FCOMMENT is not a column of COLUMNS table). I guess the old version of datanucleus didn't check MFieldSchema mapping in package.jdo, by only retrieving the columns mentioned in the <embedded> elements. The other changes are to make the legacy mappings to confirm to the current relational schema (e.g., MFieldSchema.FNAME should be mapped to COLUMNS.COLUMN_NAME). They currently does not cause any runtime exceptions, but I guess it's better to fix it proactively if we are sure the relational mapping is wrong.
          Hide
          Carl Steinbach added a comment -

          One change (at line 49) in package.jdo is to fix a bug that was not exposed by the old datanucleus version. Without the change datanucleus will throw an exception in runtime (FCOMMENT is not a column of COLUMNS table). I guess the old version of datanucleus didn't check MFieldSchema mapping in package.jdo, by only retrieving the columns mentioned in the <embedded> elements.

          Yup, looks like that's the case. It also looks like Datanucleus was ignoring the size of the FCOMMENTS field, so the older versions of TYPE_FIELDS.COMMENT and COLUMNS.COMMENT have size 256, which must be the default value. In the new schema these fields both get bumped to 4000 bytes, which is the correct size. Can you please include upgrade scripts that update the size of these columns accordingly?

          Also, as far as I can tell the change to the MOrder mapping has no effect since it is only referenced by the SORT_COLS table, which overrides the name to COLUMN_NAME instead.

          Show
          Carl Steinbach added a comment - One change (at line 49) in package.jdo is to fix a bug that was not exposed by the old datanucleus version. Without the change datanucleus will throw an exception in runtime (FCOMMENT is not a column of COLUMNS table). I guess the old version of datanucleus didn't check MFieldSchema mapping in package.jdo, by only retrieving the columns mentioned in the <embedded> elements. Yup, looks like that's the case. It also looks like Datanucleus was ignoring the size of the FCOMMENTS field, so the older versions of TYPE_FIELDS.COMMENT and COLUMNS.COMMENT have size 256, which must be the default value. In the new schema these fields both get bumped to 4000 bytes, which is the correct size. Can you please include upgrade scripts that update the size of these columns accordingly? Also, as far as I can tell the change to the MOrder mapping has no effect since it is only referenced by the SORT_COLS table, which overrides the name to COLUMN_NAME instead.
          Hide
          Mac Yang added a comment -

          Test has been running for about six hours without failure. Looks good.

          Show
          Mac Yang added a comment - Test has been running for about six hours without failure. Looks good.
          Hide
          Ning Zhang added a comment -

          Thank Mac for the update.

          @Carl, I'll change the field size if they do not match with the relational schema. BTW, we don't need to upgrade the relational schema. I'm trying to update the object-to-relational mapping in package.jdo to match with the relational schema.

          Also there are some tests failing. I'm working on that and will upload a new patch all together.

          Show
          Ning Zhang added a comment - Thank Mac for the update. @Carl, I'll change the field size if they do not match with the relational schema. BTW, we don't need to upgrade the relational schema. I'm trying to update the object-to-relational mapping in package.jdo to match with the relational schema. Also there are some tests failing. I'm working on that and will upload a new patch all together.
          Hide
          Carl Steinbach added a comment -

          DataNucleus 3.0.1 was released last month. We should probably try to go upgrade to
          that since it will make it possible to fix HIVE-2015.

          Show
          Carl Steinbach added a comment - DataNucleus 3.0.1 was released last month. We should probably try to go upgrade to that since it will make it possible to fix HIVE-2015 .
          Hide
          Andy Jefferson added a comment -

          Regarding upgrading to DN 3.0, follow this link http://www.datanucleus.org/products/accessplatform_3_0/migration.html
          3.0.2 will be out in a week, but no API changes involved from 3.0.1 to 3.0.2

          Show
          Andy Jefferson added a comment - Regarding upgrading to DN 3.0, follow this link http://www.datanucleus.org/products/accessplatform_3_0/migration.html 3.0.2 will be out in a week, but no API changes involved from 3.0.1 to 3.0.2
          Hide
          Carl Steinbach added a comment -

          Attached patch upgrades DN dependency to v3.0.1. Running tests now.

          Show
          Carl Steinbach added a comment - Attached patch upgrades DN dependency to v3.0.1. Running tests now.
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2127/
          -----------------------------------------------------------

          Review request for hive and John Sichi.

          Summary
          -------

          This patch upgrades Hive's Datanucleus dependency from v2.0.3 to v3.0.1.

          This addresses bug HIVE-2084.
          https://issues.apache.org/jira/browse/HIVE-2084

          Diffs


          conf/hive-default.xml 683b417
          ivy/libraries.properties adde3cb
          metastore/ivy.xml 2e5331d
          ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 983e1d5

          Diff: https://reviews.apache.org/r/2127/diff

          Testing
          -------

          Thanks,

          Carl

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2127/ ----------------------------------------------------------- Review request for hive and John Sichi. Summary ------- This patch upgrades Hive's Datanucleus dependency from v2.0.3 to v3.0.1. This addresses bug HIVE-2084 . https://issues.apache.org/jira/browse/HIVE-2084 Diffs conf/hive-default.xml 683b417 ivy/libraries.properties adde3cb metastore/ivy.xml 2e5331d ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 983e1d5 Diff: https://reviews.apache.org/r/2127/diff Testing ------- Thanks, Carl
          Hide
          John Sichi added a comment -

          Any stress retesting needed in case of regression from 2.2.3 to 3.0.1?

          Show
          John Sichi added a comment - Any stress retesting needed in case of regression from 2.2.3 to 3.0.1?
          Hide
          Carl Steinbach added a comment -

          @John: Sounds like a good idea. Do you know if @Ning still has the scripts he used earlier?

          Show
          Carl Steinbach added a comment - @John: Sounds like a good idea. Do you know if @Ning still has the scripts he used earlier?
          Hide
          Carl Steinbach added a comment -

          There are failures in the auth and indexing tests. I'm investigating.

          Show
          Carl Steinbach added a comment - There are failures in the auth and indexing tests. I'm investigating.
          Hide
          Ashutosh Chauhan added a comment -

          This upgrade may resolve HIVE-2473 too.

          Show
          Ashutosh Chauhan added a comment - This upgrade may resolve HIVE-2473 too.
          Hide
          Sushanth Sowmyan added a comment -

          Updated patch, and changed to Datanucleus v3.0.8. Does anyone still have any failing tests with this upgrade?

          Show
          Sushanth Sowmyan added a comment - Updated patch, and changed to Datanucleus v3.0.8. Does anyone still have any failing tests with this upgrade?
          Hide
          Carl Steinbach added a comment -

          @Sushanth: Can you please submit a review request on phabricator? Thanks. https://cwiki.apache.org/Hive/phabricatorcodereview.html

          Show
          Carl Steinbach added a comment - @Sushanth: Can you please submit a review request on phabricator? Thanks. https://cwiki.apache.org/Hive/phabricatorcodereview.html
          Hide
          Phabricator added a comment -

          khorgath requested code review of "HIVE-2084 [jira] Upgrade datanucleus from 2.0.3 to 3.0.1".
          Reviewers: JIRA

          Updated HIVE-2084 to work off DataNucleus release 3.0.8

          It seems the datanucleus 2.2.3 does a better join in caching. The time it takes to get the same set of partition objects takes about 1/4 of the time it took for the first time. While with 2.0.3, it took almost the same amount of time in the second execution. We should retest the test case mentioned in HIVE-1853, HIVE-1862.

          TEST PLAN
          existing tests (this is a library dep upgrade)

          REVISION DETAIL
          https://reviews.facebook.net/D2397

          AFFECTED FILES
          common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
          conf/hive-default.xml.template
          ivy/libraries.properties
          metastore/ivy.xml
          ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java

          MANAGE HERALD DIFFERENTIAL RULES
          https://reviews.facebook.net/herald/view/differential/

          WHY DID I GET THIS EMAIL?
          https://reviews.facebook.net/herald/transcript/5367/

          Tip: use the X-Herald-Rules header to filter Herald messages in your client.

          Show
          Phabricator added a comment - khorgath requested code review of " HIVE-2084 [jira] Upgrade datanucleus from 2.0.3 to 3.0.1". Reviewers: JIRA Updated HIVE-2084 to work off DataNucleus release 3.0.8 It seems the datanucleus 2.2.3 does a better join in caching. The time it takes to get the same set of partition objects takes about 1/4 of the time it took for the first time. While with 2.0.3, it took almost the same amount of time in the second execution. We should retest the test case mentioned in HIVE-1853 , HIVE-1862 . TEST PLAN existing tests (this is a library dep upgrade) REVISION DETAIL https://reviews.facebook.net/D2397 AFFECTED FILES common/src/java/org/apache/hadoop/hive/conf/HiveConf.java conf/hive-default.xml.template ivy/libraries.properties metastore/ivy.xml ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/5367/ Tip: use the X-Herald-Rules header to filter Herald messages in your client.
          Hide
          Sushanth Sowmyan added a comment -

          Updated : https://reviews.facebook.net/D2397

          (Also, please ignore the patch file I'd attached here before, I'd generated it from the hcatalog root dir, so it contains extra directory structure)

          Show
          Sushanth Sowmyan added a comment - Updated : https://reviews.facebook.net/D2397 (Also, please ignore the patch file I'd attached here before, I'd generated it from the hcatalog root dir, so it contains extra directory structure)
          Hide
          Alan Gates added a comment -

          When I apply this patch all of the Hive end-to-end tests I have pass, as well as the HCatalog ones (including the ones added in HCATALOG-209 to test this issue).

          Show
          Alan Gates added a comment - When I apply this patch all of the Hive end-to-end tests I have pass, as well as the HCatalog ones (including the ones added in HCATALOG-209 to test this issue).
          Hide
          Namit Jain added a comment -

          @Alan, I cant seem to find the test which was failing for the old JDO upgrade.
          There was some problem when we upgraded last time.

          Show
          Namit Jain added a comment - @Alan, I cant seem to find the test which was failing for the old JDO upgrade. There was some problem when we upgraded last time.
          Hide
          Namit Jain added a comment -

          OK, it is HIVE-1862.
          Did you run that test with the new JDO ?

          Show
          Namit Jain added a comment - OK, it is HIVE-1862 . Did you run that test with the new JDO ?
          Hide
          Sushanth Sowmyan added a comment -

          Hi Namit,

          I've run a fair number of iterations of tests against the DN versions, and I find that the test that you posted up on that jira still fails a few times with DataNucleus 3.0.8 about 0.5% of the time saying that the table was not found. With DN 2.0.3, I have no failures.

          However, if I increase the parallelism, from 5 concurrent runs, to, say, about 20, all 3 versions of DN wind up showing those kinds of 'table not found' error, and quite often (10-20% of the time) and occur even with simpler statements like 'describe table' and 'show partitions extended'. I think we may have a case of timeouts that are being incorrectly reported.

          Show
          Sushanth Sowmyan added a comment - Hi Namit, I've run a fair number of iterations of tests against the DN versions, and I find that the test that you posted up on that jira still fails a few times with DataNucleus 3.0.8 about 0.5% of the time saying that the table was not found. With DN 2.0.3, I have no failures. However, if I increase the parallelism, from 5 concurrent runs, to, say, about 20, all 3 versions of DN wind up showing those kinds of 'table not found' error, and quite often (10-20% of the time) and occur even with simpler statements like 'describe table' and 'show partitions extended'. I think we may have a case of timeouts that are being incorrectly reported.
          Hide
          Ashutosh Chauhan added a comment -

          @Sushanth,

          You have two patches up, one via phabricator another directly on jira and they have quite bit of differences among them. Which one you are proposing for review?

          Show
          Ashutosh Chauhan added a comment - @Sushanth, You have two patches up, one via phabricator another directly on jira and they have quite bit of differences among them. Which one you are proposing for review?
          Hide
          Sushanth Sowmyan added a comment -

          @Ashutosh : As per above comment of mine, ignore the one directly on jira - that was generated from hcatalog with a bunch of other unnecessary directory structures associated with it ( hive/external/ ). Please check the one on phabricator.

          Show
          Sushanth Sowmyan added a comment - @Ashutosh : As per above comment of mine, ignore the one directly on jira - that was generated from hcatalog with a bunch of other unnecessary directory structures associated with it ( hive/external/ ). Please check the one on phabricator.
          Hide
          Andy Jefferson added a comment -

          No idea what your "table not found" problem is, but if this is at startup logic would suggest passing all classes to DataNucleus via an auto-start mechanism, or using a persistence.xml. If using persistence.xml then make sure you also have persistence property "datanucleus.PersistenceUnitLoadClasses" set to true. That way, all classes are known about once started, and the store knows about these classes too (hence knows of their tables). There are no outstanding issues reported around tables not being found, so if you have something then you need to generate a reproduceable testcase and report it.

          DataNucleus 2.x versions haven't been supported for some time. DataNucleus 3.0 is now not being updated (except for commercial requests), since DataNucleus 3.1 will be out in < 2 weeks.

          Show
          Andy Jefferson added a comment - No idea what your "table not found" problem is, but if this is at startup logic would suggest passing all classes to DataNucleus via an auto-start mechanism, or using a persistence.xml. If using persistence.xml then make sure you also have persistence property "datanucleus.PersistenceUnitLoadClasses" set to true. That way, all classes are known about once started, and the store knows about these classes too (hence knows of their tables). There are no outstanding issues reported around tables not being found, so if you have something then you need to generate a reproduceable testcase and report it. DataNucleus 2.x versions haven't been supported for some time. DataNucleus 3.0 is now not being updated (except for commercial requests), since DataNucleus 3.1 will be out in < 2 weeks.
          Hide
          Carl Steinbach added a comment -

          @Sushanth: Looks like this patch fell out of sync with trunk. Would you be willing to rebase and post a new copy? Thanks.

          Show
          Carl Steinbach added a comment - @Sushanth: Looks like this patch fell out of sync with trunk. Would you be willing to rebase and post a new copy? Thanks.
          Hide
          Sushanth Sowmyan added a comment -

          @Carl : There's actually another issue I discovered with this patch, that I'm currently looking at where unit tests that use indexes start failing. (With different versions of datanucleus, I find a different number of rollbackTransaction() being called from the ObjectStore) I almost have the root cause, so I'll update this patch with that and bumping it to 3.1.0 in another day or two.

          Show
          Sushanth Sowmyan added a comment - @Carl : There's actually another issue I discovered with this patch, that I'm currently looking at where unit tests that use indexes start failing. (With different versions of datanucleus, I find a different number of rollbackTransaction() being called from the ObjectStore) I almost have the root cause, so I'll update this patch with that and bumping it to 3.1.0 in another day or two.
          Hide
          Sushanth Sowmyan added a comment -

          @Carl : As an update, I discovered that with the newer DataNucleus, what's happening is that Map types with null values cannot be persisted. This is a problem because we stamp a comment field in the parametersMap irrespective of whether a comment was provided or not, and this causes a failure during index creation.

          This is also the same issue that I refer to in HIVE-2800 where thrift has similar issues, where the fix is the same.

          Show
          Sushanth Sowmyan added a comment - @Carl : As an update, I discovered that with the newer DataNucleus, what's happening is that Map types with null values cannot be persisted. This is a problem because we stamp a comment field in the parametersMap irrespective of whether a comment was provided or not, and this causes a failure during index creation. This is also the same issue that I refer to in HIVE-2800 where thrift has similar issues, where the fix is the same.
          Hide
          Andy Jefferson added a comment -

          Obviously DataNucleus has testcases that persist Maps with null values, and they work (since all tests pass with every release), so clearly down to your map and how you're doing things.

          Show
          Andy Jefferson added a comment - Obviously DataNucleus has testcases that persist Maps with null values, and they work (since all tests pass with every release), so clearly down to your map and how you're doing things.
          Hide
          Rohith added a comment -

          Could anyone please update on this issue.?

          Show
          Rohith added a comment - Could anyone please update on this issue.?
          Hide
          Carl Steinbach added a comment -

          @Andy: I'd like to take a look at these tests that persist maps with null values. Can you please tell us where they are located? Thanks.

          Show
          Carl Steinbach added a comment - @Andy: I'd like to take a look at these tests that persist maps with null values. Can you please tell us where they are located? Thanks.
          Hide
          Andy Jefferson added a comment -

          @Carl, http://datanucleus.svn.sourceforge.net/viewvc/datanucleus/test/accessplatform/trunk/test.jdo.datastore/src/test/org/datanucleus/tests/types/SCOMapTests.java?revision=15499&content-type=text%2Fplain
          look for "checkPutNullValues". Always has passed for me since we started supporting null map values (2 or 3 years ago at a guess), most recently with datanucleus-core-3.1.2, datanucleus-api-jdo-3.1.2, datanucleus-rdbms-3.1.1. Since your problem description provides no information of the circumstances (such as log entries, persistence code etc) then cannot speculate as to what you have

          Show
          Andy Jefferson added a comment - @Carl, http://datanucleus.svn.sourceforge.net/viewvc/datanucleus/test/accessplatform/trunk/test.jdo.datastore/src/test/org/datanucleus/tests/types/SCOMapTests.java?revision=15499&content-type=text%2Fplain look for "checkPutNullValues". Always has passed for me since we started supporting null map values (2 or 3 years ago at a guess), most recently with datanucleus-core-3.1.2, datanucleus-api-jdo-3.1.2, datanucleus-rdbms-3.1.1. Since your problem description provides no information of the circumstances (such as log entries, persistence code etc) then cannot speculate as to what you have
          Hide
          Carl Steinbach added a comment -

          @Andy: Thanks for the pointer.

          Show
          Carl Steinbach added a comment - @Andy: Thanks for the pointer.
          Hide
          Phabricator added a comment -

          khorgath requested code review of "HIVE-2084 [jira] Upgrade datanucleus from 2.0.3 to 3.0.1".
          Reviewers: JIRA, cwsteinbach

          Updated for 3.1, rebasing to head, and making changes needed to continue building

          TEST PLAN
          Existing tests

          REVISION DETAIL
          https://reviews.facebook.net/D5685

          AFFECTED FILES
          common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
          conf/hive-default.xml.template
          ivy/libraries.properties
          metastore/ivy.xml
          ql/ivy.xml
          ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java

          MANAGE HERALD DIFFERENTIAL RULES
          https://reviews.facebook.net/herald/view/differential/

          WHY DID I GET THIS EMAIL?
          https://reviews.facebook.net/herald/transcript/13353/

          To: JIRA, cwsteinbach, khorgath

          Show
          Phabricator added a comment - khorgath requested code review of " HIVE-2084 [jira] Upgrade datanucleus from 2.0.3 to 3.0.1". Reviewers: JIRA, cwsteinbach Updated for 3.1, rebasing to head, and making changes needed to continue building TEST PLAN Existing tests REVISION DETAIL https://reviews.facebook.net/D5685 AFFECTED FILES common/src/java/org/apache/hadoop/hive/conf/HiveConf.java conf/hive-default.xml.template ivy/libraries.properties metastore/ivy.xml ql/ivy.xml ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/13353/ To: JIRA, cwsteinbach, khorgath
          Hide
          Sushanth Sowmyan added a comment -

          Andy, thanks for the pointer, I'll try to see what's different about our use of Maps.

          I've rebased and updated the patch, but unfortunately, I'm not entirely conversant with phabricator yet, and wound up creating a new "review" while updating. It's now up on https://reviews.facebook.net/D5685

          The easiest way to test out the difference I allude to is to run the following with and without this patch:

          ant clean && ant very-clean && ant package && ant test -Dtestcase=TestCliDriver -Dtest.silent=false -Dqfile=alter_index.q
          

          If I dig through the stack trace with prints, the eventual error that's thrown now has an error code of 056063, which localization-maps to "Null values not allowed in persistent maps."

          Show
          Sushanth Sowmyan added a comment - Andy, thanks for the pointer, I'll try to see what's different about our use of Maps. I've rebased and updated the patch, but unfortunately, I'm not entirely conversant with phabricator yet, and wound up creating a new "review" while updating. It's now up on https://reviews.facebook.net/D5685 The easiest way to test out the difference I allude to is to run the following with and without this patch: ant clean && ant very-clean && ant package && ant test -Dtestcase=TestCliDriver -Dtest.silent=false -Dqfile=alter_index.q If I dig through the stack trace with prints, the eventual error that's thrown now has an error code of 056063, which localization-maps to "Null values not allowed in persistent maps."
          Hide
          Phabricator added a comment -

          khorgath has abandoned the revision "HIVE-2084 [jira] Upgrade datanucleus from 2.0.3 to 3.0.1".

          Replaced by D5685 : https://reviews.facebook.net/D5685

          REVISION DETAIL
          https://reviews.facebook.net/D2397

          To: JIRA, khorgath

          Show
          Phabricator added a comment - khorgath has abandoned the revision " HIVE-2084 [jira] Upgrade datanucleus from 2.0.3 to 3.0.1". Replaced by D5685 : https://reviews.facebook.net/D5685 REVISION DETAIL https://reviews.facebook.net/D2397 To: JIRA, khorgath
          Hide
          Sushanth Sowmyan added a comment -

          @Andy : I guess the primary difference is in the allowNulls flag (as used in org.datanucleus.store.rdbms.scostore.AbstractMapStore), which seems to be false in our application. I see ways to turn that on through query hints, but is there any way to turn that on at an application level - i.e. through any java property?

          Show
          Sushanth Sowmyan added a comment - @Andy : I guess the primary difference is in the allowNulls flag (as used in org.datanucleus.store.rdbms.scostore.AbstractMapStore), which seems to be false in our application. I see ways to turn that on through query hints, but is there any way to turn that on at an application level - i.e. through any java property?
          Hide
          Andy Jefferson added a comment -

          "allowNulls" can either be set in metadata (XML,annotation) for the container field or it can default to what the particular "java.util" type does so, for example, HashMap/HashSet/LinkedHashSet/any-type-of-list default to allowNulls=true, and all others default to false currently. Only you know what type is your map

          Show
          Andy Jefferson added a comment - "allowNulls" can either be set in metadata (XML,annotation) for the container field or it can default to what the particular "java.util" type does so, for example, HashMap/HashSet/LinkedHashSet/any-type-of-list default to allowNulls=true, and all others default to false currently. Only you know what type is your map
          Hide
          Kevin Wilfong added a comment -

          Sushanth Sowmyan It looks like that problem is a symptom of a large one with the way Hive creates indexes from the CLI. The version of datanucleus we currently use just ignores a null value in the map so it works when the metastore is local, Thrift does not though, so this problem already causes problems when the metastore is remote.

          I filed HIVE-3722 to fix the problem, I think it should also fix the issue you're seeing.

          Show
          Kevin Wilfong added a comment - Sushanth Sowmyan It looks like that problem is a symptom of a large one with the way Hive creates indexes from the CLI. The version of datanucleus we currently use just ignores a null value in the map so it works when the metastore is local, Thrift does not though, so this problem already causes problems when the metastore is remote. I filed HIVE-3722 to fix the problem, I think it should also fix the issue you're seeing.
          Hide
          Sushanth Sowmyan added a comment -

          Kevin, yeah - the issues seem to be related.

          Show
          Sushanth Sowmyan added a comment - Kevin, yeah - the issues seem to be related.
          Hide
          Deepesh Khandelwal added a comment -

          After upgrading the datanucleus I ran into the runtime exception FCOMMENT invalid column name in COLUMNS_V2, this happens only when hive schema is pre-created with the upgrade SQL scripts (i.e. datanucleus.autoCreateSchema=false). So on MySQL, the workaround was to start with a blank schema and set datanucleus.autoCreateSchema=true but this turned out to be a problem on Oracle as the automatic creation of schema fails for the table TBLS (see HIVE-2928). To get around this on Oracle I pre-created the table TBLS and couple of other datanucleus tables (SEQUENCE_TABLE & NUCLEUS_TABLES) and let the server automatically create the other missing tables. If required, I can attach the patch for the SQL script I used for Oracle.

          Show
          Deepesh Khandelwal added a comment - After upgrading the datanucleus I ran into the runtime exception FCOMMENT invalid column name in COLUMNS_V2, this happens only when hive schema is pre-created with the upgrade SQL scripts (i.e. datanucleus.autoCreateSchema=false). So on MySQL, the workaround was to start with a blank schema and set datanucleus.autoCreateSchema=true but this turned out to be a problem on Oracle as the automatic creation of schema fails for the table TBLS (see HIVE-2928 ). To get around this on Oracle I pre-created the table TBLS and couple of other datanucleus tables (SEQUENCE_TABLE & NUCLEUS_TABLES) and let the server automatically create the other missing tables. If required, I can attach the patch for the SQL script I used for Oracle.
          Hide
          Navis added a comment -

          Is there any progress on this? With hiveserver2, I've seen 100% CPU problem (seemed to be http://www.datanucleus.org/servlet/jira/browse/NUCCORE-559) more frequently.

          Show
          Navis added a comment - Is there any progress on this? With hiveserver2, I've seen 100% CPU problem (seemed to be http://www.datanucleus.org/servlet/jira/browse/NUCCORE-559 ) more frequently.
          Hide
          Ashutosh Chauhan added a comment -

          Yeah.. I think we need to move forward on this. Based on Deepesh's comments and Ning's comment earlier in thread, it seems like we need to provide an upgrade script for folks who already have their metastore created with 2.x version of DN. Navis wondering if you have done some investigation on that ?

          Show
          Ashutosh Chauhan added a comment - Yeah.. I think we need to move forward on this. Based on Deepesh's comments and Ning's comment earlier in thread, it seems like we need to provide an upgrade script for folks who already have their metastore created with 2.x version of DN. Navis wondering if you have done some investigation on that ?
          Hide
          Navis added a comment -

          We've just upgraded datanucleus with a small step in two months ago and nothing bad happened till now.

          datanucleus-connectionpool-2.0.3.jar
          datanucleus-core-2.2.3.jar
          datanucleus-enhancer-2.1.3.jar
          datanucleus-rdbms-2.2.3.jar

          Datanucleus seemed too hard for me to investigate.

          Show
          Navis added a comment - We've just upgraded datanucleus with a small step in two months ago and nothing bad happened till now. datanucleus-connectionpool-2.0.3.jar datanucleus-core-2.2.3.jar datanucleus-enhancer-2.1.3.jar datanucleus-rdbms-2.2.3.jar Datanucleus seemed too hard for me to investigate.
          Hide
          Edward Capriolo added a comment -

          The metastore upgrades are not my favourite part of hive. Our metastore is a fairly large database in the 20GB. The upgrade scripts cause rather long read-only periods which make our applications very sad. I realize that we have to do them from time to time but they really are a bother.

          Show
          Edward Capriolo added a comment - The metastore upgrades are not my favourite part of hive. Our metastore is a fairly large database in the 20GB. The upgrade scripts cause rather long read-only periods which make our applications very sad. I realize that we have to do them from time to time but they really are a bother.
          Hide
          Edward Capriolo added a comment -

          I am actually under the option we should branch DN or switch to hibernate. I find it unbelievable that an object mapping framework is not compatible with itself between versions. What if you have large databases that can not accept downtime?

          Show
          Edward Capriolo added a comment - I am actually under the option we should branch DN or switch to hibernate. I find it unbelievable that an object mapping framework is not compatible with itself between versions. What if you have large databases that can not accept downtime?
          Hide
          Xuefu Zhang added a comment -

          I'm actually working on HIVE-3632, which is similar to this, yet having a different motivation. I'm yet to be convinced that a DB upgrade is required, and even if so, it seems unrelated to DN upgrade (but rather out of a bug in the metastore schema definition). That is yet to be seen, as review goes on for HIVE-3632.

          As far as Hibernate is concerned, I think it was excluded at very beginning due to incompatible license.

          Show
          Xuefu Zhang added a comment - I'm actually working on HIVE-3632 , which is similar to this, yet having a different motivation. I'm yet to be convinced that a DB upgrade is required, and even if so, it seems unrelated to DN upgrade (but rather out of a bug in the metastore schema definition). That is yet to be seen, as review goes on for HIVE-3632 . As far as Hibernate is concerned, I think it was excluded at very beginning due to incompatible license.
          Hide
          Sergey Shelukhin added a comment -

          As far as I saw from DN 2 and DN 3 databases, the schema upgrade might indeed be required. However, we do want to upgrade occasionally... DN 3 fixes a number of visible and less-visible bugs, so it will allow us to improve metastore in more ways.

          Show
          Sergey Shelukhin added a comment - As far as I saw from DN 2 and DN 3 databases, the schema upgrade might indeed be required. However, we do want to upgrade occasionally... DN 3 fixes a number of visible and less-visible bugs, so it will allow us to improve metastore in more ways.
          Hide
          Ashutosh Chauhan added a comment -

          This has been fixed via HIVE-3632

          Show
          Ashutosh Chauhan added a comment - This has been fixed via HIVE-3632
          Hide
          Ashutosh Chauhan added a comment -

          This issue has been fixed and released as part of 0.12 release. If you find further issues, please create a new jira and link it to this one.

          Show
          Ashutosh Chauhan added a comment - This issue has been fixed and released as part of 0.12 release. If you find further issues, please create a new jira and link it to this one.

            People

            • Assignee:
              Sushanth Sowmyan
              Reporter:
              Ning Zhang
            • Votes:
              0 Vote for this issue
              Watchers:
              26 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development