Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
None
-
None
Description
A heap dump obtained from one of our users shows that 15% of memory is wasted on duplicate strings, despite the recent optimizations that I made. The problematic strings just come from different sources this time. See the excerpt from the jxray (www.jxray.com) analysis attached.
Adding String.intern() calls in the appropriate places reduces the overhead of duplicate strings with this workload to ~6%. The remaining duplicates come mostly from JDK internal and MapReduce data structures, and thus are more difficult to fix.
Attachments
Attachments
- ch_2_excerpt.txt
- 5 kB
- Misha Dmitriev
- HIVE-16166.01.patch
- 10 kB
- Misha Dmitriev
- HIVE-16166.02.patch
- 11 kB
- Misha Dmitriev
Issue Links
- relates to
-
HIVE-15882 HS2 generating high memory pressure with many partitions and concurrent queries
- Resolved
Activity
misha@cloudera.com Could you click on the Submit Patch button so that it triggers the HiveQA?
misha@cloudera.com HiveQA already finished, but it could not post the results on the JIRA.
Here are the results:
https://builds.apache.org/view/H-L/view/Hive/job/PreCommit-HIVE-Build/4164/
Test Result (10 failures / +10)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[dynamic_partition_pruning_2]
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_partitioned]
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_dynamic_partition_pruning]
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_part]
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynamic_partition_pruning]
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_part_varchar]
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynamic_semijoin_reduction]
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[special_character_in_tabnames_1]
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[metadataonly1]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[vector_join_part_col_char]
spena I ran all these tests locally, and they passed for me. So this looks like flakiness in the Hive build, which is not uncommon.
I was looking into HIVE-15058 to see if those flaky tests were already reported, but they're not. I will retry HiveQA again to see if those flaky tests go away. I know these might not related to the patch, but seems we have green tests, I would like to give it another shot just to see a clean test result.
I just trigger the jenkins job (#4197).
See https://builds.apache.org/view/H-L/view/Hive/job/PreCommit-HIVE-Build/
Hm, the same tests failed again. Now it starts to look suspicious. I'll investigate...
misha@cloudera.com Here are the logs in case you want to look at them:
http://104.198.109.242/logs/PreCommit-HIVE-Build-4197/
spena thank you very much for the logs. I found that the failures occur in my code with the stack trace like this:
java.lang.UnsupportedOperationException at java.util.AbstractList.set(AbstractList.java:132) at java.util.AbstractList$ListItr.set(AbstractList.java:426) at org.apache.hadoop.hive.common.StringInternUtils.internStringsInList(StringInternUtils.java:112) at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory.getStandardStructObjectInspector(ObjectInspectorFactory.java:320) at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory.getStandardStructObjectInspector(ObjectInspectorFactory.java:312) ...
The piece of StringUtils.java where this is thrown looks like this:
ListIterator<String> it = list.listIterator(); while (it.hasNext()) { it.set(it.next().intern()); }
This is the standard, official way which one can use to replace elements in any List implemented in JDK, e.g. ArrayList or LinkedList. For both of them, listIterator() returns an Iterator that correctly implements the set() operation. So if the code throws an exception, my guess is that it received some List (probably not from JDK) that doesn't provide the proper Iterator implementation.
Now, I think there are two alternatives for dealing with this problem. The first is to try to find the problematic List implementation (if it's in Hive) and fix it. This is complicated, given that the stack trace doesn't show the problematic List subclass upfront, and I for some reason cannot reproduce this problem locally. But in any case, even if this is fixed, it doesn't guarantee that in the future somebody will not write another incomplete List implementation that will cause this problem again. So probably a better solution is to just catch the UnsupportedOperationException in my code and return as if nothing happened. After all, string interning is a performance optimization, it doesn't affect the application semantics, so if it doesn't always work as expected, it's not a serious problem. What do you think?
Fixed the problem with a List not providing the proper Iterator.set() functionality as explained above, i.e. by catching and ignoring UnsupportedOperationException.
Thanks. As you mentioned, this is a performance optimization, and if it does not work for some reason, then we can still rely on the normal way without using intern strings.
+1
Let's wait for HiveQA.
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12859609/HIVE-16166.02.patch
ERROR: -1 due to no test(s) being added or modified.
ERROR: -1 due to 2 failed/errored test(s), 10480 tests executed
Failed tests:
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[comments] (batchId=35) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr] (batchId=141)
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4250/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4250/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4250/
Messages:
Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 2 tests failed
This message is automatically generated.
ATTACHMENT ID: 12859609 - PreCommit-HIVE-Build
The testCliDriver[comments] test fails on other patches as well. This is not related to your patch, and is already reported.
Could you check only if {[testCliDriver[vector_if_expr]}} is related? I don't see it failing on previous Jenkins builds.
I ran 'mvn test -Dtest=TestMiniLlapLocalCliDriver -Dqfile=vector_if_expr.q' locally, and it passed.
I then checked the hive log at http://104.198.109.242/logs/PreCommit-HIVE-Build-4250/failed/141-TestMiniLlapLocalCliDriver-skewjoinopt15.q-vector_coalesce.q-orc_ppd_decimal.q-and-27-more/logs/hive.log It does have a bunch of exception stack traces, but it doesn't look like they are related with my changes. At least I don't see 'StringInternUtils' (my class where an NPE or some such is most likely to happen), and a bunch of NPEs all across this log are all of the same type and have no traces of the code that I've modified. I can't see where in this log the problematic test (vector_if_expr) starts, or do all the tests run in parallel?
I found this jira HIVE-15776 that mentions the vector_if_expr is a flaky test.
misha@cloudera.com Thanks for your contribution. I committed this to master.
Results of jxray analysis for duplicate strings, baseline code.