[IMPALA-2814] Impala daemons crash with large Snappy compressed text files

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Duplicate
• Affects Version/s: Impala 2.2
• Component/s: Backend
• Labels: crash resource-management
• Assignee: Unassigned
• Reporter: Ian Buss
• Created: 06/Jan/16 16:27
• Updated: 26/Nov/19 09:37
• Resolved: 20/Jan/16 00:50

Description

When querying a text based table with large Snappy compressed text files (leaving aside the fact that this is an anti-pattern and bad practice for now), the query will either fail with the following error (single 1GB file):

Decompressor: block size is too big. Data is likely corrupt. Size: 4796145851

Or, if the files are big enough will crash Impala daemons, as per the following scenario.

A test table with 4 large Snappy compressed text files (from a Sqoop import in this case).

ibuss@testhost:~$ hadoop fs -ls /user/hive/warehouse/test
Found 5 items
-rw-r-----+  2 testuser hadoop          0 2016-01-06 10:50 /user/hive/warehouse/test/_SUCCESS
-rw-r-----+  2 testuser hadoop 5983742424 2016-01-06 10:47 /user/hive/warehouse/test/part-m-00000
-rw-r-----+  2 testuser hadoop 5794945077 2016-01-06 10:49 /user/hive/warehouse/test/part-m-00001
-rw-r-----+  2 testuser hadoop 5713911732 2016-01-06 10:48 /user/hive/warehouse/test/part-m-00002
-rw-r-----+  2 testuser hadoop 6086993909 2016-01-06 10:50 /user/hive/warehouse/test/part-m-00003

With a simple select count( * ) query the Impala daemons crash. With the debug build the following is in the INFO log.

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007fa932c13d10, pid=18533, tid=140361029306112
#
# JRE version: Java(TM) SE Runtime Environment (7.0_67-b01) (build 1.7.0_67-b01)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (24.65-b04 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C  [libc.so.6+0x89d10]  memcpy+0x3c0
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /var/run/cloudera-scm-agent/process/7406-impala-IMPALAD/hs_err_pid18533.log
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.sun.com/bugreport/crash.jsp
#

There are workarounds such as reimporting the data from Sqoop with more mappers or rewriting the data in Hive per below, but ideally Impala should not crash.

Workaround in Hive:

SET hive.exec.compress.output=true;
SET mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.SnappyCodec;
SET mapreduce.output.fileoutputformat.compress=true;
SET hive.exec.reducers.bytes.per.reducer=134217728;}}
 
insert overwrite table default.test_dist select * from default.test distribute by abs(hash(uniquefield)) % 55;

 
ibuss@testhost:~$ hadoop fs -ls /user/hive/warehouse/test_dist
Found 54 items
-rwxrwxrwx+  2 hive hive  277937880 2016-01-06 15:33 /user/hive/warehouse/test_dist/000000_0.snappy
-rwxrwxrwx+  2 hive hive  139569820 2016-01-06 15:32 /user/hive/warehouse/test_dist/000001_0.snappy
-rwxrwxrwx+  2 hive hive  139309664 2016-01-06 15:32 /user/hive/warehouse/test_dist/000002_1.snappy
-rwxrwxrwx+  2 hive hive  138868524 2016-01-06 15:30 /user/hive/warehouse/test_dist/000003_0.snappy
-rwxrwxrwx+  2 hive hive  138779985 2016-01-06 15:33 /user/hive/warehouse/test_dist/000004_0.snappy
-rwxrwxrwx+  2 hive hive  139158798 2016-01-06 15:31 /user/hive/warehouse/test_dist/000005_0.snappy
-rwxrwxrwx+  2 hive hive  139348267 2016-01-06 15:33 /user/hive/warehouse/test_dist/000006_0.snappy
-rwxrwxrwx+  2 hive hive  139060534 2016-01-06 15:31 /user/hive/warehouse/test_dist/000007_1.snappy
-rwxrwxrwx+  2 hive hive  139158887 2016-01-06 15:32 /user/hive/warehouse/test_dist/000008_0.snappy
-rwxrwxrwx+  2 hive hive  139033052 2016-01-06 15:31 /user/hive/warehouse/test_dist/000009_0.snappy
-rwxrwxrwx+  2 hive hive  139032714 2016-01-06 15:32 /user/hive/warehouse/test_dist/000010_1.snappy
-rwxrwxrwx+  2 hive hive  138863015 2016-01-06 15:31 /user/hive/warehouse/test_dist/000011_0.snappy
-rwxrwxrwx+  2 hive hive  138683137 2016-01-06 15:33 /user/hive/warehouse/test_dist/000012_0.snappy
-rwxrwxrwx+  2 hive hive  139172288 2016-01-06 15:32 /user/hive/warehouse/test_dist/000013_0.snappy
-rwxrwxrwx+  2 hive hive  139266352 2016-01-06 15:31 /user/hive/warehouse/test_dist/000014_0.snappy
-rwxrwxrwx+  2 hive hive  139611179 2016-01-06 15:30 /user/hive/warehouse/test_dist/000015_0.snappy
-rwxrwxrwx+  2 hive hive  139389695 2016-01-06 15:32 /user/hive/warehouse/test_dist/000016_1.snappy
-rwxrwxrwx+  2 hive hive  138995277 2016-01-06 15:31 /user/hive/warehouse/test_dist/000017_0.snappy
-rwxrwxrwx+  2 hive hive  139499673 2016-01-06 15:33 /user/hive/warehouse/test_dist/000018_0.snappy
-rwxrwxrwx+  2 hive hive  139556837 2016-01-06 15:32 /user/hive/warehouse/test_dist/000019_0.snappy
-rwxrwxrwx+  2 hive hive  139144560 2016-01-06 15:32 /user/hive/warehouse/test_dist/000020_0.snappy
-rwxrwxrwx+  2 hive hive  139155260 2016-01-06 15:31 /user/hive/warehouse/test_dist/000021_0.snappy
-rwxrwxrwx+  2 hive hive  139698863 2016-01-06 15:33 /user/hive/warehouse/test_dist/000022_0.snappy
-rwxrwxrwx+  2 hive hive  139030297 2016-01-06 15:31 /user/hive/warehouse/test_dist/000023_0.snappy
-rwxrwxrwx+  2 hive hive  139438501 2016-01-06 15:33 /user/hive/warehouse/test_dist/000024_0.snappy
-rwxrwxrwx+  2 hive hive  139102292 2016-01-06 15:32 /user/hive/warehouse/test_dist/000025_1.snappy
-rwxrwxrwx+  2 hive hive  139223258 2016-01-06 15:32 /user/hive/warehouse/test_dist/000026_0.snappy
-rwxrwxrwx+  2 hive hive  139000792 2016-01-06 15:31 /user/hive/warehouse/test_dist/000027_0.snappy
-rwxrwxrwx+  2 hive hive  139459790 2016-01-06 15:32 /user/hive/warehouse/test_dist/000028_0.snappy
-rwxrwxrwx+  2 hive hive  139299463 2016-01-06 15:31 /user/hive/warehouse/test_dist/000029_1.snappy
-rwxrwxrwx+  2 hive hive  139157316 2016-01-06 15:33 /user/hive/warehouse/test_dist/000030_0.snappy
-rwxrwxrwx+  2 hive hive  139313649 2016-01-06 15:32 /user/hive/warehouse/test_dist/000031_0.snappy
-rwxrwxrwx+  2 hive hive  139450113 2016-01-06 15:31 /user/hive/warehouse/test_dist/000032_1.snappy
-rwxrwxrwx+  2 hive hive  139485608 2016-01-06 15:31 /user/hive/warehouse/test_dist/000033_0.snappy
-rwxrwxrwx+  2 hive hive  139079411 2016-01-06 15:33 /user/hive/warehouse/test_dist/000034_0.snappy
-rwxrwxrwx+  2 hive hive  139110358 2016-01-06 15:31 /user/hive/warehouse/test_dist/000035_0.snappy
-rwxrwxrwx+  2 hive hive  139123786 2016-01-06 15:33 /user/hive/warehouse/test_dist/000036_0.snappy
-rwxrwxrwx+  2 hive hive  139728110 2016-01-06 15:32 /user/hive/warehouse/test_dist/000037_0.snappy
-rwxrwxrwx+  2 hive hive  138634329 2016-01-06 15:31 /user/hive/warehouse/test_dist/000038_0.snappy
-rwxrwxrwx+  2 hive hive  139197631 2016-01-06 15:30 /user/hive/warehouse/test_dist/000039_0.snappy
-rwxrwxrwx+  2 hive hive  139506852 2016-01-06 15:33 /user/hive/warehouse/test_dist/000040_0.snappy
-rwxrwxrwx+  2 hive hive  139219899 2016-01-06 15:31 /user/hive/warehouse/test_dist/000041_0.snappy
-rwxrwxrwx+  2 hive hive  139075418 2016-01-06 15:33 /user/hive/warehouse/test_dist/000042_0.snappy
-rwxrwxrwx+  2 hive hive  139400592 2016-01-06 15:32 /user/hive/warehouse/test_dist/000043_0.snappy
-rwxrwxrwx+  2 hive hive  139109680 2016-01-06 15:32 /user/hive/warehouse/test_dist/000044_0.snappy
-rwxrwxrwx+  2 hive hive  139055164 2016-01-06 15:31 /user/hive/warehouse/test_dist/000045_0.snappy
-rwxrwxrwx+  2 hive hive  139403664 2016-01-06 15:33 /user/hive/warehouse/test_dist/000046_0.snappy
-rwxrwxrwx+  2 hive hive  139456941 2016-01-06 15:31 /user/hive/warehouse/test_dist/000047_0.snappy
-rwxrwxrwx+  2 hive hive  138798569 2016-01-06 15:33 /user/hive/warehouse/test_dist/000048_0.snappy
-rwxrwxrwx+  2 hive hive  139164351 2016-01-06 15:32 /user/hive/warehouse/test_dist/000049_0.snappy
-rwxrwxrwx+  2 hive hive  139248702 2016-01-06 15:32 /user/hive/warehouse/test_dist/000050_0.snappy
-rwxrwxrwx+  2 hive hive  139402638 2016-01-06 15:31 /user/hive/warehouse/test_dist/000051_0.snappy
-rwxrwxrwx+  2 hive hive  139111620 2016-01-06 15:33 /user/hive/warehouse/test_dist/000052_0.snappy
-rwxrwxrwx+  2 hive hive  139366083 2016-01-06 15:31 /user/hive/warehouse/test_dist/000053_0.snappy

In Impala after invalidate metadata:

Query: select count(*) from test_dist
+----------+
| count(*) |
+----------+
| 16905575 |
+----------+

Of course there are the following (correct) warnings. These are interim tables in an ETL process - Parquet is the ultimate destination.

For better performance, snappy, gzip and bzip-compressed files should not be split into multiple hdfs-blocks. file=hdfs://testhost2:8020/user/hive/warehouse/test_dist/000048_0.snappy offset 134217728

Issue Links

• duplicates: IMPALA-1619 Support single allocations larger 1GB