[IMPALA-10316] load_nested.py failed due to out of memory during Jenkins GVO

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Component/s: Infrastructure
• Labels: broken-build flaky
• Epic Color: ghx-label-12
• Assignee: Michael Smith
• Reporter: Zoltán Borók-Nagy
• Created: 11/Nov/20 09:55
• Updated: 21/Jun/22 17:35
• Resolved: 20/Jun/22 23:00

Description

The following job failed due to out of memory:

https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/12588 (please click on "Don't keep this build forever" once this issue is resolved)

Relevant log lines:

02:33:42 Loading nested orc data (logging to /home/ubuntu/Impala/logs/data_loading/load-nested.log)... 
02:35:39     FAILED (Took: 1 min 57 sec)
02:35:39     '/home/ubuntu/Impala/testdata/bin/load_nested.py -t tpch_nested_orc_def -f orc/def' failed. Tail of log:
02:35:39 2020-11-11 02:35:06,225 INFO:load_nested[348]:Executing: 
02:35:39 
02:35:39       CREATE EXTERNAL TABLE supplier
02:35:39       STORED AS orc
02:35:39       TBLPROPERTIES('orc.compress' = 'ZLIB','external.table.purge'='TRUE')
02:35:39       AS SELECT * FROM tmp_supplier
02:35:39 Traceback (most recent call last):
02:35:39   File "/home/ubuntu/Impala/testdata/bin/load_nested.py", line 415, in <module>
02:35:39     load()
02:35:39   File "/home/ubuntu/Impala/testdata/bin/load_nested.py", line 349, in load
02:35:39     hive.execute(stmt)
02:35:39   File "/home/ubuntu/Impala/tests/comparison/db_connection.py", line 206, in execute
02:35:39     return self._cursor.execute(sql, *args, **kwargs)
02:35:39   File "/home/ubuntu/Impala/infra/python/env-gcc7.5.0/lib/python2.7/site-packages/impala/hiveserver2.py", line 331, in execute
02:35:39     self._wait_to_finish()  # make execute synchronous
02:35:39   File "/home/ubuntu/Impala/infra/python/env-gcc7.5.0/lib/python2.7/site-packages/impala/hiveserver2.py", line 413, in _wait_to_finish
02:35:39     raise OperationalError(resp.errorMessage)
02:35:39 impala.error.OperationalError: Error while compiling statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 1, vertexId=vertex_1605060173780_0039_2_00, diagnostics=[Task failed, taskId=task_1605060173780_0039_2_00_000000, diagnostics=[TaskAttempt 0 failed, info=[Container container_1605060173780_0039_01_000002 finished with diagnostics set to [Container failed, exitCode=-104. [2020-11-11 02:35:11.768]Container [pid=16810,containerID=container_1605060173780_0039_01_000002] is running 7729152B beyond the 'PHYSICAL' memory limit. Current usage: 1.0 GB of 1 GB physical memory used; 2.5 GB of 2.1 GB virtual memory used. Killing container.

Issue Links

• is duplicated by: IMPALA-10669 Loading nested ORC data is flaky during Docker-based tests
• is related to: IMPALA-10503 testdata load hits hive memory limit errors during hive inserts
• is related to: IMPALA-10669 Loading nested ORC data is flaky during Docker-based tests
• relates to: HIVE-26346 Default Tez memory limits occasionally result in killing container