[IMPALA-8003] Dataload should detect diskspace issues and give a reasonable error

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: Impala 3.2.0
• Component/s: Infrastructure
• Epic Color: ghx-label-3
• Assignee: Joe McDonnell
• Reporter: Joe McDonnell
• Created: 19/Dec/18 01:12
• Updated: 19/Dec/18 01:12

Description

The minicluster dataload can fail when IMPALA_HOME runs out of disk space (or rather, HDFS believes it has run out of unused disk space). The output from bin/load-data.py tells people to look at the SQL logs in data_loading/sql/. The errors that show up in the SQL logs are useless:

Error: Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask (state=08S01,code=2)
java.sql.SQLException: Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask

There is no indication of where to look or what might be wrong. This is especially confusing because it can happen when it looks like there is a lot of free disk space.

The actual useful information for this case is in the HDFS NameNode log, which contains lines like:

3688:2018-12-18 14:59:29,594 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy: Not enough replicas was chosen. Reason:{NOT_ENOUGH_STORAGE_SPACE=2}
3689:2018-12-18 14:59:29,594 WARN org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy: Failed to place enough replicas, still in need of 3 to reach 3 (unavailableStorages=[], storagePolicy=BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]}, newBlock=true) For more information, please enable DEBUG log level on org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy and org.apache.hadoop.net.NetworkTopology

On failure, dataload should check to see if it hit this type of diskspace issue and provide intelligible output.