[HIVE-8595] Slow IO when stdout directed to NAS with large blocksize

• Type: Bug
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 0.13.1
• Component/s: Clients

Environment

nfs4 rsize=1048576,wsize=1048576

• Assignee: Unassigned
• Reporter: Kevin English
• Created: 24/Oct/14 16:01
• Updated: 24/Oct/14 16:01

Description

Very slow IO when executing a SQL command file using the following command line when the target file system is an nfs4 mounted NAS with a large blocksize:

hive -f sqlscript.sql 2>results.log >results.tab

Work around (thousands of times faster):

hive -f sqlscript.sql 2>results.log | cat >results.tab

For instance I had a command finish 10 hours ago and I forgot to use cat and it is still writing out the output which after 10 hours is in the 180 GB range.