[MAPREDUCE-5125] TestDFSIO should write less compressible data

• Type: Bug
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 2.0.3-alpha, 1.1.2
• Component/s: test
• Target Version/s: 2.1.0-beta, 1.3.0
• Assignee: Unassigned
• Reporter: Todd Lipcon
• Created: 03/Apr/13 17:38
• Updated: 09/Sep/13 05:28

Description

Currently, TestDFSIO writes a short repeating string of sequential (byte)0 through (byte)50. This makes its output very compressible (I measured 250:1 by LZOing the resulting file). This makes the results of TestDFSIO very hard to compare when running on HDFS vs other file systems which may include some compression on the network, disk, or both – what is ostensibly a benchmark of IO throughput yields completely skewed results towards the system with compression.

Issue Links

• relates to: MAPREDUCE-5491 DFSIO do not initialize write buffer correctly