[PARQUET-668] Provide option to disable auto crop feature in DumpCommand output

• Type: Improvement
• Status: Resolved
• Priority: Trivial
• Resolution: Fixed
• Fix Version/s: 1.9.0
• Component/s: parquet-mr
• Flags: Patch
• Assignee: Unassigned
• Reporter: Daniel Harper
• Created: 31/Jul/16 17:02
• Updated: 03/Aug/16 21:15
• Resolved: 03/Aug/16 21:15

Description

Problem

When using the dump command in parquet-tools, the output will sometimes be truncated based on the width of your console, especially on smaller displays.

Example:

row group 0
--------------------------------------------------------------------------------
id:          INT32 SNAPPY DO:0 FPO:4 SZ:44668/920538/20.61 VC:7240100  [more]...
name:        BINARY SNAPPY DO:0 FPO:44672 SZ:89464018/1031768430/11.53 [more]...
event_time:  INT64 SNAPPY DO:0 FPO:89508690 SZ:43600235/57923935/1.33 VC:7240100 [more]...

    id TV=7240100 RL=0 DL=0 DS: 2 DE:PLAIN_DICTIONARY
    ----------------------------------------------------------------------------
    page 0:                      DLE:BIT_PACKED RLE:BIT_PACKED VLE:PLA [more]... SZ:33291

This is especially annoying if you pipe the output to a file as the truncation remains in place.

Proposed fix

Provide the flag --disable-crop for the dump command. Truncation is enabled by default and will only be disabled when this flag is provided,

This will output the full content to standard out, for example:

row group 0
--------------------------------------------------------------------------------
id:          INT32 SNAPPY DO:0 FPO:4 SZ:44668/920538/20.61 VC:7240100 ENC:BIT_PACKED,PLAIN_DICTIONARY
name:        BINARY SNAPPY DO:0 FPO:44672 SZ:89464018/1031768430/11.53 VC:7240100 ENC:PLAIN,BIT_PACKED
event_time:  INT64 SNAPPY DO:0 FPO:89508690 SZ:43600235/57923935/1.33 VC:7240100 ENC:PLAIN,BIT_PACKED,RLE

    id TV=7240100 RL=0 DL=0 DS: 2 DE:PLAIN_DICTIONARY
    ----------------------------------------------------------------------------
    page 0:                      DLE:BIT_PACKED RLE:BIT_PACKED VLE:PLAIN_DICTIONARY ST:[min: 0, max: 1, num_nulls: 0] SZ:33291 VC:262146
    page 1:                      DLE:BIT_PACKED RLE:BIT_PACKED VLE:PLAIN_DICTIONARY ST:[min: 0, max: 1, num_nulls: 0] SZ:33291 VC:262145

Issue Links

• links to: Pull Request