Uploaded image for project: 'Parquet'
  1. Parquet
  2. PARQUET-668

Provide option to disable auto crop feature in DumpCommand output

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Trivial
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.9.0
    • Component/s: parquet-mr
    • Labels:
      None
    • Flags:
      Patch

      Description

      Problem

      When using the dump command in parquet-tools, the output will sometimes be truncated based on the width of your console, especially on smaller displays.

      Example:

      row group 0
      --------------------------------------------------------------------------------
      id:          INT32 SNAPPY DO:0 FPO:4 SZ:44668/920538/20.61 VC:7240100  [more]...
      name:        BINARY SNAPPY DO:0 FPO:44672 SZ:89464018/1031768430/11.53 [more]...
      event_time:  INT64 SNAPPY DO:0 FPO:89508690 SZ:43600235/57923935/1.33 VC:7240100 [more]...
      
          id TV=7240100 RL=0 DL=0 DS: 2 DE:PLAIN_DICTIONARY
          ----------------------------------------------------------------------------
          page 0:                      DLE:BIT_PACKED RLE:BIT_PACKED VLE:PLA [more]... SZ:33291
      

      This is especially annoying if you pipe the output to a file as the truncation remains in place.

      Proposed fix

      Provide the flag --disable-crop for the dump command. Truncation is enabled by default and will only be disabled when this flag is provided,

      This will output the full content to standard out, for example:

      row group 0
      --------------------------------------------------------------------------------
      id:          INT32 SNAPPY DO:0 FPO:4 SZ:44668/920538/20.61 VC:7240100 ENC:BIT_PACKED,PLAIN_DICTIONARY
      name:        BINARY SNAPPY DO:0 FPO:44672 SZ:89464018/1031768430/11.53 VC:7240100 ENC:PLAIN,BIT_PACKED
      event_time:  INT64 SNAPPY DO:0 FPO:89508690 SZ:43600235/57923935/1.33 VC:7240100 ENC:PLAIN,BIT_PACKED,RLE
      
          id TV=7240100 RL=0 DL=0 DS: 2 DE:PLAIN_DICTIONARY
          ----------------------------------------------------------------------------
          page 0:                      DLE:BIT_PACKED RLE:BIT_PACKED VLE:PLAIN_DICTIONARY ST:[min: 0, max: 1, num_nulls: 0] SZ:33291 VC:262146
          page 1:                      DLE:BIT_PACKED RLE:BIT_PACKED VLE:PLAIN_DICTIONARY ST:[min: 0, max: 1, num_nulls: 0] SZ:33291 VC:262145
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                djhworld Daniel Harper
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: