Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.8.0
    • Component/s: IO
    • Labels:
      None

      Description

      Add a source and target for Parquet files.

      1. CRUNCH-277-Add-licensing-info-for-parquet.patch
        1.0 kB
        Matthias Friedrich
      2. CRUNCH-277.patch
        33 kB
        Tom White
      3. CRUNCH-277.patch
        35 kB
        Tom White

        Activity

        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Resolved Resolved
        1d 14h 15m 1 Tom White 09/Oct/13 12:06
        Resolved Resolved Closed Closed
        30d 10h 18m 1 Josh Wills 08/Nov/13 21:24
        Josh Wills made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Hide
        Tom White added a comment -

        Thanks for fixing it Matthias.

        Show
        Tom White added a comment - Thanks for fixing it Matthias.
        Hide
        Matthias Friedrich added a comment -

        OK, pushed to master. Thanks, Tom and Josh!

        Show
        Matthias Friedrich added a comment - OK, pushed to master. Thanks, Tom and Josh!
        Hide
        Josh Wills added a comment -

        +1-- thanks Matthias!

        Show
        Josh Wills added a comment - +1-- thanks Matthias!
        Matthias Friedrich made changes -
        Hide
        Matthias Friedrich added a comment -

        Yep, NOTICE needs to change and there's also a somewhat hidden section in LICENSE (see attached patch).

        Show
        Matthias Friedrich added a comment - Yep, NOTICE needs to change and there's also a somewhat hidden section in LICENSE (see attached patch).
        Hide
        Tom White added a comment -

        I didn't - thanks for the reminder, Matthias! I think we need to add the following to NOTICE:

        This product includes Parquet under the Apache License, Version 2.0
        See http://parquet.io/
        
        Show
        Tom White added a comment - I didn't - thanks for the reminder, Matthias! I think we need to add the following to NOTICE: This product includes Parquet under the Apache License, Version 2.0 See http://parquet.io/
        Hide
        Matthias Friedrich added a comment -

        Sorry for being late, but since this introduces a new dependency: Did you check whether we need to update our binary distribution's licensing information?

        Show
        Matthias Friedrich added a comment - Sorry for being late, but since this introduces a new dependency: Did you check whether we need to update our binary distribution's licensing information?
        Tom White made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Fix Version/s 0.8.0 [ 12324802 ]
        Resolution Fixed [ 1 ]
        Hide
        Tom White added a comment -

        I just committed this.

        Show
        Tom White added a comment - I just committed this.
        Hide
        Josh Wills added a comment -

        +1-- still LG.

        Show
        Josh Wills added a comment - +1-- still LG.
        Tom White made changes -
        Attachment CRUNCH-277.patch [ 12607346 ]
        Hide
        Tom White added a comment -

        Thanks for reviewing the patch!

        I found that the Parquet source is not compatible with CombineFileInputFormat, since the ParquetRecordReader expects a ParquetInputSplit (which encodes Parquet block information), rather than a regular FileSplit. To fix this I've disabled the use of combine files for the Parquet source, and added a new test to verify it works.

        Show
        Tom White added a comment - Thanks for reviewing the patch! I found that the Parquet source is not compatible with CombineFileInputFormat, since the ParquetRecordReader expects a ParquetInputSplit (which encodes Parquet block information), rather than a regular FileSplit. To fix this I've disabled the use of combine files for the Parquet source, and added a new test to verify it works.
        Hide
        Gabriel Reid added a comment -

        +1

        Show
        Gabriel Reid added a comment - +1
        Hide
        Josh Wills added a comment -

        +1-- many thanks, Tom!

        Show
        Josh Wills added a comment - +1-- many thanks, Tom!
        Tom White made changes -
        Field Original Value New Value
        Attachment CRUNCH-277.patch [ 12607232 ]
        Hide
        Tom White added a comment -

        Here's a patch for Parquet support for in-memory Avro types.

        Show
        Tom White added a comment - Here's a patch for Parquet support for in-memory Avro types.
        Tom White created issue -

          People

          • Assignee:
            Tom White
            Reporter:
            Tom White
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development