Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-8120 Umbrella JIRA tracking Parquet improvements
  3. HIVE-11131

Get row information on DataWritableWriter once for better writing performance

Log workAgile BoardRank to TopRank to BottomVotersWatch issueWatchersConvert to IssueMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.2.0
    • Fix Version/s: 2.0.0
    • Component/s: None
    • Labels:
      None

      Description

      DataWritableWriter is a class used to write Hive records to Parquet files. This class is getting all the information about how to parse a record, such as schema and object inspector, every time a record is written (or write() is called).

      We can make this class perform better by initializing some writers per data
      type once, and saving all object inspectors on each writer.

      The class expects that the next records written will have the same object inspectors and schema, so there is no need to have conditions for that. When a new schema is written, DataWritableWriter is created again by Parquet.

        Attachments

        1. HIVE-11131.2.patch
          28 kB
          Sergio Peña
        2. HIVE-11131.3.patch
          29 kB
          Sergio Peña
        3. HIVE-11131.4.patch
          29 kB
          Sergio Peña

          Activity

          $i18n.getText('security.level.explanation', $currentSelection) Viewable by All Users
          Cancel

            People

            • Assignee:
              spena Sergio Peña Assign to me
              Reporter:
              spena Sergio Peña

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment