Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-8120 Umbrella JIRA tracking Parquet improvements
  3. HIVE-9692

Allocate only parquet selected columns in HiveStructConverter class

Log workAgile BoardRank to TopRank to BottomBulk Copy AttachmentsBulk Move AttachmentsAdd voteVotersWatch issueWatchersConvert to IssueMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Patch Available
    • Major
    • Resolution: Unresolved
    • None
    • None
    • None
    • None

    Description

      HiveStructConverter class is where Hive converts parquet objects to hive writable objects that will be later parsed by object inspectors. This class is allocating enough writable objects as number of columns of the file schema.

      ublic HiveStructConverter(final GroupType requestedSchema, final GroupType tableSchema, Map<String, String> metadata) {
      ...
      this.writables = new Writable[fileSchema.getFieldCount()];
      ...
      }
      

      This is always allocated even if we only select a specific number of columns. Let's say 2 columns from a table of 50 columns. 50 objects are allocated. Only 2 are used, and 48 are unused.

      We should be able to allocate only the requested number of columns in order to save memory usage.

      Attachments

        1. HIVE-9692.1.patch
          24 kB
          Sergio Peña

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            spena Sergio Peña Assign to me
            spena Sergio Peña

            Dates

              Created:
              Updated:

              Slack

                Issue deployment