Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.2.0
    • Fix Version/s: 1.3.0, 2.0.0
    • Component/s: None
    • Labels:
      None

      Description

      Parquet nested types are using an extra wrapper object (ArrayWritable) as a wrapper of map and list elements. This extra object is not needed and causing unnecessary memory allocations.

      An example of code is on HiveCollectionConverter.java:

      public void end() {
          parent.set(index, wrapList(new ArrayWritable(
              Writable.class, list.toArray(new Writable[list.size()]))));
      }
      

      This object is later unwrapped on AbstractParquetMapInspector, i.e.:

      final Writable[] mapContainer = ((ArrayWritable) data).get();
      final Writable[] mapArray = ((ArrayWritable) mapContainer[0]).get();
      for (final Writable obj : mapArray) {
        ...
      }
      

      We should get rid of this wrapper object to save time and memory.

        Attachments

        1. HIVE-9605.3.patch
          35 kB
          Sergio Peña
        2. HIVE-9605.4.patch
          28 kB
          Sergio Peña
        3. HIVE-9605.5.patch
          28 kB
          Sergio Peña
        4. HIVE-9605.6.patch
          30 kB
          Sergio Peña

          Issue Links

            Activity

              People

              • Assignee:
                spena Sergio Peña
                Reporter:
                spena Sergio Peña
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: