Uploaded image for project: 'Parquet'
  1. Parquet
  2. PARQUET-1491

Conditional debug logging in InternalParquetRecordReader to reduce GC

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Not A Problem
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: parquet-avro

      Description

      Currently there is no check for the log level in https://github.com/apache/parquet-mr/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/InternalParquetRecordReader.javaL249 which causes a lot of memory allocation and performance degradation.

      Link to parquet file which was used: https://drive.google.com/open?id=1xCMZrUPWvlS4KOFO8m9EmtkvDy-SiRHq

      Screenshot of Java Mission Control comparison with fix and without (link to the JFR files https://drive.google.com/open?id=1blSeF-AyAhQyRYaqVsihyzy7pJCJt7U3):

        Attachments

        1. image-2019-01-12-04-03-48-005.png
          278 kB
          Artavazd Balaian
        2. image-2019-01-12-04-09-18-359.png
          247 kB
          Artavazd Balaian
        3. image-2019-01-12-04-10-49-230.png
          248 kB
          Artavazd Balaian

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                REASY Artavazd Balaian
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: