Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-4412

Parquet logger cannot be configured

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Incomplete
    • Affects Version/s: 1.2.0, 1.3.1
    • Fix Version/s: None
    • Component/s: SQL
    • Labels:

      Description

      The Spark ParquetRelation.scala code makes the assumption that the parquet.Log class has already been loaded. If ParquetRelation.enableLogForwarding executes prior to the parquet.Log class being loaded then the code in enableLogForwarding has no affect.

      ParquetRelation.scala attempts to override the parquet logger but, at least currently (and if your application simply reads a parquet file before it does anything else with Parquet), the parquet.Log class hasn't been loaded yet. Therefore the code in ParquetRelation.enableLogForwarding has no affect. If you look at the code in parquet.Log there's a static initializer that needs to be called prior to enableLogForwarding or whatever enableLogForwarding does gets undone by this static initializer.

      The "fix" would be to force the static initializer to get called in parquet.Log as part of enableForwardLogging.

      PR will be forthcomming.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              jimfcarroll Jim Carroll
            • Votes:
              5 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: