[SPARK-30168] Eliminate warnings in Parquet datasource

• Type: Sub-task
• Status: In Progress
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 3.1.0
• Component/s: SQL
• Assignee: Unassigned
• Reporter: Max Gekk
• Created: 08/Dec/19 20:24
• Updated: 09/May/20 08:12

Description

1. sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/parquet/ParquetPartitionReaderFactory.scala

   Warning:Warning:line (120)class ParquetInputSplit in package hadoop is deprecated: see corresponding Javadoc for more information.
             Option[TimeZone]) => RecordReader[Void, T]): RecordReader[Void, T] = {
       Warning:Warning:line (125)class ParquetInputSplit in package hadoop is deprecated: see corresponding Javadoc for more information.
         new org.apache.parquet.hadoop.ParquetInputSplit(
       Warning:Warning:line (134)method readFooter in class ParquetFileReader is deprecated: see corresponding Javadoc for more information.
         ParquetFileReader.readFooter(conf, filePath, SKIP_ROW_GROUPS).getFileMetaData
       Warning:Warning:line (183)class ParquetInputSplit in package hadoop is deprecated: see corresponding Javadoc for more information.
         split: ParquetInputSplit,
       Warning:Warning:line (212)class ParquetInputSplit in package hadoop is deprecated: see corresponding Javadoc for more information.
         split: ParquetInputSplit,
   

2. sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/SpecificParquetRecordReaderBase.java

   Warning:Warning:line (55)java: org.apache.parquet.hadoop.ParquetInputSplit in org.apache.parquet.hadoop has been deprecated
       Warning:Warning:line (95)java: org.apache.parquet.hadoop.ParquetInputSplit in org.apache.parquet.hadoop has been deprecated
       Warning:Warning:line (95)java: org.apache.parquet.hadoop.ParquetInputSplit in org.apache.parquet.hadoop has been deprecated
       Warning:Warning:line (97)java: getRowGroupOffsets() in org.apache.parquet.hadoop.ParquetInputSplit has been deprecated
       Warning:Warning:line (105)java: readFooter(org.apache.hadoop.conf.Configuration,org.apache.hadoop.fs.Path,org.apache.parquet.format.converter.ParquetMetadataConverter.MetadataFilter) in org.apache.parquet.hadoop.ParquetFileReader has been deprecated
       Warning:Warning:line (108)java: filterRowGroups(org.apache.parquet.filter2.compat.FilterCompat.Filter,java.util.List<org.apache.parquet.hadoop.metadata.BlockMetaData>,org.apache.parquet.schema.MessageType) in org.apache.parquet.filter2.compat.RowGroupFilter has been deprecated
       Warning:Warning:line (111)java: readFooter(org.apache.hadoop.conf.Configuration,org.apache.hadoop.fs.Path,org.apache.parquet.format.converter.ParquetMetadataConverter.MetadataFilter) in org.apache.parquet.hadoop.ParquetFileReader has been deprecated
       Warning:Warning:line (147)java: ParquetFileReader(org.apache.hadoop.conf.Configuration,org.apache.parquet.hadoop.metadata.FileMetaData,org.apache.hadoop.fs.Path,java.util.List<org.apache.parquet.hadoop.metadata.BlockMetaData>,java.util.List<org.apache.parquet.column.ColumnDescriptor>) in org.apache.parquet.hadoop.ParquetFileReader has been deprecated
       Warning:Warning:line (203)java: readFooter(org.apache.hadoop.conf.Configuration,org.apache.hadoop.fs.Path,org.apache.parquet.format.converter.ParquetMetadataConverter.MetadataFilter) in org.apache.parquet.hadoop.ParquetFileReader has been deprecated
       Warning:Warning:line (226)java: ParquetFileReader(org.apache.hadoop.conf.Configuration,org.apache.parquet.hadoop.metadata.FileMetaData,org.apache.hadoop.fs.Path,java.util.List<org.apache.parquet.hadoop.metadata.BlockMetaData>,java.util.List<org.apache.parquet.column.ColumnDescriptor>) in org.apache.parquet.hadoop.ParquetFileReader has been deprecated
   

3. sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetCompatibilityTest.scala
4. sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetInteroperabilitySuite.scala
5. sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetTest.scala
6. sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala

Issue Links

• is duplicated by: SPARK-30174 Eliminate warnings: part 4
• links to: [Github] Pull Request #28484 (PavithraRamachandran)