Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-9285

Maven compile error on testdata module

    XMLWordPrintableJSON

    Details

    • Type: Task
    • Status: Resolved
    • Priority: Major
    • Resolution: Cannot Reproduce
    • Affects Version/s: None
    • Fix Version/s: Not Applicable
    • Component/s: Infrastructure
    • Labels:
      None
    • Epic Color:
      ghx-label-2

      Description

      I continuely hit this error when loading the test data:

      Running mvn  package
      Directory /home/quanlong/workspace/Impala/testdata
      ========================================================================
      [WARNING] Could not transfer metadata com.cloudera.cdh:cdh-root:6.x-SNAPSHOT/maven-metadata.xml from/to ${distMgmtSnapshotsId} (${distMgmtSnapshotsUrl}): Cannot access ${distMgmtSnapshotsUrl} with type default using the available connector factories: BasicRepositoryConnectorFactory
      [WARNING] The POM for org.apache.parquet:parquet-avro:jar:1.10.99-cdh6.x-20191031.203926-1582079 is invalid, transitive dependencies (if any) will not be available, enable debug logging for more details
      [ERROR] COMPILATION ERROR : 
      [ERROR] /home/quanlong/workspace/Impala/testdata/src/main/java/org/apache/impala/datagenerator/RandomNestedDataGenerator.java:[39,42] package org.apache.parquet.hadoop.metadata does not exist
      [ERROR] /home/quanlong/workspace/Impala/testdata/src/main/java/org/apache/impala/datagenerator/JsonToParquetConverter.java:[79,46] cannot access org.apache.parquet.hadoop.ParquetWriter
      [ERROR] /home/quanlong/workspace/Impala/testdata/src/main/java/org/apache/impala/datagenerator/JsonToParquetConverter.java:[80,26] cannot find symbol
      [ERROR] /home/quanlong/workspace/Impala/testdata/src/main/java/org/apache/impala/datagenerator/JsonToParquetConverter.java:[81,26] cannot find symbol
      [ERROR] /home/quanlong/workspace/Impala/testdata/src/main/java/org/apache/impala/datagenerator/JsonToParquetConverter.java:[78,47] cannot access org.apache.parquet.hadoop.api.WriteSupport
      [ERROR] /home/quanlong/workspace/Impala/testdata/src/main/java/org/apache/impala/datagenerator/JsonToParquetConverter.java:[87,15] cannot find symbol
      [ERROR] /home/quanlong/workspace/Impala/testdata/src/main/java/org/apache/impala/datagenerator/JsonToParquetConverter.java:[90,13] cannot find symbol
      [ERROR] /home/quanlong/workspace/Impala/testdata/src/main/java/org/apache/impala/datagenerator/RandomNestedDataGenerator.java:[66,9] cannot find symbol
      [ERROR] /home/quanlong/workspace/Impala/testdata/src/main/java/org/apache/impala/datagenerator/RandomNestedDataGenerator.java:[67,26] cannot find symbol
      [ERROR] /home/quanlong/workspace/Impala/testdata/src/main/java/org/apache/impala/datagenerator/RandomNestedDataGenerator.java:[68,26] cannot find symbol
      [ERROR] /home/quanlong/workspace/Impala/testdata/src/main/java/org/apache/impala/datagenerator/RandomNestedDataGenerator.java:[75,15] cannot find symbol
      [ERROR] /home/quanlong/workspace/Impala/testdata/src/main/java/org/apache/impala/datagenerator/RandomNestedDataGenerator.java:[78,13] cannot find symbol
      [INFO] BUILD FAILURE
      [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.3:compile (default-compile) on project impala-testdata: Compilation failure: Compilation failure: 
      [ERROR] /home/quanlong/workspace/Impala/testdata/src/main/java/org/apache/impala/datagenerator/RandomNestedDataGenerator.java:[39,42] package org.apache.parquet.hadoop.metadata does not exist
      [ERROR] /home/quanlong/workspace/Impala/testdata/src/main/java/org/apache/impala/datagenerator/JsonToParquetConverter.java:[79,46] cannot access org.apache.parquet.hadoop.ParquetWriter
      [ERROR]   class file for org.apache.parquet.hadoop.ParquetWriter not found
      [ERROR] /home/quanlong/workspace/Impala/testdata/src/main/java/org/apache/impala/datagenerator/JsonToParquetConverter.java:[80,26] cannot find symbol
      [ERROR]   symbol:   variable DEFAULT_BLOCK_SIZE
      [ERROR]   location: class org.apache.parquet.avro.AvroParquetWriter
      [ERROR] /home/quanlong/workspace/Impala/testdata/src/main/java/org/apache/impala/datagenerator/JsonToParquetConverter.java:[81,26] cannot find symbol
      [ERROR]   symbol:   variable DEFAULT_PAGE_SIZE
      [ERROR]   location: class org.apache.parquet.avro.AvroParquetWriter
      [ERROR] /home/quanlong/workspace/Impala/testdata/src/main/java/org/apache/impala/datagenerator/JsonToParquetConverter.java:[78,47] cannot access org.apache.parquet.hadoop.api.WriteSupport
      [ERROR]   class file for org.apache.parquet.hadoop.api.WriteSupport not found
      [ERROR] /home/quanlong/workspace/Impala/testdata/src/main/java/org/apache/impala/datagenerator/JsonToParquetConverter.java:[87,15] cannot find symbol
      [ERROR]   symbol:   method write(org.apache.avro.generic.GenericRecord)
      [ERROR]   location: variable writer of type org.apache.parquet.avro.AvroParquetWriter<org.apache.avro.generic.GenericRecord>
      [ERROR] /home/quanlong/workspace/Impala/testdata/src/main/java/org/apache/impala/datagenerator/JsonToParquetConverter.java:[90,13] cannot find symbol
      [ERROR]   symbol:   method close()
      [ERROR]   location: variable writer of type org.apache.parquet.avro.AvroParquetWriter<org.apache.avro.generic.GenericRecord>
      [ERROR] /home/quanlong/workspace/Impala/testdata/src/main/java/org/apache/impala/datagenerator/RandomNestedDataGenerator.java:[66,9] cannot find symbol
      [ERROR]   symbol:   variable CompressionCodecName
      [ERROR]   location: class org.apache.impala.datagenerator.RandomNestedDataGenerator
      [ERROR] /home/quanlong/workspace/Impala/testdata/src/main/java/org/apache/impala/datagenerator/RandomNestedDataGenerator.java:[67,26] cannot find symbol
      [ERROR]   symbol:   variable DEFAULT_BLOCK_SIZE
      [ERROR]   location: class org.apache.parquet.avro.AvroParquetWriter
      [ERROR] /home/quanlong/workspace/Impala/testdata/src/main/java/org/apache/impala/datagenerator/RandomNestedDataGenerator.java:[68,26] cannot find symbol
      [ERROR]   symbol:   variable DEFAULT_PAGE_SIZE
      [ERROR]   location: class org.apache.parquet.avro.AvroParquetWriter
      [ERROR] /home/quanlong/workspace/Impala/testdata/src/main/java/org/apache/impala/datagenerator/RandomNestedDataGenerator.java:[75,15] cannot find symbol
      [ERROR]   symbol:   method write(org.apache.avro.generic.GenericData.Record)
      [ERROR]   location: variable writer of type org.apache.parquet.avro.AvroParquetWriter<org.apache.avro.generic.GenericRecord>
      [ERROR] /home/quanlong/workspace/Impala/testdata/src/main/java/org/apache/impala/datagenerator/RandomNestedDataGenerator.java:[78,13] cannot find symbol
      [ERROR]   symbol:   method close()
      [ERROR]   location: variable writer of type org.apache.parquet.avro.AvroParquetWriter<org.apache.avro.generic.GenericRecord>
      [ERROR] -> [Help 1]
      [ERROR] 
      [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
      [ERROR] Re-run Maven using the -X switch to enable full debug logging.
      [ERROR] 
      [ERROR] For more information about the errors and possible solutions, please read the following articles:
      [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
      mvn  package exited with code 0
      ERROR in /home/quanlong/workspace/Impala/bin/create_testdata.sh at line 34: ${IMPALA_HOME}/bin/mvn-quiet.sh package
      Generated: /home/quanlong/workspace/Impala-CDH/logs/extra_junit_xml_logs/generate_junitxml.buildall.create_testdata.20200110_01_52_43.xml

      I can work around it by adding these dependencies in testdata/pom.xml:

           <dependency>
            <groupId>org.apache.parquet</groupId>
            <artifactId>parquet-hadoop</artifactId>
            <version>${parquet.version}</version>
          </dependency>
      
          <dependency>
            <groupId>org.apache.parquet</groupId>
            <artifactId>parquet-hadoop-bundle</artifactId>
            <version>${parquet.version}</version>
          </dependency>
      

      Will spend time to dig into it later. The output of this JIRA may be a wiki for work around or a bug fix patch.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                stigahuang Quanlong Huang
                Reporter:
                stigahuang Quanlong Huang
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: