Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-10539

s3 build failures caused by the conflicts of two Jenkins jobs running on the same bucket

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Not A Bug
    • Affects Version/s: Impala 4.0.0
    • Fix Version/s: None
    • Component/s: Infrastructure
    • Labels:
    • Environment:
      s3 build
    • Epic Color:
      ghx-label-1

      Description

      This issue happened a handful of times in the last few days for different s3 builds. 

      Log messages:

      22:28:09 Creating tpcds testcase data (logging to /data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/logs/data_loading/create-tpcds-testcase-data.log)...
      22:28:09 FAILED (Took: 0 min 3 sec)
      22:28:12 '/data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/testdata/bin/create-tpcds-testcase-files.sh' failed. Tail of log:
      22:28:12 at org.apache.hadoop.fs.s3a.S3AFileSystem$OperationCallbacksImpl.copyFile(S3AFileSystem.java:1653)
      22:28:12 at org.apache.hadoop.fs.s3a.impl.RenameOperation.copySourceAndUpdateTracker(RenameOperation.java:636)
      22:28:12 at org.apache.hadoop.fs.s3a.impl.RenameOperation.lambda$initiateCopy$1(RenameOperation.java:600)
      22:28:12 at org.apache.hadoop.fs.s3a.impl.CallableSupplier.get(CallableSupplier.java:62)
      22:28:12 at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)
      22:28:12 at org.apache.hadoop.util.SemaphoredDelegatingExecutor$RunnableWithPermitRelease.run(SemaphoredDelegatingExecutor.java:197)
      22:28:12 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      22:28:12 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      22:28:12 at java.lang.Thread.run(Thread.java:748)
      22:28:12 Caused by: java.io.FileNotFoundException: copyFile(test-warehouse/tpcds-testcase-data/impala-testcase-data-7cf4cea0-932c-47fb-a4b0-00985d26528d, user/jenkins/.Trash/Current/test-warehouse/tpcds-testcase-data1614061691533/impala-testcase-data-7cf4cea0-932c-47fb-a4b0-00985d26528d) on test-warehouse/tpcds-testcase-data/impala-testcase-data-7cf4cea0-932c-47fb-a4b0-00985d26528d: com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: DA4BB61317C3469E; S3 Extended Request ID: oSFXrBiZ7pDu1pQISlVzxsKL59aTgIA5KR6iTT77rEglPp9TFudtSkMlfWPZ+JX7zXQRPk1xwec=; Proxy: null), S3 Extended Request ID: oSFXrBiZ7pDu1pQISlVzxsKL59aTgIA5KR6iTT77rEglPp9TFudtSkMlfWPZ+JX7zXQRPk1xwec=:404 Not Found
      22:28:12 at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:260)
      22:28:12 at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:117)
      22:28:12 at org.apache.hadoop.fs.s3a.S3AFileSystem.copyFile(S3AFileSystem.java:3698)
      22:28:12 ... 10 more
      22:28:12 Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: DA4BB61317C3469E; S3 Extended Request ID: oSFXrBiZ7pDu1pQISlVzxsKL59aTgIA5KR6iTT77rEglPp9TFudtSkMlfWPZ+JX7zXQRPk1xwec=; Proxy: null), S3 Extended Request ID: oSFXrBiZ7pDu1pQISlVzxsKL59aTgIA5KR6iTT77rEglPp9TFudtSkMlfWPZ+JX7zXQRPk1xwec=
      22:28:12 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1828)
      22:28:12 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1412)
      22:28:12 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1374)
      22:28:12 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1145)
      22:28:12 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:802)
      22:28:12 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770)
      22:28:12 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744)
      22:28:12 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704)
      22:28:12 at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686)
      22:28:12 at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:550)
      22:28:12 at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:530)
      22:28:12 at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5227)
      22:28:12 at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5173)
      22:28:12 at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1360)
      22:28:12 at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$getObjectMetadata$6(S3AFileSystem.java:2027)
      22:28:12 at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:412)
      22:28:12 at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:375)
      22:28:12 at org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:2017)
      22:28:12 at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$copyFile$18(S3AFileSystem.java:3700)
      22:28:12 at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:115)
      22:28:12 ... 11 more
      22:28:12 rm: Failed to move to trash: s3a://impala-test-uswest2-1/test-warehouse/tpcds-testcase-data: No such file or directory: s3a://impala-test-uswest2-1/test-warehouse/tpcds-testcase-data. Consider using -skipTrash option
      22:28:12 21/02/22 22:28:12 INFO impl.MetricsSystemImpl: Stopping s3a-file-system metrics system...
      22:28:12 21/02/22 22:28:12 INFO impl.MetricsSystemImpl: s3a-file-system metrics system stopped.
      22:28:12 21/02/22 22:28:12 INFO impl.MetricsSystemImpl: s3a-file-system metrics system shutdown complete.
      22:28:12 ++ report_build_error 39
      22:28:12 +++ cd /data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala
      22:28:12 +++ awk 'NR == 39' /data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/testdata/bin/create-tpcds-testcase-files.sh
      22:28:12 ++ ERROR_MSG='hadoop fs -rm -r -f $TESTCASE_DATA_DIR'
      22:28:12 +++ basename – /data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/testdata/bin/create-tpcds-testcase-files.sh
      22:28:12 ++ FILENAME=create-tpcds-testcase-files.sh
      22:28:12 ++ echo ERROR in /data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/testdata/bin/create-tpcds-testcase-files.sh at line 39: hadoop fs -rm -r -f '$TESTCASE_DATA_DIR'
      22:28:12 ERROR in /data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/testdata/bin/create-tpcds-testcase-files.sh at line 39: hadoop fs -rm -r -f $TESTCASE_DATA_DIR
      22:28:12 ++ /data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/bin/generate_junitxml.py --step create-tpcds-testcase-files --error 'Error in /data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/testdata/bin/create-tpcds-testcase-files.sh at line 39: hadoop fs -rm -r -f $TESTCASE_DATA_DIR'
      22:28:12 Generated: /data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/logs/extra_junit_xml_logs/generate_junitxml.buildall.create-tpcds-testcase-files.20210223_06_28_12.xml
      22:28:12 ERROR in /data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/testdata/bin/create-load-data.sh at line 48:
      22:28:12 Generated: /data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/logs/extra_junit_xml_logs/generate_junitxml.buildall.create-load-data.20210223_06_28_12.xml
      22:28:12 + echo 'buildall.sh ' -format '-snapshot_file /data/jenkins/workspace/impala-cdpd-master-core-s3/testdata/test-warehouse-SNAPSHOT/test-warehouse-cdh7-651-SNAPSHOT.tar.gz' '-metastore_snapshot_file /data/jenkins/workspace/impala-cdpd-master-core-s3/testdata/hive_metastore_snapshot/hive_impala_dump_cdh7-651.txt failed.'
      22:28:12 buildall.sh -format -snapshot_file /data/jenkins/workspace/impala-cdpd-master-core-s3/testdata/test-warehouse-SNAPSHOT/test-warehouse-cdh7-651-SNAPSHOT.tar.gz -metastore_snapshot_file /data/jenkins/workspace/impala-cdpd-master-core-s3/testdata/hive_metastore_snapshot/hive_impala_dump_cdh7-651.txt failed.
      22:28:12 + RET_VAL=1
      22:28:12 + [[ DEBUG = \A\S\A\N ]]
      22:28:12 + [[ DEBUG = \S\T\R\E\S\S ]]
      22:28:12 + [[ DEBUG = \T\S\A\N ]]
      22:28:12 + '[' 1 -ne 0 ']'
      22:28:12 + echo
      22:28:12
      22:28:12 + echo
      22:28:12
      22:28:12 + echo '**** BUILD FAILED! ****'
      22:28:12 **** BUILD FAILED! ****

        Attachments

          Activity

            People

            • Assignee:
              laszlog Laszlo Gaal
              Reporter:
              wzhou Wenzhe Zhou
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: