Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-7798

Analyze table stored in S3 storage fails

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Metadata
    • Labels:
      None

      Description

      When I execute ANALYZE TABLE against file stored in S3 storage, it fails with "No route to host" error.
      S3 storage plugin is configured in console.
      It is either AWS S3 or Minio storage.
      Query executed:

      analyze table s3_aws.tpch_other.`nation.parquet` REFRESH METADATA;
      

      It does not work with CSV, PARQUET or any other format.
      From the stack trace it looks like ANALYZE TABLE cannot read S3 configuration from the plugin configuration stored through console.

      Stack trace:

      [Error Id: b4f1b482-9ce5-4e1a-b6b3-401774d1fbc5 on 65ab90fbb036:31010]
      	at org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:534)
      	at org.apache.drill.jdbc.impl.DrillCursor.loadInitialSchema(DrillCursor.java:599)
      	at org.apache.drill.jdbc.impl.DrillResultSetImpl.execute(DrillResultSetImpl.java:1278)
      	at org.apache.drill.jdbc.impl.DrillResultSetImpl.execute(DrillResultSetImpl.java:58)
      	at org.apache.calcite.avatica.AvaticaConnection$1.execute(AvaticaConnection.java:667)
      	at org.apache.drill.jdbc.impl.DrillMetaImpl.prepareAndExecute(DrillMetaImpl.java:1102)
      	at org.apache.drill.jdbc.impl.DrillMetaImpl.prepareAndExecute(DrillMetaImpl.java:1113)
      	at org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:675)
      	at org.apache.drill.jdbc.impl.DrillConnectionImpl.prepareAndExecuteInternal(DrillConnectionImpl.java:200)
      	at org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:156)
      	at org.apache.calcite.avatica.AvaticaStatement.execute(AvaticaStatement.java:217)
      	at sqlline.Commands.executeSingleQuery(Commands.java:1054)
      	at sqlline.Commands.execute(Commands.java:1003)
      	at sqlline.Commands.sql(Commands.java:967)
      	at sqlline.SqlLine.dispatch(SqlLine.java:734)
      	at sqlline.SqlLine.begin(SqlLine.java:541)
      	at sqlline.SqlLine.start(SqlLine.java:267)
      	at sqlline.SqlLine.main(SqlLine.java:206)
      Caused by: org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR: NoRouteToHostException: No route to host (Host unreachable)
      
      
      Please, refer to logs for more information.
      
      [Error Id: b4f1b482-9ce5-4e1a-b6b3-401774d1fbc5 on 65ab90fbb036:31010]
      	at org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:125)
      	at org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:422)
      	at org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:96)
      	at org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:273)
      	at org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:243)
      	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
      	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
      	at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
      	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
      	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
      	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
      	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:312)
      	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:286)
      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
      	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
      	at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
      	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
      	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1294)
      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
      	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:911)
      	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
      	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
      	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
      	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
      	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
      	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
      	at java.lang.Thread.run(Thread.java:748)
      Caused by: org.apache.drill.exec.planner.sql.QueryInputException: Failure handling SQL.
      	at org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:106)
      	at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:593)
      	at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:274)
      	at .......(:0)
      Caused by: java.lang.Exception: gdc-tiger-test-data-eu-central: org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: No AWS Credentials provided by SimpleAWSCredentialsProvider EnvironmentVariableCredentialsProvider InstanceProfileCredentialsProvider : com.amazonaws.SdkClientException: Unable to load credentials from service endpoint
      	at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:187)
      	at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:111)
      	at org.apache.hadoop.fs.s3a.Invoker.lambda$retry$3(Invoker.java:265)
      	at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:322)
      	at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:261)
      	at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:236)
      	at org.apache.hadoop.fs.s3a.S3AFileSystem.verifyBucketExists(S3AFileSystem.java:380)
      	at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:314)
      	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3303)
      	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124)
      	at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3352)
      	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3320)
      	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:479)
      	at org.apache.hadoop.fs.Path.getFileSystem(Path.java:365)
      	at org.apache.drill.exec.metastore.analyze.FileMetadataInfoCollector.getExpandedFileSelection(FileMetadataInfoCollector.java:405)
      	at org.apache.drill.exec.metastore.analyze.AnalyzeFileInfoProvider.getSegmentColumns(AnalyzeFileInfoProvider.java:54)
      	at org.apache.drill.exec.metastore.analyze.AnalyzeFileInfoProvider.getProjectionFields(AnalyzeFileInfoProvider.java:64)
      	at org.apache.drill.exec.metastore.analyze.AnalyzeParquetInfoProvider.getProjectionFields(AnalyzeParquetInfoProvider.java:42)
      	at org.apache.drill.exec.planner.sql.handlers.MetastoreAnalyzeTableHandler.getPlan(MetastoreAnalyzeTableHandler.java:124)
      	at org.apache.drill.exec.planner.sql.DrillSqlWorker.getQueryPlan(DrillSqlWorker.java:283)
      	at org.apache.drill.exec.planner.sql.DrillSqlWorker.getPhysicalPlan(DrillSqlWorker.java:163)
      	at org.apache.drill.exec.planner.sql.DrillSqlWorker.convertPlan(DrillSqlWorker.java:128)
      	at org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:93)
      	... 3 more
      Caused by: org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: No AWS Credentials provided by SimpleAWSCredentialsProvider EnvironmentVariableCredentialsProvider InstanceProfileCredentialsProvider : com.amazonaws.SdkClientException: Unable to load credentials from service endpoint
      	at org.apache.hadoop.fs.s3a.AWSCredentialProviderList.getCredentials(AWSCredentialProviderList.java:159)
      	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1166)
      	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:762)
      	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:724)
      	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)
      	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
      	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
      	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
      	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
      	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4368)
      	at com.amazonaws.services.s3.AmazonS3Client.getBucketRegionViaHeadRequest(AmazonS3Client.java:5129)
      	at com.amazonaws.services.s3.AmazonS3Client.fetchRegionFromCache(AmazonS3Client.java:5103)
      	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4352)
      	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4315)
      	at com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1344)
      	at com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:1284)
      	at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$verifyBucketExists$1(S3AFileSystem.java:381)
      	at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:109)
      	... 24 more
      Caused by: com.amazonaws.SdkClientException: Unable to load credentials from service endpoint
      	at com.amazonaws.auth.EC2CredentialsFetcher.handleError(EC2CredentialsFetcher.java:183)
      	at com.amazonaws.auth.EC2CredentialsFetcher.fetchCredentials(EC2CredentialsFetcher.java:162)
      	at com.amazonaws.auth.EC2CredentialsFetcher.getCredentials(EC2CredentialsFetcher.java:82)
      	at com.amazonaws.auth.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:164)
      	at org.apache.hadoop.fs.s3a.AWSCredentialProviderList.getCredentials(AWSCredentialProviderList.java:137)
      	... 41 more
      Caused by: java.net.NoRouteToHostException: No route to host (Host unreachable)
      	at .......(:0)
      	at com.amazonaws.internal.ConnectionUtils.connectToEndpoint(ConnectionUtils.java:54)
      	at com.amazonaws.internal.EC2CredentialsUtils.readResource(EC2CredentialsUtils.java:116)
      	at com.amazonaws.internal.EC2CredentialsUtils.readResource(EC2CredentialsUtils.java:87)
      	at com.amazonaws.auth.InstanceProfileCredentialsProvider$InstanceMetadataCredentialsEndpointProvider.getCredentialsEndpoint(InstanceProfileCredentialsProvider.java:189)
      	at com.amazonaws.auth.EC2CredentialsFetcher.fetchCredentials(EC2CredentialsFetcher.java:122)
      	... 44 more
      a
      

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              jaceksan Jan Soubusta
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: