Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-11487

FileNotFound on distcp to s3n/s3a due to creation inconsistency

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: 2.7.2
    • Fix Version/s: 2.8.0
    • Component/s: fs, fs/s3
    • Labels:
      None

      Description

      I'm trying to copy a large amount of files from HDFS to S3 via distcp and I'm getting the following exception:

      2015-01-16 20:53:18,187 ERROR [main] org.apache.hadoop.tools.mapred.CopyMapper: Failure in copying hdfs://10.165.35.216/hdfsFolder/file.gz to s3n://s3-bucket/file.gz
      java.io.FileNotFoundException: No such file or directory 's3n://s3-bucket/file.gz'
      	at org.apache.hadoop.fs.s3native.NativeS3FileSystem.getFileStatus(NativeS3FileSystem.java:445)
      	at org.apache.hadoop.tools.util.DistCpUtils.preserve(DistCpUtils.java:187)
      	at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:233)
      	at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:45)
      	at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
      	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
      	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
      	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:422)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
      	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
      2015-01-16 20:53:18,276 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.io.FileNotFoundException: No such file or directory 's3n://s3-bucket/file.gz'
      	at org.apache.hadoop.fs.s3native.NativeS3FileSystem.getFileStatus(NativeS3FileSystem.java:445)
      	at org.apache.hadoop.tools.util.DistCpUtils.preserve(DistCpUtils.java:187)
      	at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:233)
      	at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:45)
      	at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
      	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
      	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
      	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:422)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
      	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
      

      However, when I try hadoop fs -ls s3n://s3-bucket/file.gz the file is there. So probably due to Amazon's S3 eventual consistency the job failure.

      In my opinion, in order to fix this problem NativeS3FileSystem.getFileStatus must use fs.s3.maxRetries property in order to avoid failures like this.

        Issue Links

          Activity

          Hide
          stevel@apache.org Steve Loughran added a comment -

          The specific codepath here is addressed by HADOOP-13145; other inconsistency issues are different and addresssed elsewhere (HADOOP-13345, HADOOP-13786). Closing as fixed

          Show
          stevel@apache.org Steve Loughran added a comment - The specific codepath here is addressed by HADOOP-13145 ; other inconsistency issues are different and addresssed elsewhere ( HADOOP-13345 , HADOOP-13786 ). Closing as fixed
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Linking to HADOOP-13950, as it may be a lower cost solution.

          AWS caches negative GETs on a path, and FS.create() does an existence check as part of its getFileStatus call, which is always done to look for the dest being a directory.

          if overwrite=true, we don't care if a dir exists, so only need the 2 directory probes, not the direct path HEAD, so will skip the possibility of a -ve HEAD result being cached. This may be all that is needed

          Show
          stevel@apache.org Steve Loughran added a comment - Linking to HADOOP-13950 , as it may be a lower cost solution. AWS caches negative GETs on a path, and FS.create() does an existence check as part of its getFileStatus call, which is always done to look for the dest being a directory. if overwrite=true, we don't care if a dir exists, so only need the 2 directory probes, not the direct path HEAD, so will skip the possibility of a -ve HEAD result being cached. This may be all that is needed
          Hide
          $iddhe$h $iddhe$h Divekar added a comment -

          Np, will start watching it.
          Thanks for the help.

          Show
          $iddhe$h $iddhe$h Divekar added a comment - Np, will start watching it. Thanks for the help.
          Hide
          cnauroth Chris Nauroth added a comment -

          Is patch for HADOOP-13345 available for general use ?

          No, that's just a prototype right now. It's still under development. I can't recommend running it.

          Show
          cnauroth Chris Nauroth added a comment - Is patch for HADOOP-13345 available for general use ? No, that's just a prototype right now. It's still under development. I can't recommend running it.
          Hide
          $iddhe$h $iddhe$h Divekar added a comment -

          Cool thanks, will take a look.
          Is patch for HADOOP-13345 available for general use ?

          Show
          $iddhe$h $iddhe$h Divekar added a comment - Cool thanks, will take a look. Is patch for HADOOP-13345 available for general use ?
          Hide
          cnauroth Chris Nauroth added a comment -

          Does listStatus falls outside above consistency ?

          Yes, it does. FileSystem#listStatus maps to an operation listing the keys in an S3 bucket. For that listing operation, the consistency model you quoted does not apply. Instead, it follows an eventual consistency model. There may be propagation delays between creating a key and that key becoming visible in listings.

          There are more details on this behavior in the AWS S3 consistency model doc:

          http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#ConsistencyModel

          Show
          cnauroth Chris Nauroth added a comment - Does listStatus falls outside above consistency ? Yes, it does. FileSystem#listStatus maps to an operation listing the keys in an S3 bucket. For that listing operation, the consistency model you quoted does not apply. Instead, it follows an eventual consistency model. There may be propagation delays between creating a key and that key becoming visible in listings. There are more details on this behavior in the AWS S3 consistency model doc: http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#ConsistencyModel
          Hide
          $iddhe$h $iddhe$h Divekar added a comment -

          Hi Chris,
          Thanks for replying.
          As per AWS forum all of the S3 regions now support read-after-write consistency for new objects added to Amazon s3.
          https://forums.aws.amazon.com/ann.jspa?annID=3112
          Does listStatus falls outside above consistency ?

          For Hadoop 2.7 we started using s3a as per spark recommendations but
          but after moving to s3a we started using 3x degradation, hence moved backed to s3n.

          When will be the patch available for general use ?

          Show
          $iddhe$h $iddhe$h Divekar added a comment - Hi Chris, Thanks for replying. As per AWS forum all of the S3 regions now support read-after-write consistency for new objects added to Amazon s3. https://forums.aws.amazon.com/ann.jspa?annID=3112 Does listStatus falls outside above consistency ? For Hadoop 2.7 we started using s3a as per spark recommendations but but after moving to s3a we started using 3x degradation, hence moved backed to s3n. When will be the patch available for general use ?
          Hide
          cnauroth Chris Nauroth added a comment -

          Hello $iddhe$h Divekar. The stack trace indicates a problem during a FileSystem#listStatus call. The listing calls against S3 are subject to eventual consistency. The goals of the S3Guard project, tracked in issue HADOOP-13345, would help address this scenario. However, please note that this effort is targeted to the S3A file system, which is where our ongoing development effort on Hadoop S3 integration is happening. (Your stack trace indicates you are currently using S3N.)

          Show
          cnauroth Chris Nauroth added a comment - Hello $iddhe$h Divekar . The stack trace indicates a problem during a FileSystem#listStatus call. The listing calls against S3 are subject to eventual consistency. The goals of the S3Guard project, tracked in issue HADOOP-13345 , would help address this scenario. However, please note that this effort is targeted to the S3A file system, which is where our ongoing development effort on Hadoop S3 integration is happening. (Your stack trace indicates you are currently using S3N.)
          Hide
          $iddhe$h $iddhe$h Divekar added a comment - - edited

          Hi,
          We are processing data on US west and still seeing consistency issue.
          As per forums US west should not be having consistency issue but we
          are doing update of a table. Not sure if 'read-after-write' consistency
          will take care of 'read-after-update' consistency also.

          Will 9565 help us here.

          Below is the back trace of the issue we are seeing when we
          write some tables in parquet format from Apache Spark to S3n.

          org.apache.spark.SparkException: Job aborted.
                  at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1.apply$mcV$sp(InsertIntoHadoopFsRelation.scala:154)
                  at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1.apply(InsertIntoHadoopFsRelation.scala:106)
                  at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1.apply(InsertIntoHadoopFsRelation.scala:106)
                  at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:56)
                  at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation.run(InsertIntoHadoopFsRelation.scala:106)
                  at org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult$lzycompute(commands.scala:58)
                  at org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult(commands.scala:56)
                  at org.apache.spark.sql.execution.ExecutedCommand.doExecute(commands.scala:70)
                  at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$5.apply(SparkPlan.scala:132)
                  at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$5.apply(SparkPlan.scala:130)
                  at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150)
                  at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:130)
                  at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:55)
                  at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:55)
                  at org.apache.spark.sql.DataFrameWriter.insertInto(DataFrameWriter.scala:189)
                  at org.apache.spark.sql.DataFrameWriter.saveAsTable(DataFrameWriter.scala:239)
                  at org.apache.spark.sql.DataFrameWriter.saveAsTable(DataFrameWriter.scala:221)
                  at com.foo.vAnalytics.xyz_load$.main(xyz_load.scala:130)
                  at com.foo.vAnalytics.xyz_load.main(xyz_load.scala)
                  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
                  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                  at java.lang.reflect.Method.invoke(Method.java:606)
                  at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
                  at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
                  at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
                  at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
                  at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
                  at org.apache.oozie.action.hadoop.SparkMain.runSpark(SparkMain.java:104)
                  at org.apache.oozie.action.hadoop.SparkMain.run(SparkMain.java:95)
                  at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:47)
                  at org.apache.oozie.action.hadoop.SparkMain.main(SparkMain.java:38)
                  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
                  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                  at java.lang.reflect.Method.invoke(Method.java:606)
                  at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:236)
                  at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
                  at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
                  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
                  at org.apache.hadoop.mapred.LocalContainerLauncher$SubtaskRunner.runSubtask(LocalContainerLauncher.java:317)
                  at org.apache.hadoop.mapred.LocalContainerLauncher$SubtaskRunner.run(LocalContainerLauncher.java:232)
                  at java.lang.Thread.run(Thread.java:745)
          Caused by: java.io.FileNotFoundException: File s3n://foo-hive/warehouse/fooabcxyz0719/_temporary/0/task_201607210010_0005_m_000041 does not exist.
                  at org.apache.hadoop.fs.s3native.NativeS3FileSystem.listStatus(NativeS3FileSystem.java:506)
                  at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.mergePaths(FileOutputCommitter.java:360)
                  at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitJob(FileOutputCommitter.java:310)
                  at org.apache.parquet.hadoop.ParquetOutputCommitter.commitJob(ParquetOutputCommitter.java:46)
                  at org.apache.spark.sql.execution.datasources.BaseWriterContainer.commitJob(WriterContainer.scala:230)
                  at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1.apply$mcV$sp(InsertIntoHadoopFsRelation.scala:149)
                  ... 42 more
          
          2016-07-21 00:22:54,370  WARN ParameterVerifier:523 - SERVER[ip-10-0-0-136.us-west-2.compute.internal] USER[root] GROUP[-] TOKEN[] APP[spark-coord] JOB[0000096-160719225831887-oozie-root-C] ACTION[0000096-1607192\
          25831887-oozie-root-C@3] The application does not define formal parameters in its XML definition
          
          Show
          $iddhe$h $iddhe$h Divekar added a comment - - edited Hi, We are processing data on US west and still seeing consistency issue. As per forums US west should not be having consistency issue but we are doing update of a table. Not sure if 'read-after-write' consistency will take care of 'read-after-update' consistency also. Will 9565 help us here. Below is the back trace of the issue we are seeing when we write some tables in parquet format from Apache Spark to S3n. org.apache.spark.SparkException: Job aborted. at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1.apply$mcV$sp(InsertIntoHadoopFsRelation.scala:154) at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1.apply(InsertIntoHadoopFsRelation.scala:106) at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1.apply(InsertIntoHadoopFsRelation.scala:106) at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:56) at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation.run(InsertIntoHadoopFsRelation.scala:106) at org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult$lzycompute(commands.scala:58) at org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult(commands.scala:56) at org.apache.spark.sql.execution.ExecutedCommand.doExecute(commands.scala:70) at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$5.apply(SparkPlan.scala:132) at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$5.apply(SparkPlan.scala:130) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150) at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:130) at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:55) at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:55) at org.apache.spark.sql.DataFrameWriter.insertInto(DataFrameWriter.scala:189) at org.apache.spark.sql.DataFrameWriter.saveAsTable(DataFrameWriter.scala:239) at org.apache.spark.sql.DataFrameWriter.saveAsTable(DataFrameWriter.scala:221) at com.foo.vAnalytics.xyz_load$.main(xyz_load.scala:130) at com.foo.vAnalytics.xyz_load.main(xyz_load.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) at org.apache.oozie.action.hadoop.SparkMain.runSpark(SparkMain.java:104) at org.apache.oozie.action.hadoop.SparkMain.run(SparkMain.java:95) at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:47) at org.apache.oozie.action.hadoop.SparkMain.main(SparkMain.java:38) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:236) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342) at org.apache.hadoop.mapred.LocalContainerLauncher$SubtaskRunner.runSubtask(LocalContainerLauncher.java:317) at org.apache.hadoop.mapred.LocalContainerLauncher$SubtaskRunner.run(LocalContainerLauncher.java:232) at java.lang. Thread .run( Thread .java:745) Caused by: java.io.FileNotFoundException: File s3n: //foo-hive/warehouse/fooabcxyz0719/_temporary/0/task_201607210010_0005_m_000041 does not exist. at org.apache.hadoop.fs.s3native.NativeS3FileSystem.listStatus(NativeS3FileSystem.java:506) at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.mergePaths(FileOutputCommitter.java:360) at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitJob(FileOutputCommitter.java:310) at org.apache.parquet.hadoop.ParquetOutputCommitter.commitJob(ParquetOutputCommitter.java:46) at org.apache.spark.sql.execution.datasources.BaseWriterContainer.commitJob(WriterContainer.scala:230) at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1.apply$mcV$sp(InsertIntoHadoopFsRelation.scala:149) ... 42 more 2016-07-21 00:22:54,370 WARN ParameterVerifier:523 - SERVER[ip-10-0-0-136.us-west-2.compute.internal] USER[root] GROUP[-] TOKEN[] APP[spark-coord] JOB[0000096-160719225831887-oozie-root-C] ACTION[0000096-1607192\ 25831887-oozie-root-C@3] The application does not define formal parameters in its XML definition
          Hide
          cnauroth Chris Nauroth added a comment -

          I just attached a patch to HADOOP-13145 to prevent this getFileInfo call when DistCp is run without the -p option. I suspect that patch can help us get past this problem with eventual consistency on DistCp to a destination on S3A, at least when DistCp is run without the -p option.

          I filed that patch on a new JIRA instead of here, because the discussion here indicates that it may expand to a larger scope for addressing a wider set of eventual consistency concerns. HADOOP-13145 is more of a spot performance enhancement.

          Show
          cnauroth Chris Nauroth added a comment - I just attached a patch to HADOOP-13145 to prevent this getFileInfo call when DistCp is run without the -p option. I suspect that patch can help us get past this problem with eventual consistency on DistCp to a destination on S3A, at least when DistCp is run without the -p option. I filed that patch on a new JIRA instead of here, because the discussion here indicates that it may expand to a larger scope for addressing a wider set of eventual consistency concerns. HADOOP-13145 is more of a spot performance enhancement.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          eu-central is consistent, so its not create inconsistency. But s3 everywhere lags in listing consistency: if you GET a file it should be there, but if you list the pattern, it may not

          Show
          stevel@apache.org Steve Loughran added a comment - eu-central is consistent, so its not create inconsistency. But s3 everywhere lags in listing consistency: if you GET a file it should be there, but if you list the pattern, it may not
          Hide
          dekken Philip Deegan added a comment -

          Are you sure? It's a read only op from the same region (eu-central)

          Show
          dekken Philip Deegan added a comment - Are you sure? It's a read only op from the same region (eu-central)
          Hide
          stevel@apache.org Steve Loughran added a comment -

          no, that looks like creation inconsistency, which is an AWS architecture issue

          Amazon S3 buckets in the US Standard Region only provide read-after-write consistency when accessed through the Northern Virginia endpoint (s3-external-1.amazonaws.com).

          Show
          stevel@apache.org Steve Loughran added a comment - no, that looks like creation inconsistency, which is an AWS architecture issue Amazon S3 buckets in the US Standard Region only provide read-after-write consistency when accessed through the Northern Virginia endpoint (s3-external-1.amazonaws.com).
          Hide
          dekken Philip Deegan added a comment -

          Is it possible DRILL-3546 is similar?

          Show
          dekken Philip Deegan added a comment - Is it possible DRILL-3546 is similar?
          Hide
          stevel@apache.org Steve Loughran added a comment -

          marking as a dependency on HADOOP-9565 and changing JIRA to problem, not (initial) proposed solution

          Show
          stevel@apache.org Steve Loughran added a comment - marking as a dependency on HADOOP-9565 and changing JIRA to problem, not (initial) proposed solution
          Hide
          thodemoor Thomas Demoor added a comment -

          Nice find (unfortunately for you). I think the most likely culprit is org.apache.hadoop.tools.mapred.RetriableFileCopyCommand#promoteTmpToTarget which uses rename(): another "commiting by renaming" operation identified which we might be able to resolve in HADOOP-9565.

          Show
          thodemoor Thomas Demoor added a comment - Nice find (unfortunately for you). I think the most likely culprit is org.apache.hadoop.tools.mapred.RetriableFileCopyCommand#promoteTmpToTarget which uses rename(): another "commiting by renaming" operation identified which we might be able to resolve in HADOOP-9565 .
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Looking at this stack trace a bit more, it's actually the logic where the mapper is trying to propagate use & group, ACL & extended permissions. None of which s3 has.

          once HADOOP-9565 is in, this bit of distcp could be checking to see if there is create/update consistency at the destination, and if not, spinning a bit on failure. This would have it handle the problem of no-entry-to-update against s3 or other FS, without having to hide polling & waiting in the s3 clients themselves.

          Show
          stevel@apache.org Steve Loughran added a comment - Looking at this stack trace a bit more, it's actually the logic where the mapper is trying to propagate use & group, ACL & extended permissions. None of which s3 has. once HADOOP-9565 is in, this bit of distcp could be checking to see if there is create/update consistency at the destination, and if not, spinning a bit on failure. This would have it handle the problem of no-entry-to-update against s3 or other FS, without having to hide polling & waiting in the s3 clients themselves.
          Hide
          stevel@apache.org Steve Loughran added a comment -
          1. Which version of hadoop?
          2. Which S3 zone? Only US-east lacks create consistency

          Blobstores are the bane of our lives. They aren't real filesystems...really code around it needs to recognise this and act on it, though as they all have standard expectations of files and their metadata, that's not easy

          It's not enough to retry on FS status as there are other inconsistencies: directory renames and deletes, blob updates, etc.

          There's a new FS client, s3a, in hadoop 2.6 which is where all future fs/s3 work is going on. Try it to see if it is any better, though I doubt it.

          If we were to fix it, the route would be to go with something derived off NetFlix S3mper. Retrying on a 404 is not sufficient.

          Show
          stevel@apache.org Steve Loughran added a comment - Which version of hadoop? Which S3 zone? Only US-east lacks create consistency Blobstores are the bane of our lives. They aren't real filesystems...really code around it needs to recognise this and act on it, though as they all have standard expectations of files and their metadata, that's not easy It's not enough to retry on FS status as there are other inconsistencies: directory renames and deletes, blob updates, etc. There's a new FS client, s3a, in hadoop 2.6 which is where all future fs/s3 work is going on. Try it to see if it is any better, though I doubt it. If we were to fix it, the route would be to go with something derived off NetFlix S3mper. Retrying on a 404 is not sufficient.

            People

            • Assignee:
              jzhuge John Zhuge
              Reporter:
              pauloricardomg Paulo Motta
            • Votes:
              0 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development