Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: HADOOP-13345
    • Component/s: fs/s3
    • Labels:
      None

      Description

      Provide an implementation of the metadata store backed by DynamoDB.

      1. HADOOP-13449-HADOOP-13345.000.patch
        50 kB
        Mingliang Liu
      2. HADOOP-13449-HADOOP-13345.001.patch
        69 kB
        Mingliang Liu
      3. HADOOP-13449-HADOOP-13345.002.patch
        79 kB
        Mingliang Liu
      4. HADOOP-13449-HADOOP-13345.003.patch
        81 kB
        Mingliang Liu
      5. HADOOP-13449-HADOOP-13345.004.patch
        81 kB
        Mingliang Liu
      6. HADOOP-13449-HADOOP-13345.005.patch
        86 kB
        Mingliang Liu
      7. HADOOP-13449-HADOOP-13345.006.patch
        86 kB
        Mingliang Liu
      8. HADOOP-13449-HADOOP-13345.007.patch
        87 kB
        Mingliang Liu
      9. HADOOP-13449-HADOOP-13345.008.patch
        87 kB
        Mingliang Liu
      10. HADOOP-13449-HADOOP-13345.009.patch
        83 kB
        Mingliang Liu
      11. HADOOP-13449-HADOOP-13345.010.patch
        84 kB
        Mingliang Liu
      12. HADOOP-13449-HADOOP-13345.011.patch
        89 kB
        Mingliang Liu
      13. HADOOP-13449-HADOOP-13345.012.patch
        89 kB
        Mingliang Liu
      14. HADOOP-13449-HADOOP-13345.013.patch
        89 kB
        Mingliang Liu

        Issue Links

          Activity

          Hide
          liuml07 Mingliang Liu added a comment -

          Feel free to assign it to me if your working queue is too long. Thanks Chris Nauroth.

          Show
          liuml07 Mingliang Liu added a comment - Feel free to assign it to me if your working queue is too long. Thanks Chris Nauroth .
          Hide
          cnauroth Chris Nauroth added a comment -

          Hello Mingliang Liu. Thank you for volunteering to help. I am tentatively assigning this one to you. I think HADOOP-13448 will be a pre-requisite (defining the interface).

          Show
          cnauroth Chris Nauroth added a comment - Hello Mingliang Liu . Thank you for volunteering to help. I am tentatively assigning this one to you. I think HADOOP-13448 will be a pre-requisite (defining the interface).
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          Hi Mingliang Liu. Is there any progress here? We'd love to start work on the following implementations, which are depended on this JIRA.

          Show
          eddyxu Lei (Eddy) Xu added a comment - Hi Mingliang Liu . Is there any progress here? We'd love to start work on the following implementations, which are depended on this JIRA.
          Hide
          liuml07 Mingliang Liu added a comment -

          Hi Lei (Eddy) Xu, I expect progress once HADOOP-13448 is committed and/or almost done. We can first link the dependent JIRAs here so related efforts can be easily tracked. Thanks.

          Show
          liuml07 Mingliang Liu added a comment - Hi Lei (Eddy) Xu , I expect progress once HADOOP-13448 is committed and/or almost done. We can first link the dependent JIRAs here so related efforts can be easily tracked. Thanks.
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          Hi, Mingliang Liu

          Would you mind give some insights about the progress, giving HADOOP-13448 been committed?

          Much appreciated.

          Show
          eddyxu Lei (Eddy) Xu added a comment - Hi, Mingliang Liu Would you mind give some insights about the progress, giving HADOOP-13448 been committed? Much appreciated.
          Hide
          liuml07 Mingliang Liu added a comment -

          Will post a WIP patch in one week.

          Show
          liuml07 Mingliang Liu added a comment - Will post a WIP patch in one week.
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          Great.Thanks a lot!

          Show
          eddyxu Lei (Eddy) Xu added a comment - Great.Thanks a lot!
          Hide
          liuml07 Mingliang Liu added a comment -

          I just came from a short vacation. Will post the patch early next week. Thanks.

          Show
          liuml07 Mingliang Liu added a comment - I just came from a short vacation. Will post the patch early next week. Thanks.
          Hide
          liuml07 Mingliang Liu added a comment -

          Attach a demo patch. I'm working on a real patch and will post it soon. Thanks.

          Show
          liuml07 Mingliang Liu added a comment - Attach a demo patch. I'm working on a real patch and will post it soon. Thanks.
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          Thanks for posting the patch,Mingliang Liu.

          The concept looks very reasonable in general. And I like the schema. I understand that this is a WIP patch. So I'd list the following suggestions for reference.

          • Using local dynamodb local mode in test is really nice.
          • We should store more metadata beside is_directory in metadata. so that we can reconstruct S3AFileStatus in itemToPathMetadata(). Especially, do you think it is worth to store S3AFileStatus#isEmptyDirectory as well?
          • To this extend, I think PathMetadata should take S3AFileStatus() instead of FileStatus. So that S3 specific attributes are more easily to be obtained.
          • DynamoDBMetadataStore should have a deleteTable on par with initTable. Both initTable and deleteTable should be package-wide visible so that it can be used from CLI tools.
          • try {
               table.waitForActive();
            } catch (InterruptedException e) {
               LOG.warn("Interrupted while waiting for DynamoDB table {} active",
                  tableName, e);
            }
            

          Should it throw IOE to indicate the failure?

          • When do table.query() in listChildren(), the query might return partial results because the returned dataset is large. You can use {{ QueryResult#LastEvaluatedKey()}} for the following calls.
          • DynamoDB tableName should be able to be specified in configuration, i.e., considering that multiple ETL jobs might running against the same dataset with different purposes and different lifetimes, using different tables could allow such jobs managed the lifetime of dynamodb tables by themself.

          Thanks for the nice work!

          Show
          eddyxu Lei (Eddy) Xu added a comment - Thanks for posting the patch, Mingliang Liu . The concept looks very reasonable in general. And I like the schema. I understand that this is a WIP patch. So I'd list the following suggestions for reference. Using local dynamodb local mode in test is really nice. We should store more metadata beside is_directory in metadata. so that we can reconstruct S3AFileStatus in itemToPathMetadata() . Especially, do you think it is worth to store S3AFileStatus#isEmptyDirectory as well? To this extend, I think PathMetadata should take S3AFileStatus() instead of FileStatus . So that S3 specific attributes are more easily to be obtained. DynamoDBMetadataStore should have a deleteTable on par with initTable . Both initTable and deleteTable should be package-wide visible so that it can be used from CLI tools. try { table.waitForActive(); } catch (InterruptedException e) { LOG.warn( "Interrupted while waiting for DynamoDB table {} active" , tableName, e); } Should it throw IOE to indicate the failure? When do table.query() in listChildren() , the query might return partial results because the returned dataset is large. You can use {{ QueryResult#LastEvaluatedKey()}} for the following calls. DynamoDB tableName should be able to be specified in configuration, i.e., considering that multiple ETL jobs might running against the same dataset with different purposes and different lifetimes, using different tables could allow such jobs managed the lifetime of dynamodb tables by themself. Thanks for the nice work!
          Hide
          liuml07 Mingliang Liu added a comment -

          Thank you Lei (Eddy) Xu for your review! I'll address them in the v0 patch. I'm oncall this week so hopefully early next week I can post a working patch.

          The unit test can not pass by now, as the UT assumes read (e.g. get) operations return a very valid S3AFileStatus which DynamoDBMetadataStaore doesn't provide yet. I have to read related patches in parallel to address this.

          Show
          liuml07 Mingliang Liu added a comment - Thank you Lei (Eddy) Xu for your review! I'll address them in the v0 patch. I'm oncall this week so hopefully early next week I can post a working patch. The unit test can not pass by now, as the UT assumes read (e.g. get ) operations return a very valid S3AFileStatus which DynamoDBMetadataStaore doesn't provide yet. I have to read related patches in parallel to address this.
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          Thanks!

          For the MS contract test, I am also playing that: I think that it should not verify the fields that S3AFileStatus does not support, i.e., atime. We should also adjust contract test for that.

          Show
          eddyxu Lei (Eddy) Xu added a comment - Thanks! For the MS contract test, I am also playing that: I think that it should not verify the fields that S3AFileStatus does not support, i.e., atime . We should also adjust contract test for that.
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          Ping Mingliang Liu. Do you have any updates on this?

          As started to work on HADOOP-13650, just realized that there is no one MetadataStore implementation actually being checked in...I'd much appreciate if we can have a patch soon, so we can work on HADOOP-13650 soon.

          Show
          eddyxu Lei (Eddy) Xu added a comment - Ping Mingliang Liu . Do you have any updates on this? As started to work on HADOOP-13650 , just realized that there is no one MetadataStore implementation actually being checked in...I'd much appreciate if we can have a patch soon, so we can work on HADOOP-13650 soon.
          Hide
          liuml07 Mingliang Liu added a comment -

          I noticed the work HADOOP-13650 is blocked. Sorry about that. The v1 patch will be ready by this week.

          Show
          liuml07 Mingliang Liu added a comment - I noticed the work HADOOP-13650 is blocked. Sorry about that. The v1 patch will be ready by this week.
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          Ping, Mingliang Liu, in case that you did not HADOOP-13736, could you take a look of whether that approach can make your patch easier to implement?

          Show
          eddyxu Lei (Eddy) Xu added a comment - Ping, Mingliang Liu , in case that you did not HADOOP-13736 , could you take a look of whether that approach can make your patch easier to implement?
          Hide
          fabbri Aaron Fabbri added a comment -

          I'm curious what you guys are doing about path normalization.. For LocalMetadataStore, I'm using Path as the hashtable key. We already agreed to require all incoming paths to be absolute, but this is not quite enough to ensure consistency. Some callers may hand in an absolute path without scheme and bucket (a.k.a URI host) qualification, and others may hand in a path to the same object with both of those things.

          I first tried stripping scheme and bucket away (via

          {Path#getPathWithoutSchemeAndAuthority()}

          ). This causes a problem, though, if multiple FileSystem instances share a metadata store singleton. In this case, we actually need the host portion of the URI to distinguish between the different buckets that may be sharing a single MetadataStore.

          So, now I'm thinking that MetadataStore implementations, and DirListingMetadata API, need to do something like fs.qualify(path) for incoming paths.

          Questions
          1. Does this sound right (always using qualified paths as lookup keys)?
          2. Is there something less heavyweight we should do besides fs.qualify(path)? I'd like common case to be fast.
          3. Should MetadataStore impls. and DirListingMetadata be responsible for this path conversion, or should we assert that the caller already did it?
          3.b. If the former, are we ok with DirListingMetadata keeping a reference to an "owning" FileSystem instance via its constructor?

          Show
          fabbri Aaron Fabbri added a comment - I'm curious what you guys are doing about path normalization.. For LocalMetadataStore, I'm using Path as the hashtable key. We already agreed to require all incoming paths to be absolute, but this is not quite enough to ensure consistency. Some callers may hand in an absolute path without scheme and bucket (a.k.a URI host) qualification, and others may hand in a path to the same object with both of those things. I first tried stripping scheme and bucket away (via {Path#getPathWithoutSchemeAndAuthority()} ). This causes a problem, though, if multiple FileSystem instances share a metadata store singleton. In this case, we actually need the host portion of the URI to distinguish between the different buckets that may be sharing a single MetadataStore. So, now I'm thinking that MetadataStore implementations, and DirListingMetadata API, need to do something like fs.qualify(path) for incoming paths. Questions 1. Does this sound right (always using qualified paths as lookup keys)? 2. Is there something less heavyweight we should do besides fs.qualify(path)? I'd like common case to be fast. 3. Should MetadataStore impls. and DirListingMetadata be responsible for this path conversion, or should we assert that the caller already did it? 3.b. If the former, are we ok with DirListingMetadata keeping a reference to an "owning" FileSystem instance via its constructor?
          Hide
          fabbri Aaron Fabbri added a comment -

          Chatted with a friend about this some and we both are leaning towards requiring callers of MetadataStore and DirListingMetadata to pass in fully-qualified paths (with scheme+bucket). Considering that DirListingMetadata is , conceptually, a sort of simple POJO that folks will serialize/deserialize, it makes sense to have it consume complete paths instead of "calling back" to the FS client for clarification on what it means.

          Hope this discussion helps with your efforts on this JIRA Lei (Eddy) Xu and Mingliang Liu. Let me know if you disagree at all..

          Show
          fabbri Aaron Fabbri added a comment - Chatted with a friend about this some and we both are leaning towards requiring callers of MetadataStore and DirListingMetadata to pass in fully-qualified paths (with scheme+bucket). Considering that DirListingMetadata is , conceptually, a sort of simple POJO that folks will serialize/deserialize, it makes sense to have it consume complete paths instead of "calling back" to the FS client for clarification on what it means. Hope this discussion helps with your efforts on this JIRA Lei (Eddy) Xu and Mingliang Liu . Let me know if you disagree at all..
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          Hi, Aaron Fabbri

          One S3AFileSystem only works for one S3 bucket, I think we can safely assume that even multiple file system instances (i assume what you mean is that they are running on different clusters for different jobs (i.e., ETL steps)), they are running against the same bucket. And the scheme is always be s3a://. So Path should work well.

          Thanks.

          Show
          eddyxu Lei (Eddy) Xu added a comment - Hi, Aaron Fabbri One S3AFileSystem only works for one S3 bucket, I think we can safely assume that even multiple file system instances (i assume what you mean is that they are running on different clusters for different jobs (i.e., ETL steps)), they are running against the same bucket. And the scheme is always be s3a:// . So Path should work well. Thanks.
          Hide
          fabbri Aaron Fabbri added a comment - - edited

          Thanks for feedback Lei (Eddy) Xu. Turns out a user can provide URIs for any number of buckets. They will each have their own S3AFileSystem of course.

          I actually ran into this running integration tests (this is the last failure I have for my S3AFileSystem integration work). Some of the S3A Scale integration tests hit s3a://landsat-pds in addition to the test filesystem. This caused a failure when I had paths from both buckets in my MetadataStore, without the leading scheme+host (bucket) to differentiate between the two.

          I'm proposing that any FS client that supplies a Path without a host (bucket) component will get an exception, if it should have one (i.e. S3AFileSystem should, RawLocalFilesystem should not). I'll call it out when I post my next patch on HADOOP-13651.

          Show
          fabbri Aaron Fabbri added a comment - - edited Thanks for feedback Lei (Eddy) Xu . Turns out a user can provide URIs for any number of buckets. They will each have their own S3AFileSystem of course. I actually ran into this running integration tests (this is the last failure I have for my S3AFileSystem integration work). Some of the S3A Scale integration tests hit s3a://landsat-pds in addition to the test filesystem. This caused a failure when I had paths from both buckets in my MetadataStore, without the leading scheme+host (bucket) to differentiate between the two. I'm proposing that any FS client that supplies a Path without a host (bucket) component will get an exception, if it should have one (i.e. S3AFileSystem should, RawLocalFilesystem should not). I'll call it out when I post my next patch on HADOOP-13651 .
          Hide
          liuml07 Mingliang Liu added a comment -

          When do table.query() in listChildren(), the query might return partial results because the returned dataset is large. You can use {{ QueryResult#LastEvaluatedKey()}} for the following calls.

          The table.query() is using the document model, whose response includes an ItemCollection object that provides all items returned by the query. This is simpler and clearer than old AmazonDynamoDBClient#query(). See http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/QueryingJavaDocumentAPI.html

          We should store more metadata beside is_directory in metadata.

          Agreed. There are 3 unit tests failing because of this. I'm wondering whether we should also save access time, owner, group, permission, size etc in the metadata, as verifyBasicFileStatus and verifyDirStatus test a few.

          Other comments are addressed in the v0 patch.

          I still need to pass the 4/15 failing unit tests before the patch is in a good shape. Meanwhile, I'll review related patches and rebase the patch for FileStatus. The DirListingMetadata#isAuthoritative flag is not yet taken care of when putting/listing children. I'll post a refined v1 patch to address the above concerns this week.

          Thanks,

          Show
          liuml07 Mingliang Liu added a comment - When do table.query() in listChildren(), the query might return partial results because the returned dataset is large. You can use {{ QueryResult#LastEvaluatedKey()}} for the following calls. The table.query() is using the document model, whose response includes an ItemCollection object that provides all items returned by the query. This is simpler and clearer than old AmazonDynamoDBClient#query() . See http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/QueryingJavaDocumentAPI.html We should store more metadata beside is_directory in metadata. Agreed. There are 3 unit tests failing because of this. I'm wondering whether we should also save access time, owner, group, permission, size etc in the metadata, as verifyBasicFileStatus and verifyDirStatus test a few. Other comments are addressed in the v0 patch. I still need to pass the 4/15 failing unit tests before the patch is in a good shape. Meanwhile, I'll review related patches and rebase the patch for FileStatus . The DirListingMetadata#isAuthoritative flag is not yet taken care of when putting/listing children. I'll post a refined v1 patch to address the above concerns this week. Thanks,
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          . I'm wondering whether we should also save access time, owner, group, permission, size etc in the metadata,

          Yes, in theory, we can save them. However, for S3AFileStatus, these fields are not set, while the tests are not test the special cases of S3AFIleStatus (i.e., whether isEmptyDirectory() == true && DirMetadata.isEmpty() ). IMO, we should modify the tests for these invariants.

          I am working on reviewing the rest of the patch. Will post a review soon. Thanks for the good work.

          Show
          eddyxu Lei (Eddy) Xu added a comment - . I'm wondering whether we should also save access time, owner, group, permission, size etc in the metadata, Yes, in theory, we can save them. However, for S3AFileStatus, these fields are not set, while the tests are not test the special cases of S3AFIleStatus (i.e., whether isEmptyDirectory() == true && DirMetadata.isEmpty() ). IMO, we should modify the tests for these invariants. I am working on reviewing the rest of the patch. Will post a review soon. Thanks for the good work.
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          . I'm wondering whether we should also save access time, owner, group, permission, size etc in the metadata,

          Yes, in theory, we can save them. However, for S3AFileStatus, these fields are not set, while the tests are not test the special cases of S3AFIleStatus (i.e., whether isEmptyDirectory() == true && DirMetadata.isEmpty() ). IMO, we should modify the tests for these invariants.

          I am working on reviewing the rest of the patch. Will post a review soon. Thanks for the good work.

          Show
          eddyxu Lei (Eddy) Xu added a comment - . I'm wondering whether we should also save access time, owner, group, permission, size etc in the metadata, Yes, in theory, we can save them. However, for S3AFileStatus, these fields are not set, while the tests are not test the special cases of S3AFIleStatus (i.e., whether isEmptyDirectory() == true && DirMetadata.isEmpty() ). IMO, we should modify the tests for these invariants. I am working on reviewing the rest of the patch. Will post a review soon. Thanks for the good work.
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          Hi, Mingliang Liu

          The patch looks good to me overall. Looking forward to fill the gaps of tests.

          PathMetadataToDynamoDBTranslation.java
          final FileStatus fileStatus = isDir ? new S3AFileStatus(true, false, path) : new S3AFileStatus(0, 0, path, 0);
          

          Here, it seems that it only has the path be correctly populated. It assumes that S3AFileSystem only checks the existence of file in MS. It is different to InMemoryMetadataStore. I feel that we should have a consistent contract for all the stores.

          • Regarding isEmptyDirectory. should we store it in dynamodb as well? The drawback is that we should update this field in DynamoDB in S3AFileStatus#finishWrite for every file.
          • DynamoDBMetadataStore should have a getTableName() or getTable(). The table name is parsed within initialize(), so from the caller (i.e., CLI tool) point of view, it is difficult to get the table name to call deleteTable(String tableName);.
          • For authoritative(), do you think storing it as a flag in the DynamoDB is a good idea?
          Show
          eddyxu Lei (Eddy) Xu added a comment - Hi, Mingliang Liu The patch looks good to me overall. Looking forward to fill the gaps of tests. PathMetadataToDynamoDBTranslation.java final FileStatus fileStatus = isDir ? new S3AFileStatus( true , false , path) : new S3AFileStatus(0, 0, path, 0); Here, it seems that it only has the path be correctly populated. It assumes that S3AFileSystem only checks the existence of file in MS . It is different to InMemoryMetadataStore . I feel that we should have a consistent contract for all the stores. Regarding isEmptyDirectory . should we store it in dynamodb as well? The drawback is that we should update this field in DynamoDB in S3AFileStatus#finishWrite for every file. DynamoDBMetadataStore should have a getTableName() or getTable() . The table name is parsed within initialize() , so from the caller (i.e., CLI tool) point of view, it is difficult to get the table name to call deleteTable(String tableName); . For authoritative() , do you think storing it as a flag in the DynamoDB is a good idea?
          Hide
          liuml07 Mingliang Liu added a comment - - edited

          I feel that we should have a consistent contract for all the stores.

          Yes we should fix this.

          Regarding isEmptyDirectory. should we store it in dynamodb as well?

          I plan to store this in the v1 patch. We can optimize this in the future if too many requests to DynamoDB.

          DynamoDBMetadataStore should have a getTableName() or getTable().

          That makes perfect sense as well. Perhaps a Table is better so that we can (will we?) operate it directly (query/scan etc) in the CLI tool?

          For authoritative(), do you think storing it as a flag in the DynamoDB is a good idea?

          That's a good suggestion. I can try this.

          Show
          liuml07 Mingliang Liu added a comment - - edited I feel that we should have a consistent contract for all the stores. Yes we should fix this. Regarding isEmptyDirectory. should we store it in dynamodb as well? I plan to store this in the v1 patch. We can optimize this in the future if too many requests to DynamoDB. DynamoDBMetadataStore should have a getTableName() or getTable(). That makes perfect sense as well. Perhaps a Table is better so that we can (will we?) operate it directly (query/scan etc) in the CLI tool? For authoritative(), do you think storing it as a flag in the DynamoDB is a good idea? That's a good suggestion. I can try this.
          Hide
          liuml07 Mingliang Liu added a comment -

          Thansk for the review, Lei (Eddy) Xu. Will post new patches addressing these.

          Show
          liuml07 Mingliang Liu added a comment - Thansk for the review, Lei (Eddy) Xu . Will post new patches addressing these.
          Hide
          fabbri Aaron Fabbri added a comment -

          Heads-up I just posted my latest patch to HADOOP-13651. Integration with S3AFileSystem is looking pretty good. I did make some changes to the way Paths are handled. I have two patches outstanding that could go in: HADOOP-13631 (move implementation) and HADOOP-13651 (S3AFileSystem integration). We may want to start reviewing and committing these to avoid merge hell when this one gets done.

          The work in my latest patch should make your life easier on this JIRA when you get to running all the S3A integration tests. I'm available to help with that, i.e. if you want some test refactoring to make the MetadataStore contract tests easier to apply to S3AFileStatus (where not all FileSystem fields need to be persisted), just let me know.

          Show
          fabbri Aaron Fabbri added a comment - Heads-up I just posted my latest patch to HADOOP-13651 . Integration with S3AFileSystem is looking pretty good. I did make some changes to the way Paths are handled. I have two patches outstanding that could go in: HADOOP-13631 (move implementation) and HADOOP-13651 (S3AFileSystem integration). We may want to start reviewing and committing these to avoid merge hell when this one gets done. The work in my latest patch should make your life easier on this JIRA when you get to running all the S3A integration tests. I'm available to help with that, i.e. if you want some test refactoring to make the MetadataStore contract tests easier to apply to S3AFileStatus (where not all FileSystem fields need to be persisted), just let me know.
          Hide
          liuml07 Mingliang Liu added a comment -

          Thanks! I'll review that patch this week.

          Show
          liuml07 Mingliang Liu added a comment - Thanks! I'll review that patch this week.
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          Ping Mingliang Liu. Just wondering is there any update on this?

          Thanks!

          Show
          eddyxu Lei (Eddy) Xu added a comment - Ping Mingliang Liu . Just wondering is there any update on this? Thanks!
          Hide
          liuml07 Mingliang Liu added a comment -

          Thanks for asking Lei (Eddy) Xu. I attach the v1 patch for quick feedback.

          1. I changed the base unit test as the owner, group and permission etc are not part of the metadata we're interested in by now.
          2. We store the is_empty for directory in the DynamoDB (DDB) metadata store now. We have to update this information in a consistent and efficient way. We don't want to check the parent directory every time we delete/put a file item. At least we can optimize this when deleting a subtree.
          3. The contract assumes we create the direct parent directory (other ancestors should be taken care of by the clients/callers) when putting a new file item. I checked the in-memory local metadata store and it implements this idea. This may be not efficient to DDB. Basically for putting X items, we have to issue 2X~3X DDB requests (X for putting file, X for checking its parent directories, and possible X for updating its parent directories). I'm wondering if we can also let the client/caller pre-create the direct parent directory as other ancestors.
            This is root cause of the only left 2 of 16 failing unit tests, i.e. testPutDirListing and testPutNew.
          4. As to replacing FileStatus with S3AFileStatus in PathMetadata, I'm +0 for the idea. If we do agree on the switch, HADOOP-13736 is basically good to me. If not, I can live with the similar way to S3AFileSystem vs. FileSystem in the MetadataStore#initialize().
          5. I need to review HADOOP-13651 and revisit the patch after catching up the current discussion. Will post v2 patch in one week. I will also handle the isAuthoritative in the next patch. Storing an extra field is a good and simple idea. Any idea how client sets/gets this value?

          Thanks,

          Show
          liuml07 Mingliang Liu added a comment - Thanks for asking Lei (Eddy) Xu . I attach the v1 patch for quick feedback. I changed the base unit test as the owner , group and permission etc are not part of the metadata we're interested in by now. We store the is_empty for directory in the DynamoDB (DDB) metadata store now. We have to update this information in a consistent and efficient way. We don't want to check the parent directory every time we delete/put a file item. At least we can optimize this when deleting a subtree. The contract assumes we create the direct parent directory (other ancestors should be taken care of by the clients/callers) when putting a new file item. I checked the in-memory local metadata store and it implements this idea. This may be not efficient to DDB. Basically for putting X items, we have to issue 2X~3X DDB requests (X for putting file, X for checking its parent directories, and possible X for updating its parent directories). I'm wondering if we can also let the client/caller pre-create the direct parent directory as other ancestors. This is root cause of the only left 2 of 16 failing unit tests, i.e. testPutDirListing and testPutNew . As to replacing FileStatus with S3AFileStatus in PathMetadata , I'm +0 for the idea. If we do agree on the switch, HADOOP-13736 is basically good to me. If not, I can live with the similar way to S3AFileSystem vs. FileSystem in the MetadataStore#initialize() . I need to review HADOOP-13651 and revisit the patch after catching up the current discussion. Will post v2 patch in one week. I will also handle the isAuthoritative in the next patch. Storing an extra field is a good and simple idea. Any idea how client sets/gets this value? Thanks,
          Hide
          fabbri Aaron Fabbri added a comment - - edited

          Exciting stuff, thanks for update.

          I changed the base unit test as the owner, group and permission etc are not part of the metadata we're interested in by now.

          Good. We could have a helper function that all tests could use, e.g. doesMetadataStorePersistOwnerGroupPermission() which returns false if MetadataStore instanceof DynamoDBMetadataStore. This is also another spot it might be nice to add a function getProperty() for MetadataStore, so we could getProperty(PERSISTS_PERMISSIONS etc. We could do that later on.

          We store the is_empty for directory in the DynamoDB (DDB) metadata store now. We have to update this information in a consistent and efficient way. We don't want to check the parent directory every time we delete/put a file item. At least we can optimize this when deleting a subtree.

          This part is a pain. We should revisit the whole S3AFileStatus#isEmptyDirectory idea in the future.

          In case it helps, my algorithm is here:

          In put(PathMetadata meta):

            if we have PathMetadata for meta's parent path:
                parentMeta.setIsEmpty(false)
          

          The harder case, when we are removing an entry:

          
                // If we have cached a FileStatus for the parent...
                DirListingMetadata dir = dirHash.get(parent);
                if (dir != null) {
                  LOG.debug("removing parent's entry for {} ", path);
          
                  // Remove our path from the parent dir
                  dir.remove(path);
          
                  // S3A-specific logic dealing with S3AFileStatus#isEmptyDirectory()
                  if (isS3A) {
                    if (dir.isAuthoritative() && dir.numEntries() == 0) {
                      setS3AIsEmpty(parent, true);
                    } else if (dir.numEntries() == 0) {
                      // We do not know of any remaining entries in parent directory.
                      // However, we do not have authoritative listing, so there may
                      // still be some entries in the dir.  Since we cannot know the
                      // proper state of the parent S3AFileStatus#isEmptyDirectory, we
                      // will invalidate our entries for it.
                      // Better than deleting entries would be marking them as "missing
                      // metadata".  Deleting them means we lose consistent listing and
                      // ability to retry for eventual consistency for the parent path.
          
                      // TODO implement missing metadata feature
                      invalidateFileStatus(parent);
                    }
                    // else parent directory still has entries in it, isEmptyDirectory
                    // does not change
                  }
          

          Fixing the loss of consistency on the parent could be achieved by leaving an empty PathMetadata for the parent that does not contain a FileStatus in it. That "missing metadata" PathMetadata would indicate to future getFileStatus() or listStatus() calls that the file does exist (so retry if S3 is eventually consistent), but the FileStatus needs to be recreated (the regular getFileStatus() logic) , since we cannot know the value of its isEmptyDirectory()

          I added a TODO because we can tackle this later if we want.

          The contract assumes we create the direct parent directory (other ancestors should be taken care of by the clients/callers) when putting a new file item

          Yeah this is for consistent listing on the parent after the child is created. I'm wondering if we can relax this or make it configurable? When fs.s3a.metadatastore.authoritative is true, the performance hit on create could be offset by a performance gain on subsequent listing of the parent directory.

          Looks like good progress! Please shout if I can help at all.

          Show
          fabbri Aaron Fabbri added a comment - - edited Exciting stuff, thanks for update. I changed the base unit test as the owner, group and permission etc are not part of the metadata we're interested in by now. Good. We could have a helper function that all tests could use, e.g. doesMetadataStorePersistOwnerGroupPermission() which returns false if MetadataStore instanceof DynamoDBMetadataStore. This is also another spot it might be nice to add a function getProperty() for MetadataStore, so we could getProperty(PERSISTS_PERMISSIONS etc. We could do that later on. We store the is_empty for directory in the DynamoDB (DDB) metadata store now. We have to update this information in a consistent and efficient way. We don't want to check the parent directory every time we delete/put a file item. At least we can optimize this when deleting a subtree. This part is a pain. We should revisit the whole S3AFileStatus#isEmptyDirectory idea in the future. In case it helps, my algorithm is here: In put(PathMetadata meta): if we have PathMetadata for meta's parent path: parentMeta.setIsEmpty( false ) The harder case, when we are removing an entry: // If we have cached a FileStatus for the parent... DirListingMetadata dir = dirHash.get(parent); if (dir != null ) { LOG.debug( "removing parent's entry for {} " , path); // Remove our path from the parent dir dir.remove(path); // S3A-specific logic dealing with S3AFileStatus#isEmptyDirectory() if (isS3A) { if (dir.isAuthoritative() && dir.numEntries() == 0) { setS3AIsEmpty(parent, true ); } else if (dir.numEntries() == 0) { // We do not know of any remaining entries in parent directory. // However, we do not have authoritative listing, so there may // still be some entries in the dir. Since we cannot know the // proper state of the parent S3AFileStatus#isEmptyDirectory, we // will invalidate our entries for it. // Better than deleting entries would be marking them as "missing // metadata". Deleting them means we lose consistent listing and // ability to retry for eventual consistency for the parent path. // TODO implement missing metadata feature invalidateFileStatus(parent); } // else parent directory still has entries in it, isEmptyDirectory // does not change } Fixing the loss of consistency on the parent could be achieved by leaving an empty PathMetadata for the parent that does not contain a FileStatus in it. That "missing metadata" PathMetadata would indicate to future getFileStatus() or listStatus() calls that the file does exist (so retry if S3 is eventually consistent), but the FileStatus needs to be recreated (the regular getFileStatus() logic) , since we cannot know the value of its isEmptyDirectory() I added a TODO because we can tackle this later if we want. The contract assumes we create the direct parent directory (other ancestors should be taken care of by the clients/callers) when putting a new file item Yeah this is for consistent listing on the parent after the child is created. I'm wondering if we can relax this or make it configurable? When fs.s3a.metadatastore.authoritative is true, the performance hit on create could be offset by a performance gain on subsequent listing of the parent directory. Looks like good progress! Please shout if I can help at all.
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          Good discussion, Mingliang Liu and Aaron Fabbri

          The contract assumes we create the direct parent directory (other ancestors should be taken care of by the clients/callers) when putting a new file item. I checked the in-memory local metadata store and it implements this idea. This may be not efficient to DDB. Basically for putting X items, we have to issue 2X~3X DDB requests (X for putting file, X for checking its parent directories, and possible X for updating its parent directories). I'm wondering if we can also let the client/caller pre-create the direct parent directory as other ancestors.

          I suggest to consider this into two aspects:

          • Checking parents directories in normal S3AFileSystem operations (i.e., create / mkdirs ). In such case, S3AFileSystem should already ensure the invariant of the contracts(the parent directories existed before S3AFileSystem starts to create files on S3).
          • Loading files and directories outside of normal S3AFileSystem operations, e.g., load a non-cached directory or from CLI tool, in such cases, would a small local "dentry_cache" types of data structure be sufficient for a batch operation? Because these operations can ensure that the namespace structure exists on S3 already.

          The last resort is, if S3AFileSystem considers that it is safe to create / mkdir on a path. You can always create all its parent directories in a single batch to dynamodb. In short, I'd suggest to let S3AFileSystem ensure the contract.

          We store the is_empty for directory in the DynamoDB (DDB) metadata store now. We have to update this information in a consistent and efficient way. We don't want to check the parent directory every time we delete/put a file item. At least we can optimize this when deleting a subtree.

          Another way to do it is letting the isEmpty() flag being set by issuing a small additional query on the directory with a Limit=1. So if it returns more than 1 result, the isEmpty flag is false, otherwise, the flag is true. And this value can be cached with the lifetime of S3AFileStatus, as it can not reliably reflect the changes in S3 anyway. So the query cost only occurs when you call the IsEmpty() for the first time. And you don't need to update this flag for any S3 writes.

          Hope that works.

          Show
          eddyxu Lei (Eddy) Xu added a comment - Good discussion, Mingliang Liu and Aaron Fabbri The contract assumes we create the direct parent directory (other ancestors should be taken care of by the clients/callers) when putting a new file item. I checked the in-memory local metadata store and it implements this idea. This may be not efficient to DDB. Basically for putting X items, we have to issue 2X~3X DDB requests (X for putting file, X for checking its parent directories, and possible X for updating its parent directories). I'm wondering if we can also let the client/caller pre-create the direct parent directory as other ancestors. I suggest to consider this into two aspects: Checking parents directories in normal S3AFileSystem operations (i.e., create / mkdirs ). In such case, S3AFileSystem should already ensure the invariant of the contracts(the parent directories existed before S3AFileSystem starts to create files on S3). Loading files and directories outside of normal S3AFileSystem operations, e.g., load a non-cached directory or from CLI tool, in such cases, would a small local "dentry_cache" types of data structure be sufficient for a batch operation? Because these operations can ensure that the namespace structure exists on S3 already. The last resort is, if S3AFileSystem considers that it is safe to create / mkdir on a path. You can always create all its parent directories in a single batch to dynamodb. In short, I'd suggest to let S3AFileSystem ensure the contract. We store the is_empty for directory in the DynamoDB (DDB) metadata store now. We have to update this information in a consistent and efficient way. We don't want to check the parent directory every time we delete/put a file item. At least we can optimize this when deleting a subtree. Another way to do it is letting the isEmpty() flag being set by issuing a small additional query on the directory with a Limit=1 . So if it returns more than 1 result, the isEmpty flag is false, otherwise, the flag is true. And this value can be cached with the lifetime of S3AFileStatus , as it can not reliably reflect the changes in S3 anyway. So the query cost only occurs when you call the IsEmpty() for the first time. And you don't need to update this flag for any S3 writes. Hope that works.
          Hide
          stevel@apache.org Steve Loughran added a comment - - edited

          I see this patch bumps up the AWS version. Could that change be self contained in HADOOP-13050; that way the change is more visible & easier to cherry pick. This also implies HADOOP-12705.

          Also: that dynamo DB dependency MUST be at <provided> scope. We don't want to force it on people.

          Show
          stevel@apache.org Steve Loughran added a comment - - edited I see this patch bumps up the AWS version. Could that change be self contained in HADOOP-13050 ; that way the change is more visible & easier to cherry pick. This also implies HADOOP-12705 . Also: that dynamo DB dependency MUST be at <provided> scope. We don't want to force it on people.
          Hide
          cnauroth Chris Nauroth added a comment -

          Also: that dynamo DB dependency MUST be at <provided> scope. We don't want to force it on people.

          Steve Loughran, the DynamoDB client adds no transitive dependencies that hadoop-aws is not already picking up through the AWS SDK. Here we can see the only dependencies are on aws-java-sdk-s3 and aws-java-sdk-core:

          https://github.com/aws/aws-sdk-java/blob/1.10.6/aws-java-sdk-dynamodb/pom.xml#L19-L45

          Everything else is a test-only dependency.

          In this case, I wonder if the right trade-off is for us to allow the dependency, so that downstream projects can pick up S3Guard functionality without needing to add the aws-java-sdk-dynamodb dependency explicitly. Those projects would likely need to keep synchronized with whatever AWS SDK version number we're using in Hadoop, so as to avoid internal version conflicts around things like aws-java-sdk-core.

          Show
          cnauroth Chris Nauroth added a comment - Also: that dynamo DB dependency MUST be at <provided> scope. We don't want to force it on people. Steve Loughran , the DynamoDB client adds no transitive dependencies that hadoop-aws is not already picking up through the AWS SDK. Here we can see the only dependencies are on aws-java-sdk-s3 and aws-java-sdk-core: https://github.com/aws/aws-sdk-java/blob/1.10.6/aws-java-sdk-dynamodb/pom.xml#L19-L45 Everything else is a test-only dependency. In this case, I wonder if the right trade-off is for us to allow the dependency, so that downstream projects can pick up S3Guard functionality without needing to add the aws-java-sdk-dynamodb dependency explicitly. Those projects would likely need to keep synchronized with whatever AWS SDK version number we're using in Hadoop, so as to avoid internal version conflicts around things like aws-java-sdk-core.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          I'm just catching up with this, apologies if I say things that are clearly wrong to anyone who knows the code or its history: I don't, yet.

          Build

          1. Chris Nauroth I see your point about declaring the dependency; you are correct. It does need to be something published for downstream users.
          2. I do still want the AWS update to be a standalone patch, and with a matching Jackson update. Those can perhaps be done to trunk/ itself, and merged in here, so that any/all other trunk work will be with the upgraded artifacts.

          Code

          Anything in source marked TODO scares me. There's a lot here. Presumably the plan is to have them addressed by the time the patch goes in? Or at least pulled out into explicit followup JIRAs?

          DynamoDBMetadataStore

          • just use .* on the static imports of the s3a constant, util, PathMetadataDynamoDBTranslation entries
          • L108: no need to mix @code and <pre> tags. For multiline, <pre> should suffice ... check with the generated javadocs to see
          • L218: that endpoint map/convert logic should be pulled into a static s3a util method, with tests.
          • L465: what if close() is called twice? If a re-entrant call is made?
          • L492, 531: Throw InterruptedIOException, or set the thread's interrupted bit again. We don't want that thread interrupt to be lost if at all possible.
          • Most of those info-level per-operation logs should be at Debug
          • Operation param to calls of translateException could be more informative. Consider: what info would you need there in order to debug this from the logs.

          Tests

          As well as the unit tests, I need to be able to run the entire existing suite with s3guard enabled. This could be done with a new maven profile which would enable it, or simply a property passed down through the build. That's what's done in the scale tests in trunk, using methods in S3ATestUtils to allow a maven-defined property to override one in the core-site.xml, allowing you to enable it permanently in your test/resources/auth-keys.xml reference, or via maven.

          I see that the tests are using java 8 language features. That is going to make backporting to branch-2 in future harder. Is everyone happy there (i.e. willing to do the effort to downgrade the code if the need arises)?

          MetadataStoreTestBase

          • L234: I know it's not in this patch, but I think the path name should be changed to something else.

          TestDynamoDBMetadataStore

          I'll need to spend some time looking/playing with this.

          • There's an inevitable risk the native libs aren't around/going to work with the native OS running the build. What policy is good there? Fail? or downgrade to skip? It's probably easiest to leave it as it is now (fail) and see what needs to change as/when failures surface.
          • Add S3AFS.close() call in tearDownAfterClass just to make sure threads all get cleaned up.
          Show
          stevel@apache.org Steve Loughran added a comment - I'm just catching up with this, apologies if I say things that are clearly wrong to anyone who knows the code or its history: I don't, yet. Build Chris Nauroth I see your point about declaring the dependency; you are correct. It does need to be something published for downstream users. I do still want the AWS update to be a standalone patch, and with a matching Jackson update. Those can perhaps be done to trunk/ itself, and merged in here, so that any/all other trunk work will be with the upgraded artifacts. Code Anything in source marked TODO scares me. There's a lot here. Presumably the plan is to have them addressed by the time the patch goes in? Or at least pulled out into explicit followup JIRAs? DynamoDBMetadataStore just use .* on the static imports of the s3a constant, util, PathMetadataDynamoDBTranslation entries L108: no need to mix @code and <pre> tags. For multiline, <pre> should suffice ... check with the generated javadocs to see L218: that endpoint map/convert logic should be pulled into a static s3a util method, with tests. L465: what if close() is called twice? If a re-entrant call is made? L492, 531: Throw InterruptedIOException , or set the thread's interrupted bit again. We don't want that thread interrupt to be lost if at all possible. Most of those info-level per-operation logs should be at Debug Operation param to calls of translateException could be more informative. Consider: what info would you need there in order to debug this from the logs. Tests As well as the unit tests, I need to be able to run the entire existing suite with s3guard enabled. This could be done with a new maven profile which would enable it, or simply a property passed down through the build. That's what's done in the scale tests in trunk, using methods in S3ATestUtils to allow a maven-defined property to override one in the core-site.xml, allowing you to enable it permanently in your test/resources/auth-keys.xml reference, or via maven. I see that the tests are using java 8 language features. That is going to make backporting to branch-2 in future harder. Is everyone happy there (i.e. willing to do the effort to downgrade the code if the need arises)? MetadataStoreTestBase L234: I know it's not in this patch, but I think the path name should be changed to something else. TestDynamoDBMetadataStore I'll need to spend some time looking/playing with this. There's an inevitable risk the native libs aren't around/going to work with the native OS running the build. What policy is good there? Fail? or downgrade to skip? It's probably easiest to leave it as it is now (fail) and see what needs to change as/when failures surface. Add S3AFS.close() call in tearDownAfterClass just to make sure threads all get cleaned up.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          regarding create() on a path. S3A isn't strict enough (HADOOP-13321) . It checks the dest path is there and not a directory, but doesn't go up the tree. It should. we know it should, but we also know how much slower things would be. I'm hoping to make this something done asynchronously between creating the file and actually committing the write in the final close(). That is, fail later, rather than sooner —but do fail before anything is materialized (HADOOP-13654).

          Show
          stevel@apache.org Steve Loughran added a comment - regarding create() on a path. S3A isn't strict enough ( HADOOP-13321 ) . It checks the dest path is there and not a directory, but doesn't go up the tree. It should. we know it should, but we also know how much slower things would be. I'm hoping to make this something done asynchronously between creating the file and actually committing the write in the final close(). That is, fail later, rather than sooner —but do fail before anything is materialized ( HADOOP-13654 ).
          Hide
          liuml07 Mingliang Liu added a comment -

          Thanks all for the insightful comments! I changed the patch accordingly, with some comments to address in the next patch (I'm actively working on v3 patch now):

          1. We assume the client/callers have pre-create all the parents of the path. This is different from in-memory local store. Put itself will not check this and will not fail. As we should fail if the parent directories are missing, we fail later (e.g. in S3A#close).
          2. Sorry the "isAuthorative" is not stored yet. I'm confused when to set/get the value. Is this purely controlled by the client?
          3. Bumping the AWS SDK version is a prerequisite. This patch bumps for test purpose.
          4. Will make the "DDBMetadataStore" 1:1 mapping with the table. So createTable, deleteTable and provisionTable will all take no table names.
          5. Will add more logic to guard the isEmpty field.
          Show
          liuml07 Mingliang Liu added a comment - Thanks all for the insightful comments! I changed the patch accordingly, with some comments to address in the next patch (I'm actively working on v3 patch now): We assume the client/callers have pre-create all the parents of the path. This is different from in-memory local store. Put itself will not check this and will not fail. As we should fail if the parent directories are missing, we fail later (e.g. in S3A#close). Sorry the "isAuthorative" is not stored yet. I'm confused when to set/get the value. Is this purely controlled by the client? Bumping the AWS SDK version is a prerequisite. This patch bumps for test purpose. Will make the "DDBMetadataStore" 1:1 mapping with the table. So createTable , deleteTable and provisionTable will all take no table names. Will add more logic to guard the isEmpty field.
          Hide
          liuml07 Mingliang Liu added a comment -

          Will make the "DDBMetadataStore" 1:1 mapping with the table. So createTable, deleteTable and provisionTable will all take no table names.

          Now in v3 patch, a DynamoDBMetatadataStore object is associated with only one Table after initialization. It's not able to delete/provision other tables by accident.

          There's an inevitable risk the native libs aren't around/going to work with the native OS running the build. What policy is good there? Fail? or downgrade to skip? It's probably easiest to leave it as it is now (fail) and see what needs to change as/when failures surface.

          That's a very good point. In the v3 patch, the test will fail early in setUpBeforeClass method if the local server is not working (e.g. native libs are not loaded correctly). All the test cases will be ignored then.

          In short, I'd suggest to let S3AFileSystem ensure the contract.

          I'm with this point as well. The MetadataStore assumes all ancestor directories (including direct parent directory) have been pre-created by the caller/user. I have to change the base test MetadataStoreTestBase to make all the tests pass. We have to change LocalMetadataStore accordingly. Ping Aaron Fabbri and Steve Loughran for inputs.

          One question I have in the implementation is that, for initialize() / destroy functions, can we provide a version of such functions that do not take S3FIleSystem as parameters (i.e., taking Configuration instead)?

          Yes, now we have such functionality, see DynamoDBMetadataStore#initialize(Configuration conf). I did not test this thoroughly but the basic idea is feasible. I hope this will help other tools (e.g. command line) that operate the metadata store without initializing an S3FileSystem, which will check the bucket is there, unnecessarily. Ping Lei (Eddy) Xu for more input about this.

          I also tried bumping the AWS SDK version to higher value than 1.11.0, say 1.11.45 (see HADOOP-13050), only to find that the DynamoDBLocal was not included yet. We may have to use different version for DynamoDBLocal and other AWS SDK modules, or use mocked objects as replacements (which is not ideal). To easy this, I moved the logic to create a DDBClient code to the MockS3ClientFactory, which may be helpful.

          Provisioning table is supported.

          TODO:

          1. to make isEmpty() more efficient, and easy to integrate with file system
          2. implement isAuthoritative() related methods
          3. Make changes for S3AFileSystem integration (see HADOOP-13651).
          Show
          liuml07 Mingliang Liu added a comment - Will make the "DDBMetadataStore" 1:1 mapping with the table. So createTable, deleteTable and provisionTable will all take no table names. Now in v3 patch, a DynamoDBMetatadataStore object is associated with only one Table after initialization. It's not able to delete/provision other tables by accident. There's an inevitable risk the native libs aren't around/going to work with the native OS running the build. What policy is good there? Fail? or downgrade to skip? It's probably easiest to leave it as it is now (fail) and see what needs to change as/when failures surface. That's a very good point. In the v3 patch, the test will fail early in setUpBeforeClass method if the local server is not working (e.g. native libs are not loaded correctly). All the test cases will be ignored then. In short, I'd suggest to let S3AFileSystem ensure the contract. I'm with this point as well. The MetadataStore assumes all ancestor directories (including direct parent directory) have been pre-created by the caller/user. I have to change the base test MetadataStoreTestBase to make all the tests pass. We have to change LocalMetadataStore accordingly. Ping Aaron Fabbri and Steve Loughran for inputs. One question I have in the implementation is that, for initialize() / destroy functions, can we provide a version of such functions that do not take S3FIleSystem as parameters (i.e., taking Configuration instead)? Yes, now we have such functionality, see DynamoDBMetadataStore#initialize(Configuration conf) . I did not test this thoroughly but the basic idea is feasible. I hope this will help other tools (e.g. command line) that operate the metadata store without initializing an S3FileSystem, which will check the bucket is there, unnecessarily. Ping Lei (Eddy) Xu for more input about this. I also tried bumping the AWS SDK version to higher value than 1.11.0 , say 1.11.45 (see HADOOP-13050 ), only to find that the DynamoDBLocal was not included yet. We may have to use different version for DynamoDBLocal and other AWS SDK modules, or use mocked objects as replacements (which is not ideal). To easy this, I moved the logic to create a DDBClient code to the MockS3ClientFactory , which may be helpful. Provisioning table is supported. TODO: to make isEmpty() more efficient, and easy to integrate with file system implement isAuthoritative() related methods Make changes for S3AFileSystem integration (see HADOOP-13651 ).
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 22s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 6 new or modified test files.
          0 mvndep 2m 32s Maven dependency ordering for branch
          +1 mvninstall 6m 54s HADOOP-13345 passed
          +1 compile 6m 53s HADOOP-13345 passed
          +1 checkstyle 1m 29s HADOOP-13345 passed
          +1 mvnsite 1m 34s HADOOP-13345 passed
          +1 mvneclipse 0m 50s HADOOP-13345 passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project
          +1 findbugs 1m 47s HADOOP-13345 passed
          +1 javadoc 1m 12s HADOOP-13345 passed
          0 mvndep 0m 36s Maven dependency ordering for patch
          -1 mvninstall 0m 22s hadoop-aws in the patch failed.
          +1 compile 7m 3s the patch passed
          -1 javac 7m 3s root generated 2 new + 695 unchanged - 0 fixed = 697 total (was 695)
          -0 checkstyle 1m 34s root: The patch generated 24 new + 4 unchanged - 0 fixed = 28 total (was 4)
          +1 mvnsite 1m 45s the patch passed
          +1 mvneclipse 1m 1s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 4s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project
          -1 findbugs 0m 49s hadoop-tools/hadoop-aws generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
          -1 javadoc 0m 19s hadoop-tools_hadoop-aws generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)
          +1 unit 0m 12s hadoop-project in the patch passed.
          +1 unit 8m 48s hadoop-common in the patch passed.
          -1 unit 0m 38s hadoop-aws in the patch failed.
          +1 asflicense 0m 28s The patch does not generate ASF License warnings.
          73m 51s



          Reason Tests
          FindBugs module:hadoop-tools/hadoop-aws
            Arguments in wrong order for invocation of checkNotNull in org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.initialize(Configuration) At DynamoDBMetadataStore.java:invocation of checkNotNull in org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.initialize(Configuration) At DynamoDBMetadataStore.java:[line 213]
          Failed junit tests hadoop.fs.s3a.s3guard.TestLocalMetadataStore



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HADOOP-13449
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838488/HADOOP-13449-HADOOP-13345.003.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux ab950f5879d2 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision HADOOP-13345 / 56b715f
          Default Java 1.8.0_101
          findbugs v3.0.0
          mvninstall https://builds.apache.org/job/PreCommit-HADOOP-Build/11050/artifact/patchprocess/patch-mvninstall-hadoop-tools_hadoop-aws.txt
          javac https://builds.apache.org/job/PreCommit-HADOOP-Build/11050/artifact/patchprocess/diff-compile-javac-root.txt
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11050/artifact/patchprocess/diff-checkstyle-root.txt
          findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/11050/artifact/patchprocess/new-findbugs-hadoop-tools_hadoop-aws.html
          javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/11050/artifact/patchprocess/diff-javadoc-javadoc-hadoop-tools_hadoop-aws.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11050/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-aws.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11050/testReport/
          modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11050/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 22s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 6 new or modified test files. 0 mvndep 2m 32s Maven dependency ordering for branch +1 mvninstall 6m 54s HADOOP-13345 passed +1 compile 6m 53s HADOOP-13345 passed +1 checkstyle 1m 29s HADOOP-13345 passed +1 mvnsite 1m 34s HADOOP-13345 passed +1 mvneclipse 0m 50s HADOOP-13345 passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project +1 findbugs 1m 47s HADOOP-13345 passed +1 javadoc 1m 12s HADOOP-13345 passed 0 mvndep 0m 36s Maven dependency ordering for patch -1 mvninstall 0m 22s hadoop-aws in the patch failed. +1 compile 7m 3s the patch passed -1 javac 7m 3s root generated 2 new + 695 unchanged - 0 fixed = 697 total (was 695) -0 checkstyle 1m 34s root: The patch generated 24 new + 4 unchanged - 0 fixed = 28 total (was 4) +1 mvnsite 1m 45s the patch passed +1 mvneclipse 1m 1s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 4s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project -1 findbugs 0m 49s hadoop-tools/hadoop-aws generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) -1 javadoc 0m 19s hadoop-tools_hadoop-aws generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) +1 unit 0m 12s hadoop-project in the patch passed. +1 unit 8m 48s hadoop-common in the patch passed. -1 unit 0m 38s hadoop-aws in the patch failed. +1 asflicense 0m 28s The patch does not generate ASF License warnings. 73m 51s Reason Tests FindBugs module:hadoop-tools/hadoop-aws   Arguments in wrong order for invocation of checkNotNull in org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.initialize(Configuration) At DynamoDBMetadataStore.java:invocation of checkNotNull in org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.initialize(Configuration) At DynamoDBMetadataStore.java: [line 213] Failed junit tests hadoop.fs.s3a.s3guard.TestLocalMetadataStore Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HADOOP-13449 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838488/HADOOP-13449-HADOOP-13345.003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux ab950f5879d2 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision HADOOP-13345 / 56b715f Default Java 1.8.0_101 findbugs v3.0.0 mvninstall https://builds.apache.org/job/PreCommit-HADOOP-Build/11050/artifact/patchprocess/patch-mvninstall-hadoop-tools_hadoop-aws.txt javac https://builds.apache.org/job/PreCommit-HADOOP-Build/11050/artifact/patchprocess/diff-compile-javac-root.txt checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11050/artifact/patchprocess/diff-checkstyle-root.txt findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/11050/artifact/patchprocess/new-findbugs-hadoop-tools_hadoop-aws.html javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/11050/artifact/patchprocess/diff-javadoc-javadoc-hadoop-tools_hadoop-aws.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11050/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-aws.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11050/testReport/ modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11050/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          liuml07 Mingliang Liu added a comment -

          v4 patch is to address the javac, findbugs, checkstyle warnings.

          The maven install warning and test failure are very related.

          1. Yes we should bump the aws sdk version in a separate patch. See my above comments.
          2. We should fix the LocalMetadataStore, see my above comments as well.
          Show
          liuml07 Mingliang Liu added a comment - v4 patch is to address the javac, findbugs, checkstyle warnings. The maven install warning and test failure are very related. Yes we should bump the aws sdk version in a separate patch. See my above comments. We should fix the LocalMetadataStore, see my above comments as well.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 19s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 6 new or modified test files.
          0 mvndep 0m 15s Maven dependency ordering for branch
          +1 mvninstall 6m 47s HADOOP-13345 passed
          +1 compile 6m 57s HADOOP-13345 passed
          +1 checkstyle 1m 28s HADOOP-13345 passed
          +1 mvnsite 1m 33s HADOOP-13345 passed
          +1 mvneclipse 0m 41s HADOOP-13345 passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project
          +1 findbugs 1m 47s HADOOP-13345 passed
          +1 javadoc 1m 12s HADOOP-13345 passed
          0 mvndep 0m 17s Maven dependency ordering for patch
          -1 mvninstall 0m 21s hadoop-aws in the patch failed.
          +1 compile 6m 55s the patch passed
          +1 javac 6m 55s the patch passed
          +1 checkstyle 2m 53s the patch passed
          +1 mvnsite 1m 58s the patch passed
          +1 mvneclipse 0m 59s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 3s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project
          +1 findbugs 2m 10s the patch passed
          -1 javadoc 0m 20s hadoop-tools_hadoop-aws generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)
          +1 unit 0m 15s hadoop-project in the patch passed.
          +1 unit 8m 14s hadoop-common in the patch passed.
          -1 unit 0m 37s hadoop-aws in the patch failed.
          +1 asflicense 0m 28s The patch does not generate ASF License warnings.
          71m 52s



          Reason Tests
          Failed junit tests hadoop.fs.s3a.s3guard.TestLocalMetadataStore



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HADOOP-13449
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838628/HADOOP-13449-HADOOP-13345.004.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux 94ee94c329d4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision HADOOP-13345 / 56b715f
          Default Java 1.8.0_101
          findbugs v3.0.0
          mvninstall https://builds.apache.org/job/PreCommit-HADOOP-Build/11051/artifact/patchprocess/patch-mvninstall-hadoop-tools_hadoop-aws.txt
          javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/11051/artifact/patchprocess/diff-javadoc-javadoc-hadoop-tools_hadoop-aws.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11051/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-aws.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11051/testReport/
          modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11051/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 6 new or modified test files. 0 mvndep 0m 15s Maven dependency ordering for branch +1 mvninstall 6m 47s HADOOP-13345 passed +1 compile 6m 57s HADOOP-13345 passed +1 checkstyle 1m 28s HADOOP-13345 passed +1 mvnsite 1m 33s HADOOP-13345 passed +1 mvneclipse 0m 41s HADOOP-13345 passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project +1 findbugs 1m 47s HADOOP-13345 passed +1 javadoc 1m 12s HADOOP-13345 passed 0 mvndep 0m 17s Maven dependency ordering for patch -1 mvninstall 0m 21s hadoop-aws in the patch failed. +1 compile 6m 55s the patch passed +1 javac 6m 55s the patch passed +1 checkstyle 2m 53s the patch passed +1 mvnsite 1m 58s the patch passed +1 mvneclipse 0m 59s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 3s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project +1 findbugs 2m 10s the patch passed -1 javadoc 0m 20s hadoop-tools_hadoop-aws generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) +1 unit 0m 15s hadoop-project in the patch passed. +1 unit 8m 14s hadoop-common in the patch passed. -1 unit 0m 37s hadoop-aws in the patch failed. +1 asflicense 0m 28s The patch does not generate ASF License warnings. 71m 52s Reason Tests Failed junit tests hadoop.fs.s3a.s3guard.TestLocalMetadataStore Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HADOOP-13449 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838628/HADOOP-13449-HADOOP-13345.004.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 94ee94c329d4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision HADOOP-13345 / 56b715f Default Java 1.8.0_101 findbugs v3.0.0 mvninstall https://builds.apache.org/job/PreCommit-HADOOP-Build/11051/artifact/patchprocess/patch-mvninstall-hadoop-tools_hadoop-aws.txt javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/11051/artifact/patchprocess/diff-javadoc-javadoc-hadoop-tools_hadoop-aws.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11051/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-aws.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11051/testReport/ modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11051/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          Hi, Mingliang Liu

          The patch LGTM overall. +1. I am OK to commit the last patch in the feature branch.

          Some nits that are not necessary to be addressed in this patch:

          • void deleteTable() should be an interface method for MetadataStore, and maybe rename it to destroy() / tearDown()? Each MS should have this function and not all of them are implemented "table" concept.
          • Can we move createTable() from initialization() ? or add a flag like CREATE_IF_NOT_EXISTS in configuration? Because for the s3guard destroy -m URI command, it needs to initialize the DynamoDBMetadataStore instance first and then call destroy() / deleteTable().
          Show
          eddyxu Lei (Eddy) Xu added a comment - Hi, Mingliang Liu The patch LGTM overall. +1. I am OK to commit the last patch in the feature branch. Some nits that are not necessary to be addressed in this patch: void deleteTable() should be an interface method for MetadataStore , and maybe rename it to destroy() / tearDown() ? Each MS should have this function and not all of them are implemented "table" concept. Can we move createTable() from initialization() ? or add a flag like CREATE_IF_NOT_EXISTS in configuration? Because for the s3guard destroy -m URI command, it needs to initialize the DynamoDBMetadataStore instance first and then call destroy() / deleteTable() .
          Hide
          liuml07 Mingliang Liu added a comment - - edited

          Thanks Lei (Eddy) Xu for your review and insightful comments.

          1. Yes we can define a new method for destroying in MS interface. At least, that makes sense to DDB/MySQL stores. I'll upload a refined patch with this addressed along with isAuthoritative.
          2. As to not creating the table in initialization or adding a flag indicating the CREATE_IF_NOT_EXISTS , I did consider this, but was not sure about this.
            • The 1st concern I have is about the mapping. I propose to map one MetadataStore to one S3 bucket. MetadataStore does not have utility or general functions for operating metadatas elsewhere except the specific S3 bucket (and thus MetadataStore, say, a Table). After initialization all operations will operate against the same S3 bucket; we don't need to specify the associated Table for put/get/list etc operations; we don't need to specify the name for creation and destroy either.
            • The second point is about the initialization() semantic. In S3AFileSystem#initialize, it checks the bucket exists or not. It associate the file system object with the bucket explicitly. If we initialize the DDBMetadataStore from an S3AFileSystem, we associate the Table with the bucket as well. Once initialized successfully, we know that the table is there, and incoming operations are free to go. That's why I assumed we always create table if not exists. If the table exists, the createTable() will use the existing one instead of creating a new one. For destroy/deleteTable operations, the target table being existent at initialization time is even more meaningful. The overhead by now is considered small; we can of course optimize this part later. Do this make sense?
          Show
          liuml07 Mingliang Liu added a comment - - edited Thanks Lei (Eddy) Xu for your review and insightful comments. Yes we can define a new method for destroying in MS interface. At least, that makes sense to DDB/MySQL stores. I'll upload a refined patch with this addressed along with isAuthoritative . As to not creating the table in initialization or adding a flag indicating the CREATE_IF_NOT_EXISTS , I did consider this, but was not sure about this. The 1st concern I have is about the mapping. I propose to map one MetadataStore to one S3 bucket. MetadataStore does not have utility or general functions for operating metadatas elsewhere except the specific S3 bucket (and thus MetadataStore, say, a Table). After initialization all operations will operate against the same S3 bucket; we don't need to specify the associated Table for put/get/list etc operations; we don't need to specify the name for creation and destroy either. The second point is about the initialization() semantic. In S3AFileSystem#initialize , it checks the bucket exists or not. It associate the file system object with the bucket explicitly. If we initialize the DDBMetadataStore from an S3AFileSystem, we associate the Table with the bucket as well. Once initialized successfully, we know that the table is there, and incoming operations are free to go. That's why I assumed we always create table if not exists. If the table exists, the createTable() will use the existing one instead of creating a new one. For destroy/deleteTable operations, the target table being existent at initialization time is even more meaningful. The overhead by now is considered small; we can of course optimize this part later. Do this make sense?
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          Mingliang Liu If you is going to make a new patch. Would you also please add two configuration keys for Dynamodb write / read capacity unit?

          The CLI tool needs a way to customize these.

          Thanks.

          Show
          eddyxu Lei (Eddy) Xu added a comment - Mingliang Liu If you is going to make a new patch. Would you also please add two configuration keys for Dynamodb write / read capacity unit? The CLI tool needs a way to customize these. Thanks.
          Hide
          liuml07 Mingliang Liu added a comment -

          If the user is gonna destroy a non-existent table, that will be interesting. We don't have to create a new table, wait it to become active, and then delete the table, wait the table become deleted... that's not good.

          Let me add an internal (not for users) config key CREATE_IF_NOT_EXISTS, whose default value will be true; in the command tool for deletion/destroy work, the config will be set false. How about this?

          p.s. I just noticed but yes write/read capacity unit can be configured. If we're using an existing table, should we provision it according to the config keys? Or we only use this one for newly created tables?

          Thanks.

          Show
          liuml07 Mingliang Liu added a comment - If the user is gonna destroy a non-existent table, that will be interesting. We don't have to create a new table, wait it to become active, and then delete the table, wait the table become deleted... that's not good. Let me add an internal (not for users) config key CREATE_IF_NOT_EXISTS, whose default value will be true; in the command tool for deletion/destroy work, the config will be set false. How about this? p.s. I just noticed but yes write/read capacity unit can be configured. If we're using an existing table, should we provision it according to the config keys? Or we only use this one for newly created tables? Thanks.
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          If we're using an existing table, should we provision it according to the config keys? Or we only use this one for newly created tables?

          From what I understand, we can only specify read/write capacity when create the table? I think it is ok to just use these for newly created tables.

          Show
          eddyxu Lei (Eddy) Xu added a comment - If we're using an existing table, should we provision it according to the config keys? Or we only use this one for newly created tables? From what I understand, we can only specify read/write capacity when create the table? I think it is ok to just use these for newly created tables.
          Hide
          liuml07 Mingliang Liu added a comment -

          We can provision the capacity of an existing table in DDB, and DynamoDBMetadataStore also supports this via DynamoDBMetadataStore#provisionTable. This may be useful to the command line tools.

          In initialization, it's also works for me to only consider the config keys while creating table.

          Show
          liuml07 Mingliang Liu added a comment - We can provision the capacity of an existing table in DDB, and DynamoDBMetadataStore also supports this via DynamoDBMetadataStore#provisionTable . This may be useful to the command line tools. In initialization , it's also works for me to only consider the config keys while creating table.
          Hide
          fabbri Aaron Fabbri added a comment -

          Hey I've been travelling but I'm back and will try to get you a review on this shortly. I had a couple minor comments so far, that could be addressed in subsequent JIRA if needed.

          Show
          fabbri Aaron Fabbri added a comment - Hey I've been travelling but I'm back and will try to get you a review on this shortly. I had a couple minor comments so far, that could be addressed in subsequent JIRA if needed.
          Hide
          fabbri Aaron Fabbri added a comment -

          Thanks for your hard work on this Mingliang Liu!

          -    <aws-java-sdk.version>1.10.6</aws-java-sdk.version>
          +    <aws-java-sdk.version>1.11.0</aws-java-sdk.version>
          

          We should apply this to trunk first, separately, and merge back to feature branch, as Steve Loughran suggested. Let me know if you want help w/ that.

          +  @Test
          +  public void testDescendantsIterator() throws IOException {
          

          Thanks for the extra test code! Should we move this to MetadataStoreTestBase? That is, should this test run for any MetadataStore?

            
          -  private void verifyBasicFileStatus(PathMetadata meta) {
          -    FileStatus status = meta.getFileStatus();
          +  void verifyFileStatus(FileStatus status, long size) {
               assertFalse("Not a dir", status.isDirectory());
          -    assertEquals("Replication value", REPLICATION, status.getReplication());
          -    assertEquals("Access time", accessTime, status.getAccessTime());
               assertEquals("Mod time", modTime, status.getModificationTime());
          +    assertEquals("File size", size, status.getLen());
               assertEquals("Block size", BLOCK_SIZE, status.getBlockSize());
          -    assertEquals("Owner", OWNER, status.getOwner());
          -    assertEquals("Group", GROUP, status.getGroup());
          -    assertEquals("Permission", PERMISSION, status.getPermission());
             }
            
          -  private FileStatus makeDirStatus(String pathStr) {
          +  private FileStatus makeDirStatus(String pathStr)
          +      throws IOException {
               return basicFileStatus(new Path(pathStr), 0, true);
             }
            
          -  private void verifyDirStatus(PathMetadata meta) {
          -    FileStatus status = meta.getFileStatus();
          +  /**
          +   * Verify the directory file status. Subclass may verify additional fields.
          +   */
          +  void verifyDirStatus(FileStatus status) {
               assertTrue("Is a dir", status.isDirectory());
               assertEquals("zero length", 0, status.getLen());
          -    assertEquals("Replication value", REPLICATION, status.getReplication());
          -    assertEquals("Access time", accessTime, status.getAccessTime());
          -    assertEquals("Mod time", modTime, status.getModificationTime());
          -    assertEquals("Owner", OWNER, status.getOwner());
          -    assertEquals("Group", GROUP, status.getGroup());
          -    assertEquals("Permission", PERMISSION, status.getPermission());
          +  }
          +
          

          Glad you got this working. I'd like to keep this test code though. Can we extend the MetadataStoreTestBase instead of changing it? I made some suggestions here.

          Similarly, the changes in TestLocalMetadataStore should not be needed. Also I think that code has changed in the latest version posted in HADOOP-13651, so those changes would conflict on merge.

          -    public MetadataStore getMetadataStore() throws IOException {
          -      LocalMetadataStore lms = new LocalMetadataStore();
          -      return lms;
          +    public MetadataStore getMs() {
          

          This function rename will probably cause merge headaches for both of us as well.

          Other than that, looks good. Have you ran any of the integration / contract tests with it yet? I suppose not since you'll need HADOOP-13651 first.

          Show
          fabbri Aaron Fabbri added a comment - Thanks for your hard work on this Mingliang Liu ! - <aws-java-sdk.version>1.10.6</aws-java-sdk.version> + <aws-java-sdk.version>1.11.0</aws-java-sdk.version> We should apply this to trunk first, separately, and merge back to feature branch, as Steve Loughran suggested. Let me know if you want help w/ that. + @Test + public void testDescendantsIterator() throws IOException { Thanks for the extra test code! Should we move this to MetadataStoreTestBase? That is, should this test run for any MetadataStore? - private void verifyBasicFileStatus(PathMetadata meta) { - FileStatus status = meta.getFileStatus(); + void verifyFileStatus(FileStatus status, long size) { assertFalse( "Not a dir" , status.isDirectory()); - assertEquals( "Replication value" , REPLICATION, status.getReplication()); - assertEquals( "Access time" , accessTime, status.getAccessTime()); assertEquals( "Mod time" , modTime, status.getModificationTime()); + assertEquals( "File size" , size, status.getLen()); assertEquals( "Block size" , BLOCK_SIZE, status.getBlockSize()); - assertEquals( "Owner" , OWNER, status.getOwner()); - assertEquals( "Group" , GROUP, status.getGroup()); - assertEquals( "Permission" , PERMISSION, status.getPermission()); } - private FileStatus makeDirStatus( String pathStr) { + private FileStatus makeDirStatus( String pathStr) + throws IOException { return basicFileStatus( new Path(pathStr), 0, true ); } - private void verifyDirStatus(PathMetadata meta) { - FileStatus status = meta.getFileStatus(); + /** + * Verify the directory file status. Subclass may verify additional fields. + */ + void verifyDirStatus(FileStatus status) { assertTrue( "Is a dir" , status.isDirectory()); assertEquals( "zero length" , 0, status.getLen()); - assertEquals( "Replication value" , REPLICATION, status.getReplication()); - assertEquals( "Access time" , accessTime, status.getAccessTime()); - assertEquals( "Mod time" , modTime, status.getModificationTime()); - assertEquals( "Owner" , OWNER, status.getOwner()); - assertEquals( "Group" , GROUP, status.getGroup()); - assertEquals( "Permission" , PERMISSION, status.getPermission()); + } + Glad you got this working. I'd like to keep this test code though. Can we extend the MetadataStoreTestBase instead of changing it? I made some suggestions here . Similarly, the changes in TestLocalMetadataStore should not be needed. Also I think that code has changed in the latest version posted in HADOOP-13651 , so those changes would conflict on merge. - public MetadataStore getMetadataStore() throws IOException { - LocalMetadataStore lms = new LocalMetadataStore(); - return lms; + public MetadataStore getMs() { This function rename will probably cause merge headaches for both of us as well. Other than that, looks good. Have you ran any of the integration / contract tests with it yet? I suppose not since you'll need HADOOP-13651 first.
          Hide
          liuml07 Mingliang Liu added a comment -

          Thanks for your review Aaron Fabbri. Quick reply (will consider more carefully before I submit a new patch this week):

          1. Yes, the AWS SDK version bump should be separated out. However, I found the DynamoDBLocal is not included in version >1.11.0.1 Steve does have a patch in HADOOP-13050, which bumps up the version to 1.11.45. I guess we may have to use different DynamoDBLocal version from AWS SDK (and thus DynamoDB) version. By the way, the headache of jackson2 version, I got 2.5.5 working. Next patch will be easier to separate the version bump code out.
          2. I think moving testDescendantsIterator to MetadataStoreTestBase is a very good idea. I'll try to consolidate this in next patch. It should work.
          3. As to the changes in the test base and local metadata store. I'm sorry for that. I was selfish to make my test pass only. I was indeed make the test methods in base class common/general enough so that both DDB and local MS can use this. Plus, the local MS can override them:
            from the patch
            	61	
            62	  @Override
            63	  protected void verifyFileStatus(FileStatus status, long size) {
            64	    super.verifyFileStatus(status, size);
            65	
            66	    assertEquals("Replication value", REPLICATION, status.getReplication());
            67	    assertEquals("Access time", getAccessTime(), status.getAccessTime());
            68	    assertEquals("Owner", OWNER, status.getOwner());
            69	    assertEquals("Group", GROUP, status.getGroup());
            70	    assertEquals("Permission", PERMISSION, status.getPermission());
            71	  }
            72	
            73	  @Override
            74	  protected void verifyDirStatus(FileStatus status) {
            75	    super.verifyDirStatus(status);
            76	
            77	    assertEquals("Mod time", getModTime(), status.getModificationTime());
            78	    assertEquals("Replication value", REPLICATION, status.getReplication());
            79	    assertEquals("Access time", getAccessTime(), status.getAccessTime());
            80	    assertEquals("Owner", OWNER, status.getOwner());
            81	    assertEquals("Group", GROUP, status.getGroup());
            82	    assertEquals("Permission", PERMISSION, status.getPermission());
            83	  }
            

            Do you want me to make the base test methods untouched (as well as local), and make DDB test override the super methods? That is also feasible, but the base class will not be general/common enough; plus we may have duplicate code.

          4. public abstract MetadataStore getMs(); in AbstractMSContract was unexpected. I must have been fooled by my IDE.
          5. No I'll finish my review (sorry for the delay) for HADOOP-13651 before I run any integration/contract tests. We may have a few of follow-up JIRAs to make end2end tests work.

          The comments are driving this patch in the right direction. But.... did I promise too much about the next version patch? I may post a mid-way patch for quick feedback.

          My final concern is that, the MetadataStore assumes all ancestor directories (including direct parent directory) have been pre-created by the caller/user. I have to change the base test MetadataStoreTestBase to make all the tests pass. We have to change LocalMetadataStore accordingly. See my above comments.

          Show
          liuml07 Mingliang Liu added a comment - Thanks for your review Aaron Fabbri . Quick reply (will consider more carefully before I submit a new patch this week): Yes, the AWS SDK version bump should be separated out. However, I found the DynamoDBLocal is not included in version >1.11.0.1 Steve does have a patch in HADOOP-13050 , which bumps up the version to 1.11.45. I guess we may have to use different DynamoDBLocal version from AWS SDK (and thus DynamoDB) version. By the way, the headache of jackson2 version, I got 2.5.5 working. Next patch will be easier to separate the version bump code out. I think moving testDescendantsIterator to MetadataStoreTestBase is a very good idea. I'll try to consolidate this in next patch. It should work. As to the changes in the test base and local metadata store. I'm sorry for that. I was selfish to make my test pass only. I was indeed make the test methods in base class common/general enough so that both DDB and local MS can use this. Plus, the local MS can override them: from the patch 61 62 @Override 63 protected void verifyFileStatus(FileStatus status, long size) { 64 super .verifyFileStatus(status, size); 65 66 assertEquals( "Replication value" , REPLICATION, status.getReplication()); 67 assertEquals( "Access time" , getAccessTime(), status.getAccessTime()); 68 assertEquals( "Owner" , OWNER, status.getOwner()); 69 assertEquals( "Group" , GROUP, status.getGroup()); 70 assertEquals( "Permission" , PERMISSION, status.getPermission()); 71 } 72 73 @Override 74 protected void verifyDirStatus(FileStatus status) { 75 super .verifyDirStatus(status); 76 77 assertEquals( "Mod time" , getModTime(), status.getModificationTime()); 78 assertEquals( "Replication value" , REPLICATION, status.getReplication()); 79 assertEquals( "Access time" , getAccessTime(), status.getAccessTime()); 80 assertEquals( "Owner" , OWNER, status.getOwner()); 81 assertEquals( "Group" , GROUP, status.getGroup()); 82 assertEquals( "Permission" , PERMISSION, status.getPermission()); 83 } Do you want me to make the base test methods untouched (as well as local), and make DDB test override the super methods? That is also feasible, but the base class will not be general/common enough; plus we may have duplicate code. public abstract MetadataStore getMs(); in AbstractMSContract was unexpected. I must have been fooled by my IDE. No I'll finish my review (sorry for the delay) for HADOOP-13651 before I run any integration/contract tests. We may have a few of follow-up JIRAs to make end2end tests work. The comments are driving this patch in the right direction. But.... did I promise too much about the next version patch? I may post a mid-way patch for quick feedback. My final concern is that, the MetadataStore assumes all ancestor directories (including direct parent directory) have been pre-created by the caller/user. I have to change the base test MetadataStoreTestBase to make all the tests pass. We have to change LocalMetadataStore accordingly. See my above comments.
          Hide
          fabbri Aaron Fabbri added a comment -

          Thanks Mingliang Liu, sounds good. Can't wait to try this stuff out.

          My final concern is that, the MetadataStore assumes all ancestor directories (including direct parent directory) have been pre-created by the caller/user. I have to change the base test MetadataStoreTestBase to make all the tests pass. We have to change LocalMetadataStore accordingly. See my above comments.

          I looked in the test code and did not see the bug you mention. Feel free to call out a particular line of code: I may have missed it.

          This is the intention: Say you start by doing a MetadataStore.put(PathMetadata(/a/b/file)).

          That file should appear when listing /a/b.

          That is, MetadataStore.listChildren(/a/b) should return

          {[/a/b/file], isAuthoritative=false}

          . This is used by S3AFileSystem#listStatus() to get list consistency (e.g. if /a/b/file is not visible in s3 yet, we will add that entry).

          However, there is no requirement that MetadataStore.get(/a/b) should return anything. So the parent directory does not need to be created, per se, but the new file does need to appear when listing the parent.

          If the MetadataStore contract tests do not reflect this, I am very happy to change them. Also you might have missed some other test code (helper functions) that explicitly create parent dirs. We don't want to assume creation of ancestors at all, just file to show up in parent listing

          The LocalMetadataStore ends up creating a parent dir internally, but that should be implementation detail, and not required as above.

          Does that help clarify?

          Show
          fabbri Aaron Fabbri added a comment - Thanks Mingliang Liu , sounds good. Can't wait to try this stuff out. My final concern is that, the MetadataStore assumes all ancestor directories (including direct parent directory) have been pre-created by the caller/user. I have to change the base test MetadataStoreTestBase to make all the tests pass. We have to change LocalMetadataStore accordingly. See my above comments. I looked in the test code and did not see the bug you mention. Feel free to call out a particular line of code: I may have missed it. This is the intention: Say you start by doing a MetadataStore.put(PathMetadata(/a/b/file)). That file should appear when listing /a/b. That is, MetadataStore.listChildren(/a/b) should return {[/a/b/file], isAuthoritative=false} . This is used by S3AFileSystem#listStatus() to get list consistency (e.g. if /a/b/file is not visible in s3 yet, we will add that entry). However, there is no requirement that MetadataStore.get(/a/b) should return anything. So the parent directory does not need to be created, per se, but the new file does need to appear when listing the parent. If the MetadataStore contract tests do not reflect this, I am very happy to change them. Also you might have missed some other test code (helper functions) that explicitly create parent dirs. We don't want to assume creation of ancestors at all, just file to show up in parent listing The LocalMetadataStore ends up creating a parent dir internally, but that should be implementation detail, and not required as above. Does that help clarify?
          Hide
          liuml07 Mingliang Liu added a comment -

          When we put /a/b/file, /a/b/ does not need to be there. MetadataStore will not enforce this. Either the caller has to pre-create the parent, or she knows about this and do not make any assumption about it in application. I agree that the put is a simple request for one exact entry to the metadata store.

          But for listStatus(), my understanding is that we assume the entry per se is there. Or else, if we query the DDB and no entries having parent as this path, is the directory nonexistent, or the directory is empty? DDBMetadataStore should return DirListingMetadata accordingly. Thanks,

          Show
          liuml07 Mingliang Liu added a comment - When we put /a/b/file , /a/b/ does not need to be there. MetadataStore will not enforce this. Either the caller has to pre-create the parent, or she knows about this and do not make any assumption about it in application. I agree that the put is a simple request for one exact entry to the metadata store. But for listStatus() , my understanding is that we assume the entry per se is there. Or else, if we query the DDB and no entries having parent as this path, is the directory nonexistent, or the directory is empty? DDBMetadataStore should return DirListingMetadata accordingly. Thanks,
          Hide
          fabbri Aaron Fabbri added a comment -

          But for listStatus(), my understanding is that we assume the entry per se is there.

          The only thing that matters is the semantics or API contract. As long as you have the behavior I outlined, it is correct.

          We cannot require client to put(parent) before put(child), since we may run on an existing bucket where the directory was already created before we started our cluster.

          Or else, if we query the DDB and no entries having parent as this path, is the directory nonexistent, or the directory is empty? DDBMetadataStore should return DirListingMetadata accordingly. Thanks,

          Ok, I think this is an implementation detail for DynamoDB. Two ideas. #1 seems pretty good:

          1. Do a prefix scan.. I thought DynamoDB had built-in support for looking up values by key prefix. I.e. begins_with. When you do a listChildren(parent), you can just query for key begins_with parent?

          2. Create the parent path when you create the child so you can implement listChildren() properly (what I did for initial LocalMetadataStore).

          Also, can't you use prefix queries to eliminate the whole DescendantsIterator thing for recursive delete?

          Thanks for the discussion Mingliang Liu.

          Show
          fabbri Aaron Fabbri added a comment - But for listStatus(), my understanding is that we assume the entry per se is there. The only thing that matters is the semantics or API contract. As long as you have the behavior I outlined, it is correct. We cannot require client to put(parent) before put(child), since we may run on an existing bucket where the directory was already created before we started our cluster. Or else, if we query the DDB and no entries having parent as this path, is the directory nonexistent, or the directory is empty? DDBMetadataStore should return DirListingMetadata accordingly. Thanks, Ok, I think this is an implementation detail for DynamoDB. Two ideas. #1 seems pretty good: 1. Do a prefix scan.. I thought DynamoDB had built-in support for looking up values by key prefix. I.e. begins_with . When you do a listChildren(parent), you can just query for key begins_with parent ? 2. Create the parent path when you create the child so you can implement listChildren() properly (what I did for initial LocalMetadataStore). Also, can't you use prefix queries to eliminate the whole DescendantsIterator thing for recursive delete? Thanks for the discussion Mingliang Liu .
          Hide
          fabbri Aaron Fabbri added a comment -

          Sorry, I was a little hasty to comment without the details of your DynamoDB schema.

          Since you use the parent dir as the key for each entry, listChildren(/a/b) can just return all items with key=/a/b, right?

          You shouldn't have to create any separate entry for parent (unless client specifically does put(PathMetadata(/a/b)))

          Still curious if you can use begins_with query to implement recursive delete. If so, that could be a future optimization.

          Next time you run the existing test code, and you have a failure related to this, feel free to email/JIRA mention me and I'll work with you on it. I'll take another look at the MetadataStoreTestBase now.

          Show
          fabbri Aaron Fabbri added a comment - Sorry, I was a little hasty to comment without the details of your DynamoDB schema. Since you use the parent dir as the key for each entry, listChildren(/a/b) can just return all items with key=/a/b, right? You shouldn't have to create any separate entry for parent (unless client specifically does put(PathMetadata(/a/b)) ) Still curious if you can use begins_with query to implement recursive delete. If so, that could be a future optimization. Next time you run the existing test code, and you have a failure related to this, feel free to email/JIRA mention me and I'll work with you on it. I'll take another look at the MetadataStoreTestBase now.
          Hide
          fabbri Aaron Fabbri added a comment -

          One more answer.. Sorry for spamming but I have a hard time parsing the questions sometimes (need a whiteboard).

          But for listStatus(), my understanding is that we assume the entry per se is there. Or else, if we query the DDB and no entries having parent as this path, is the directory nonexistent, or the directory is empty?

          I think in this case you return null. This means "I do not have any state for that directory". The client has to check S3.

          Once we have delete tracking, we might look for an entry for the directory itself with isDeleted=true, and return file not found, but we are not doing delete tracking initially (just focus on list consistency).

          Show
          fabbri Aaron Fabbri added a comment - One more answer.. Sorry for spamming but I have a hard time parsing the questions sometimes (need a whiteboard). But for listStatus(), my understanding is that we assume the entry per se is there. Or else, if we query the DDB and no entries having parent as this path, is the directory nonexistent, or the directory is empty? I think in this case you return null. This means "I do not have any state for that directory". The client has to check S3. Once we have delete tracking, we might look for an entry for the directory itself with isDeleted=true, and return file not found, but we are not doing delete tracking initially (just focus on list consistency).
          Hide
          liuml07 Mingliang Liu added a comment -

          Thanks for the ideas. Now we're getting close to the depth.

          The client has to check S3.

          I see the difference that we have now. After import, I was treating the DDB as the consistent store and authoritative by default. Suppose we have /a, /a/b, /a/c, for delete operations, we delete /a/ first; if other thread/process comes to list /a, it returns null (because /a is not there) indicating the subtree does not exist, though /a/b/ and /a/c/ are there. We may need to cover corner cases. Another question is that, if another process bypasses S3Guard and puts a new entry to /a/d/, do we have to make sure /a/d be added to the store by checking S3? I was thinking not; S3Guard guards those who buy it.

          My previous thoughts: scan was not acceptable. If we should use scan; we have to redesign the DDB schema. Creating the parent path while creating the child is not efficient; for putting X files in the same directory, we don't want to check X times ancestors which bring heavy overhead. For empty directories, I didn't have to go to S3 as we may need time to get latest state. There must be feasible solutions/workarounds for DDBMetadataStore implementation if we agree on the list contract; I'll post my ideas later.

          Now I have to think about the difference on the "contract" for list. As to the concern of not strictly having parent directories pre-created, is importing the only one? Cluster not being started is OK; DDB persists the data. We can import data via tools (e.g. command line) first on Day 1.

          Show
          liuml07 Mingliang Liu added a comment - Thanks for the ideas. Now we're getting close to the depth. The client has to check S3. I see the difference that we have now. After import, I was treating the DDB as the consistent store and authoritative by default. Suppose we have /a, /a/b, /a/c, for delete operations, we delete /a/ first; if other thread/process comes to list /a, it returns null (because /a is not there) indicating the subtree does not exist, though /a/b/ and /a/c/ are there. We may need to cover corner cases. Another question is that, if another process bypasses S3Guard and puts a new entry to /a/d/, do we have to make sure /a/d be added to the store by checking S3? I was thinking not; S3Guard guards those who buy it. My previous thoughts: scan was not acceptable. If we should use scan; we have to redesign the DDB schema. Creating the parent path while creating the child is not efficient; for putting X files in the same directory, we don't want to check X times ancestors which bring heavy overhead. For empty directories, I didn't have to go to S3 as we may need time to get latest state. There must be feasible solutions/workarounds for DDBMetadataStore implementation if we agree on the list contract; I'll post my ideas later. Now I have to think about the difference on the "contract" for list. As to the concern of not strictly having parent directories pre-created, is importing the only one? Cluster not being started is OK; DDB persists the data. We can import data via tools (e.g. command line) first on Day 1.
          Hide
          fabbri Aaron Fabbri added a comment -

          I see the difference that we have now. After import, I was treating the DDB as the consistent store and authoritative by default.

          An import process could set all directory entries as authoritative.. Or, for v1, you could always return authoritative = false. The ideal third option is to set directories as authoritative (fully-cached) only when the client does so via put(DirListingMetadata). The interface allows any of these behaviors. None of them should require changing the API semantics or the MS Contract tests, unless there is a bug in my tests

          For the ideal option #3, where would you store the isAuthoritative bit without extra queries? Maybe that is future work. I have some ideas.

          Suppose we have /a, /a/b, /a/c, for delete operations, we delete /a/ first; if other thread/process comes to list /a, it returns null (because /a is not there) indicating the subtree does not exist, though /a/b/ and /a/c/ are there.

          That sounds like valid behavior. There are many race conditions for concurrent directory updates, both with, and without, S3Guard.

          (Slightly related: The filesystem is responsible for ensuring that the delete to /a must be recursive since it is not empty. MetadataStore explicitly does not do that.)

          Another question is that, if another process bypasses S3Guard and puts a new entry to /a/d/, do we have to make sure /a/d be added to the store by checking S3? I was thinking not; S3Guard guards those who buy it.

          Correct. Consistency only works for those that use S3Guard. If fs.s3a.metadatastore.authoritative config is false (default) the client will ignore the isAuthoritative return value from listChildren(), and will always check S3 in addition to the MetadataStore. In this configuration, clients will discover files added outside of S3Guard. Those will be subject to eventual consistency, of course.

          As to the concern of not strictly having parent directories pre-created, is importing the only one?

          You either have to (A) pay money to store an extra copy of your metadata forever, or (B) spend money and time hydrating the MetadataStore each time you start a cluster.

          Also, if there are failures (DynamoDB is famous for throttling users and being difficult to provision properly), and we don't assume everything is always in DynamoDB, it makes recovery much easier. In general, you can invalidate state in the MetadataStore by just removing paths or subtrees, and the client will adaptively reload those parts from S3.

          The other concern is that I just don't understand why you would want to do the preloading.

          Can you elaborate a bit on why do you want to have parent directories pre-created? Which operation does it help with?

          JIRA communication is a bit difficult. I'm happy to host a public conference call if that would help.

          Show
          fabbri Aaron Fabbri added a comment - I see the difference that we have now. After import, I was treating the DDB as the consistent store and authoritative by default. An import process could set all directory entries as authoritative.. Or, for v1, you could always return authoritative = false. The ideal third option is to set directories as authoritative (fully-cached) only when the client does so via put(DirListingMetadata). The interface allows any of these behaviors. None of them should require changing the API semantics or the MS Contract tests, unless there is a bug in my tests For the ideal option #3, where would you store the isAuthoritative bit without extra queries? Maybe that is future work. I have some ideas. Suppose we have /a, /a/b, /a/c, for delete operations, we delete /a/ first; if other thread/process comes to list /a, it returns null (because /a is not there) indicating the subtree does not exist, though /a/b/ and /a/c/ are there. That sounds like valid behavior. There are many race conditions for concurrent directory updates, both with, and without, S3Guard. (Slightly related: The filesystem is responsible for ensuring that the delete to /a must be recursive since it is not empty. MetadataStore explicitly does not do that.) Another question is that, if another process bypasses S3Guard and puts a new entry to /a/d/, do we have to make sure /a/d be added to the store by checking S3? I was thinking not; S3Guard guards those who buy it. Correct. Consistency only works for those that use S3Guard. If fs.s3a.metadatastore.authoritative config is false (default) the client will ignore the isAuthoritative return value from listChildren(), and will always check S3 in addition to the MetadataStore. In this configuration, clients will discover files added outside of S3Guard. Those will be subject to eventual consistency, of course. As to the concern of not strictly having parent directories pre-created, is importing the only one? You either have to (A) pay money to store an extra copy of your metadata forever, or (B) spend money and time hydrating the MetadataStore each time you start a cluster. Also, if there are failures (DynamoDB is famous for throttling users and being difficult to provision properly), and we don't assume everything is always in DynamoDB, it makes recovery much easier. In general, you can invalidate state in the MetadataStore by just removing paths or subtrees, and the client will adaptively reload those parts from S3. The other concern is that I just don't understand why you would want to do the preloading. Can you elaborate a bit on why do you want to have parent directories pre-created? Which operation does it help with? JIRA communication is a bit difficult. I'm happy to host a public conference call if that would help.
          Hide
          liuml07 Mingliang Liu added a comment -

          Thanks for the discussion, Aaron Fabbri. That's very helpful.

          for v1, you could always return authoritative = false.

          Yes, it's the current patch. Let's address this as a follow-up JIRA after the HADOOP-13651 and this both be committed.

          The interface allows any of these behaviors.... The filesystem is responsible for ensuring that the delete to /a must be recursive since it is not empty. MetadataStore explicitly does not do that.

          Agreed. For example, delete(path) does not check the directory path being empty.

          You either have to (A) pay money to store an extra copy of your metadata forever, or (B) spend money and time hydrating the MetadataStore each time you start a cluster.

          The metadata size is considered small and the price of DDB storage is low comparing with read/write operations pricing. If I have to choose, (A) makes more sense.

          and we don't assume everything is always in DynamoDB, it makes recovery much easier

          That's very valid. Altering S3 and MetadataStore is not atomic.

          The other concern is that I just don't understand why you would want to do the preloading.

          You mean import? I suppose not. For read/write existing s3 buckets, importing the structure first seems a prerequisite unless we assume it discovers/converges fast or we reach little consistency.
          I guess you mean the constrictions on the pre-creating parent directories. I re-read the design doc and HADOOP-13651 patch, and think you made a good point about this. Let S3AFileSystem ensure the contract.

          Moreover, I now think storing the is_empty bit in DynamoDB is not ideal. Maintaining it needs non-trivial effort and it's easy to make it wrong. Perhaps we can query via parent directories as HASH key when we need this information. This is non-trivial either; I'll think about this as my next work. We can either fix this in next patch, or I'll work on a follow-up JIRA.

          If this patch is still in question, a conference call will be very helpful. Let's schedule next week. Steve Loughran is traveling this week.

          Lei (Eddy) Xu you have more comments since I revised the latest patch?

          Thank you,

          Show
          liuml07 Mingliang Liu added a comment - Thanks for the discussion, Aaron Fabbri . That's very helpful. for v1, you could always return authoritative = false. Yes, it's the current patch. Let's address this as a follow-up JIRA after the HADOOP-13651 and this both be committed. The interface allows any of these behaviors.... The filesystem is responsible for ensuring that the delete to /a must be recursive since it is not empty. MetadataStore explicitly does not do that. Agreed. For example, delete(path) does not check the directory path being empty. You either have to (A) pay money to store an extra copy of your metadata forever, or (B) spend money and time hydrating the MetadataStore each time you start a cluster. The metadata size is considered small and the price of DDB storage is low comparing with read/write operations pricing. If I have to choose, (A) makes more sense. and we don't assume everything is always in DynamoDB, it makes recovery much easier That's very valid. Altering S3 and MetadataStore is not atomic. The other concern is that I just don't understand why you would want to do the preloading. You mean import? I suppose not. For read/write existing s3 buckets, importing the structure first seems a prerequisite unless we assume it discovers/converges fast or we reach little consistency. I guess you mean the constrictions on the pre-creating parent directories. I re-read the design doc and HADOOP-13651 patch, and think you made a good point about this. Let S3AFileSystem ensure the contract. Moreover, I now think storing the is_empty bit in DynamoDB is not ideal. Maintaining it needs non-trivial effort and it's easy to make it wrong. Perhaps we can query via parent directories as HASH key when we need this information. This is non-trivial either; I'll think about this as my next work. We can either fix this in next patch, or I'll work on a follow-up JIRA. If this patch is still in question, a conference call will be very helpful. Let's schedule next week. Steve Loughran is traveling this week. Lei (Eddy) Xu you have more comments since I revised the latest patch? Thank you,
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 20s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 6 new or modified test files.
          0 mvndep 1m 39s Maven dependency ordering for branch
          +1 mvninstall 6m 58s HADOOP-13345 passed
          +1 compile 6m 53s HADOOP-13345 passed
          +1 checkstyle 1m 27s HADOOP-13345 passed
          +1 mvnsite 1m 34s HADOOP-13345 passed
          +1 mvneclipse 0m 40s HADOOP-13345 passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project
          +1 findbugs 1m 50s HADOOP-13345 passed
          +1 javadoc 1m 13s HADOOP-13345 passed
          0 mvndep 0m 31s Maven dependency ordering for patch
          -1 mvninstall 0m 22s hadoop-aws in the patch failed.
          +1 compile 6m 55s the patch passed
          -1 javac 6m 55s root generated 5 new + 695 unchanged - 0 fixed = 700 total (was 695)
          -0 checkstyle 1m 30s root: The patch generated 10 new + 4 unchanged - 0 fixed = 14 total (was 4)
          +1 mvnsite 1m 42s the patch passed
          +1 mvneclipse 1m 1s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 4s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project
          +1 findbugs 2m 32s the patch passed
          +1 javadoc 1m 26s the patch passed
          +1 unit 0m 14s hadoop-project in the patch passed.
          +1 unit 9m 7s hadoop-common in the patch passed.
          +1 unit 0m 39s hadoop-aws in the patch passed.
          +1 asflicense 0m 29s The patch does not generate ASF License warnings.
          73m 24s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HADOOP-13449
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12839489/HADOOP-13449-HADOOP-13345.005.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux 9c2561317d10 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision HADOOP-13345 / 56b715f
          Default Java 1.8.0_111
          findbugs v3.0.0
          mvninstall https://builds.apache.org/job/PreCommit-HADOOP-Build/11095/artifact/patchprocess/patch-mvninstall-hadoop-tools_hadoop-aws.txt
          javac https://builds.apache.org/job/PreCommit-HADOOP-Build/11095/artifact/patchprocess/diff-compile-javac-root.txt
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11095/artifact/patchprocess/diff-checkstyle-root.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11095/testReport/
          modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11095/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 20s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 6 new or modified test files. 0 mvndep 1m 39s Maven dependency ordering for branch +1 mvninstall 6m 58s HADOOP-13345 passed +1 compile 6m 53s HADOOP-13345 passed +1 checkstyle 1m 27s HADOOP-13345 passed +1 mvnsite 1m 34s HADOOP-13345 passed +1 mvneclipse 0m 40s HADOOP-13345 passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project +1 findbugs 1m 50s HADOOP-13345 passed +1 javadoc 1m 13s HADOOP-13345 passed 0 mvndep 0m 31s Maven dependency ordering for patch -1 mvninstall 0m 22s hadoop-aws in the patch failed. +1 compile 6m 55s the patch passed -1 javac 6m 55s root generated 5 new + 695 unchanged - 0 fixed = 700 total (was 695) -0 checkstyle 1m 30s root: The patch generated 10 new + 4 unchanged - 0 fixed = 14 total (was 4) +1 mvnsite 1m 42s the patch passed +1 mvneclipse 1m 1s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 4s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project +1 findbugs 2m 32s the patch passed +1 javadoc 1m 26s the patch passed +1 unit 0m 14s hadoop-project in the patch passed. +1 unit 9m 7s hadoop-common in the patch passed. +1 unit 0m 39s hadoop-aws in the patch passed. +1 asflicense 0m 29s The patch does not generate ASF License warnings. 73m 24s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HADOOP-13449 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12839489/HADOOP-13449-HADOOP-13345.005.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 9c2561317d10 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision HADOOP-13345 / 56b715f Default Java 1.8.0_111 findbugs v3.0.0 mvninstall https://builds.apache.org/job/PreCommit-HADOOP-Build/11095/artifact/patchprocess/patch-mvninstall-hadoop-tools_hadoop-aws.txt javac https://builds.apache.org/job/PreCommit-HADOOP-Build/11095/artifact/patchprocess/diff-compile-javac-root.txt checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11095/artifact/patchprocess/diff-checkstyle-root.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11095/testReport/ modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11095/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          +1 for the current patch. LGTM.

          Thanks Mingliang Liu

          Show
          eddyxu Lei (Eddy) Xu added a comment - +1 for the current patch. LGTM. Thanks Mingliang Liu
          Hide
          liuml07 Mingliang Liu added a comment -

          Thanks Lei (Eddy) Xu for your review. The v6 simply addresses the checkstyle warnings and mvn install errors. For the javac warnings, that's caused by the AWS SDK version bump; considered existing ones and should be fixed separately.

          For bumping the version, if we need a separate patch, we can simply extract the changes in pom.xml files; its version number is very close to HADOOP-13050. Unfortunately that patch is not committed yet.
          If we don't need a separate patch for bumping sdk version, we can resolve that when we merge from trunk (after HADOOP-13050) later.

          Show
          liuml07 Mingliang Liu added a comment - Thanks Lei (Eddy) Xu for your review. The v6 simply addresses the checkstyle warnings and mvn install errors. For the javac warnings, that's caused by the AWS SDK version bump; considered existing ones and should be fixed separately. For bumping the version, if we need a separate patch, we can simply extract the changes in pom.xml files; its version number is very close to HADOOP-13050 . Unfortunately that patch is not committed yet. If we don't need a separate patch for bumping sdk version, we can resolve that when we merge from trunk (after HADOOP-13050 ) later.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 17s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 6 new or modified test files.
          0 mvndep 0m 17s Maven dependency ordering for branch
          +1 mvninstall 8m 15s HADOOP-13345 passed
          +1 compile 8m 6s HADOOP-13345 passed
          +1 checkstyle 1m 37s HADOOP-13345 passed
          +1 mvnsite 1m 43s HADOOP-13345 passed
          +1 mvneclipse 0m 43s HADOOP-13345 passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project
          +1 findbugs 2m 8s HADOOP-13345 passed
          +1 javadoc 1m 19s HADOOP-13345 passed
          0 mvndep 0m 41s Maven dependency ordering for patch
          +1 mvninstall 1m 13s the patch passed
          +1 compile 8m 14s the patch passed
          -1 javac 8m 14s root generated 5 new + 695 unchanged - 0 fixed = 700 total (was 695)
          -0 checkstyle 1m 37s root: The patch generated 1 new + 4 unchanged - 0 fixed = 5 total (was 4)
          +1 mvnsite 1m 51s the patch passed
          +1 mvneclipse 1m 7s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 3s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project
          +1 findbugs 2m 10s the patch passed
          +1 javadoc 1m 23s the patch passed
          +1 unit 0m 15s hadoop-project in the patch passed.
          +1 unit 8m 14s hadoop-common in the patch passed.
          +1 unit 0m 35s hadoop-aws in the patch passed.
          +1 asflicense 0m 28s The patch does not generate ASF License warnings.
          75m 38s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HADOOP-13449
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12839604/HADOOP-13449-HADOOP-13345.006.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux 4948552e7f11 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision HADOOP-13345 / 56b715f
          Default Java 1.8.0_111
          findbugs v3.0.0
          javac https://builds.apache.org/job/PreCommit-HADOOP-Build/11100/artifact/patchprocess/diff-compile-javac-root.txt
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11100/artifact/patchprocess/diff-checkstyle-root.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11100/testReport/
          modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11100/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 6 new or modified test files. 0 mvndep 0m 17s Maven dependency ordering for branch +1 mvninstall 8m 15s HADOOP-13345 passed +1 compile 8m 6s HADOOP-13345 passed +1 checkstyle 1m 37s HADOOP-13345 passed +1 mvnsite 1m 43s HADOOP-13345 passed +1 mvneclipse 0m 43s HADOOP-13345 passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project +1 findbugs 2m 8s HADOOP-13345 passed +1 javadoc 1m 19s HADOOP-13345 passed 0 mvndep 0m 41s Maven dependency ordering for patch +1 mvninstall 1m 13s the patch passed +1 compile 8m 14s the patch passed -1 javac 8m 14s root generated 5 new + 695 unchanged - 0 fixed = 700 total (was 695) -0 checkstyle 1m 37s root: The patch generated 1 new + 4 unchanged - 0 fixed = 5 total (was 4) +1 mvnsite 1m 51s the patch passed +1 mvneclipse 1m 7s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 3s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project +1 findbugs 2m 10s the patch passed +1 javadoc 1m 23s the patch passed +1 unit 0m 15s hadoop-project in the patch passed. +1 unit 8m 14s hadoop-common in the patch passed. +1 unit 0m 35s hadoop-aws in the patch passed. +1 asflicense 0m 28s The patch does not generate ASF License warnings. 75m 38s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HADOOP-13449 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12839604/HADOOP-13449-HADOOP-13345.006.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 4948552e7f11 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision HADOOP-13345 / 56b715f Default Java 1.8.0_111 findbugs v3.0.0 javac https://builds.apache.org/job/PreCommit-HADOOP-Build/11100/artifact/patchprocess/diff-compile-javac-root.txt checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11100/artifact/patchprocess/diff-checkstyle-root.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11100/testReport/ modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11100/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          fabbri Aaron Fabbri added a comment -

          Hey Mingliang Liu. I am available early next week to help work on this. I could help with:

          • The test refactoring needed to keep existing code but extend behavior for S3A/DDB-specific behavior.
          • Running the S3A integration tests with your MetadataStore and fixing bugs.
          • Working on the SDK revision bump for trunk.
          • Anything else you think of (TODOs etc).

          If you are interested in chatting on Monday about this JIRA, please email me times that work for you and I can host a public call. Monday afternoon (pacific time) is good for me.

          Also in response to your last comment, I feel like the patch is good enough to start integration testing and get list consistency (without authoritative / high-performance caching) for a V1 of the feature.

          Show
          fabbri Aaron Fabbri added a comment - Hey Mingliang Liu . I am available early next week to help work on this. I could help with: The test refactoring needed to keep existing code but extend behavior for S3A/DDB-specific behavior. Running the S3A integration tests with your MetadataStore and fixing bugs. Working on the SDK revision bump for trunk. Anything else you think of (TODOs etc). If you are interested in chatting on Monday about this JIRA, please email me times that work for you and I can host a public call. Monday afternoon (pacific time) is good for me. Also in response to your last comment, I feel like the patch is good enough to start integration testing and get list consistency (without authoritative / high-performance caching) for a V1 of the feature.
          Hide
          liuml07 Mingliang Liu added a comment -

          Thanks for the comments, aaron fabbri.

          The v7 patch rebased from the feature branch and resolved conflicts.

          The test refactoring needed to keep existing code but extend behavior for S3A/DDB-specific behavior.

          The v7 patch refactored the test code but did not remove any case; the base class validates common fields while subclass can extend and test more (e.g. in local).

          Running the S3A integration tests with your MetadataStore and fixing bugs.

          That will be great.

          Working on the SDK revision bump for trunk.

          HADOOP-13050 may need more time to commit to trunk (and backport to S3Guard). I'd suggest we temporarily bump the sdk version here and get the code in for moving forward to test / bug fixes, and resolve conflicts later when we merge from trunk (after HADOOP-13050). For this patch, the major problem is about DDBLocal and for HADOOP-13050 I think it's Jackson. But if you guys insist, we can work on HADOOP-13050 first.

          I don't have specific questions that need conference-call discussion; but we can meet if you think we should talk about the current patch. For next steps e.g. integration tests, we will have more discussion for sure; but we can do it later. Steve's in UK so we may have to find a good time of a day.

          Show
          liuml07 Mingliang Liu added a comment - Thanks for the comments, aaron fabbri . The v7 patch rebased from the feature branch and resolved conflicts. The test refactoring needed to keep existing code but extend behavior for S3A/DDB-specific behavior. The v7 patch refactored the test code but did not remove any case; the base class validates common fields while subclass can extend and test more (e.g. in local). Running the S3A integration tests with your MetadataStore and fixing bugs. That will be great. Working on the SDK revision bump for trunk. HADOOP-13050 may need more time to commit to trunk (and backport to S3Guard). I'd suggest we temporarily bump the sdk version here and get the code in for moving forward to test / bug fixes, and resolve conflicts later when we merge from trunk (after HADOOP-13050 ). For this patch, the major problem is about DDBLocal and for HADOOP-13050 I think it's Jackson. But if you guys insist, we can work on HADOOP-13050 first. I don't have specific questions that need conference-call discussion; but we can meet if you think we should talk about the current patch. For next steps e.g. integration tests, we will have more discussion for sure; but we can do it later. Steve's in UK so we may have to find a good time of a day.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.
          0 mvndep 1m 28s Maven dependency ordering for branch
          +1 mvninstall 6m 48s HADOOP-13345 passed
          +1 compile 6m 54s HADOOP-13345 passed
          +1 checkstyle 1m 27s HADOOP-13345 passed
          +1 mvnsite 1m 34s HADOOP-13345 passed
          +1 mvneclipse 0m 41s HADOOP-13345 passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project
          +1 findbugs 1m 49s HADOOP-13345 passed
          +1 javadoc 1m 14s HADOOP-13345 passed
          0 mvndep 0m 17s Maven dependency ordering for patch
          -1 mvninstall 0m 18s hadoop-aws in the patch failed.
          -1 compile 6m 37s root in the patch failed.
          -1 javac 6m 37s root in the patch failed.
          -0 checkstyle 1m 29s root: The patch generated 7 new + 5 unchanged - 0 fixed = 12 total (was 5)
          -1 mvnsite 0m 23s hadoop-aws in the patch failed.
          +1 mvneclipse 0m 50s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 4s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project
          -1 findbugs 0m 20s hadoop-aws in the patch failed.
          +1 javadoc 1m 23s the patch passed
          +1 unit 0m 15s hadoop-project in the patch passed.
          +1 unit 8m 15s hadoop-common in the patch passed.
          -1 unit 0m 24s hadoop-aws in the patch failed.
          +1 asflicense 0m 28s The patch does not generate ASF License warnings.
          70m 19s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HADOOP-13449
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12839677/HADOOP-13449-HADOOP-13345.007.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux 07b328f0b616 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision HADOOP-13345 / f4c37a5
          Default Java 1.8.0_111
          findbugs v3.0.0
          mvninstall https://builds.apache.org/job/PreCommit-HADOOP-Build/11103/artifact/patchprocess/patch-mvninstall-hadoop-tools_hadoop-aws.txt
          compile https://builds.apache.org/job/PreCommit-HADOOP-Build/11103/artifact/patchprocess/patch-compile-root.txt
          javac https://builds.apache.org/job/PreCommit-HADOOP-Build/11103/artifact/patchprocess/patch-compile-root.txt
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11103/artifact/patchprocess/diff-checkstyle-root.txt
          mvnsite https://builds.apache.org/job/PreCommit-HADOOP-Build/11103/artifact/patchprocess/patch-mvnsite-hadoop-tools_hadoop-aws.txt
          findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/11103/artifact/patchprocess/patch-findbugs-hadoop-tools_hadoop-aws.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11103/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-aws.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11103/testReport/
          modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11103/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 5 new or modified test files. 0 mvndep 1m 28s Maven dependency ordering for branch +1 mvninstall 6m 48s HADOOP-13345 passed +1 compile 6m 54s HADOOP-13345 passed +1 checkstyle 1m 27s HADOOP-13345 passed +1 mvnsite 1m 34s HADOOP-13345 passed +1 mvneclipse 0m 41s HADOOP-13345 passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project +1 findbugs 1m 49s HADOOP-13345 passed +1 javadoc 1m 14s HADOOP-13345 passed 0 mvndep 0m 17s Maven dependency ordering for patch -1 mvninstall 0m 18s hadoop-aws in the patch failed. -1 compile 6m 37s root in the patch failed. -1 javac 6m 37s root in the patch failed. -0 checkstyle 1m 29s root: The patch generated 7 new + 5 unchanged - 0 fixed = 12 total (was 5) -1 mvnsite 0m 23s hadoop-aws in the patch failed. +1 mvneclipse 0m 50s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 4s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project -1 findbugs 0m 20s hadoop-aws in the patch failed. +1 javadoc 1m 23s the patch passed +1 unit 0m 15s hadoop-project in the patch passed. +1 unit 8m 15s hadoop-common in the patch passed. -1 unit 0m 24s hadoop-aws in the patch failed. +1 asflicense 0m 28s The patch does not generate ASF License warnings. 70m 19s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HADOOP-13449 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12839677/HADOOP-13449-HADOOP-13345.007.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 07b328f0b616 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision HADOOP-13345 / f4c37a5 Default Java 1.8.0_111 findbugs v3.0.0 mvninstall https://builds.apache.org/job/PreCommit-HADOOP-Build/11103/artifact/patchprocess/patch-mvninstall-hadoop-tools_hadoop-aws.txt compile https://builds.apache.org/job/PreCommit-HADOOP-Build/11103/artifact/patchprocess/patch-compile-root.txt javac https://builds.apache.org/job/PreCommit-HADOOP-Build/11103/artifact/patchprocess/patch-compile-root.txt checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11103/artifact/patchprocess/diff-checkstyle-root.txt mvnsite https://builds.apache.org/job/PreCommit-HADOOP-Build/11103/artifact/patchprocess/patch-mvnsite-hadoop-tools_hadoop-aws.txt findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/11103/artifact/patchprocess/patch-findbugs-hadoop-tools_hadoop-aws.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11103/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-aws.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11103/testReport/ modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11103/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          I can talk morning Palo Alto time from 09:00 to 11:00. It's good to open the calls to all. I have a webex conf #, or we can use google+ hangouts (which I prefer, though they will block anyone from China trying to dial in)

          Show
          stevel@apache.org Steve Loughran added a comment - I can talk morning Palo Alto time from 09:00 to 11:00. It's good to open the calls to all. I have a webex conf #, or we can use google+ hangouts (which I prefer, though they will block anyone from China trying to dial in)
          Hide
          stevel@apache.org Steve Loughran added a comment -

          correction; I can't talk monday, other commitments.

          w.r.t SDK and jackson, we're going to have to upgrade jackson and the SDK on branch-2, using a jackson version that is compatible at the API level with existing code. This is going to cause problems downstream, but we don't really have a choice any more.

          the jackson update should go in as HADOOP-12705

          I'm also going to have to update the aws SDK in 2.7 and 2.8; troublesome that. We need to get rid of org.json artifacts embedded in the existing AWS SDK, which means update SDK, which means Jsckson update, unless I can do one of: (a) shade everything or (b) swap in tdunning's org.json replacement (more specifically: cut the org.json lib off the AWS JAR and explicitly add ted's replacement)

          Show
          stevel@apache.org Steve Loughran added a comment - correction; I can't talk monday, other commitments. w.r.t SDK and jackson, we're going to have to upgrade jackson and the SDK on branch-2, using a jackson version that is compatible at the API level with existing code. This is going to cause problems downstream, but we don't really have a choice any more. the jackson update should go in as HADOOP-12705 I'm also going to have to update the aws SDK in 2.7 and 2.8; troublesome that. We need to get rid of org.json artifacts embedded in the existing AWS SDK, which means update SDK, which means Jsckson update, unless I can do one of: (a) shade everything or (b) swap in tdunning's org.json replacement (more specifically: cut the org.json lib off the AWS JAR and explicitly add ted's replacement)
          Hide
          liuml07 Mingliang Liu added a comment -

          The v8 patch included the changes in HADOOP-12705 which upgraded the Jackson version to 2.7.8. I think we can rebase this feature branch from trunk to include this Jackson upgrade. Moreover, we have to include the AWS SDK upgrade in HADOOP-13050. I'll review the patch shortly.

          Hopefully this will get a better Jenkins run.

          Thanks,

          Show
          liuml07 Mingliang Liu added a comment - The v8 patch included the changes in HADOOP-12705 which upgraded the Jackson version to 2.7.8. I think we can rebase this feature branch from trunk to include this Jackson upgrade. Moreover, we have to include the AWS SDK upgrade in HADOOP-13050 . I'll review the patch shortly. Hopefully this will get a better Jenkins run. Thanks,
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.
          0 mvndep 0m 17s Maven dependency ordering for branch
          +1 mvninstall 6m 53s HADOOP-13345 passed
          +1 compile 7m 56s HADOOP-13345 passed
          +1 checkstyle 1m 32s HADOOP-13345 passed
          +1 mvnsite 1m 38s HADOOP-13345 passed
          +1 mvneclipse 0m 41s HADOOP-13345 passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project
          +1 findbugs 1m 54s HADOOP-13345 passed
          +1 javadoc 1m 14s HADOOP-13345 passed
          0 mvndep 0m 22s Maven dependency ordering for patch
          +1 mvninstall 1m 17s the patch passed
          +1 compile 7m 48s the patch passed
          -1 javac 7m 48s root generated 6 new + 695 unchanged - 0 fixed = 701 total (was 695)
          +1 checkstyle 1m 32s the patch passed
          +1 mvnsite 1m 53s the patch passed
          +1 mvneclipse 0m 55s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 4s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project
          +1 findbugs 2m 39s the patch passed
          +1 javadoc 1m 27s the patch passed
          +1 unit 0m 15s hadoop-project in the patch passed.
          -1 unit 8m 35s hadoop-common in the patch failed.
          -1 unit 0m 38s hadoop-aws in the patch failed.
          +1 asflicense 0m 28s The patch does not generate ASF License warnings.
          73m 35s



          Reason Tests
          Failed junit tests hadoop.security.TestGroupsCaching
            hadoop.fs.s3a.s3guard.TestNullMetadataStore



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HADOOP-13449
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12839928/HADOOP-13449-HADOOP-13345.008.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux 907eddc6f41a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision HADOOP-13345 / f4c37a5
          Default Java 1.8.0_111
          findbugs v3.0.0
          javac https://builds.apache.org/job/PreCommit-HADOOP-Build/11111/artifact/patchprocess/diff-compile-javac-root.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11111/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11111/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-aws.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11111/testReport/
          modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11111/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 5 new or modified test files. 0 mvndep 0m 17s Maven dependency ordering for branch +1 mvninstall 6m 53s HADOOP-13345 passed +1 compile 7m 56s HADOOP-13345 passed +1 checkstyle 1m 32s HADOOP-13345 passed +1 mvnsite 1m 38s HADOOP-13345 passed +1 mvneclipse 0m 41s HADOOP-13345 passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project +1 findbugs 1m 54s HADOOP-13345 passed +1 javadoc 1m 14s HADOOP-13345 passed 0 mvndep 0m 22s Maven dependency ordering for patch +1 mvninstall 1m 17s the patch passed +1 compile 7m 48s the patch passed -1 javac 7m 48s root generated 6 new + 695 unchanged - 0 fixed = 701 total (was 695) +1 checkstyle 1m 32s the patch passed +1 mvnsite 1m 53s the patch passed +1 mvneclipse 0m 55s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 4s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project +1 findbugs 2m 39s the patch passed +1 javadoc 1m 27s the patch passed +1 unit 0m 15s hadoop-project in the patch passed. -1 unit 8m 35s hadoop-common in the patch failed. -1 unit 0m 38s hadoop-aws in the patch failed. +1 asflicense 0m 28s The patch does not generate ASF License warnings. 73m 35s Reason Tests Failed junit tests hadoop.security.TestGroupsCaching   hadoop.fs.s3a.s3guard.TestNullMetadataStore Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HADOOP-13449 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12839928/HADOOP-13449-HADOOP-13345.008.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 907eddc6f41a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision HADOOP-13345 / f4c37a5 Default Java 1.8.0_111 findbugs v3.0.0 javac https://builds.apache.org/job/PreCommit-HADOOP-Build/11111/artifact/patchprocess/diff-compile-javac-root.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11111/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11111/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-aws.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11111/testReport/ modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11111/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          liuml07 Mingliang Liu added a comment -

          The patch that bumps the AWS SDK version is committed to trunk. To help us here, I suggest we merge from trunk again; Aaron Fabbri, thoughts? Last time we did rebase IIRC?

          After that, I'll upload the patch against the changes there; I don't expect major conflicts (if any) for this patch.

          Show
          liuml07 Mingliang Liu added a comment - The patch that bumps the AWS SDK version is committed to trunk . To help us here, I suggest we merge from trunk again; Aaron Fabbri , thoughts? Last time we did rebase IIRC? After that, I'll upload the patch against the changes there; I don't expect major conflicts (if any) for this patch.
          Hide
          fabbri Aaron Fabbri added a comment -

          Sounds good Mingliang Liu. Last time I did a rebase but Steve Loughran suggested we just do merge commit to avoid the force-push and associated issues. I'm fine either way. I can do this if you like.

          Show
          fabbri Aaron Fabbri added a comment - Sounds good Mingliang Liu . Last time I did a rebase but Steve Loughran suggested we just do merge commit to avoid the force-push and associated issues. I'm fine either way. I can do this if you like.
          Hide
          liuml07 Mingliang Liu added a comment -

          I guess a git checkout HADOOP-13345 and git merge trunk will work? I tested here and it was a clean merge with commit message saying "Merge branch 'trunk' into HADOOP-13345". Then we simply git push to the remote repo? If I get it wrongly, please take care of this; thank you very much.

          Show
          liuml07 Mingliang Liu added a comment - I guess a git checkout HADOOP-13345 and git merge trunk will work? I tested here and it was a clean merge with commit message saying "Merge branch 'trunk' into HADOOP-13345 ". Then we simply git push to the remote repo? If I get it wrongly, please take care of this; thank you very much.
          Hide
          fabbri Aaron Fabbri added a comment - - edited

          Yes, I believe that is what Steve L was suggesting. Go ahead, I'm +1 on
          the merge.

          Show
          fabbri Aaron Fabbri added a comment - - edited Yes, I believe that is what Steve L was suggesting. Go ahead, I'm +1 on the merge.
          Hide
          liuml07 Mingliang Liu added a comment -

          I just merged the trunk on this feature branch HADOOP-13345 and pushed to the repo. It was clean. Please also pull for changes. Thanks Aaron Fabbri and Steve Loughran for discussion.

          The v9 patch removes its change to bump the AWS SDK version along with Jackson dependencies; we still need to add the DynamoDB in this patch though.

          The v9 patch also skips the assertion in testDescendantsIterator if it allows missing (NullMetadataStore); this should fix the failing UT.

          Show
          liuml07 Mingliang Liu added a comment - I just merged the trunk on this feature branch HADOOP-13345 and pushed to the repo. It was clean. Please also pull for changes. Thanks Aaron Fabbri and Steve Loughran for discussion. The v9 patch removes its change to bump the AWS SDK version along with Jackson dependencies; we still need to add the DynamoDB in this patch though. The v9 patch also skips the assertion in testDescendantsIterator if it allows missing (NullMetadataStore); this should fix the failing UT.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.
          0 mvndep 0m 22s Maven dependency ordering for branch
          +1 mvninstall 11m 49s HADOOP-13345 passed
          +1 compile 13m 16s HADOOP-13345 passed
          +1 checkstyle 1m 58s HADOOP-13345 passed
          +1 mvnsite 2m 22s HADOOP-13345 passed
          +1 mvneclipse 1m 18s HADOOP-13345 passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project
          +1 findbugs 2m 29s HADOOP-13345 passed
          +1 javadoc 1m 54s HADOOP-13345 passed
          0 mvndep 0m 40s Maven dependency ordering for patch
          -1 mvninstall 0m 26s hadoop-aws in the patch failed.
          +1 compile 13m 46s the patch passed
          +1 javac 13m 46s the patch passed
          +1 checkstyle 2m 2s the patch passed
          +1 mvnsite 2m 29s the patch passed
          +1 mvneclipse 2m 7s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 4s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project
          +1 findbugs 2m 35s the patch passed
          +1 javadoc 1m 55s the patch passed
          +1 unit 0m 17s hadoop-project in the patch passed.
          +1 unit 9m 54s hadoop-common in the patch passed.
          +1 unit 1m 7s hadoop-aws in the patch passed.
          +1 asflicense 0m 45s The patch does not generate ASF License warnings.
          98m 59s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-13449
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12840161/HADOOP-13449-HADOOP-13345.009.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux 99976286ab13 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision HADOOP-13345 / 86a67ff
          Default Java 1.8.0_111
          findbugs v3.0.0
          mvninstall https://builds.apache.org/job/PreCommit-HADOOP-Build/11118/artifact/patchprocess/patch-mvninstall-hadoop-tools_hadoop-aws.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11118/testReport/
          modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11118/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 5 new or modified test files. 0 mvndep 0m 22s Maven dependency ordering for branch +1 mvninstall 11m 49s HADOOP-13345 passed +1 compile 13m 16s HADOOP-13345 passed +1 checkstyle 1m 58s HADOOP-13345 passed +1 mvnsite 2m 22s HADOOP-13345 passed +1 mvneclipse 1m 18s HADOOP-13345 passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project +1 findbugs 2m 29s HADOOP-13345 passed +1 javadoc 1m 54s HADOOP-13345 passed 0 mvndep 0m 40s Maven dependency ordering for patch -1 mvninstall 0m 26s hadoop-aws in the patch failed. +1 compile 13m 46s the patch passed +1 javac 13m 46s the patch passed +1 checkstyle 2m 2s the patch passed +1 mvnsite 2m 29s the patch passed +1 mvneclipse 2m 7s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 4s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project +1 findbugs 2m 35s the patch passed +1 javadoc 1m 55s the patch passed +1 unit 0m 17s hadoop-project in the patch passed. +1 unit 9m 54s hadoop-common in the patch passed. +1 unit 1m 7s hadoop-aws in the patch passed. +1 asflicense 0m 45s The patch does not generate ASF License warnings. 98m 59s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-13449 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12840161/HADOOP-13449-HADOOP-13345.009.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 99976286ab13 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision HADOOP-13345 / 86a67ff Default Java 1.8.0_111 findbugs v3.0.0 mvninstall https://builds.apache.org/job/PreCommit-HADOOP-Build/11118/artifact/patchprocess/patch-mvninstall-hadoop-tools_hadoop-aws.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11118/testReport/ modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11118/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.
          0 mvndep 0m 18s Maven dependency ordering for branch
          +1 mvninstall 8m 41s HADOOP-13345 passed
          +1 compile 10m 3s HADOOP-13345 passed
          +1 checkstyle 1m 36s HADOOP-13345 passed
          +1 mvnsite 1m 53s HADOOP-13345 passed
          +1 mvneclipse 1m 1s HADOOP-13345 passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project
          +1 findbugs 2m 7s HADOOP-13345 passed
          +1 javadoc 1m 28s HADOOP-13345 passed
          0 mvndep 0m 18s Maven dependency ordering for patch
          +1 mvninstall 1m 16s the patch passed
          +1 compile 9m 49s the patch passed
          +1 javac 9m 49s the patch passed
          +1 checkstyle 1m 38s the patch passed
          +1 mvnsite 2m 2s the patch passed
          +1 mvneclipse 1m 6s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 4s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project
          +1 findbugs 2m 27s the patch passed
          +1 javadoc 1m 39s the patch passed
          +1 unit 0m 19s hadoop-project in the patch passed.
          +1 unit 8m 45s hadoop-common in the patch passed.
          +1 unit 0m 47s hadoop-aws in the patch passed.
          +1 asflicense 0m 37s The patch does not generate ASF License warnings.
          81m 56s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-13449
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12840184/HADOOP-13449-HADOOP-13345.010.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux 1d67e385a192 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision HADOOP-13345 / 86a67ff
          Default Java 1.8.0_111
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11120/testReport/
          modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11120/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 5 new or modified test files. 0 mvndep 0m 18s Maven dependency ordering for branch +1 mvninstall 8m 41s HADOOP-13345 passed +1 compile 10m 3s HADOOP-13345 passed +1 checkstyle 1m 36s HADOOP-13345 passed +1 mvnsite 1m 53s HADOOP-13345 passed +1 mvneclipse 1m 1s HADOOP-13345 passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project +1 findbugs 2m 7s HADOOP-13345 passed +1 javadoc 1m 28s HADOOP-13345 passed 0 mvndep 0m 18s Maven dependency ordering for patch +1 mvninstall 1m 16s the patch passed +1 compile 9m 49s the patch passed +1 javac 9m 49s the patch passed +1 checkstyle 1m 38s the patch passed +1 mvnsite 2m 2s the patch passed +1 mvneclipse 1m 6s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 4s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project +1 findbugs 2m 27s the patch passed +1 javadoc 1m 39s the patch passed +1 unit 0m 19s hadoop-project in the patch passed. +1 unit 8m 45s hadoop-common in the patch passed. +1 unit 0m 47s hadoop-aws in the patch passed. +1 asflicense 0m 37s The patch does not generate ASF License warnings. 81m 56s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-13449 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12840184/HADOOP-13449-HADOOP-13345.010.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 1d67e385a192 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision HADOOP-13345 / 86a67ff Default Java 1.8.0_111 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11120/testReport/ modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11120/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          liuml07 Mingliang Liu added a comment -

          I merged again from the trunk locally and resolved minor conflicts with HADOOP-13823. If you guys support the merge, I'll push it to the remote feature branch.

          Show
          liuml07 Mingliang Liu added a comment - I merged again from the trunk locally and resolved minor conflicts with HADOOP-13823 . If you guys support the merge, I'll push it to the remote feature branch.
          Hide
          fabbri Aaron Fabbri added a comment -

          Congrats on the clean jenkins run Mingliang Liu.

          I'm running some tests with this configured as the MetadataStore today. First thing I'm noticing is a failure in TestS3AGetFileStatus. This led me to this part of the v10 patch:

          +public class MockS3ClientFactory extends Configured implements S3ClientFactory {
          +
          +  @Override
          +  public AmazonDynamoDBClient createDynamoDBClient(
          +      URI uri, com.amazonaws.regions.Region region) throws IOException {
          +    final DefaultS3ClientFactory factory = new DefaultS3ClientFactory();
          +    factory.setConf(getConf());
          +    return factory.createDynamoDBClient(uri, region);
          +  }
          

          I believe the goal of the mock s3 client is to be able to run non-integration (unit) tests without S3 configured. It looks like you are creating an actual S3 client from the mock client. Doesn't this break the ability of unit tests to run without S3?

          It seems like all the unit tests should only use MetadataStores which can run locally (Null or LocalMetadataStore). So, maybe we do not need this code at all. Maybe MockS3ClientFactory#createDynamoDBClient() just throws a runtime exception "Failing to create DynamoDB for unit test", and then we fall back to the NullMetadataStore automatically in S3Guard#getMetadataStore()?

          I'm also wondering if, instead of having S3ClientFactory expose a createDynamoDBClient() method, we should just add getters to S3ClientFactory (getAwsConfig() and maybe getCredentials()), and then move createDynamoDBClient() to inside the DynamoDBMetadataStore? The DynamoDBMetadataStore can then call the getters on the client to get what it needs to construct a DynamoDB client. The goal here would be to keep dynamodb specifics encapsulated in DynamoDBMetadataStore. This would allow, for example, removing the Dynamo dependency from s3a if we ever want to create a separate submodule for the DynamoDBMetadataStore.

          Show
          fabbri Aaron Fabbri added a comment - Congrats on the clean jenkins run Mingliang Liu . I'm running some tests with this configured as the MetadataStore today. First thing I'm noticing is a failure in TestS3AGetFileStatus . This led me to this part of the v10 patch: + public class MockS3ClientFactory extends Configured implements S3ClientFactory { + + @Override + public AmazonDynamoDBClient createDynamoDBClient( + URI uri, com.amazonaws.regions.Region region) throws IOException { + final DefaultS3ClientFactory factory = new DefaultS3ClientFactory(); + factory.setConf(getConf()); + return factory.createDynamoDBClient(uri, region); + } I believe the goal of the mock s3 client is to be able to run non-integration (unit) tests without S3 configured. It looks like you are creating an actual S3 client from the mock client. Doesn't this break the ability of unit tests to run without S3? It seems like all the unit tests should only use MetadataStores which can run locally (Null or LocalMetadataStore). So, maybe we do not need this code at all. Maybe MockS3ClientFactory#createDynamoDBClient() just throws a runtime exception "Failing to create DynamoDB for unit test", and then we fall back to the NullMetadataStore automatically in S3Guard#getMetadataStore()? I'm also wondering if, instead of having S3ClientFactory expose a createDynamoDBClient() method, we should just add getters to S3ClientFactory (getAwsConfig() and maybe getCredentials()), and then move createDynamoDBClient() to inside the DynamoDBMetadataStore? The DynamoDBMetadataStore can then call the getters on the client to get what it needs to construct a DynamoDB client. The goal here would be to keep dynamodb specifics encapsulated in DynamoDBMetadataStore. This would allow, for example, removing the Dynamo dependency from s3a if we ever want to create a separate submodule for the DynamoDBMetadataStore.
          Hide
          fabbri Aaron Fabbri added a comment -

          This fixes that failed unit test for me:

          --- a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/MockS3ClientFactory.java
          +++ b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/MockS3ClientFactory.java
          @@ -41,9 +41,8 @@
             @Override
             public AmazonDynamoDBClient createDynamoDBClient(
                 URI uri, com.amazonaws.regions.Region region) throws IOException {
          -    final DefaultS3ClientFactory factory = new DefaultS3ClientFactory();
          -    factory.setConf(getConf());
          -    return factory.createDynamoDBClient(uri, region);
          +    throw new IOException("Purposely failing to create DynamoDB client"
          +      + " for unit test.");
             }
          

          Also noticed a spot we need to fix the exception thrown (supposed to be an IOException):

              @Override
              public AmazonDynamoDBClient createDynamoDBClient(URI fsUri, Region region)
                  throws IOException {
          

          ...

                    String msg = "Incorrect DynamoDB endpoint: "  + endPoint;
                    LOG.error(msg, e);
                    throw new IllegalArgumentException(msg, e);
                  }
                }
          

          I have a number of integration test failures I'll be working through next. BTW, I'm happy to submit a follow-up (v11) patch with these things if that would help, just shout.

          Show
          fabbri Aaron Fabbri added a comment - This fixes that failed unit test for me: --- a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/MockS3ClientFactory.java +++ b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/MockS3ClientFactory.java @@ -41,9 +41,8 @@ @Override public AmazonDynamoDBClient createDynamoDBClient( URI uri, com.amazonaws.regions.Region region) throws IOException { - final DefaultS3ClientFactory factory = new DefaultS3ClientFactory(); - factory.setConf(getConf()); - return factory.createDynamoDBClient(uri, region); + throw new IOException( "Purposely failing to create DynamoDB client" + + " for unit test." ); } Also noticed a spot we need to fix the exception thrown (supposed to be an IOException): @Override public AmazonDynamoDBClient createDynamoDBClient(URI fsUri, Region region) throws IOException { ... String msg = "Incorrect DynamoDB endpoint: " + endPoint; LOG.error(msg, e); throw new IllegalArgumentException(msg, e); } } I have a number of integration test failures I'll be working through next. BTW, I'm happy to submit a follow-up (v11) patch with these things if that would help, just shout.
          Hide
          fabbri Aaron Fabbri added a comment -

          Finished a run of the S3A integration tests. I see that fixing the MockS3Client factory is not as simple as my last comment, as you use it for the DynamoDBMetadataStore unit test. We can revisit this here or on HADOOP-13589.

          Here are the integration test failures I see when I configure the DynamoDBMetadataStore via core-site.xml:

          Failed tests: 
            ITestS3AContractDelete>AbstractContractDeleteTest.testDeleteNonEmptyDirNonRecursive:78->Assert.fail:88 non recursive delete should have raised an exception, but completed with exit code true
            ITestS3AContractDelete>AbstractContractDeleteTest.testDeleteNonEmptyDirRecursive:94->AbstractFSContractTestBase.assertDeleted:349->Assert.fail:88 Deleted file: unexpectedly found s3a://fabbri-dev/test/testDeleteNonEmptyDirNonRecursive as  S3AFileStatus{path=s3a://fabbri-dev/test/testDeleteNonEmptyDirNonRecursive; isDirectory=true; modification_time=0; access_time=0; owner=fabbri; group=fabbri; permission=rwxrwxrwx; isSymlink=false} isEmptyDirectory=false
            ITestS3AConfiguration.testUsernameFromUGI:481 owner in S3AFileStatus{path=s3a://fabbri-dev/; isDirectory=true; modification_time=0; access_time=0; owner=fabbri; group=fabbri; permission=rwxrwxrwx; isSymlink=false} isEmptyDirectory=false expected:<[alice]> but was:<[fabbri]>
            ITestS3AFileOperationCost.testFakeDirectoryDeletion:254->Assert.assertEquals:555->Assert.assertEquals:118->Assert.failNotEquals:743->Assert.fail:88 after rename(srcFilePath, destFilePath): directories_created expected:<1> but was:<0>
            ITestS3AFileOperationCost.testCostOfGetFileStatusOnNonEmptyDir:139->Assert.fail:88 FileStatus says directory isempty: S3AFileStatus{path=s3a://fabbri-dev/test/empty; isDirectory=true; modification_time=0; access_time=0; owner=fabbri; group=fabbri; permission=rwxrwxrwx; isSymlink=false} isEmptyDirectory=true
          ls s3a://fabbri-dev/test/empty [00] S3AFileStatus{path=s3a://fabbri-dev/test/empty/simple.txt; isDirectory=false; length=0; replication=1; blocksize=33554432; modification_time=1480497225005; access_time=0; owner=fabbri; group=fabbri; permission=rw-rw-rw-; isSymlink=false} isEmptyDirectory=false
          
          Tests in error: 
            ITestS3AContractRootDir>AbstractContractRootDirectoryTest.testRmEmptyRootDirNonRecursive:116 » PathIO
            ITestS3AFileContextMainOperations>FileContextMainOperationsBaseTest.testRenameDirectoryAsNonExistentDirectory:1038->FileContextMainOperationsBaseTest.testRenameDirectoryAsNonExistentDirectory:1052->FileContextMainOperationsBaseTest.rename:1197 » IO
            ITestS3AAWSCredentialsProvider.testAnonymousProvider:133 » AWSServiceIO initia...
            ITestS3AAWSCredentialsProvider.testBadCredentials:102->createFailingFS:76 » AWSServiceIO
            ITestS3ACredentialsInURL.testInstantiateFromURL:86 » AWSClientIO initializing ...
            ITestS3AFileSystemContract>FileSystemContractBaseTest.testWriteReadAndDeleteOneBlock:266->FileSystemContractBaseTest.writeReadAndDelete:285->FileSystemContractBaseTest.writeAndRead:815 » FileAlreadyExists
            ITestS3AFileSystemContract>FileSystemContractBaseTest.testRenameToDirWithSamePrefixAllowed:656->FileSystemContractBaseTest.rename:512 » AWSServiceIO
          
          Show
          fabbri Aaron Fabbri added a comment - Finished a run of the S3A integration tests. I see that fixing the MockS3Client factory is not as simple as my last comment, as you use it for the DynamoDBMetadataStore unit test. We can revisit this here or on HADOOP-13589 . Here are the integration test failures I see when I configure the DynamoDBMetadataStore via core-site.xml: Failed tests: ITestS3AContractDelete>AbstractContractDeleteTest.testDeleteNonEmptyDirNonRecursive:78->Assert.fail:88 non recursive delete should have raised an exception, but completed with exit code true ITestS3AContractDelete>AbstractContractDeleteTest.testDeleteNonEmptyDirRecursive:94->AbstractFSContractTestBase.assertDeleted:349->Assert.fail:88 Deleted file: unexpectedly found s3a: //fabbri-dev/test/testDeleteNonEmptyDirNonRecursive as S3AFileStatus{path=s3a://fabbri-dev/test/testDeleteNonEmptyDirNonRecursive; isDirectory= true ; modification_time=0; access_time=0; owner=fabbri; group=fabbri; permission=rwxrwxrwx; isSymlink= false } isEmptyDirectory= false ITestS3AConfiguration.testUsernameFromUGI:481 owner in S3AFileStatus{path=s3a: //fabbri-dev/; isDirectory= true ; modification_time=0; access_time=0; owner=fabbri; group=fabbri; permission=rwxrwxrwx; isSymlink= false } isEmptyDirectory= false expected:<[alice]> but was:<[fabbri]> ITestS3AFileOperationCost.testFakeDirectoryDeletion:254->Assert.assertEquals:555->Assert.assertEquals:118->Assert.failNotEquals:743->Assert.fail:88 after rename(srcFilePath, destFilePath): directories_created expected:<1> but was:<0> ITestS3AFileOperationCost.testCostOfGetFileStatusOnNonEmptyDir:139->Assert.fail:88 FileStatus says directory isempty: S3AFileStatus{path=s3a: //fabbri-dev/test/empty; isDirectory= true ; modification_time=0; access_time=0; owner=fabbri; group=fabbri; permission=rwxrwxrwx; isSymlink= false } isEmptyDirectory= true ls s3a: //fabbri-dev/test/empty [00] S3AFileStatus{path=s3a://fabbri-dev/test/empty/simple.txt; isDirectory= false ; length=0; replication=1; blocksize=33554432; modification_time=1480497225005; access_time=0; owner=fabbri; group=fabbri; permission=rw-rw-rw-; isSymlink= false } isEmptyDirectory= false Tests in error: ITestS3AContractRootDir>AbstractContractRootDirectoryTest.testRmEmptyRootDirNonRecursive:116 » PathIO ITestS3AFileContextMainOperations>FileContextMainOperationsBaseTest.testRenameDirectoryAsNonExistentDirectory:1038->FileContextMainOperationsBaseTest.testRenameDirectoryAsNonExistentDirectory:1052->FileContextMainOperationsBaseTest.rename:1197 » IO ITestS3AAWSCredentialsProvider.testAnonymousProvider:133 » AWSServiceIO initia... ITestS3AAWSCredentialsProvider.testBadCredentials:102->createFailingFS:76 » AWSServiceIO ITestS3ACredentialsInURL.testInstantiateFromURL:86 » AWSClientIO initializing ... ITestS3AFileSystemContract>FileSystemContractBaseTest.testWriteReadAndDeleteOneBlock:266->FileSystemContractBaseTest.writeReadAndDelete:285->FileSystemContractBaseTest.writeAndRead:815 » FileAlreadyExists ITestS3AFileSystemContract>FileSystemContractBaseTest.testRenameToDirWithSamePrefixAllowed:656->FileSystemContractBaseTest.rename:512 » AWSServiceIO
          Hide
          fabbri Aaron Fabbri added a comment -

          FYI so we don't duplicate effort: I'm looking at ITestS3AFileSystemContract failure right now. Looks like it may be a failure to delete from DDB metadata store.

          Show
          fabbri Aaron Fabbri added a comment - FYI so we don't duplicate effort: I'm looking at ITestS3AFileSystemContract failure right now. Looks like it may be a failure to delete from DDB metadata store.
          Hide
          liuml07 Mingliang Liu added a comment -

          Sorry for late reply. Thank you Aaron Fabbri very much for running integration tests and analyze the failure. I can reproduce the unit test failure TestS3AGetFileStatus#testNotFound. I can also reproduce the integration failures on US-standard region. I'll work on them this tomorrow. Thanks for taking care of ITestS3AFileSystemContract.

          -------------------------------------------------------
           T E S T S
          -------------------------------------------------------
          
          -------------------------------------------------------
           T E S T S
          -------------------------------------------------------
          Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractGetFileStatus
          Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractMkdir
          Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractSeek
          Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractRename
          Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractDelete
          Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractOpen
          Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractCreate
          Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractDistCp
          Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 63.946 sec - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractMkdir
          Running org.apache.hadoop.fs.contract.s3n.ITestS3NContractCreate
          Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 64.332 sec - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractOpen
          Tests run: 10, Failures: 0, Errors: 0, Skipped: 10, Time elapsed: 0.372 sec - in org.apache.hadoop.fs.contract.s3n.ITestS3NContractCreate
          Running org.apache.hadoop.fs.contract.s3n.ITestS3NContractDelete
          Running org.apache.hadoop.fs.contract.s3n.ITestS3NContractMkdir
          Tests run: 8, Failures: 0, Errors: 0, Skipped: 8, Time elapsed: 0.455 sec - in org.apache.hadoop.fs.contract.s3n.ITestS3NContractDelete
          Tests run: 5, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 0.375 sec - in org.apache.hadoop.fs.contract.s3n.ITestS3NContractMkdir
          Running org.apache.hadoop.fs.contract.s3n.ITestS3NContractOpen
          Running org.apache.hadoop.fs.contract.s3n.ITestS3NContractRename
          Tests run: 6, Failures: 0, Errors: 0, Skipped: 6, Time elapsed: 0.406 sec - in org.apache.hadoop.fs.contract.s3n.ITestS3NContractRename
          Tests run: 6, Failures: 0, Errors: 0, Skipped: 6, Time elapsed: 0.478 sec - in org.apache.hadoop.fs.contract.s3n.ITestS3NContractOpen
          Running org.apache.hadoop.fs.contract.s3n.ITestS3NContractSeek
          Running org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContext
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.313 sec - in org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContext
          Tests run: 18, Failures: 0, Errors: 0, Skipped: 18, Time elapsed: 0.655 sec - in org.apache.hadoop.fs.contract.s3n.ITestS3NContractSeek
          Running org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextCreateMkdir
          Running org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextMainOperations
          Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 72.987 sec - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractRename
          Running org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextURI
          Tests run: 10, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 73.829 sec - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractCreate
          Running org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextUtil
          Tests run: 8, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 75.878 sec <<< FAILURE! - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractDelete
          testDeleteNonEmptyDirNonRecursive(org.apache.hadoop.fs.contract.s3a.ITestS3AContractDelete)  Time elapsed: 28.759 sec  <<< FAILURE!
          java.lang.AssertionError: non recursive delete should have raised an exception, but completed with exit code true
          	at org.junit.Assert.fail(Assert.java:88)
          	at org.apache.hadoop.fs.contract.AbstractContractDeleteTest.testDeleteNonEmptyDirNonRecursive(AbstractContractDeleteTest.java:78)
          	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          	at java.lang.reflect.Method.invoke(Method.java:497)
          	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
          	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
          	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
          	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
          	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
          	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
          	at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
          	at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
          
          testDeleteNonEmptyDirRecursive(org.apache.hadoop.fs.contract.s3a.ITestS3AContractDelete)  Time elapsed: 4.349 sec  <<< FAILURE!
          java.lang.AssertionError: Deleted file: unexpectedly found s3a://mliu-test-aws-s3a/fork-2/test/testDeleteNonEmptyDirNonRecursive as  S3AFileStatus{path=s3a://mliu-test-aws-s3a/fork-2/test/testDeleteNonEmptyDirNonRecursive; isDirectory=true; modification_time=0; access_time=0; owner=mliu; group=mliu; permission=rwxrwxrwx; isSymlink=false} isEmptyDirectory=false
          	at org.junit.Assert.fail(Assert.java:88)
          	at org.apache.hadoop.fs.contract.ContractTestUtils.assertPathDoesNotExist(ContractTestUtils.java:754)
          	at org.apache.hadoop.fs.contract.ContractTestUtils.assertDeleted(ContractTestUtils.java:612)
          	at org.apache.hadoop.fs.contract.ContractTestUtils.assertDeleted(ContractTestUtils.java:590)
          	at org.apache.hadoop.fs.contract.AbstractFSContractTestBase.assertDeleted(AbstractFSContractTestBase.java:349)
          	at org.apache.hadoop.fs.contract.AbstractContractDeleteTest.testDeleteNonEmptyDirRecursive(AbstractContractDeleteTest.java:94)
          	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          	at java.lang.reflect.Method.invoke(Method.java:497)
          	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
          	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
          	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
          	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
          	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
          	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
          	at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
          	at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
          
          Running org.apache.hadoop.fs.s3a.ITestBlockingThreadPoolExecutorService
          Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.869 sec - in org.apache.hadoop.fs.s3a.ITestBlockingThreadPoolExecutorService
          Running org.apache.hadoop.fs.s3a.ITestS3AAWSCredentialsProvider
          Tests run: 4, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 11.613 sec <<< FAILURE! - in org.apache.hadoop.fs.s3a.ITestS3AAWSCredentialsProvider
          testAnonymousProvider(org.apache.hadoop.fs.s3a.ITestS3AAWSCredentialsProvider)  Time elapsed: 0.91 sec  <<< ERROR!
          org.apache.hadoop.fs.s3a.AWSServiceIOException: initializing  on s3a://landsat-pds/scene_list.gz: com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException: Request is missing Authentication Token (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: MissingAuthenticationTokenException; Request ID: NS80UK0G6OKHI6IR7KCIV1VRONVV4KQNSO5AEMVJF66Q9ASUAAJG): Request is missing Authentication Token (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: MissingAuthenticationTokenException; Request ID: NS80UK0G6OKHI6IR7KCIV1VRONVV4KQNSO5AEMVJF66Q9ASUAAJG)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1529)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1167)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:948)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:661)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:635)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:618)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$300(AmazonHttpClient.java:586)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:573)
          	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:445)
          	at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.doInvoke(AmazonDynamoDBClient.java:1722)
          	at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:1698)
          	at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.createTable(AmazonDynamoDBClient.java:743)
          	at com.amazonaws.services.dynamodbv2.document.DynamoDB.createTable(DynamoDB.java:96)
          	at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.createTable(DynamoDBMetadataStore.java:413)
          	at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.initialize(DynamoDBMetadataStore.java:187)
          	at org.apache.hadoop.fs.s3a.s3guard.S3Guard.getMetadataStore(S3Guard.java:85)
          	at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:252)
          	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3246)
          	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:123)
          	at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3295)
          	at org.apache.hadoop.fs.FileSystem$Cache.getUnique(FileSystem.java:3269)
          	at org.apache.hadoop.fs.FileSystem.newInstance(FileSystem.java:529)
          	at org.apache.hadoop.fs.s3a.ITestS3AAWSCredentialsProvider.testAnonymousProvider(ITestS3AAWSCredentialsProvider.java:133)
          
          testBadCredentials(org.apache.hadoop.fs.s3a.ITestS3AAWSCredentialsProvider)  Time elapsed: 0.82 sec  <<< ERROR!
          org.apache.hadoop.fs.s3a.AWSServiceIOException: initializing  on s3a://mliu-test-aws-s3a/: com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException: The security token included in the request is invalid. (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: UnrecognizedClientException; Request ID: UUBHUTU01895I8AH4CGS72R24FVV4KQNSO5AEMVJF66Q9ASUAAJG): The security token included in the request is invalid. (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: UnrecognizedClientException; Request ID: UUBHUTU01895I8AH4CGS72R24FVV4KQNSO5AEMVJF66Q9ASUAAJG)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1529)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1167)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:948)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:661)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:635)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:618)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$300(AmazonHttpClient.java:586)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:573)
          	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:445)
          	at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.doInvoke(AmazonDynamoDBClient.java:1722)
          	at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:1698)
          	at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.createTable(AmazonDynamoDBClient.java:743)
          	at com.amazonaws.services.dynamodbv2.document.DynamoDB.createTable(DynamoDB.java:96)
          	at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.createTable(DynamoDBMetadataStore.java:413)
          	at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.initialize(DynamoDBMetadataStore.java:187)
          	at org.apache.hadoop.fs.s3a.s3guard.S3Guard.getMetadataStore(S3Guard.java:85)
          	at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:252)
          	at org.apache.hadoop.fs.s3a.S3ATestUtils.createTestFileSystem(S3ATestUtils.java:103)
          	at org.apache.hadoop.fs.s3a.S3ATestUtils.createTestFileSystem(S3ATestUtils.java:63)
          	at org.apache.hadoop.fs.s3a.ITestS3AAWSCredentialsProvider.createFailingFS(ITestS3AAWSCredentialsProvider.java:76)
          	at org.apache.hadoop.fs.s3a.ITestS3AAWSCredentialsProvider.testBadCredentials(ITestS3AAWSCredentialsProvider.java:102)
          
          Running org.apache.hadoop.fs.s3a.ITestS3ABlockOutputArray
          Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.847 sec - in org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextUtil
          Running org.apache.hadoop.fs.s3a.ITestS3ABlockOutputByteBuffer
          Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 21.233 sec - in org.apache.hadoop.fs.s3a.ITestS3ABlockOutputArray
          Running org.apache.hadoop.fs.s3a.ITestS3ABlockOutputDisk
          Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 18.82 sec - in org.apache.hadoop.fs.s3a.ITestS3ABlockOutputByteBuffer
          Running org.apache.hadoop.fs.s3a.ITestS3ABlocksize
          Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.371 sec - in org.apache.hadoop.fs.s3a.ITestS3ABlocksize
          Running org.apache.hadoop.fs.s3a.ITestS3AConfiguration
          Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 60.955 sec - in org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextCreateMkdir
          Running org.apache.hadoop.fs.s3a.ITestS3ACredentialsInURL
          Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 18.991 sec - in org.apache.hadoop.fs.s3a.ITestS3ABlockOutputDisk
          Running org.apache.hadoop.fs.s3a.ITestS3AEncryption
          Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 3.518 sec <<< FAILURE! - in org.apache.hadoop.fs.s3a.ITestS3ACredentialsInURL
          testInvalidCredentialsFail(org.apache.hadoop.fs.s3a.ITestS3ACredentialsInURL)  Time elapsed: 0.95 sec  <<< FAILURE!
          java.lang.AssertionError: Expected an AccessDeniedException, got S3AFileStatus{path=s3a://mliu-test-aws-s3a/; isDirectory=true; modification_time=0; access_time=0; owner=mliu; group=mliu; permission=rwxrwxrwx; isSymlink=false} isEmptyDirectory=false
          	at org.junit.Assert.fail(Assert.java:88)
          	at org.apache.hadoop.fs.s3a.ITestS3ACredentialsInURL.testInvalidCredentialsFail(ITestS3ACredentialsInURL.java:130)
          
          Running org.apache.hadoop.fs.s3a.ITestS3AEncryptionAlgorithmPropagation
          Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.72 sec - in org.apache.hadoop.fs.s3a.ITestS3AEncryptionAlgorithmPropagation
          Running org.apache.hadoop.fs.s3a.ITestS3AEncryptionBlockOutputStream
          Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 142.197 sec - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractSeek
          Running org.apache.hadoop.fs.s3a.ITestS3AFailureHandling
          Tests run: 19, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 16.14 sec <<< FAILURE! - in org.apache.hadoop.fs.s3a.ITestS3AConfiguration
          testUsernameFromUGI(org.apache.hadoop.fs.s3a.ITestS3AConfiguration)  Time elapsed: 0.923 sec  <<< FAILURE!
          org.junit.ComparisonFailure: owner in S3AFileStatus{path=s3a://mliu-test-aws-s3a/; isDirectory=true; modification_time=0; access_time=0; owner=mliu; group=mliu; permission=rwxrwxrwx; isSymlink=false} isEmptyDirectory=false expected:<[alice]> but was:<[mliu]>
          	at org.junit.Assert.assertEquals(Assert.java:115)
          	at org.apache.hadoop.fs.s3a.ITestS3AConfiguration.testUsernameFromUGI(ITestS3AConfiguration.java:481)
          
          Running org.apache.hadoop.fs.s3a.ITestS3AFileOperationCost
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.446 sec - in org.apache.hadoop.fs.s3a.ITestS3AFailureHandling
          Running org.apache.hadoop.fs.s3a.ITestS3AFileSystemContract
          Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 50.437 sec - in org.apache.hadoop.fs.s3a.ITestS3AEncryption
          Running org.apache.hadoop.fs.s3a.ITestS3AMiscOperations
          Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 51.966 sec - in org.apache.hadoop.fs.s3a.ITestS3AEncryptionBlockOutputStream
          Running org.apache.hadoop.fs.s3a.ITestS3ATemporaryCredentials
          Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.832 sec - in org.apache.hadoop.fs.s3a.ITestS3AMiscOperations
          Tests run: 7, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 52.913 sec <<< FAILURE! - in org.apache.hadoop.fs.s3a.ITestS3AFileOperationCost
          testFakeDirectoryDeletion(org.apache.hadoop.fs.s3a.ITestS3AFileOperationCost)  Time elapsed: 19.243 sec  <<< FAILURE!
          java.lang.AssertionError: after rename(srcFilePath, destFilePath): directories_created expected:<1> but was:<0>
          	at org.junit.Assert.fail(Assert.java:88)
          	at org.junit.Assert.failNotEquals(Assert.java:743)
          	at org.junit.Assert.assertEquals(Assert.java:118)
          	at org.junit.Assert.assertEquals(Assert.java:555)
          	at org.apache.hadoop.fs.s3a.S3ATestUtils$MetricDiff.assertDiffEquals(S3ATestUtils.java:431)
          	at org.apache.hadoop.fs.s3a.ITestS3AFileOperationCost.testFakeDirectoryDeletion(ITestS3AFileOperationCost.java:254)
          
          testCostOfGetFileStatusOnNonEmptyDir(org.apache.hadoop.fs.s3a.ITestS3AFileOperationCost)  Time elapsed: 5.892 sec  <<< FAILURE!
          java.lang.AssertionError: FileStatus says directory isempty: S3AFileStatus{path=s3a://mliu-test-aws-s3a/fork-6/test/empty; isDirectory=true; modification_time=0; access_time=0; owner=mliu; group=mliu; permission=rwxrwxrwx; isSymlink=false} isEmptyDirectory=true
          ls s3a://mliu-test-aws-s3a/fork-6/test/empty [00] S3AFileStatus{path=s3a://mliu-test-aws-s3a/fork-6/test/empty/simple.txt; isDirectory=false; length=0; replication=1; blocksize=33554432; modification_time=1480569669039; access_time=0; owner=mliu; group=mliu; permission=rw-rw-rw-; isSymlink=false} isEmptyDirectory=false
          
          S3AFileSystem{uri=s3a://mliu-test-aws-s3a, workingDir=s3a://mliu-test-aws-s3a/user/mliu, inputPolicy=normal, partSize=104857600, enableMultiObjectsDelete=true, maxKeys=5000, readAhead=65536, blockSize=33554432, multiPartThreshold=2147483647, executor=BlockingThreadPoolExecutorService{SemaphoredDelegatingExecutor{permitCount=25, available=25, waiting=0}, activeCount=0}, statistics {10240 bytes read, 10240 bytes written, 26 read ops, 0 large read ops, 66 write ops}, metrics {{Context=S3AFileSystem} {FileSystemId=66ae0ffd-8746-4911-88df-d73e3b217dab-mliu-test-aws-s3a} {fsURI=s3a://mliu-test-aws-s3a/} {files_created=1} {files_copied=0} {files_copied_bytes=0} {files_deleted=0} {fake_directories_deleted=3} {directories_created=2} {directories_deleted=0} {ignored_errors=0} {op_copy_from_local_file=0} {op_exists=0} {op_get_file_status=6} {op_glob_status=0} {op_is_directory=0} {op_is_file=0} {op_list_files=0} {op_list_located_status=0} {op_list_status=0} {op_mkdirs=2} {op_rename=0} {object_copy_requests=0} {object_delete_requests=1} {object_list_requests=3} {object_continue_list_requests=0} {object_metadata_requests=6} {object_multipart_aborted=0} {object_put_bytes=0} {object_put_requests=3} {object_put_requests_completed=3} {stream_write_failures=0} {stream_write_block_uploads=0} {stream_write_block_uploads_committed=0} {stream_write_block_uploads_aborted=0} {stream_write_total_time=0} {stream_write_total_data=0} {object_put_requests_active=0} {object_put_bytes_pending=0} {stream_write_block_uploads_active=0} {stream_write_block_uploads_pending=0} {stream_write_block_uploads_data_pending=0} {stream_read_fully_operations=0} {stream_bytes_skipped_on_seek=0} {stream_bytes_backwards_on_seek=0} {stream_bytes_read=0} {streamOpened=0} {stream_read_operations_incomplete=0} {stream_bytes_discarded_in_abort=0} {stream_close_operations=0} {stream_read_operations=0} {stream_aborted=0} {stream_forward_seek_operations=0} {stream_backward_seek_operations=0} {streamClosed=0} {stream_seek_operations=0} {stream_bytes_read_in_close=0} {stream_read_exceptions=0} }}
          	at org.junit.Assert.fail(Assert.java:88)
          	at org.apache.hadoop.fs.s3a.ITestS3AFileOperationCost.testCostOfGetFileStatusOnNonEmptyDir(ITestS3AFileOperationCost.java:139)
          
          Running org.apache.hadoop.fs.s3a.scale.ITestS3ADeleteFilesOneByOne
          Running org.apache.hadoop.fs.s3a.ITestS3ATestUtils
          Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.19 sec - in org.apache.hadoop.fs.s3a.ITestS3ATestUtils
          Running org.apache.hadoop.fs.s3a.scale.ITestS3ADeleteManyFiles
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 6.948 sec - in org.apache.hadoop.fs.s3a.scale.ITestS3ADeleteFilesOneByOne
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 6.651 sec - in org.apache.hadoop.fs.s3a.scale.ITestS3ADeleteManyFiles
          Running org.apache.hadoop.fs.s3a.scale.ITestS3ADirectoryPerformance
          Running org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance
          Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 17.942 sec - in org.apache.hadoop.fs.s3a.ITestS3ATemporaryCredentials
          Running org.apache.hadoop.fs.s3a.yarn.ITestS3A
          Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.445 sec - in org.apache.hadoop.fs.s3a.yarn.ITestS3A
          Running org.apache.hadoop.fs.s3a.yarn.ITestS3AMiniYarnCluster
          Tests run: 5, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 19.071 sec - in org.apache.hadoop.fs.s3a.scale.ITestS3ADirectoryPerformance
          

          For MockS3ClientFactory, my idea was that having createDynamoDBClient over DynamoDBLocal for unit test will help us find bugs easier and earlier than mocked objects. For integration tests, it will go to AWS DynamoDB service as expected. If I can not find an easy approach now, we can address this along with HADOOP-13589.

          By the way, when I run the integration tests myself, the s3n tests were included by default. Is there a way to exclude it?

          Show
          liuml07 Mingliang Liu added a comment - Sorry for late reply. Thank you Aaron Fabbri very much for running integration tests and analyze the failure. I can reproduce the unit test failure TestS3AGetFileStatus#testNotFound . I can also reproduce the integration failures on US-standard region. I'll work on them this tomorrow. Thanks for taking care of ITestS3AFileSystemContract . ------------------------------------------------------- T E S T S ------------------------------------------------------- ------------------------------------------------------- T E S T S ------------------------------------------------------- Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractGetFileStatus Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractMkdir Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractSeek Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractRename Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractDelete Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractOpen Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractCreate Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractDistCp Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 63.946 sec - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractMkdir Running org.apache.hadoop.fs.contract.s3n.ITestS3NContractCreate Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 64.332 sec - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractOpen Tests run: 10, Failures: 0, Errors: 0, Skipped: 10, Time elapsed: 0.372 sec - in org.apache.hadoop.fs.contract.s3n.ITestS3NContractCreate Running org.apache.hadoop.fs.contract.s3n.ITestS3NContractDelete Running org.apache.hadoop.fs.contract.s3n.ITestS3NContractMkdir Tests run: 8, Failures: 0, Errors: 0, Skipped: 8, Time elapsed: 0.455 sec - in org.apache.hadoop.fs.contract.s3n.ITestS3NContractDelete Tests run: 5, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 0.375 sec - in org.apache.hadoop.fs.contract.s3n.ITestS3NContractMkdir Running org.apache.hadoop.fs.contract.s3n.ITestS3NContractOpen Running org.apache.hadoop.fs.contract.s3n.ITestS3NContractRename Tests run: 6, Failures: 0, Errors: 0, Skipped: 6, Time elapsed: 0.406 sec - in org.apache.hadoop.fs.contract.s3n.ITestS3NContractRename Tests run: 6, Failures: 0, Errors: 0, Skipped: 6, Time elapsed: 0.478 sec - in org.apache.hadoop.fs.contract.s3n.ITestS3NContractOpen Running org.apache.hadoop.fs.contract.s3n.ITestS3NContractSeek Running org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContext Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.313 sec - in org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContext Tests run: 18, Failures: 0, Errors: 0, Skipped: 18, Time elapsed: 0.655 sec - in org.apache.hadoop.fs.contract.s3n.ITestS3NContractSeek Running org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextCreateMkdir Running org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextMainOperations Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 72.987 sec - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractRename Running org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextURI Tests run: 10, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 73.829 sec - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractCreate Running org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextUtil Tests run: 8, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 75.878 sec <<< FAILURE! - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractDelete testDeleteNonEmptyDirNonRecursive(org.apache.hadoop.fs.contract.s3a.ITestS3AContractDelete) Time elapsed: 28.759 sec <<< FAILURE! java.lang.AssertionError: non recursive delete should have raised an exception, but completed with exit code true at org.junit.Assert.fail(Assert.java:88) at org.apache.hadoop.fs.contract.AbstractContractDeleteTest.testDeleteNonEmptyDirNonRecursive(AbstractContractDeleteTest.java:78) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) testDeleteNonEmptyDirRecursive(org.apache.hadoop.fs.contract.s3a.ITestS3AContractDelete) Time elapsed: 4.349 sec <<< FAILURE! java.lang.AssertionError: Deleted file: unexpectedly found s3a: //mliu-test-aws-s3a/fork-2/test/testDeleteNonEmptyDirNonRecursive as S3AFileStatus{path=s3a://mliu-test-aws-s3a/fork-2/test/testDeleteNonEmptyDirNonRecursive; isDirectory= true ; modification_time=0; access_time=0; owner=mliu; group=mliu; permission=rwxrwxrwx; isSymlink= false } isEmptyDirectory= false at org.junit.Assert.fail(Assert.java:88) at org.apache.hadoop.fs.contract.ContractTestUtils.assertPathDoesNotExist(ContractTestUtils.java:754) at org.apache.hadoop.fs.contract.ContractTestUtils.assertDeleted(ContractTestUtils.java:612) at org.apache.hadoop.fs.contract.ContractTestUtils.assertDeleted(ContractTestUtils.java:590) at org.apache.hadoop.fs.contract.AbstractFSContractTestBase.assertDeleted(AbstractFSContractTestBase.java:349) at org.apache.hadoop.fs.contract.AbstractContractDeleteTest.testDeleteNonEmptyDirRecursive(AbstractContractDeleteTest.java:94) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Running org.apache.hadoop.fs.s3a.ITestBlockingThreadPoolExecutorService Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.869 sec - in org.apache.hadoop.fs.s3a.ITestBlockingThreadPoolExecutorService Running org.apache.hadoop.fs.s3a.ITestS3AAWSCredentialsProvider Tests run: 4, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 11.613 sec <<< FAILURE! - in org.apache.hadoop.fs.s3a.ITestS3AAWSCredentialsProvider testAnonymousProvider(org.apache.hadoop.fs.s3a.ITestS3AAWSCredentialsProvider) Time elapsed: 0.91 sec <<< ERROR! org.apache.hadoop.fs.s3a.AWSServiceIOException: initializing on s3a: //landsat-pds/scene_list.gz: com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException: Request is missing Authentication Token (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: MissingAuthenticationTokenException; Request ID: NS80UK0G6OKHI6IR7KCIV1VRONVV4KQNSO5AEMVJF66Q9ASUAAJG): Request is missing Authentication Token (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: MissingAuthenticationTokenException; Request ID: NS80UK0G6OKHI6IR7KCIV1VRONVV4KQNSO5AEMVJF66Q9ASUAAJG) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1529) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1167) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:948) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:661) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:635) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:618) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$300(AmazonHttpClient.java:586) at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:573) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:445) at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.doInvoke(AmazonDynamoDBClient.java:1722) at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:1698) at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.createTable(AmazonDynamoDBClient.java:743) at com.amazonaws.services.dynamodbv2.document.DynamoDB.createTable(DynamoDB.java:96) at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.createTable(DynamoDBMetadataStore.java:413) at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.initialize(DynamoDBMetadataStore.java:187) at org.apache.hadoop.fs.s3a.s3guard.S3Guard.getMetadataStore(S3Guard.java:85) at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:252) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3246) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:123) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3295) at org.apache.hadoop.fs.FileSystem$Cache.getUnique(FileSystem.java:3269) at org.apache.hadoop.fs.FileSystem.newInstance(FileSystem.java:529) at org.apache.hadoop.fs.s3a.ITestS3AAWSCredentialsProvider.testAnonymousProvider(ITestS3AAWSCredentialsProvider.java:133) testBadCredentials(org.apache.hadoop.fs.s3a.ITestS3AAWSCredentialsProvider) Time elapsed: 0.82 sec <<< ERROR! org.apache.hadoop.fs.s3a.AWSServiceIOException: initializing on s3a: //mliu-test-aws-s3a/: com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException: The security token included in the request is invalid. (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: UnrecognizedClientException; Request ID: UUBHUTU01895I8AH4CGS72R24FVV4KQNSO5AEMVJF66Q9ASUAAJG): The security token included in the request is invalid. (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: UnrecognizedClientException; Request ID: UUBHUTU01895I8AH4CGS72R24FVV4KQNSO5AEMVJF66Q9ASUAAJG) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1529) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1167) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:948) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:661) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:635) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:618) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$300(AmazonHttpClient.java:586) at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:573) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:445) at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.doInvoke(AmazonDynamoDBClient.java:1722) at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:1698) at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.createTable(AmazonDynamoDBClient.java:743) at com.amazonaws.services.dynamodbv2.document.DynamoDB.createTable(DynamoDB.java:96) at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.createTable(DynamoDBMetadataStore.java:413) at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.initialize(DynamoDBMetadataStore.java:187) at org.apache.hadoop.fs.s3a.s3guard.S3Guard.getMetadataStore(S3Guard.java:85) at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:252) at org.apache.hadoop.fs.s3a.S3ATestUtils.createTestFileSystem(S3ATestUtils.java:103) at org.apache.hadoop.fs.s3a.S3ATestUtils.createTestFileSystem(S3ATestUtils.java:63) at org.apache.hadoop.fs.s3a.ITestS3AAWSCredentialsProvider.createFailingFS(ITestS3AAWSCredentialsProvider.java:76) at org.apache.hadoop.fs.s3a.ITestS3AAWSCredentialsProvider.testBadCredentials(ITestS3AAWSCredentialsProvider.java:102) Running org.apache.hadoop.fs.s3a.ITestS3ABlockOutputArray Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.847 sec - in org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextUtil Running org.apache.hadoop.fs.s3a.ITestS3ABlockOutputByteBuffer Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 21.233 sec - in org.apache.hadoop.fs.s3a.ITestS3ABlockOutputArray Running org.apache.hadoop.fs.s3a.ITestS3ABlockOutputDisk Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 18.82 sec - in org.apache.hadoop.fs.s3a.ITestS3ABlockOutputByteBuffer Running org.apache.hadoop.fs.s3a.ITestS3ABlocksize Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.371 sec - in org.apache.hadoop.fs.s3a.ITestS3ABlocksize Running org.apache.hadoop.fs.s3a.ITestS3AConfiguration Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 60.955 sec - in org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextCreateMkdir Running org.apache.hadoop.fs.s3a.ITestS3ACredentialsInURL Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 18.991 sec - in org.apache.hadoop.fs.s3a.ITestS3ABlockOutputDisk Running org.apache.hadoop.fs.s3a.ITestS3AEncryption Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 3.518 sec <<< FAILURE! - in org.apache.hadoop.fs.s3a.ITestS3ACredentialsInURL testInvalidCredentialsFail(org.apache.hadoop.fs.s3a.ITestS3ACredentialsInURL) Time elapsed: 0.95 sec <<< FAILURE! java.lang.AssertionError: Expected an AccessDeniedException, got S3AFileStatus{path=s3a: //mliu-test-aws-s3a/; isDirectory= true ; modification_time=0; access_time=0; owner=mliu; group=mliu; permission=rwxrwxrwx; isSymlink= false } isEmptyDirectory= false at org.junit.Assert.fail(Assert.java:88) at org.apache.hadoop.fs.s3a.ITestS3ACredentialsInURL.testInvalidCredentialsFail(ITestS3ACredentialsInURL.java:130) Running org.apache.hadoop.fs.s3a.ITestS3AEncryptionAlgorithmPropagation Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.72 sec - in org.apache.hadoop.fs.s3a.ITestS3AEncryptionAlgorithmPropagation Running org.apache.hadoop.fs.s3a.ITestS3AEncryptionBlockOutputStream Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 142.197 sec - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractSeek Running org.apache.hadoop.fs.s3a.ITestS3AFailureHandling Tests run: 19, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 16.14 sec <<< FAILURE! - in org.apache.hadoop.fs.s3a.ITestS3AConfiguration testUsernameFromUGI(org.apache.hadoop.fs.s3a.ITestS3AConfiguration) Time elapsed: 0.923 sec <<< FAILURE! org.junit.ComparisonFailure: owner in S3AFileStatus{path=s3a: //mliu-test-aws-s3a/; isDirectory= true ; modification_time=0; access_time=0; owner=mliu; group=mliu; permission=rwxrwxrwx; isSymlink= false } isEmptyDirectory= false expected:<[alice]> but was:<[mliu]> at org.junit.Assert.assertEquals(Assert.java:115) at org.apache.hadoop.fs.s3a.ITestS3AConfiguration.testUsernameFromUGI(ITestS3AConfiguration.java:481) Running org.apache.hadoop.fs.s3a.ITestS3AFileOperationCost Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.446 sec - in org.apache.hadoop.fs.s3a.ITestS3AFailureHandling Running org.apache.hadoop.fs.s3a.ITestS3AFileSystemContract Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 50.437 sec - in org.apache.hadoop.fs.s3a.ITestS3AEncryption Running org.apache.hadoop.fs.s3a.ITestS3AMiscOperations Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 51.966 sec - in org.apache.hadoop.fs.s3a.ITestS3AEncryptionBlockOutputStream Running org.apache.hadoop.fs.s3a.ITestS3ATemporaryCredentials Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.832 sec - in org.apache.hadoop.fs.s3a.ITestS3AMiscOperations Tests run: 7, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 52.913 sec <<< FAILURE! - in org.apache.hadoop.fs.s3a.ITestS3AFileOperationCost testFakeDirectoryDeletion(org.apache.hadoop.fs.s3a.ITestS3AFileOperationCost) Time elapsed: 19.243 sec <<< FAILURE! java.lang.AssertionError: after rename(srcFilePath, destFilePath): directories_created expected:<1> but was:<0> at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:743) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:555) at org.apache.hadoop.fs.s3a.S3ATestUtils$MetricDiff.assertDiffEquals(S3ATestUtils.java:431) at org.apache.hadoop.fs.s3a.ITestS3AFileOperationCost.testFakeDirectoryDeletion(ITestS3AFileOperationCost.java:254) testCostOfGetFileStatusOnNonEmptyDir(org.apache.hadoop.fs.s3a.ITestS3AFileOperationCost) Time elapsed: 5.892 sec <<< FAILURE! java.lang.AssertionError: FileStatus says directory isempty: S3AFileStatus{path=s3a: //mliu-test-aws-s3a/fork-6/test/empty; isDirectory= true ; modification_time=0; access_time=0; owner=mliu; group=mliu; permission=rwxrwxrwx; isSymlink= false } isEmptyDirectory= true ls s3a: //mliu-test-aws-s3a/fork-6/test/empty [00] S3AFileStatus{path=s3a://mliu-test-aws-s3a/fork-6/test/empty/simple.txt; isDirectory= false ; length=0; replication=1; blocksize=33554432; modification_time=1480569669039; access_time=0; owner=mliu; group=mliu; permission=rw-rw-rw-; isSymlink= false } isEmptyDirectory= false S3AFileSystem{uri=s3a: //mliu-test-aws-s3a, workingDir=s3a://mliu-test-aws-s3a/user/mliu, inputPolicy=normal, partSize=104857600, enableMultiObjectsDelete= true , maxKeys=5000, readAhead=65536, blockSize=33554432, multiPartThreshold=2147483647, executor=BlockingThreadPoolExecutorService{SemaphoredDelegatingExecutor{permitCount=25, available=25, waiting=0}, activeCount=0}, statistics {10240 bytes read, 10240 bytes written, 26 read ops, 0 large read ops, 66 write ops}, metrics {{Context=S3AFileSystem} {FileSystemId=66ae0ffd-8746-4911-88df-d73e3b217dab-mliu-test-aws-s3a} {fsURI=s3a://mliu-test-aws-s3a/} {files_created=1} {files_copied=0} {files_copied_bytes=0} {files_deleted=0} {fake_directories_deleted=3} {directories_created=2} {directories_deleted=0} {ignored_errors=0} {op_copy_from_local_file=0} {op_exists=0} {op_get_file_status=6} {op_glob_status=0} {op_is_directory=0} {op_is_file=0} {op_list_files=0} {op_list_located_status=0} {op_list_status=0} {op_mkdirs=2} {op_rename=0} {object_copy_requests=0} {object_delete_requests=1} {object_list_requests=3} {object_continue_list_requests=0} {object_metadata_requests=6} {object_multipart_aborted=0} {object_put_bytes=0} {object_put_requests=3} {object_put_requests_completed=3} {stream_write_failures=0} {stream_write_block_uploads=0} {stream_write_block_uploads_committed=0} {stream_write_block_uploads_aborted=0} {stream_write_total_time=0} {stream_write_total_data=0} {object_put_requests_active=0} {object_put_bytes_pending=0} {stream_write_block_uploads_active=0} {stream_write_block_uploads_pending=0} {stream_write_block_uploads_data_pending=0} {stream_read_fully_operations=0} {stream_bytes_skipped_on_seek=0} {stream_bytes_backwards_on_seek=0} {stream_bytes_read=0} {streamOpened=0} {stream_read_operations_incomplete=0} {stream_bytes_discarded_in_abort=0} {stream_close_operations=0} {stream_read_operations=0} {stream_aborted=0} {stream_forward_seek_operations=0} {stream_backward_seek_operations=0} {streamClosed=0} {stream_seek_operations=0} {stream_bytes_read_in_close=0} {stream_read_exceptions=0} }} at org.junit.Assert.fail(Assert.java:88) at org.apache.hadoop.fs.s3a.ITestS3AFileOperationCost.testCostOfGetFileStatusOnNonEmptyDir(ITestS3AFileOperationCost.java:139) Running org.apache.hadoop.fs.s3a.scale.ITestS3ADeleteFilesOneByOne Running org.apache.hadoop.fs.s3a.ITestS3ATestUtils Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.19 sec - in org.apache.hadoop.fs.s3a.ITestS3ATestUtils Running org.apache.hadoop.fs.s3a.scale.ITestS3ADeleteManyFiles Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 6.948 sec - in org.apache.hadoop.fs.s3a.scale.ITestS3ADeleteFilesOneByOne Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 6.651 sec - in org.apache.hadoop.fs.s3a.scale.ITestS3ADeleteManyFiles Running org.apache.hadoop.fs.s3a.scale.ITestS3ADirectoryPerformance Running org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 17.942 sec - in org.apache.hadoop.fs.s3a.ITestS3ATemporaryCredentials Running org.apache.hadoop.fs.s3a.yarn.ITestS3A Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.445 sec - in org.apache.hadoop.fs.s3a.yarn.ITestS3A Running org.apache.hadoop.fs.s3a.yarn.ITestS3AMiniYarnCluster Tests run: 5, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 19.071 sec - in org.apache.hadoop.fs.s3a.scale.ITestS3ADirectoryPerformance For MockS3ClientFactory , my idea was that having createDynamoDBClient over DynamoDBLocal for unit test will help us find bugs easier and earlier than mocked objects. For integration tests, it will go to AWS DynamoDB service as expected. If I can not find an easy approach now, we can address this along with HADOOP-13589 . By the way, when I run the integration tests myself, the s3n tests were included by default. Is there a way to exclude it?
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Some of these tests are failing because dynamo db is trying to init, even when anonymously authenticating with an external read only store.

          we need to consider how to support deployment where some object stores are read only + no dynamo db, and perhaps fallback to no db if auth fails. And also the situation where one store has an authoritative DB, another none...it'll have to be on a per-object store basis

          
          testAnonymousProvider(org.apache.hadoop.fs.s3a.ITestS3AAWSCredentialsProvider)  Time elapsed: 0.91 sec  <<< ERROR!
          org.apache.hadoop.fs.s3a.AWSServiceIOException: initializing  on s3a://landsat-pds/scene_list.gz: com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException: Request is missing Authentication Token (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: MissingAuthenticationTokenException; Request ID: NS80UK0G6OKHI6IR7KCIV1VRONVV4KQNSO5AEMVJF66Q9ASUAAJG): Request is missing Authentication Token (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: MissingAuthenticationTokenException; Request ID: NS80UK0G6OKHI6IR7KCIV1VRONVV4KQNSO5AEMVJF66Q9ASUAAJG)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1529)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1167)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:948)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:661)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:635)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:618)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$300(AmazonHttpClient.java:586)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:573)
          	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:445)
          	at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.doInvoke(AmazonDynamoDBClient.java:1722)
          	at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:1698)
          	at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.createTable(AmazonDynamoDBClient.java:743)
          	at com.amazonaws.services.dynamodbv2.document.DynamoDB.createTable(DynamoDB.java:96)
          	at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.createTable(DynamoDBMetadataStore.java:413)
          	at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.initialize(DynamoDBMetadataStore.java:187)
          	at org.apache.hadoop.fs.s3a.s3guard.S3Guard.getMetadataStore(S3Guard.java:85)
          	at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:252)
          	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3246)
          	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:123)
          	at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3295)
          	at org.apache.hadoop.fs.FileSystem$Cache.getUnique(FileSystem.java:3269)
          	at org.apache.hadoop.fs.FileSystem.newInstance(FileSystem.java:529)
          	at org.apache.hadoop.fs.s3a.ITestS3AAWSCredentialsProvider.testAnonymousProvider(ITestS3AAWSCredenti
          
          Show
          stevel@apache.org Steve Loughran added a comment - Some of these tests are failing because dynamo db is trying to init, even when anonymously authenticating with an external read only store. we need to consider how to support deployment where some object stores are read only + no dynamo db, and perhaps fallback to no db if auth fails. And also the situation where one store has an authoritative DB, another none...it'll have to be on a per-object store basis testAnonymousProvider(org.apache.hadoop.fs.s3a.ITestS3AAWSCredentialsProvider) Time elapsed: 0.91 sec <<< ERROR! org.apache.hadoop.fs.s3a.AWSServiceIOException: initializing on s3a: //landsat-pds/scene_list.gz: com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException: Request is missing Authentication Token (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: MissingAuthenticationTokenException; Request ID: NS80UK0G6OKHI6IR7KCIV1VRONVV4KQNSO5AEMVJF66Q9ASUAAJG): Request is missing Authentication Token (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: MissingAuthenticationTokenException; Request ID: NS80UK0G6OKHI6IR7KCIV1VRONVV4KQNSO5AEMVJF66Q9ASUAAJG) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1529) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1167) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:948) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:661) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:635) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:618) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$300(AmazonHttpClient.java:586) at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:573) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:445) at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.doInvoke(AmazonDynamoDBClient.java:1722) at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:1698) at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.createTable(AmazonDynamoDBClient.java:743) at com.amazonaws.services.dynamodbv2.document.DynamoDB.createTable(DynamoDB.java:96) at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.createTable(DynamoDBMetadataStore.java:413) at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.initialize(DynamoDBMetadataStore.java:187) at org.apache.hadoop.fs.s3a.s3guard.S3Guard.getMetadataStore(S3Guard.java:85) at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:252) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3246) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:123) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3295) at org.apache.hadoop.fs.FileSystem$Cache.getUnique(FileSystem.java:3269) at org.apache.hadoop.fs.FileSystem.newInstance(FileSystem.java:529) at org.apache.hadoop.fs.s3a.ITestS3AAWSCredentialsProvider.testAnonymousProvider(ITestS3AAWSCredenti
          Hide
          fabbri Aaron Fabbri added a comment -

          That use case for Mock S3 Client + Real DDB client (for DynamoDBLocal) makes sense. We also need to be able to ensure that DDB metadatastore is disabled for unit tests, even if it is configured in the Hadoop configuration. That could be solved as part of HADOOP-13589.

          In my working tree I have a patch on top of your v10 here that separates out the DynamoDB Client Factory into a separate class DynamoDBClientFactory. That would allow us to use a Mock S3 client without a real DDB client. It is an easy change but depends on (or conflicts with) my outstanding patch for HADOOP-13793 (which we should get in soon).

          As for disabling s3n integration tests, you should be able to add a couple of lines to your pom to exclude those.. Google the Maven Failsafe options for details. I personally run just integration tests like so:

          mvn clean test-compile failsafe:integration-test

          and then find one failure that I want to debug, and run that one alone like this:

          mvn clean test-compile failsafe:integration-test -Dit.test=ITestS3AFileSystemContract

          Show
          fabbri Aaron Fabbri added a comment - That use case for Mock S3 Client + Real DDB client (for DynamoDBLocal) makes sense. We also need to be able to ensure that DDB metadatastore is disabled for unit tests, even if it is configured in the Hadoop configuration. That could be solved as part of HADOOP-13589 . In my working tree I have a patch on top of your v10 here that separates out the DynamoDB Client Factory into a separate class DynamoDBClientFactory . That would allow us to use a Mock S3 client without a real DDB client. It is an easy change but depends on (or conflicts with) my outstanding patch for HADOOP-13793 (which we should get in soon). As for disabling s3n integration tests, you should be able to add a couple of lines to your pom to exclude those.. Google the Maven Failsafe options for details. I personally run just integration tests like so: mvn clean test-compile failsafe:integration-test and then find one failure that I want to debug, and run that one alone like this: mvn clean test-compile failsafe:integration-test -Dit.test=ITestS3AFileSystemContract
          Hide
          liuml07 Mingliang Liu added a comment -

          Thanks for the tip of disabling s3n integration tests. I find the command mvn -Dparallel-tests -DtestsThreadCount=8 -Dit.test='ITestS3A*' -Dtest=none clean verify is also helpful.

          I'll review and/or commit HADOOP-13793 today. You're right we have to disable the DDB metadatastore is disabled for unit tests even it's configured. For the DynamoDBClientFactory, that sounds good if both S3 client and DDB client are mocked except TestDynamoDBMetadataStore which will create one itself against the DynamoDBLocal.

          I'm working on fixing other failing tests. Per offline discussion with Steve, he suggested we by now ignore the failing anonymous auth tests for this patch. We do have to support that though.

          Show
          liuml07 Mingliang Liu added a comment - Thanks for the tip of disabling s3n integration tests. I find the command mvn -Dparallel-tests -DtestsThreadCount=8 -Dit.test='ITestS3A*' -Dtest=none clean verify is also helpful. I'll review and/or commit HADOOP-13793 today. You're right we have to disable the DDB metadatastore is disabled for unit tests even it's configured. For the DynamoDBClientFactory , that sounds good if both S3 client and DDB client are mocked except TestDynamoDBMetadataStore which will create one itself against the DynamoDBLocal. I'm working on fixing other failing tests. Per offline discussion with Steve, he suggested we by now ignore the failing anonymous auth tests for this patch. We do have to support that though.
          Hide
          fabbri Aaron Fabbri added a comment -

          I think this is the list of outstanding items to get integration tests passing.

          1. Dealing with anonymous / reduced privilege bucket credentials (Steve Loughran's previous comment). We should discuss separately... maybe separate JIRA? I have some other related requirements around table <-> bucket mappings.
          2. Updating S3AFileStatus#isEmptyDirectory(). move(), put(), delete(), and deleteSubtree() will need to maintain the parent dir's empty bit and/or invalidate it's state. I think the basic logic used in LocalMetadataStore should work fine for now.
          3. deleteSubtree(path) assumes that any deleted subtree is fully recorded in the MetadataStore. Best solution, IMO, is to query for all entries that have 'path' as an ancestor. Hoping we can use a prefix scan to keep that efficient. Mingliang Liu would love to hear your DynamoDB expertise on that idea?

          I'm working on #2 at the moment. I wrote a new integration test ITestS3AEmptyDirectory that exercises a directory going from empty->non-empty and vice-versa. Much easier to debug that case in isolation! It passes for LocalMetadataStore but, as expected, fails for DDB still.

          Show
          fabbri Aaron Fabbri added a comment - I think this is the list of outstanding items to get integration tests passing. 1. Dealing with anonymous / reduced privilege bucket credentials ( Steve Loughran 's previous comment). We should discuss separately... maybe separate JIRA? I have some other related requirements around table <-> bucket mappings. 2. Updating S3AFileStatus#isEmptyDirectory() . move(), put(), delete(), and deleteSubtree() will need to maintain the parent dir's empty bit and/or invalidate it's state. I think the basic logic used in LocalMetadataStore should work fine for now. 3. deleteSubtree(path) assumes that any deleted subtree is fully recorded in the MetadataStore. Best solution, IMO, is to query for all entries that have 'path' as an ancestor. Hoping we can use a prefix scan to keep that efficient. Mingliang Liu would love to hear your DynamoDB expertise on that idea? I'm working on #2 at the moment. I wrote a new integration test ITestS3AEmptyDirectory that exercises a directory going from empty->non-empty and vice-versa. Much easier to debug that case in isolation! It passes for LocalMetadataStore but, as expected, fails for DDB still.
          Hide
          liuml07 Mingliang Liu added a comment -

          Thanks Aaron Fabbri. Quick reply (I'm working on this as well, will keep posted):

          1. For point 1, let's track elsewhere.
          2. For point 2, the explanation makes sense. My current in-progress change is to remove the "isEmpty" field from DynamoDB (DDB) table for directories, and to issue a query DDB request whose "parent" field is the current directory. Then I realized that, there may be items in the table whose ancestor (parent of parent, say) is the given directory, but their parent directories are missing. e.g for /a, /a/b/c, /a/b/d, /a is not empty. This has some similar problem to point 3. A simply query seems not enough.
          3. For point 3, yes we may have to use scan as the hash key is not known. Let's figure out the best solution.
          Show
          liuml07 Mingliang Liu added a comment - Thanks Aaron Fabbri . Quick reply (I'm working on this as well, will keep posted): For point 1, let's track elsewhere. For point 2, the explanation makes sense. My current in-progress change is to remove the "isEmpty" field from DynamoDB (DDB) table for directories, and to issue a query DDB request whose "parent" field is the current directory. Then I realized that, there may be items in the table whose ancestor (parent of parent, say) is the given directory, but their parent directories are missing. e.g for /a, /a/b/c, /a/b/d , /a is not empty. This has some similar problem to point 3. A simply query seems not enough. For point 3, yes we may have to use scan as the hash key is not known. Let's figure out the best solution.
          Hide
          fabbri Aaron Fabbri added a comment -

          I did a little research on #3. It looks like you cannot do a prefix scan on a partition key for DynamoDB. This seems to imply that, considering an operation deleteSubtree(delete_path), a simple search by prefix to find all entries with paths that begin with delete_path would actually be a full table scan. If I'm right, that is unfortunate.

          The problem with the existing deleteSubtree(delete_path) implementation is that all the children under delete_path might not be reachable from delete_path by doing a simple tree walk over the state in the MetadataStore. The algorithm would work, however, if, when we created a file, we also created all its ancestor directories up to the root. This would establish an invariant that

          For any path p in DDB MetadataStore
          For each ancestor a_i from p to the root
          a_i is in DDB MetadataStore

          This actually sounds reasonable. Can we do it without changing the MetadataStore interface? I think we can: when we create(path), we always have the full absolute 'path', so we know the names of the ancestors all the way to the root.

          Thoughts?

          Show
          fabbri Aaron Fabbri added a comment - I did a little research on #3. It looks like you cannot do a prefix scan on a partition key for DynamoDB. This seems to imply that, considering an operation deleteSubtree(delete_path) , a simple search by prefix to find all entries with paths that begin with delete_path would actually be a full table scan. If I'm right, that is unfortunate. The problem with the existing deleteSubtree(delete_path) implementation is that all the children under delete_path might not be reachable from delete_path by doing a simple tree walk over the state in the MetadataStore. The algorithm would work, however, if, when we created a file, we also created all its ancestor directories up to the root. This would establish an invariant that For any path p in DDB MetadataStore For each ancestor a_i from p to the root a_i is in DDB MetadataStore This actually sounds reasonable. Can we do it without changing the MetadataStore interface? I think we can: when we create(path), we always have the full absolute 'path', so we know the names of the ancestors all the way to the root. Thoughts?
          Hide
          liuml07 Mingliang Liu added a comment -

          Yes I did consider putting all the ancestors to the metadata store when putting a single path. Another benefit is that, isEmpty will be much easier: simply issue a query request (limit return size 1) whose hash key ("parent" field) is the specific directory, and if there is any data returned, the directory is non-empty; else empty. Then the case that /a, /a/b/c, /a/b/d yet /a is not empty, does not exist. Plus we don't have to store/maintain the isEmpty field any longer.

          I gave up this constraints when implementing DDB and let the file system enforces this for the sake of performance. Consider a simple case: to put(PathMetadata meta) 1K files in a deep directory (say 10 layers), every put operation will check if all the ancestors exist, and 1K operation becomes 10K operations to DDB. For put(DirListingMetadata meta), it will be efficient so we can blame users for not using this one instead.

          So overall, not changing MetadataStore is possible and we can change this in the DynamoDBMetadataStore implementation. I'll post a patch (may be a wip one) soon.

          So we did find real bugs/problems/limitation via integration tests; and they're helpful. Thanks,

          Show
          liuml07 Mingliang Liu added a comment - Yes I did consider putting all the ancestors to the metadata store when putting a single path. Another benefit is that, isEmpty will be much easier: simply issue a query request (limit return size 1) whose hash key ("parent" field) is the specific directory, and if there is any data returned, the directory is non-empty; else empty. Then the case that /a, /a/b/c, /a/b/d yet /a is not empty, does not exist. Plus we don't have to store/maintain the isEmpty field any longer. I gave up this constraints when implementing DDB and let the file system enforces this for the sake of performance. Consider a simple case: to put(PathMetadata meta) 1K files in a deep directory (say 10 layers), every put operation will check if all the ancestors exist, and 1K operation becomes 10K operations to DDB. For put(DirListingMetadata meta) , it will be efficient so we can blame users for not using this one instead. So overall, not changing MetadataStore is possible and we can change this in the DynamoDBMetadataStore implementation. I'll post a patch (may be a wip one) soon. So we did find real bugs/problems/limitation via integration tests; and they're helpful. Thanks,
          Hide
          liuml07 Mingliang Liu added a comment -

          The v11 patch:

          1. Rebases from the feature branch, and also enforces that when putting a new item, it also put all its ancestors to the DDB table. One downside of this change is that, testPutNew() will fail as it assumes we don't create&put its ancestors when we put a new item. We have to update that.
          2. Removes the isEmpty field in the DDB table; it queries the DDB for determining whether directory items are empty.
          3. Simply borrowes the idea of a new class DynamoDBClientFactor. The unit test will always use a mocked S3 client and null metadata store, except the TestDynamoDBMetadataStore which starts a local DDB server for comprehensive test.

          I know Aaron Fabbri has also some in-progress changes. I hope you don't mind rebasing and uploading a new one after that.

          Will run and pass more integration tests soon.

          -------------------------------------------------------
           T E S T S
          -------------------------------------------------------
          Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractGetFileStatus
          Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractDistCp
          Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractDelete
          Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractCreate
          Tests run: 10, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 64.509 sec - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractCreate
          Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractMkdir
          Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 72.058 sec - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractDelete
          Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractOpen
          Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 36.762 sec - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractOpen
          Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractRename
          Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 46.983 sec - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractMkdir
          Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractRootDir
          Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 57.878 sec - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractRename
          Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractSeek
          Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 247.155 sec - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractDistCp
          Running org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContext
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.202 sec - in org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContext
          Running org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextCreateMkdir
          Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 250.842 sec - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractGetFileStatus
          Running org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextMainOperations
          Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 139.282 sec - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractSeek
          Running org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextStatistics
          Tests run: 9, Failures: 3, Errors: 1, Skipped: 0, Time elapsed: 208.542 sec <<< FAILURE! - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractRootDir
          testRecursiveRootListing(org.apache.hadoop.fs.contract.s3a.ITestS3AContractRootDir)  Time elapsed: 119.093 sec  <<< FAILURE!
          java.lang.AssertionError: files mismatch: between
            "s3a://mliu-test-aws-s3guard/Users/mliu/Workspace/hadoop/hadoop-tools/hadoop-aws/target/test-dir/2/qw4OLiKxn8/file"
            "s3a://mliu-test-aws-s3guard/file.txt"
            "s3a://mliu-test-aws-s3guard/fork-2/test/ITestS3AContractDistCp/largeFilesFromRemote/inputDir/file2"
            "s3a://mliu-test-aws-s3guard/fork-2/test/ITestS3AContractDistCp/largeFilesFromRemote/inputDir/file1"
            "s3a://mliu-test-aws-s3guard/fork-2/test/ITestS3AContractDistCp/largeFilesFromRemote/inputDir/file3"
            "s3a://mliu-test-aws-s3guard/user/mliu/test/parentdir/child"
            "s3a://mliu-test-aws-s3guard/user/mliu/test/file"
          ] and
            "s3a://mliu-test-aws-s3guard/Users/mliu/Workspace/hadoop/hadoop-tools/hadoop-aws/target/test-dir/2/qw4OLiKxn8/file"
            "s3a://mliu-test-aws-s3guard/file.txt"
            "s3a://mliu-test-aws-s3guard/fork-2/test/ITestS3AContractDistCp/largeFilesToRemote/outputDir/inputDir/file3"
            "s3a://mliu-test-aws-s3guard/fork-4/test/testComplexDirActions/dir-0-0000/file--0-0000-0000.txt"
            "s3a://mliu-test-aws-s3guard/fork-4/test/testComplexDirActions/dir-0-0000/file--0-0000-0001.txt"
            "s3a://mliu-test-aws-s3guard/fork-4/test/testComplexDirActions/dir-0-0000/file--0-0000-0002.txt"
            "s3a://mliu-test-aws-s3guard/fork-4/test/testComplexDirActions/dir-0-0000/file--0-0000-0003.txt"
            "s3a://mliu-test-aws-s3guard/fork-4/test/testComplexDirActions/file--0-0000.txt"
            "s3a://mliu-test-aws-s3guard/fork-4/test/testComplexDirActions/file--0-0001.txt"
            "s3a://mliu-test-aws-s3guard/fork-4/test/testComplexDirActions/file--0-0002.txt"
            "s3a://mliu-test-aws-s3guard/fork-4/test/testComplexDirActions/file--0-0003.txt"
            "s3a://mliu-test-aws-s3guard/user/mliu/test/file"
            "s3a://mliu-test-aws-s3guard/user/mliu/test/parentdir/child"
          ]
          	at org.junit.Assert.fail(Assert.java:88)
          	at org.junit.Assert.assertTrue(Assert.java:41)
          	at org.apache.hadoop.fs.contract.ContractTestUtils$TreeScanResults.assertFieldsEquivalent(ContractTestUtils.java:1369)
          	at org.apache.hadoop.fs.contract.AbstractContractRootDirectoryTest.testRecursiveRootListing(AbstractContractRootDirectoryTest.java:222)
          	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          	at java.lang.reflect.Method.invoke(Method.java:497)
          	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
          	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
          	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
          	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
          	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
          	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
          	at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
          	at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
          
          testRmEmptyRootDirNonRecursive(org.apache.hadoop.fs.contract.s3a.ITestS3AContractRootDir)  Time elapsed: 35.451 sec  <<< FAILURE!
          java.lang.AssertionError: After 1 attempts: listing after rm /* not empty
          final [00] S3AFileStatus{path=s3a://mliu-test-aws-s3guard/Users; isDirectory=true; modification_time=0; access_time=0; owner=mliu; group=mliu; permission=rwxrwxrwx; isSymlink=false} isEmptyDirectory=true
          [01] S3AFileStatus{path=s3a://mliu-test-aws-s3guard/fork-3; isDirectory=true; modification_time=0; access_time=0; owner=mliu; group=mliu; permission=rwxrwxrwx; isSymlink=false} isEmptyDirectory=true
          
          deleted [00] S3AFileStatus{path=s3a://mliu-test-aws-s3guard/Users; isDirectory=true; modification_time=0; access_time=0; owner=mliu; group=mliu; permission=rwxrwxrwx; isSymlink=false} isEmptyDirectory=true
          [01] S3AFileStatus{path=s3a://mliu-test-aws-s3guard/fork-3; isDirectory=true; modification_time=0; access_time=0; owner=mliu; group=mliu; permission=rwxrwxrwx; isSymlink=false} isEmptyDirectory=true
          [02] S3AFileStatus{path=s3a://mliu-test-aws-s3guard/path; isDirectory=true; modification_time=0; access_time=0; owner=mliu; group=mliu; permission=rwxrwxrwx; isSymlink=false} isEmptyDirectory=true
          [03] S3AFileStatus{path=s3a://mliu-test-aws-s3guard/fork-4; isDirectory=true; modification_time=0; access_time=0; owner=mliu; group=mliu; permission=rwxrwxrwx; isSymlink=false} isEmptyDirectory=true
          [04] S3AFileStatus{path=s3a://mliu-test-aws-s3guard/fork-1; isDirectory=true; modification_time=0; access_time=0; owner=mliu; group=mliu; permission=rwxrwxrwx; isSymlink=false} isEmptyDirectory=true
          [05] S3AFileStatus{path=s3a://mliu-test-aws-s3guard/file.txt; isDirectory=false; length=0; replication=1; blocksize=33554432; modification_time=1481010208231; access_time=0; owner=mliu; group=mliu; permission=rw-rw-rw-; isSymlink=false} isEmptyDirectory=false
          [06] S3AFileStatus{path=s3a://mliu-test-aws-s3guard/fork-2; isDirectory=true; modification_time=0; access_time=0; owner=mliu; group=mliu; permission=rwxrwxrwx; isSymlink=false} isEmptyDirectory=true
          [07] S3AFileStatus{path=s3a://mliu-test-aws-s3guard/user; isDirectory=true; modification_time=0; access_time=0; owner=mliu; group=mliu; permission=rwxrwxrwx; isSymlink=false} isEmptyDirectory=true
          
          original [00] S3AFileStatus{path=s3a://mliu-test-aws-s3guard/Users; isDirectory=true; modification_time=0; access_time=0; owner=mliu; group=mliu; permission=rwxrwxrwx; isSymlink=false} isEmptyDirectory=true
          [01] S3AFileStatus{path=s3a://mliu-test-aws-s3guard/fork-3; isDirectory=true; modification_time=0; access_time=0; owner=mliu; group=mliu; permission=rwxrwxrwx; isSymlink=false} isEmptyDirectory=true
          [02] S3AFileStatus{path=s3a://mliu-test-aws-s3guard/path; isDirectory=true; modification_time=0; access_time=0; owner=mliu; group=mliu; permission=rwxrwxrwx; isSymlink=false} isEmptyDirectory=true
          [03] S3AFileStatus{path=s3a://mliu-test-aws-s3guard/fork-4; isDirectory=true; modification_time=0; access_time=0; owner=mliu; group=mliu; permission=rwxrwxrwx; isSymlink=false} isEmptyDirectory=true
          [04] S3AFileStatus{path=s3a://mliu-test-aws-s3guard/fork-1; isDirectory=true; modification_time=0; access_time=0; owner=mliu; group=mliu; permission=rwxrwxrwx; isSymlink=false} isEmptyDirectory=true
          [05] S3AFileStatus{path=s3a://mliu-test-aws-s3guard/file.txt; isDirectory=false; length=0; replication=1; blocksize=33554432; modification_time=1481010208231; access_time=0; owner=mliu; group=mliu; permission=rw-rw-rw-; isSymlink=false} isEmptyDirectory=false
          [06] S3AFileStatus{path=s3a://mliu-test-aws-s3guard/fork-2; isDirectory=true; modification_time=0; access_time=0; owner=mliu; group=mliu; permission=rwxrwxrwx; isSymlink=false} isEmptyDirectory=true
          [07] S3AFileStatus{path=s3a://mliu-test-aws-s3guard/user; isDirectory=true; modification_time=0; access_time=0; owner=mliu; group=mliu; permission=rwxrwxrwx; isSymlink=false} isEmptyDirectory=true
          
          	at org.junit.Assert.fail(Assert.java:88)
          	at org.apache.hadoop.fs.contract.AbstractContractRootDirectoryTest$1.call(AbstractContractRootDirectoryTest.java:103)
          	at org.apache.hadoop.fs.contract.AbstractContractRootDirectoryTest$1.call(AbstractContractRootDirectoryTest.java:97)
          	at org.apache.hadoop.test.LambdaTestUtils.eventually(LambdaTestUtils.java:234)
          	at org.apache.hadoop.fs.contract.AbstractContractRootDirectoryTest.testRmEmptyRootDirNonRecursive(AbstractContractRootDirectoryTest.java:95)
          	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          	at java.lang.reflect.Method.invoke(Method.java:497)
          	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
          	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
          	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
          	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
          	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
          	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
          	at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
          	at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
          
          testListEmptyRootDirectory(org.apache.hadoop.fs.contract.s3a.ITestS3AContractRootDir)  Time elapsed: 34.391 sec  <<< ERROR!
          java.lang.NullPointerException: null
          	at org.apache.hadoop.fs.s3a.s3guard.DescendantsIterator.next(DescendantsIterator.java:133)
          	at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.deleteSubtree(DynamoDBMetadataStore.java:262)
          	at org.apache.hadoop.fs.s3a.S3AFileSystem.innerDelete(S3AFileSystem.java:1340)
          	at org.apache.hadoop.fs.s3a.S3AFileSystem.delete(S3AFileSystem.java:1261)
          	at org.apache.hadoop.fs.contract.ContractTestUtils.assertDeleted(ContractTestUtils.java:609)
          	at org.apache.hadoop.fs.contract.ContractTestUtils.assertDeleted(ContractTestUtils.java:590)
          	at org.apache.hadoop.fs.contract.AbstractContractRootDirectoryTest.testListEmptyRootDirectory(AbstractContractRootDirectoryTest.java:186)
          	at org.apache.hadoop.fs.contract.s3a.ITestS3AContractRootDir.testListEmptyRootDirectory(ITestS3AContractRootDir.java:49)
          
          testSimpleRootListing(org.apache.hadoop.fs.contract.s3a.ITestS3AContractRootDir)  Time elapsed: 2.493 sec  <<< FAILURE!
          java.lang.AssertionError: expected:<3> but was:<2>
          	at org.junit.Assert.fail(Assert.java:88)
          	at org.junit.Assert.failNotEquals(Assert.java:743)
          	at org.junit.Assert.assertEquals(Assert.java:118)
          	at org.junit.Assert.assertEquals(Assert.java:555)
          	at org.junit.Assert.assertEquals(Assert.java:542)
          	at org.apache.hadoop.fs.contract.AbstractContractRootDirectoryTest.testSimpleRootListing(AbstractContractRootDirectoryTest.java:207)
          	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          	at java.lang.reflect.Method.invoke(Method.java:497)
          	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
          	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
          	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
          	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
          	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
          	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
          	at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
          	at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
          
          Running org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextURI
          Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 27.604 sec - in org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextStatistics
          Running org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextUtil
          Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 34.459 sec - in org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextUtil
          Running org.apache.hadoop.fs.s3a.ITestS3AAWSCredentialsProvider
          Tests run: 4, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 6.343 sec <<< FAILURE! - in org.apache.hadoop.fs.s3a.ITestS3AAWSCredentialsProvider
          testAnonymousProvider(org.apache.hadoop.fs.s3a.ITestS3AAWSCredentialsProvider)  Time elapsed: 1.866 sec  <<< ERROR!
          org.apache.hadoop.fs.s3a.AWSServiceIOException: initializing  on s3a://landsat-pds/scene_list.gz: com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException: Request is missing Authentication Token (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: MissingAuthenticationTokenException; Request ID: 62EEHHLVUVIDPQ5V29A4R5O8RBVV4KQNSO5AEMVJF66Q9ASUAAJG): Request is missing Authentication Token (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: MissingAuthenticationTokenException; Request ID: 62EEHHLVUVIDPQ5V29A4R5O8RBVV4KQNSO5AEMVJF66Q9ASUAAJG)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1529)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1167)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:948)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:661)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:635)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:618)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$300(AmazonHttpClient.java:586)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:573)
          	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:445)
          	at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.doInvoke(AmazonDynamoDBClient.java:1722)
          	at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:1698)
          	at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.createTable(AmazonDynamoDBClient.java:743)
          	at com.amazonaws.services.dynamodbv2.document.DynamoDB.createTable(DynamoDB.java:96)
          	at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.createTable(DynamoDBMetadataStore.java:455)
          	at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.initialize(DynamoDBMetadataStore.java:184)
          	at org.apache.hadoop.fs.s3a.s3guard.S3Guard.getMetadataStore(S3Guard.java:138)
          	at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:252)
          	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3246)
          	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:123)
          	at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3295)
          	at org.apache.hadoop.fs.FileSystem$Cache.getUnique(FileSystem.java:3269)
          	at org.apache.hadoop.fs.FileSystem.newInstance(FileSystem.java:529)
          	at org.apache.hadoop.fs.s3a.ITestS3AAWSCredentialsProvider.testAnonymousProvider(ITestS3AAWSCredentialsProvider.java:133)
          
          testBadCredentials(org.apache.hadoop.fs.s3a.ITestS3AAWSCredentialsProvider)  Time elapsed: 1.301 sec  <<< ERROR!
          org.apache.hadoop.fs.s3a.AWSServiceIOException: initializing  on s3a://mliu-test-aws-s3guard/: com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException: The security token included in the request is invalid. (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: UnrecognizedClientException; Request ID: ICJT511MGKE0PTBRS2D5OH75FJVV4KQNSO5AEMVJF66Q9ASUAAJG): The security token included in the request is invalid. (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: UnrecognizedClientException; Request ID: ICJT511MGKE0PTBRS2D5OH75FJVV4KQNSO5AEMVJF66Q9ASUAAJG)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1529)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1167)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:948)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:661)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:635)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:618)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$300(AmazonHttpClient.java:586)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:573)
          	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:445)
          	at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.doInvoke(AmazonDynamoDBClient.java:1722)
          	at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:1698)
          	at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.createTable(AmazonDynamoDBClient.java:743)
          	at com.amazonaws.services.dynamodbv2.document.DynamoDB.createTable(DynamoDB.java:96)
          	at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.createTable(DynamoDBMetadataStore.java:455)
          	at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.initialize(DynamoDBMetadataStore.java:184)
          	at org.apache.hadoop.fs.s3a.s3guard.S3Guard.getMetadataStore(S3Guard.java:138)
          	at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:252)
          	at org.apache.hadoop.fs.s3a.S3ATestUtils.createTestFileSystem(S3ATestUtils.java:103)
          	at org.apache.hadoop.fs.s3a.S3ATestUtils.createTestFileSystem(S3ATestUtils.java:63)
          	at org.apache.hadoop.fs.s3a.ITestS3AAWSCredentialsProvider.createFailingFS(ITestS3AAWSCredentialsProvider.java:76)
          	at org.apache.hadoop.fs.s3a.ITestS3AAWSCredentialsProvider.testBadCredentials(ITestS3AAWSCredentialsProvider.java:102)
          
          Running org.apache.hadoop.fs.s3a.ITestS3ABlockOutputArray
          Tests run: 11, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 143.379 sec <<< FAILURE! - in org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextCreateMkdir
          testMkdirsRecursiveWithExistingDir(org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextCreateMkdir)  Time elapsed: 34.171 sec  <<< ERROR!
          java.io.FileNotFoundException: No such file or directory: s3a://mliu-test-aws-s3guard/Users/mliu/Workspace/hadoop/hadoop-tools/hadoop-aws/target/test-dir/2/dYvFHHBXOC/aDir/bDir/cDir
          	at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:1710)
          	at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:119)
          	at org.apache.hadoop.fs.DelegateToFileSystem.getFileStatus(DelegateToFileSystem.java:126)
          	at org.apache.hadoop.fs.FileContext$15.next(FileContext.java:1167)
          	at org.apache.hadoop.fs.FileContext$15.next(FileContext.java:1163)
          	at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90)
          	at org.apache.hadoop.fs.FileContext.getFileStatus(FileContext.java:1169)
          	at org.apache.hadoop.fs.FileContextCreateMkdirBaseTest.testMkdirsRecursiveWithExistingDir(FileContextCreateMkdirBaseTest.java:125)
          	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          	at java.lang.reflect.Method.invoke(Method.java:497)
          	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
          	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
          	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
          	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
          	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
          	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
          	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
          	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
          	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
          	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
          	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
          	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
          	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
          	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
          	at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
          	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
          	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
          	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
          	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
          	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
          	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
          
          Running org.apache.hadoop.fs.s3a.ITestS3ABlockOutputByteBuffer
          Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.146 sec - in org.apache.hadoop.fs.s3a.ITestS3ABlockOutputArray
          Running org.apache.hadoop.fs.s3a.ITestS3ABlockOutputDisk
          Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.391 sec - in org.apache.hadoop.fs.s3a.ITestS3ABlockOutputByteBuffer
          Running org.apache.hadoop.fs.s3a.ITestS3ABlocksize
          Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 24.089 sec - in org.apache.hadoop.fs.s3a.ITestS3ABlockOutputDisk
          Running org.apache.hadoop.fs.s3a.ITestS3AConfiguration
          Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.792 sec - in org.apache.hadoop.fs.s3a.ITestS3ABlocksize
          Running org.apache.hadoop.fs.s3a.ITestS3ACredentialsInURL
          Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 5.613 sec <<< FAILURE! - in org.apache.hadoop.fs.s3a.ITestS3ACredentialsInURL
          testInvalidCredentialsFail(org.apache.hadoop.fs.s3a.ITestS3ACredentialsInURL)  Time elapsed: 1.443 sec  <<< FAILURE!
          java.lang.AssertionError: Expected an AccessDeniedException, got S3AFileStatus{path=s3a://mliu-test-aws-s3guard/; isDirectory=true; modification_time=0; access_time=0; owner=mliu; group=mliu; permission=rwxrwxrwx; isSymlink=false} isEmptyDirectory=false
          	at org.junit.Assert.fail(Assert.java:88)
          	at org.apache.hadoop.fs.s3a.ITestS3ACredentialsInURL.testInvalidCredentialsFail(ITestS3ACredentialsInURL.java:130)
          
          Running org.apache.hadoop.fs.s3a.ITestS3AEncryption
          Tests run: 19, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.095 sec - in org.apache.hadoop.fs.s3a.ITestS3AConfiguration
          Running org.apache.hadoop.fs.s3a.ITestS3AEncryptionAlgorithmPropagation
          Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.756 sec - in org.apache.hadoop.fs.s3a.ITestS3AEncryptionAlgorithmPropagation
          Running org.apache.hadoop.fs.s3a.ITestS3AEncryptionBlockOutputStream
          Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 71.234 sec - in org.apache.hadoop.fs.s3a.ITestS3AEncryption
          Running org.apache.hadoop.fs.s3a.ITestS3AFailureHandling
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.71 sec - in org.apache.hadoop.fs.s3a.ITestS3AFailureHandling
          Running org.apache.hadoop.fs.s3a.ITestS3AFileOperationCost
          Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 64.232 sec - in org.apache.hadoop.fs.s3a.ITestS3AEncryptionBlockOutputStream
          Running org.apache.hadoop.fs.s3a.ITestS3AFileSystemContract
          Tests run: 7, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 69.269 sec <<< FAILURE! - in org.apache.hadoop.fs.s3a.ITestS3AFileOperationCost
          testFakeDirectoryDeletion(org.apache.hadoop.fs.s3a.ITestS3AFileOperationCost)  Time elapsed: 38.245 sec  <<< FAILURE!
          java.lang.AssertionError: after rename(srcFilePath, destFilePath): directories_created expected:<1> but was:<0>
          	at org.junit.Assert.fail(Assert.java:88)
          	at org.junit.Assert.failNotEquals(Assert.java:743)
          	at org.junit.Assert.assertEquals(Assert.java:118)
          	at org.junit.Assert.assertEquals(Assert.java:555)
          	at org.apache.hadoop.fs.s3a.S3ATestUtils$MetricDiff.assertDiffEquals(S3ATestUtils.java:431)
          	at org.apache.hadoop.fs.s3a.ITestS3AFileOperationCost.testFakeDirectoryDeletion(ITestS3AFileOperationCost.java:254)
          
          Running org.apache.hadoop.fs.s3a.ITestS3AMiscOperations
          Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.188 sec - in org.apache.hadoop.fs.s3a.ITestS3AMiscOperations
          Running org.apache.hadoop.fs.s3a.ITestS3ATemporaryCredentials
          Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.366 sec - in org.apache.hadoop.fs.s3a.ITestS3ATemporaryCredentials
          Running org.apache.hadoop.fs.s3a.ITestS3ATestUtils
          Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.123 sec - in org.apache.hadoop.fs.s3a.ITestS3ATestUtils
          Running org.apache.hadoop.fs.s3a.scale.ITestS3ADeleteFilesOneByOne
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 4.396 sec - in org.apache.hadoop.fs.s3a.scale.ITestS3ADeleteFilesOneByOne
          Running org.apache.hadoop.fs.s3a.scale.ITestS3ADeleteManyFiles
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 4.342 sec - in org.apache.hadoop.fs.s3a.scale.ITestS3ADeleteManyFiles
          Running org.apache.hadoop.fs.s3a.scale.ITestS3ADirectoryPerformance
          Tests run: 5, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 17.954 sec - in org.apache.hadoop.fs.s3a.scale.ITestS3ADirectoryPerformance
          Running org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesArrayBlocks
          Tests run: 5, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 23.965 sec - in org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesArrayBlocks
          Running org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesByteBufferBlocks
          Tests run: 17, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 353.478 sec - in org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextURI
          Running org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesClassicOutput
          Tests run: 5, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 15.458 sec - in org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesByteBufferBlocks
          Running org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesDiskBlocks
          Tests run: 5, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 15.381 sec - in org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesClassicOutput
          Running org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance
          Tests run: 5, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 16.141 sec - in org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesDiskBlocks
          Running org.apache.hadoop.fs.s3a.yarn.ITestS3A
          Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.081 sec - in org.apache.hadoop.fs.s3a.yarn.ITestS3A
          Running org.apache.hadoop.fs.s3a.yarn.ITestS3AMiniYarnCluster
          Tests run: 8, Failures: 0, Errors: 0, Skipped: 8, Time elapsed: 24.094 sec - in org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 41.678 sec - in org.apache.hadoop.fs.s3a.yarn.ITestS3AMiniYarnCluster
          Tests run: 43, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 279.272 sec <<< FAILURE! - in org.apache.hadoop.fs.s3a.ITestS3AFileSystemContract
          testRenameToDirWithSamePrefixAllowed(org.apache.hadoop.fs.s3a.ITestS3AFileSystemContract)  Time elapsed: 4.693 sec  <<< ERROR!
          org.apache.hadoop.fs.s3a.AWSServiceIOException: createTable: com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException: Provided list of item keys contains duplicates (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: ValidationException; Request ID: P128RFOF3H9IAHJTKRUR8PIHENVV4KQNSO5AEMVJF66Q9ASUAAJG): Provided list of item keys contains duplicates (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: ValidationException; Request ID: P128RFOF3H9IAHJTKRUR8PIHENVV4KQNSO5AEMVJF66Q9ASUAAJG)
          	at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:166)
          	at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.move(DynamoDBMetadataStore.java:347)
          	at org.apache.hadoop.fs.s3a.S3AFileSystem.innerRename(S3AFileSystem.java:862)
          	at org.apache.hadoop.fs.s3a.S3AFileSystem.rename(S3AFileSystem.java:655)
          	at org.apache.hadoop.fs.FileSystemContractBaseTest.rename(FileSystemContractBaseTest.java:512)
          	at org.apache.hadoop.fs.FileSystemContractBaseTest.testRenameToDirWithSamePrefixAllowed(FileSystemContractBaseTest.java:656)
          	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          	at java.lang.reflect.Method.invoke(Method.java:497)
          	at junit.framework.TestCase.runTest(TestCase.java:176)
          	at junit.framework.TestCase.runBare(TestCase.java:141)
          	at junit.framework.TestResult$1.protect(TestResult.java:122)
          	at junit.framework.TestResult.runProtected(TestResult.java:142)
          	at junit.framework.TestResult.run(TestResult.java:125)
          	at junit.framework.TestCase.run(TestCase.java:129)
          	at junit.framework.TestSuite.runTest(TestSuite.java:255)
          	at junit.framework.TestSuite.run(TestSuite.java:250)
          	at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:84)
          	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
          	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
          	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
          	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
          	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
          	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
          Caused by: com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException: Provided list of item keys contains duplicates (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: ValidationException; Request ID: P128RFOF3H9IAHJTKRUR8PIHENVV4KQNSO5AEMVJF66Q9ASUAAJG)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1529)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1167)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:948)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:661)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:635)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:618)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$300(AmazonHttpClient.java:586)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:573)
          	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:445)
          	at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.doInvoke(AmazonDynamoDBClient.java:1722)
          	at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:1698)
          	at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.batchWriteItem(AmazonDynamoDBClient.java:668)
          	at com.amazonaws.services.dynamodbv2.document.internal.BatchWriteItemImpl.doBatchWriteItem(BatchWriteItemImpl.java:111)
          	at com.amazonaws.services.dynamodbv2.document.internal.BatchWriteItemImpl.batchWriteItem(BatchWriteItemImpl.java:52)
          	at com.amazonaws.services.dynamodbv2.document.DynamoDB.batchWriteItem(DynamoDB.java:178)
          	at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.move(DynamoDBMetadataStore.java:338)
          	at org.apache.hadoop.fs.s3a.S3AFileSystem.innerRename(S3AFileSystem.java:862)
          	at org.apache.hadoop.fs.s3a.S3AFileSystem.rename(S3AFileSystem.java:655)
          	at org.apache.hadoop.fs.FileSystemContractBaseTest.rename(FileSystemContractBaseTest.java:512)
          	at org.apache.hadoop.fs.FileSystemContractBaseTest.testRenameToDirWithSamePrefixAllowed(FileSystemContractBaseTest.java:656)
          	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          	at java.lang.reflect.Method.invoke(Method.java:497)
          	at junit.framework.TestCase.runTest(TestCase.java:176)
          	at junit.framework.TestCase.runBare(TestCase.java:141)
          	at junit.framework.TestResult$1.protect(TestResult.java:122)
          	at junit.framework.TestResult.runProtected(TestResult.java:142)
          	at junit.framework.TestResult.run(TestResult.java:125)
          	at junit.framework.TestCase.run(TestCase.java:129)
          	at junit.framework.TestSuite.runTest(TestSuite.java:255)
          	at junit.framework.TestSuite.run(TestSuite.java:250)
          	at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:84)
          	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
          	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
          	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
          	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
          	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
          	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
          
          Show
          liuml07 Mingliang Liu added a comment - The v11 patch: Rebases from the feature branch, and also enforces that when putting a new item, it also put all its ancestors to the DDB table. One downside of this change is that, testPutNew() will fail as it assumes we don't create&put its ancestors when we put a new item. We have to update that. Removes the isEmpty field in the DDB table; it queries the DDB for determining whether directory items are empty. Simply borrowes the idea of a new class DynamoDBClientFactor. The unit test will always use a mocked S3 client and null metadata store, except the TestDynamoDBMetadataStore which starts a local DDB server for comprehensive test. I know Aaron Fabbri has also some in-progress changes. I hope you don't mind rebasing and uploading a new one after that. Will run and pass more integration tests soon. ------------------------------------------------------- T E S T S ------------------------------------------------------- Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractGetFileStatus Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractDistCp Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractDelete Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractCreate Tests run: 10, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 64.509 sec - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractCreate Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractMkdir Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 72.058 sec - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractDelete Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractOpen Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 36.762 sec - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractOpen Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractRename Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 46.983 sec - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractMkdir Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractRootDir Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 57.878 sec - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractRename Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractSeek Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 247.155 sec - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractDistCp Running org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContext Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.202 sec - in org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContext Running org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextCreateMkdir Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 250.842 sec - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractGetFileStatus Running org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextMainOperations Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 139.282 sec - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractSeek Running org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextStatistics Tests run: 9, Failures: 3, Errors: 1, Skipped: 0, Time elapsed: 208.542 sec <<< FAILURE! - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractRootDir testRecursiveRootListing(org.apache.hadoop.fs.contract.s3a.ITestS3AContractRootDir) Time elapsed: 119.093 sec <<< FAILURE! java.lang.AssertionError: files mismatch: between "s3a: //mliu-test-aws-s3guard/Users/mliu/Workspace/hadoop/hadoop-tools/hadoop-aws/target/test-dir/2/qw4OLiKxn8/file" "s3a: //mliu-test-aws-s3guard/file.txt" "s3a: //mliu-test-aws-s3guard/fork-2/test/ITestS3AContractDistCp/largeFilesFromRemote/inputDir/file2" "s3a: //mliu-test-aws-s3guard/fork-2/test/ITestS3AContractDistCp/largeFilesFromRemote/inputDir/file1" "s3a: //mliu-test-aws-s3guard/fork-2/test/ITestS3AContractDistCp/largeFilesFromRemote/inputDir/file3" "s3a: //mliu-test-aws-s3guard/user/mliu/test/parentdir/child" "s3a: //mliu-test-aws-s3guard/user/mliu/test/file" ] and "s3a: //mliu-test-aws-s3guard/Users/mliu/Workspace/hadoop/hadoop-tools/hadoop-aws/target/test-dir/2/qw4OLiKxn8/file" "s3a: //mliu-test-aws-s3guard/file.txt" "s3a: //mliu-test-aws-s3guard/fork-2/test/ITestS3AContractDistCp/largeFilesToRemote/outputDir/inputDir/file3" "s3a: //mliu-test-aws-s3guard/fork-4/test/testComplexDirActions/dir-0-0000/file--0-0000-0000.txt" "s3a: //mliu-test-aws-s3guard/fork-4/test/testComplexDirActions/dir-0-0000/file--0-0000-0001.txt" "s3a: //mliu-test-aws-s3guard/fork-4/test/testComplexDirActions/dir-0-0000/file--0-0000-0002.txt" "s3a: //mliu-test-aws-s3guard/fork-4/test/testComplexDirActions/dir-0-0000/file--0-0000-0003.txt" "s3a: //mliu-test-aws-s3guard/fork-4/test/testComplexDirActions/file--0-0000.txt" "s3a: //mliu-test-aws-s3guard/fork-4/test/testComplexDirActions/file--0-0001.txt" "s3a: //mliu-test-aws-s3guard/fork-4/test/testComplexDirActions/file--0-0002.txt" "s3a: //mliu-test-aws-s3guard/fork-4/test/testComplexDirActions/file--0-0003.txt" "s3a: //mliu-test-aws-s3guard/user/mliu/test/file" "s3a: //mliu-test-aws-s3guard/user/mliu/test/parentdir/child" ] at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.assertTrue(Assert.java:41) at org.apache.hadoop.fs.contract.ContractTestUtils$TreeScanResults.assertFieldsEquivalent(ContractTestUtils.java:1369) at org.apache.hadoop.fs.contract.AbstractContractRootDirectoryTest.testRecursiveRootListing(AbstractContractRootDirectoryTest.java:222) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) testRmEmptyRootDirNonRecursive(org.apache.hadoop.fs.contract.s3a.ITestS3AContractRootDir) Time elapsed: 35.451 sec <<< FAILURE! java.lang.AssertionError: After 1 attempts: listing after rm /* not empty final [00] S3AFileStatus{path=s3a: //mliu-test-aws-s3guard/Users; isDirectory= true ; modification_time=0; access_time=0; owner=mliu; group=mliu; permission=rwxrwxrwx; isSymlink= false } isEmptyDirectory= true [01] S3AFileStatus{path=s3a: //mliu-test-aws-s3guard/fork-3; isDirectory= true ; modification_time=0; access_time=0; owner=mliu; group=mliu; permission=rwxrwxrwx; isSymlink= false } isEmptyDirectory= true deleted [00] S3AFileStatus{path=s3a: //mliu-test-aws-s3guard/Users; isDirectory= true ; modification_time=0; access_time=0; owner=mliu; group=mliu; permission=rwxrwxrwx; isSymlink= false } isEmptyDirectory= true [01] S3AFileStatus{path=s3a: //mliu-test-aws-s3guard/fork-3; isDirectory= true ; modification_time=0; access_time=0; owner=mliu; group=mliu; permission=rwxrwxrwx; isSymlink= false } isEmptyDirectory= true [02] S3AFileStatus{path=s3a: //mliu-test-aws-s3guard/path; isDirectory= true ; modification_time=0; access_time=0; owner=mliu; group=mliu; permission=rwxrwxrwx; isSymlink= false } isEmptyDirectory= true [03] S3AFileStatus{path=s3a: //mliu-test-aws-s3guard/fork-4; isDirectory= true ; modification_time=0; access_time=0; owner=mliu; group=mliu; permission=rwxrwxrwx; isSymlink= false } isEmptyDirectory= true [04] S3AFileStatus{path=s3a: //mliu-test-aws-s3guard/fork-1; isDirectory= true ; modification_time=0; access_time=0; owner=mliu; group=mliu; permission=rwxrwxrwx; isSymlink= false } isEmptyDirectory= true [05] S3AFileStatus{path=s3a: //mliu-test-aws-s3guard/file.txt; isDirectory= false ; length=0; replication=1; blocksize=33554432; modification_time=1481010208231; access_time=0; owner=mliu; group=mliu; permission=rw-rw-rw-; isSymlink= false } isEmptyDirectory= false [06] S3AFileStatus{path=s3a: //mliu-test-aws-s3guard/fork-2; isDirectory= true ; modification_time=0; access_time=0; owner=mliu; group=mliu; permission=rwxrwxrwx; isSymlink= false } isEmptyDirectory= true [07] S3AFileStatus{path=s3a: //mliu-test-aws-s3guard/user; isDirectory= true ; modification_time=0; access_time=0; owner=mliu; group=mliu; permission=rwxrwxrwx; isSymlink= false } isEmptyDirectory= true original [00] S3AFileStatus{path=s3a: //mliu-test-aws-s3guard/Users; isDirectory= true ; modification_time=0; access_time=0; owner=mliu; group=mliu; permission=rwxrwxrwx; isSymlink= false } isEmptyDirectory= true [01] S3AFileStatus{path=s3a: //mliu-test-aws-s3guard/fork-3; isDirectory= true ; modification_time=0; access_time=0; owner=mliu; group=mliu; permission=rwxrwxrwx; isSymlink= false } isEmptyDirectory= true [02] S3AFileStatus{path=s3a: //mliu-test-aws-s3guard/path; isDirectory= true ; modification_time=0; access_time=0; owner=mliu; group=mliu; permission=rwxrwxrwx; isSymlink= false } isEmptyDirectory= true [03] S3AFileStatus{path=s3a: //mliu-test-aws-s3guard/fork-4; isDirectory= true ; modification_time=0; access_time=0; owner=mliu; group=mliu; permission=rwxrwxrwx; isSymlink= false } isEmptyDirectory= true [04] S3AFileStatus{path=s3a: //mliu-test-aws-s3guard/fork-1; isDirectory= true ; modification_time=0; access_time=0; owner=mliu; group=mliu; permission=rwxrwxrwx; isSymlink= false } isEmptyDirectory= true [05] S3AFileStatus{path=s3a: //mliu-test-aws-s3guard/file.txt; isDirectory= false ; length=0; replication=1; blocksize=33554432; modification_time=1481010208231; access_time=0; owner=mliu; group=mliu; permission=rw-rw-rw-; isSymlink= false } isEmptyDirectory= false [06] S3AFileStatus{path=s3a: //mliu-test-aws-s3guard/fork-2; isDirectory= true ; modification_time=0; access_time=0; owner=mliu; group=mliu; permission=rwxrwxrwx; isSymlink= false } isEmptyDirectory= true [07] S3AFileStatus{path=s3a: //mliu-test-aws-s3guard/user; isDirectory= true ; modification_time=0; access_time=0; owner=mliu; group=mliu; permission=rwxrwxrwx; isSymlink= false } isEmptyDirectory= true at org.junit.Assert.fail(Assert.java:88) at org.apache.hadoop.fs.contract.AbstractContractRootDirectoryTest$1.call(AbstractContractRootDirectoryTest.java:103) at org.apache.hadoop.fs.contract.AbstractContractRootDirectoryTest$1.call(AbstractContractRootDirectoryTest.java:97) at org.apache.hadoop.test.LambdaTestUtils.eventually(LambdaTestUtils.java:234) at org.apache.hadoop.fs.contract.AbstractContractRootDirectoryTest.testRmEmptyRootDirNonRecursive(AbstractContractRootDirectoryTest.java:95) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) testListEmptyRootDirectory(org.apache.hadoop.fs.contract.s3a.ITestS3AContractRootDir) Time elapsed: 34.391 sec <<< ERROR! java.lang.NullPointerException: null at org.apache.hadoop.fs.s3a.s3guard.DescendantsIterator.next(DescendantsIterator.java:133) at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.deleteSubtree(DynamoDBMetadataStore.java:262) at org.apache.hadoop.fs.s3a.S3AFileSystem.innerDelete(S3AFileSystem.java:1340) at org.apache.hadoop.fs.s3a.S3AFileSystem.delete(S3AFileSystem.java:1261) at org.apache.hadoop.fs.contract.ContractTestUtils.assertDeleted(ContractTestUtils.java:609) at org.apache.hadoop.fs.contract.ContractTestUtils.assertDeleted(ContractTestUtils.java:590) at org.apache.hadoop.fs.contract.AbstractContractRootDirectoryTest.testListEmptyRootDirectory(AbstractContractRootDirectoryTest.java:186) at org.apache.hadoop.fs.contract.s3a.ITestS3AContractRootDir.testListEmptyRootDirectory(ITestS3AContractRootDir.java:49) testSimpleRootListing(org.apache.hadoop.fs.contract.s3a.ITestS3AContractRootDir) Time elapsed: 2.493 sec <<< FAILURE! java.lang.AssertionError: expected:<3> but was:<2> at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:743) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:555) at org.junit.Assert.assertEquals(Assert.java:542) at org.apache.hadoop.fs.contract.AbstractContractRootDirectoryTest.testSimpleRootListing(AbstractContractRootDirectoryTest.java:207) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Running org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextURI Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 27.604 sec - in org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextStatistics Running org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextUtil Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 34.459 sec - in org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextUtil Running org.apache.hadoop.fs.s3a.ITestS3AAWSCredentialsProvider Tests run: 4, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 6.343 sec <<< FAILURE! - in org.apache.hadoop.fs.s3a.ITestS3AAWSCredentialsProvider testAnonymousProvider(org.apache.hadoop.fs.s3a.ITestS3AAWSCredentialsProvider) Time elapsed: 1.866 sec <<< ERROR! org.apache.hadoop.fs.s3a.AWSServiceIOException: initializing on s3a: //landsat-pds/scene_list.gz: com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException: Request is missing Authentication Token (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: MissingAuthenticationTokenException; Request ID: 62EEHHLVUVIDPQ5V29A4R5O8RBVV4KQNSO5AEMVJF66Q9ASUAAJG): Request is missing Authentication Token (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: MissingAuthenticationTokenException; Request ID: 62EEHHLVUVIDPQ5V29A4R5O8RBVV4KQNSO5AEMVJF66Q9ASUAAJG) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1529) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1167) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:948) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:661) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:635) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:618) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$300(AmazonHttpClient.java:586) at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:573) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:445) at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.doInvoke(AmazonDynamoDBClient.java:1722) at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:1698) at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.createTable(AmazonDynamoDBClient.java:743) at com.amazonaws.services.dynamodbv2.document.DynamoDB.createTable(DynamoDB.java:96) at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.createTable(DynamoDBMetadataStore.java:455) at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.initialize(DynamoDBMetadataStore.java:184) at org.apache.hadoop.fs.s3a.s3guard.S3Guard.getMetadataStore(S3Guard.java:138) at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:252) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3246) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:123) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3295) at org.apache.hadoop.fs.FileSystem$Cache.getUnique(FileSystem.java:3269) at org.apache.hadoop.fs.FileSystem.newInstance(FileSystem.java:529) at org.apache.hadoop.fs.s3a.ITestS3AAWSCredentialsProvider.testAnonymousProvider(ITestS3AAWSCredentialsProvider.java:133) testBadCredentials(org.apache.hadoop.fs.s3a.ITestS3AAWSCredentialsProvider) Time elapsed: 1.301 sec <<< ERROR! org.apache.hadoop.fs.s3a.AWSServiceIOException: initializing on s3a: //mliu-test-aws-s3guard/: com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException: The security token included in the request is invalid. (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: UnrecognizedClientException; Request ID: ICJT511MGKE0PTBRS2D5OH75FJVV4KQNSO5AEMVJF66Q9ASUAAJG): The security token included in the request is invalid. (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: UnrecognizedClientException; Request ID: ICJT511MGKE0PTBRS2D5OH75FJVV4KQNSO5AEMVJF66Q9ASUAAJG) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1529) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1167) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:948) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:661) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:635) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:618) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$300(AmazonHttpClient.java:586) at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:573) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:445) at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.doInvoke(AmazonDynamoDBClient.java:1722) at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:1698) at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.createTable(AmazonDynamoDBClient.java:743) at com.amazonaws.services.dynamodbv2.document.DynamoDB.createTable(DynamoDB.java:96) at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.createTable(DynamoDBMetadataStore.java:455) at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.initialize(DynamoDBMetadataStore.java:184) at org.apache.hadoop.fs.s3a.s3guard.S3Guard.getMetadataStore(S3Guard.java:138) at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:252) at org.apache.hadoop.fs.s3a.S3ATestUtils.createTestFileSystem(S3ATestUtils.java:103) at org.apache.hadoop.fs.s3a.S3ATestUtils.createTestFileSystem(S3ATestUtils.java:63) at org.apache.hadoop.fs.s3a.ITestS3AAWSCredentialsProvider.createFailingFS(ITestS3AAWSCredentialsProvider.java:76) at org.apache.hadoop.fs.s3a.ITestS3AAWSCredentialsProvider.testBadCredentials(ITestS3AAWSCredentialsProvider.java:102) Running org.apache.hadoop.fs.s3a.ITestS3ABlockOutputArray Tests run: 11, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 143.379 sec <<< FAILURE! - in org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextCreateMkdir testMkdirsRecursiveWithExistingDir(org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextCreateMkdir) Time elapsed: 34.171 sec <<< ERROR! java.io.FileNotFoundException: No such file or directory: s3a: //mliu-test-aws-s3guard/Users/mliu/Workspace/hadoop/hadoop-tools/hadoop-aws/target/test-dir/2/dYvFHHBXOC/aDir/bDir/cDir at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:1710) at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:119) at org.apache.hadoop.fs.DelegateToFileSystem.getFileStatus(DelegateToFileSystem.java:126) at org.apache.hadoop.fs.FileContext$15.next(FileContext.java:1167) at org.apache.hadoop.fs.FileContext$15.next(FileContext.java:1163) at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90) at org.apache.hadoop.fs.FileContext.getFileStatus(FileContext.java:1169) at org.apache.hadoop.fs.FileContextCreateMkdirBaseTest.testMkdirsRecursiveWithExistingDir(FileContextCreateMkdirBaseTest.java:125) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) at org.junit.runners.ParentRunner.run(ParentRunner.java:309) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124) at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103) Running org.apache.hadoop.fs.s3a.ITestS3ABlockOutputByteBuffer Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.146 sec - in org.apache.hadoop.fs.s3a.ITestS3ABlockOutputArray Running org.apache.hadoop.fs.s3a.ITestS3ABlockOutputDisk Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.391 sec - in org.apache.hadoop.fs.s3a.ITestS3ABlockOutputByteBuffer Running org.apache.hadoop.fs.s3a.ITestS3ABlocksize Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 24.089 sec - in org.apache.hadoop.fs.s3a.ITestS3ABlockOutputDisk Running org.apache.hadoop.fs.s3a.ITestS3AConfiguration Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.792 sec - in org.apache.hadoop.fs.s3a.ITestS3ABlocksize Running org.apache.hadoop.fs.s3a.ITestS3ACredentialsInURL Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 5.613 sec <<< FAILURE! - in org.apache.hadoop.fs.s3a.ITestS3ACredentialsInURL testInvalidCredentialsFail(org.apache.hadoop.fs.s3a.ITestS3ACredentialsInURL) Time elapsed: 1.443 sec <<< FAILURE! java.lang.AssertionError: Expected an AccessDeniedException, got S3AFileStatus{path=s3a: //mliu-test-aws-s3guard/; isDirectory= true ; modification_time=0; access_time=0; owner=mliu; group=mliu; permission=rwxrwxrwx; isSymlink= false } isEmptyDirectory= false at org.junit.Assert.fail(Assert.java:88) at org.apache.hadoop.fs.s3a.ITestS3ACredentialsInURL.testInvalidCredentialsFail(ITestS3ACredentialsInURL.java:130) Running org.apache.hadoop.fs.s3a.ITestS3AEncryption Tests run: 19, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.095 sec - in org.apache.hadoop.fs.s3a.ITestS3AConfiguration Running org.apache.hadoop.fs.s3a.ITestS3AEncryptionAlgorithmPropagation Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.756 sec - in org.apache.hadoop.fs.s3a.ITestS3AEncryptionAlgorithmPropagation Running org.apache.hadoop.fs.s3a.ITestS3AEncryptionBlockOutputStream Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 71.234 sec - in org.apache.hadoop.fs.s3a.ITestS3AEncryption Running org.apache.hadoop.fs.s3a.ITestS3AFailureHandling Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.71 sec - in org.apache.hadoop.fs.s3a.ITestS3AFailureHandling Running org.apache.hadoop.fs.s3a.ITestS3AFileOperationCost Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 64.232 sec - in org.apache.hadoop.fs.s3a.ITestS3AEncryptionBlockOutputStream Running org.apache.hadoop.fs.s3a.ITestS3AFileSystemContract Tests run: 7, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 69.269 sec <<< FAILURE! - in org.apache.hadoop.fs.s3a.ITestS3AFileOperationCost testFakeDirectoryDeletion(org.apache.hadoop.fs.s3a.ITestS3AFileOperationCost) Time elapsed: 38.245 sec <<< FAILURE! java.lang.AssertionError: after rename(srcFilePath, destFilePath): directories_created expected:<1> but was:<0> at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:743) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:555) at org.apache.hadoop.fs.s3a.S3ATestUtils$MetricDiff.assertDiffEquals(S3ATestUtils.java:431) at org.apache.hadoop.fs.s3a.ITestS3AFileOperationCost.testFakeDirectoryDeletion(ITestS3AFileOperationCost.java:254) Running org.apache.hadoop.fs.s3a.ITestS3AMiscOperations Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.188 sec - in org.apache.hadoop.fs.s3a.ITestS3AMiscOperations Running org.apache.hadoop.fs.s3a.ITestS3ATemporaryCredentials Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.366 sec - in org.apache.hadoop.fs.s3a.ITestS3ATemporaryCredentials Running org.apache.hadoop.fs.s3a.ITestS3ATestUtils Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.123 sec - in org.apache.hadoop.fs.s3a.ITestS3ATestUtils Running org.apache.hadoop.fs.s3a.scale.ITestS3ADeleteFilesOneByOne Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 4.396 sec - in org.apache.hadoop.fs.s3a.scale.ITestS3ADeleteFilesOneByOne Running org.apache.hadoop.fs.s3a.scale.ITestS3ADeleteManyFiles Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 4.342 sec - in org.apache.hadoop.fs.s3a.scale.ITestS3ADeleteManyFiles Running org.apache.hadoop.fs.s3a.scale.ITestS3ADirectoryPerformance Tests run: 5, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 17.954 sec - in org.apache.hadoop.fs.s3a.scale.ITestS3ADirectoryPerformance Running org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesArrayBlocks Tests run: 5, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 23.965 sec - in org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesArrayBlocks Running org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesByteBufferBlocks Tests run: 17, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 353.478 sec - in org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextURI Running org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesClassicOutput Tests run: 5, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 15.458 sec - in org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesByteBufferBlocks Running org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesDiskBlocks Tests run: 5, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 15.381 sec - in org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesClassicOutput Running org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance Tests run: 5, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 16.141 sec - in org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesDiskBlocks Running org.apache.hadoop.fs.s3a.yarn.ITestS3A Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.081 sec - in org.apache.hadoop.fs.s3a.yarn.ITestS3A Running org.apache.hadoop.fs.s3a.yarn.ITestS3AMiniYarnCluster Tests run: 8, Failures: 0, Errors: 0, Skipped: 8, Time elapsed: 24.094 sec - in org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 41.678 sec - in org.apache.hadoop.fs.s3a.yarn.ITestS3AMiniYarnCluster Tests run: 43, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 279.272 sec <<< FAILURE! - in org.apache.hadoop.fs.s3a.ITestS3AFileSystemContract testRenameToDirWithSamePrefixAllowed(org.apache.hadoop.fs.s3a.ITestS3AFileSystemContract) Time elapsed: 4.693 sec <<< ERROR! org.apache.hadoop.fs.s3a.AWSServiceIOException: createTable: com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException: Provided list of item keys contains duplicates (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: ValidationException; Request ID: P128RFOF3H9IAHJTKRUR8PIHENVV4KQNSO5AEMVJF66Q9ASUAAJG): Provided list of item keys contains duplicates (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: ValidationException; Request ID: P128RFOF3H9IAHJTKRUR8PIHENVV4KQNSO5AEMVJF66Q9ASUAAJG) at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:166) at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.move(DynamoDBMetadataStore.java:347) at org.apache.hadoop.fs.s3a.S3AFileSystem.innerRename(S3AFileSystem.java:862) at org.apache.hadoop.fs.s3a.S3AFileSystem.rename(S3AFileSystem.java:655) at org.apache.hadoop.fs.FileSystemContractBaseTest.rename(FileSystemContractBaseTest.java:512) at org.apache.hadoop.fs.FileSystemContractBaseTest.testRenameToDirWithSamePrefixAllowed(FileSystemContractBaseTest.java:656) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at junit.framework.TestCase.runTest(TestCase.java:176) at junit.framework.TestCase.runBare(TestCase.java:141) at junit.framework.TestResult$1.protect(TestResult.java:122) at junit.framework.TestResult.runProtected(TestResult.java:142) at junit.framework.TestResult.run(TestResult.java:125) at junit.framework.TestCase.run(TestCase.java:129) at junit.framework.TestSuite.runTest(TestSuite.java:255) at junit.framework.TestSuite.run(TestSuite.java:250) at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:84) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124) at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103) Caused by: com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException: Provided list of item keys contains duplicates (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: ValidationException; Request ID: P128RFOF3H9IAHJTKRUR8PIHENVV4KQNSO5AEMVJF66Q9ASUAAJG) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1529) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1167) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:948) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:661) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:635) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:618) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$300(AmazonHttpClient.java:586) at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:573) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:445) at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.doInvoke(AmazonDynamoDBClient.java:1722) at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:1698) at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.batchWriteItem(AmazonDynamoDBClient.java:668) at com.amazonaws.services.dynamodbv2.document.internal.BatchWriteItemImpl.doBatchWriteItem(BatchWriteItemImpl.java:111) at com.amazonaws.services.dynamodbv2.document.internal.BatchWriteItemImpl.batchWriteItem(BatchWriteItemImpl.java:52) at com.amazonaws.services.dynamodbv2.document.DynamoDB.batchWriteItem(DynamoDB.java:178) at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.move(DynamoDBMetadataStore.java:338) at org.apache.hadoop.fs.s3a.S3AFileSystem.innerRename(S3AFileSystem.java:862) at org.apache.hadoop.fs.s3a.S3AFileSystem.rename(S3AFileSystem.java:655) at org.apache.hadoop.fs.FileSystemContractBaseTest.rename(FileSystemContractBaseTest.java:512) at org.apache.hadoop.fs.FileSystemContractBaseTest.testRenameToDirWithSamePrefixAllowed(FileSystemContractBaseTest.java:656) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at junit.framework.TestCase.runTest(TestCase.java:176) at junit.framework.TestCase.runBare(TestCase.java:141) at junit.framework.TestResult$1.protect(TestResult.java:122) at junit.framework.TestResult.runProtected(TestResult.java:142) at junit.framework.TestResult.run(TestResult.java:125) at junit.framework.TestCase.run(TestCase.java:129) at junit.framework.TestSuite.runTest(TestSuite.java:255) at junit.framework.TestSuite.run(TestSuite.java:250) at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:84) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124) at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 37s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 6 new or modified test files.
          0 mvndep 0m 16s Maven dependency ordering for branch
          +1 mvninstall 7m 25s HADOOP-13345 passed
          +1 compile 9m 47s HADOOP-13345 passed
          +1 checkstyle 1m 33s HADOOP-13345 passed
          +1 mvnsite 1m 50s HADOOP-13345 passed
          +1 mvneclipse 0m 55s HADOOP-13345 passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project
          +1 findbugs 1m 59s HADOOP-13345 passed
          +1 javadoc 1m 27s HADOOP-13345 passed
          0 mvndep 0m 39s Maven dependency ordering for patch
          +1 mvninstall 1m 9s the patch passed
          +1 compile 9m 40s the patch passed
          +1 javac 9m 40s the patch passed
          -0 checkstyle 1m 43s root: The patch generated 1 new + 8 unchanged - 0 fixed = 9 total (was 8)
          +1 mvnsite 2m 2s the patch passed
          +1 mvneclipse 1m 15s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 4s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project
          +1 findbugs 2m 33s the patch passed
          -1 javadoc 0m 26s hadoop-aws in the patch failed.
          +1 unit 0m 17s hadoop-project in the patch passed.
          +1 unit 8m 24s hadoop-common in the patch passed.
          +1 unit 0m 41s hadoop-aws in the patch passed.
          +1 asflicense 0m 38s The patch does not generate ASF License warnings.
          80m 0s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-13449
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12841900/HADOOP-13449-HADOOP-13345.011.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux ea3d4d4c2f23 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision HADOOP-13345 / 013a3c4
          Default Java 1.8.0_111
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11202/artifact/patchprocess/diff-checkstyle-root.txt
          javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/11202/artifact/patchprocess/patch-javadoc-hadoop-tools_hadoop-aws.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11202/testReport/
          modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11202/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 37s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 6 new or modified test files. 0 mvndep 0m 16s Maven dependency ordering for branch +1 mvninstall 7m 25s HADOOP-13345 passed +1 compile 9m 47s HADOOP-13345 passed +1 checkstyle 1m 33s HADOOP-13345 passed +1 mvnsite 1m 50s HADOOP-13345 passed +1 mvneclipse 0m 55s HADOOP-13345 passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project +1 findbugs 1m 59s HADOOP-13345 passed +1 javadoc 1m 27s HADOOP-13345 passed 0 mvndep 0m 39s Maven dependency ordering for patch +1 mvninstall 1m 9s the patch passed +1 compile 9m 40s the patch passed +1 javac 9m 40s the patch passed -0 checkstyle 1m 43s root: The patch generated 1 new + 8 unchanged - 0 fixed = 9 total (was 8) +1 mvnsite 2m 2s the patch passed +1 mvneclipse 1m 15s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 4s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project +1 findbugs 2m 33s the patch passed -1 javadoc 0m 26s hadoop-aws in the patch failed. +1 unit 0m 17s hadoop-project in the patch passed. +1 unit 8m 24s hadoop-common in the patch passed. +1 unit 0m 41s hadoop-aws in the patch passed. +1 asflicense 0m 38s The patch does not generate ASF License warnings. 80m 0s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-13449 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12841900/HADOOP-13449-HADOOP-13345.011.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux ea3d4d4c2f23 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision HADOOP-13345 / 013a3c4 Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11202/artifact/patchprocess/diff-checkstyle-root.txt javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/11202/artifact/patchprocess/patch-javadoc-hadoop-tools_hadoop-aws.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11202/testReport/ modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11202/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          fabbri Aaron Fabbri added a comment -

          This looks pretty good Mingliang Liu, thank you. A couple of comments:

          1. For put(), in your loop which walks the path to the root, should we should break as soon as we find a non-null ancestor? I see you have a comment suggesting that.

          This should be correct, by induction, since the invariant for the DDB MetadataStore (MS) is:

          • For any path P in MS, all ancestors p_i are in MS.

          Thus, when we encounter an ancestor p_i on the way to the root, we know that all its ancestors must already exist.

          2. The isEmpty bit on directory FileStatus's.

           @Override
            public PathMetadata get(Path path) throws IOException {
            ... (snip) ...
                  // for directory, we query its direct children to determine isEmpty bit
          

          I think this misses the case where we have not seen all the children yet. For example, we have an existing bucket with some directory /a/b/ and two existing files /a/b/file1, /a/b/file2. We create a new MetadataStore and put(/a/b/), then get(/a/b).

          That said, we should consider merging this patch and creating follow-up JIRAs since it is good to split up the work and keep the code integrated.

          Also, we should consider reworking the isEmptyDirectory logic around S3AFileStatus as we discussed before. Meanwhile, I'd be happy to implement a similar algorithm as to what I used for LocalMetadataStore if that would be helpful.

          Testing I did on this patch:
          mvn clean verify -Dit.test="ITestS3A*"
          Failed tests:
          ITestS3ACredentialsInURL.testInvalidCredentialsFail:130->Assert.fail:88 Expected an AccessDeniedException, got S3AFileStatus

          {path=s3a://fabbri-dev/; isDirectory=true; modification_time=0; access_time=0; owner=fabbri; group=fabbri; permission=rwxrwxrwx; isSymlink=false}

          isEmptyDirectory=false
          ITestS3AFileOperationCost.testFakeDirectoryDeletion:254->Assert.assertEquals:555->Assert.assertEquals:118->Assert.failNotEquals:743->Assert.fail:88 after rename(srcFilePath, destFilePath): directories_created expected:<1> but was:<0>

          Tests in error:
          ITestS3AAWSCredentialsProvider.testAnonymousProvider:133 » AWSServiceIO initia...
          ITestS3AAWSCredentialsProvider.testBadCredentials:102->createFailingFS:76 » AWSServiceIO
          ITestS3ACredentialsInURL.testInstantiateFromURL:86 » AWSClientIO initializing ...
          ITestS3AFileSystemContract>FileSystemContractBaseTest.testRenameToDirWithSamePrefixAllowed:656->FileSystemContractBaseTest.rename:512 » AWSServiceIO

          Tests run: 321, Failures: 2, Errors: 4, Skipped: 42

          To summarize:

          • isEmptyDirectory logic for DynamoDBMetadataStore (needs JIRA--unless I'm missing something)
          • Anonymous credentials issue (needs JIRA)
          • Issue with invalid creds in URI. Creds in URI in general may not be honored by DDB MetadataStore? (needs JIRA & investigation)
          • Stats not showing directory creation from rename(): ITestS3AFileOperationCost.testFakeDirectoryDeletion (needs investigation)
          Show
          fabbri Aaron Fabbri added a comment - This looks pretty good Mingliang Liu , thank you. A couple of comments: 1. For put(), in your loop which walks the path to the root, should we should break as soon as we find a non-null ancestor? I see you have a comment suggesting that. This should be correct, by induction, since the invariant for the DDB MetadataStore (MS) is: For any path P in MS, all ancestors p_i are in MS. Thus, when we encounter an ancestor p_i on the way to the root, we know that all its ancestors must already exist. 2. The isEmpty bit on directory FileStatus's. @Override public PathMetadata get(Path path) throws IOException { ... (snip) ... // for directory, we query its direct children to determine isEmpty bit I think this misses the case where we have not seen all the children yet. For example, we have an existing bucket with some directory /a/b/ and two existing files /a/b/file1 , /a/b/file2 . We create a new MetadataStore and put(/a/b/), then get(/a/b). That said, we should consider merging this patch and creating follow-up JIRAs since it is good to split up the work and keep the code integrated. Also, we should consider reworking the isEmptyDirectory logic around S3AFileStatus as we discussed before. Meanwhile, I'd be happy to implement a similar algorithm as to what I used for LocalMetadataStore if that would be helpful. Testing I did on this patch: mvn clean verify -Dit.test="ITestS3A*" Failed tests: ITestS3ACredentialsInURL.testInvalidCredentialsFail:130->Assert.fail:88 Expected an AccessDeniedException, got S3AFileStatus {path=s3a://fabbri-dev/; isDirectory=true; modification_time=0; access_time=0; owner=fabbri; group=fabbri; permission=rwxrwxrwx; isSymlink=false} isEmptyDirectory=false ITestS3AFileOperationCost.testFakeDirectoryDeletion:254->Assert.assertEquals:555->Assert.assertEquals:118->Assert.failNotEquals:743->Assert.fail:88 after rename(srcFilePath, destFilePath): directories_created expected:<1> but was:<0> Tests in error: ITestS3AAWSCredentialsProvider.testAnonymousProvider:133 » AWSServiceIO initia... ITestS3AAWSCredentialsProvider.testBadCredentials:102->createFailingFS:76 » AWSServiceIO ITestS3ACredentialsInURL.testInstantiateFromURL:86 » AWSClientIO initializing ... ITestS3AFileSystemContract>FileSystemContractBaseTest.testRenameToDirWithSamePrefixAllowed:656->FileSystemContractBaseTest.rename:512 » AWSServiceIO Tests run: 321, Failures: 2, Errors: 4, Skipped: 42 To summarize: isEmptyDirectory logic for DynamoDBMetadataStore (needs JIRA--unless I'm missing something) Anonymous credentials issue (needs JIRA) Issue with invalid creds in URI. Creds in URI in general may not be honored by DDB MetadataStore? (needs JIRA & investigation) Stats not showing directory creation from rename(): ITestS3AFileOperationCost.testFakeDirectoryDeletion (needs investigation)
          Hide
          liuml07 Mingliang Liu added a comment -

          Thanks for your insightful comments, Aaron Fabbri. You summary looks very good to me. Having follow-up JIRAs looks a good step as most of them can be addressed separately === parallelly.

          1. For point 1, I'll change this in the new patch. I think by induction it is good enough to include this optimization which should be minor change.
          2. For the example of pre-existing items, I was thinking the import phase may have already loaded the S3 directory tree to the DDB table, e.g. by command line tool. For better implementing isEmpty, I think working with S3AFileStatus is a good idea. Steve also suggested we should query DDB lazily only when we need the isEmpty information. Obviously we can't enable this within DDBMetatdataStore itself.
          3. The integration tests you shared are very helpful. I'll have a look at the ITestS3AFileOperationCost#testFakeDirectoryDeletion first. Will get back to this the day after tomorrow.
          4. For the invalid creds in URI, the DDBClientFactory itself uses the same createAWSCredentialProviderSet as S3ClientFactory does so it should honor the creds in URI name. But after FS#initialization, S3AFS has stripped the creds and returns the scheme://host only URI to create a MetadataStore. We can refuse to support this case as it's very unsafe and deprecated. One approach though is to pass the name URI which contains the creds to S3Guard#getMetadataStore().
            S3AFileSystem#initialize()
            -      metadataStore = S3Guard.getMetadataStore(this);
            +      metadataStore = S3Guard.getMetadataStore(this, name);
            

          By the way, do you have suggestions about the failing testPutNew unit test? My current idea is to override this test method in DDB and local so they can have different behaviors; alternatively we can enforce the put API of metadata store so both DDB and local implement that for newly put item, all its ancestors are existent.

          Show
          liuml07 Mingliang Liu added a comment - Thanks for your insightful comments, Aaron Fabbri . You summary looks very good to me. Having follow-up JIRAs looks a good step as most of them can be addressed separately === parallelly. For point 1, I'll change this in the new patch. I think by induction it is good enough to include this optimization which should be minor change. For the example of pre-existing items, I was thinking the import phase may have already loaded the S3 directory tree to the DDB table, e.g. by command line tool. For better implementing isEmpty , I think working with S3AFileStatus is a good idea. Steve also suggested we should query DDB lazily only when we need the isEmpty information. Obviously we can't enable this within DDBMetatdataStore itself. The integration tests you shared are very helpful. I'll have a look at the ITestS3AFileOperationCost#testFakeDirectoryDeletion first. Will get back to this the day after tomorrow. For the invalid creds in URI, the DDBClientFactory itself uses the same createAWSCredentialProviderSet as S3ClientFactory does so it should honor the creds in URI name. But after FS#initialization, S3AFS has stripped the creds and returns the scheme://host only URI to create a MetadataStore. We can refuse to support this case as it's very unsafe and deprecated. One approach though is to pass the name URI which contains the creds to S3Guard#getMetadataStore() . S3AFileSystem#initialize() - metadataStore = S3Guard.getMetadataStore( this ); + metadataStore = S3Guard.getMetadataStore( this , name); By the way, do you have suggestions about the failing testPutNew unit test? My current idea is to override this test method in DDB and local so they can have different behaviors; alternatively we can enforce the put API of metadata store so both DDB and local implement that for newly put item, all its ancestors are existent.
          Hide
          fabbri Aaron Fabbri added a comment -

          What line does testPutNew() fail on for you? TestDynamoDBMetadataStore is passing for me.

          Show
          fabbri Aaron Fabbri added a comment - What line does testPutNew() fail on for you? TestDynamoDBMetadataStore is passing for me.
          Hide
          fabbri Aaron Fabbri added a comment - - edited

          If it is MetadataStoreTestBase line 154 "assertEmptyDirs() shown here:

              ms.put(new PathMetadata(makeFileStatus("/da1/db1/fc1", 100)));
          
              assertEmptyDirs("/da1", "/da2", "/da3");
              assertDirectorySize("/da1/db1", 1);
          

          I think we can change that to be

              assertEmptyDirs("/da2", "/da3")
          

          Why? Because it is not unreasonable to allow a MetadataStore to infer the existence of /da1/db1 from a call to put(/da1/db1/fc1).

          In fact, as we see here, it can be a helpful implementation technique when we don't have a cheap way to prefix scan everything in the MetadataStore.

          Show
          fabbri Aaron Fabbri added a comment - - edited If it is MetadataStoreTestBase line 154 "assertEmptyDirs() shown here: ms.put( new PathMetadata(makeFileStatus( "/da1/db1/fc1" , 100))); assertEmptyDirs( "/da1" , "/da2" , "/da3" ); assertDirectorySize( "/da1/db1" , 1); I think we can change that to be assertEmptyDirs( "/da2" , "/da3" ) Why? Because it is not unreasonable to allow a MetadataStore to infer the existence of /da1/db1 from a call to put(/da1/db1/fc1). In fact, as we see here, it can be a helpful implementation technique when we don't have a cheap way to prefix scan everything in the MetadataStore.
          Hide
          liuml07 Mingliang Liu added a comment -

          Sorry I marked it as @Ignored in the patch...

          TestDynamoDBMetadataStore.java
          184	  // TODO; update the test base class instead of ignoring this one
          185	  @Ignore
          186	  @Override
          187	  public void testPutNew() throws Exception {
          188	  }
          
          Show
          liuml07 Mingliang Liu added a comment - Sorry I marked it as @Ignored in the patch... TestDynamoDBMetadataStore.java 184 // TODO; update the test base class instead of ignoring this one 185 @Ignore 186 @Override 187 public void testPutNew() throws Exception { 188 }
          Hide
          liuml07 Mingliang Liu added a comment -

          Thanks Aaron Fabbri, as we discussed, I simply changed the base test case.

          Show
          liuml07 Mingliang Liu added a comment - Thanks Aaron Fabbri , as we discussed, I simply changed the base test case.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 15s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 6 new or modified test files.
          0 mvndep 1m 47s Maven dependency ordering for branch
          +1 mvninstall 6m 54s HADOOP-13345 passed
          +1 compile 10m 47s HADOOP-13345 passed
          +1 checkstyle 1m 42s HADOOP-13345 passed
          +1 mvnsite 1m 49s HADOOP-13345 passed
          +1 mvneclipse 0m 56s HADOOP-13345 passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project
          +1 findbugs 2m 11s HADOOP-13345 passed
          +1 javadoc 1m 33s HADOOP-13345 passed
          0 mvndep 0m 21s Maven dependency ordering for patch
          +1 mvninstall 1m 20s the patch passed
          +1 compile 9m 54s the patch passed
          +1 javac 9m 54s the patch passed
          -0 checkstyle 1m 41s root: The patch generated 1 new + 8 unchanged - 0 fixed = 9 total (was 8)
          +1 mvnsite 2m 10s the patch passed
          +1 mvneclipse 1m 4s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 4s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project
          +1 findbugs 2m 21s the patch passed
          +1 javadoc 1m 36s the patch passed
          +1 unit 0m 20s hadoop-project in the patch passed.
          +1 unit 8m 45s hadoop-common in the patch passed.
          +1 unit 0m 42s hadoop-aws in the patch passed.
          +1 asflicense 0m 37s The patch does not generate ASF License warnings.
          82m 25s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-13449
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12842071/HADOOP-13449-HADOOP-13345.012.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux 5ea041425062 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision HADOOP-13345 / 013a3c4
          Default Java 1.8.0_111
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11208/artifact/patchprocess/diff-checkstyle-root.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11208/testReport/
          modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11208/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 6 new or modified test files. 0 mvndep 1m 47s Maven dependency ordering for branch +1 mvninstall 6m 54s HADOOP-13345 passed +1 compile 10m 47s HADOOP-13345 passed +1 checkstyle 1m 42s HADOOP-13345 passed +1 mvnsite 1m 49s HADOOP-13345 passed +1 mvneclipse 0m 56s HADOOP-13345 passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project +1 findbugs 2m 11s HADOOP-13345 passed +1 javadoc 1m 33s HADOOP-13345 passed 0 mvndep 0m 21s Maven dependency ordering for patch +1 mvninstall 1m 20s the patch passed +1 compile 9m 54s the patch passed +1 javac 9m 54s the patch passed -0 checkstyle 1m 41s root: The patch generated 1 new + 8 unchanged - 0 fixed = 9 total (was 8) +1 mvnsite 2m 10s the patch passed +1 mvneclipse 1m 4s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 4s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project +1 findbugs 2m 21s the patch passed +1 javadoc 1m 36s the patch passed +1 unit 0m 20s hadoop-project in the patch passed. +1 unit 8m 45s hadoop-common in the patch passed. +1 unit 0m 42s hadoop-aws in the patch passed. +1 asflicense 0m 37s The patch does not generate ASF License warnings. 82m 25s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-13449 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12842071/HADOOP-13449-HADOOP-13345.012.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 5ea041425062 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision HADOOP-13345 / 013a3c4 Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11208/artifact/patchprocess/diff-checkstyle-root.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11208/testReport/ modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11208/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          fabbri Aaron Fabbri added a comment -

          I'm in the process of testing the v12 patch. Mingliang Liu and Steve Loughran I'd like to propose we get this patch in so we can split up the remaining work on DynamoDB. I'm thinking the steps are:

          1. I will finish testing and do a quick review on the v12 patch here.
          2. I will open JIRAs for outstanding issues related to DynamoDB.
          3. If you guys are +1 on this, I will commit the v12 (or latest) patch.

          One possible concern is if we wanted to try and merge the HADOOP-13345 branch to trunk before DynamoDB support is finished (we'd talked about that to deal with code churn and allow things like working on parallel rename without needing to redo all the s3guard rename code, etc). If we still wanted to to attempt this, let me know.

          Show
          fabbri Aaron Fabbri added a comment - I'm in the process of testing the v12 patch. Mingliang Liu and Steve Loughran I'd like to propose we get this patch in so we can split up the remaining work on DynamoDB. I'm thinking the steps are: 1. I will finish testing and do a quick review on the v12 patch here. 2. I will open JIRAs for outstanding issues related to DynamoDB. 3. If you guys are +1 on this, I will commit the v12 (or latest) patch. One possible concern is if we wanted to try and merge the HADOOP-13345 branch to trunk before DynamoDB support is finished (we'd talked about that to deal with code churn and allow things like working on parallel rename without needing to redo all the s3guard rename code, etc). If we still wanted to to attempt this, let me know.
          Hide
          liuml07 Mingliang Liu added a comment -

          Thanks Aaron Fabbri.

          I saw you created two follow up JIRA, which are very good. We can address them separately. I also have the v13 patch, which can be addressed separately as well as the changes are kinda isolated to v12 patch. Specially, the v13 patch addresses the trivial checkstyle warning, and changes the region of DynamoDB table to create from region of S3Client to region of Bucket. I think it makes more sense.

          For the failing integration test ITestS3AFileOperationCost#testFakeDirectoryDeletion, I found it fails with LocalMetadataStore as well, can you verify that? The reason it fails is because the number of create operations counter does not match after rename.

          ITestS3AFileOperationCost#testFakeDirectoryDeletion()
              fs.rename(srcFilePath, destFilePath);
              state = "after rename(srcFilePath, destFilePath)";
              directoriesCreated.assertDiffEquals(state, 1);    <=== fail here
          

          When the S3AFS renames a file (in this case), it will create fake parent directories after deleting the old file when necessary. This is guarded by createFakeDirectoryIfNecessary, which checks if the parent directory exits.

          createFakeDirectoryIfNecessary() called by rename()
            private void createFakeDirectoryIfNecessary(Path f)
                throws IOException, AmazonClientException {
              String key = pathToKey(f);
              if (!key.isEmpty() && !exists(f)) {   <=== only if nonexistent
                LOG.debug("Creating new fake directory at {}", f);
                createFakeDirectory(key);
              }
            }
          

          However, getFileStatus() for exists() will check the metadata store only for this. The metadatastore surely will return true and no fake directory will be created in S3 then.

            public S3AFileStatus getFileStatus(final Path f) throws IOException {
              incrementStatistic(INVOCATION_GET_FILE_STATUS);
              final Path path = qualify(f);
              String key = pathToKey(path);
              LOG.debug("Getting path status for {}  ({})", path , key);
          
              // Check MetadataStore, if any.
              PathMetadata pm = metadataStore.get(path);
              if (pm != null) {
                // HADOOP-13760: handle deleted files, i.e. PathMetadata#isDeleted() here
                return (S3AFileStatus)pm.getFileStatus();
              }
              ...
          

          This seems a bug elesewhere and can also be addressed/investigated separately.

          Show
          liuml07 Mingliang Liu added a comment - Thanks Aaron Fabbri . I saw you created two follow up JIRA, which are very good. We can address them separately. I also have the v13 patch, which can be addressed separately as well as the changes are kinda isolated to v12 patch. Specially, the v13 patch addresses the trivial checkstyle warning, and changes the region of DynamoDB table to create from region of S3Client to region of Bucket. I think it makes more sense. For the failing integration test ITestS3AFileOperationCost#testFakeDirectoryDeletion , I found it fails with LocalMetadataStore as well, can you verify that? The reason it fails is because the number of create operations counter does not match after rename. ITestS3AFileOperationCost#testFakeDirectoryDeletion() fs.rename(srcFilePath, destFilePath); state = "after rename(srcFilePath, destFilePath)" ; directoriesCreated.assertDiffEquals(state, 1); <=== fail here When the S3AFS renames a file (in this case), it will create fake parent directories after deleting the old file when necessary. This is guarded by createFakeDirectoryIfNecessary , which checks if the parent directory exits. createFakeDirectoryIfNecessary() called by rename() private void createFakeDirectoryIfNecessary(Path f) throws IOException, AmazonClientException { String key = pathToKey(f); if (!key.isEmpty() && !exists(f)) { <=== only if nonexistent LOG.debug( "Creating new fake directory at {}" , f); createFakeDirectory(key); } } However, getFileStatus() for exists() will check the metadata store only for this. The metadatastore surely will return true and no fake directory will be created in S3 then. public S3AFileStatus getFileStatus( final Path f) throws IOException { incrementStatistic(INVOCATION_GET_FILE_STATUS); final Path path = qualify(f); String key = pathToKey(path); LOG.debug( "Getting path status for {} ({})" , path , key); // Check MetadataStore, if any. PathMetadata pm = metadataStore.get(path); if (pm != null ) { // HADOOP-13760: handle deleted files, i.e. PathMetadata#isDeleted() here return (S3AFileStatus)pm.getFileStatus(); } ... This seems a bug elesewhere and can also be addressed/investigated separately.
          Hide
          fabbri Aaron Fabbri added a comment -

          Sounds good. Thanks for analysis on that failure. I will do some review + testing with the v13 patch tonight and make sure we have updated JIRAs for any issues, including the createFakeDirectoryIfNecessary() thing. I can commit the v13 patch in the morning if everyone is in favor and I don't find any new issues.

          Show
          fabbri Aaron Fabbri added a comment - Sounds good. Thanks for analysis on that failure. I will do some review + testing with the v13 patch tonight and make sure we have updated JIRAs for any issues, including the createFakeDirectoryIfNecessary() thing. I can commit the v13 patch in the morning if everyone is in favor and I don't find any new issues.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 15s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.
          0 mvndep 2m 25s Maven dependency ordering for branch
          +1 mvninstall 6m 55s HADOOP-13345 passed
          +1 compile 9m 37s HADOOP-13345 passed
          +1 checkstyle 1m 32s HADOOP-13345 passed
          +1 mvnsite 1m 47s HADOOP-13345 passed
          +1 mvneclipse 0m 55s HADOOP-13345 passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project
          +1 findbugs 1m 57s HADOOP-13345 passed
          +1 javadoc 1m 28s HADOOP-13345 passed
          0 mvndep 0m 19s Maven dependency ordering for patch
          +1 mvninstall 1m 10s the patch passed
          +1 compile 9m 12s the patch passed
          +1 javac 9m 12s the patch passed
          -0 checkstyle 1m 36s root: The patch generated 1 new + 8 unchanged - 0 fixed = 9 total (was 8)
          +1 mvnsite 1m 57s the patch passed
          +1 mvneclipse 1m 4s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 4s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project
          +1 findbugs 2m 20s the patch passed
          +1 javadoc 1m 37s the patch passed
          +1 unit 0m 20s hadoop-project in the patch passed.
          +1 unit 8m 29s hadoop-common in the patch passed.
          +1 unit 0m 42s hadoop-aws in the patch passed.
          +1 asflicense 0m 37s The patch does not generate ASF License warnings.
          80m 4s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-13449
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12842477/HADOOP-13449-HADOOP-13345.013.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux d255e124893c 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision HADOOP-13345 / 881de1f
          Default Java 1.8.0_111
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11226/artifact/patchprocess/diff-checkstyle-root.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11226/testReport/
          modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11226/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 5 new or modified test files. 0 mvndep 2m 25s Maven dependency ordering for branch +1 mvninstall 6m 55s HADOOP-13345 passed +1 compile 9m 37s HADOOP-13345 passed +1 checkstyle 1m 32s HADOOP-13345 passed +1 mvnsite 1m 47s HADOOP-13345 passed +1 mvneclipse 0m 55s HADOOP-13345 passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project +1 findbugs 1m 57s HADOOP-13345 passed +1 javadoc 1m 28s HADOOP-13345 passed 0 mvndep 0m 19s Maven dependency ordering for patch +1 mvninstall 1m 10s the patch passed +1 compile 9m 12s the patch passed +1 javac 9m 12s the patch passed -0 checkstyle 1m 36s root: The patch generated 1 new + 8 unchanged - 0 fixed = 9 total (was 8) +1 mvnsite 1m 57s the patch passed +1 mvneclipse 1m 4s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 4s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project +1 findbugs 2m 20s the patch passed +1 javadoc 1m 37s the patch passed +1 unit 0m 20s hadoop-project in the patch passed. +1 unit 8m 29s hadoop-common in the patch passed. +1 unit 0m 42s hadoop-aws in the patch passed. +1 asflicense 0m 37s The patch does not generate ASF License warnings. 80m 4s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-13449 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12842477/HADOOP-13449-HADOOP-13345.013.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux d255e124893c 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision HADOOP-13345 / 881de1f Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11226/artifact/patchprocess/diff-checkstyle-root.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11226/testReport/ modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11226/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          fabbri Aaron Fabbri added a comment -

          I committed this to the HADOOP-13345 feature branch. Thank you for all your hard work on this Mingliang Liu.

          I ran all integration tests against s3-us-west-2.amazonaws.com endpoint.

          Failures were as expected:
          ITestS3AFileOperationCost.testFakeDirectoryDeletion:254->Assert.assertEquals:555
          ITestJets3tNativeS3FileSystemContract>NativeS3FileSystemContractBaseTest.testListStatusForRoot:66 Root directory is not empty; expected:<0> but was:<3>

          ITestS3AAWSCredentialsProvider.testAnonymousProvider:133 » AWSServiceIO initia...
          ITestS3ACredentialsInURL.testInstantiateFromURL:86 » InterruptedIO initializin...
          ITestS3AFileSystemContract>FileSystemContractBaseTest.testRenameToDirWithSamePrefixAllowed:669->FileSystemContractBaseTest.rename:525 » AWSServiceIO

          Which are covered by HADOOP-13876 and HADOOP-13886.

          Show
          fabbri Aaron Fabbri added a comment - I committed this to the HADOOP-13345 feature branch. Thank you for all your hard work on this Mingliang Liu . I ran all integration tests against s3-us-west-2.amazonaws.com endpoint. Failures were as expected: ITestS3AFileOperationCost.testFakeDirectoryDeletion:254->Assert.assertEquals:555 ITestJets3tNativeS3FileSystemContract>NativeS3FileSystemContractBaseTest.testListStatusForRoot:66 Root directory is not empty; expected:<0> but was:<3> ITestS3AAWSCredentialsProvider.testAnonymousProvider:133 » AWSServiceIO initia... ITestS3ACredentialsInURL.testInstantiateFromURL:86 » InterruptedIO initializin... ITestS3AFileSystemContract>FileSystemContractBaseTest.testRenameToDirWithSamePrefixAllowed:669->FileSystemContractBaseTest.rename:525 » AWSServiceIO Which are covered by HADOOP-13876 and HADOOP-13886 .
          Hide
          liuml07 Mingliang Liu added a comment - - edited

          Thank you Aaron Fabbri for your great help, discussion, review, testing, bug fixing! Thanks Steve Loughran and Chris Nauroth for the helpful discussion and initial patch. Thank you Lei (Eddy) Xu for insightful comments to make the patch in good shape.

          Let's move on to other tasks and make this feature branch be merged to trunk early.

          Show
          liuml07 Mingliang Liu added a comment - - edited Thank you Aaron Fabbri for your great help, discussion, review, testing, bug fixing! Thanks Steve Loughran and Chris Nauroth for the helpful discussion and initial patch. Thank you Lei (Eddy) Xu for insightful comments to make the patch in good shape. Let's move on to other tasks and make this feature branch be merged to trunk early.
          Hide
          fabbri Aaron Fabbri added a comment -

          You are welcome! Thanks also to Lei (Eddy) Xu for his help getting to this point.

          Show
          fabbri Aaron Fabbri added a comment - You are welcome! Thanks also to Lei (Eddy) Xu for his help getting to this point.
          Hide
          liuml07 Mingliang Liu added a comment -

          Yes just edited the comment above. Lei helped us a lot in reviewing patches. I think we can get HADOOP-13650 in soon after review.

          Show
          liuml07 Mingliang Liu added a comment - Yes just edited the comment above. Lei helped us a lot in reviewing patches. I think we can get HADOOP-13650 in soon after review.
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          Thanks a lot for the great work here, Mingliang Liu and Aaron Fabbri. It is great that I can integrate this patch to HADOOP-13650 now. I will start to work on it today and keep you guys updated.

          Show
          eddyxu Lei (Eddy) Xu added a comment - Thanks a lot for the great work here, Mingliang Liu and Aaron Fabbri . It is great that I can integrate this patch to HADOOP-13650 now. I will start to work on it today and keep you guys updated.
          Hide
          stevel@apache.org Steve Loughran added a comment - - edited

          I've commented on HADOOP-13886 ; I'm not sure that this is a bug in the dynamo work, more a surfacing of the fact that those internal metrics, while they helped test what was going on, turn out to be brittle to change. That is what Allen Wittenauer warned me about, so he can be happy that the first people to experience it is ourselves.

          we can cut the test.

          see: https://steveloughran.blogspot.co.uk/2016/04/distributed-testing-making-use-of.html

          Show
          stevel@apache.org Steve Loughran added a comment - - edited I've commented on HADOOP-13886 ; I'm not sure that this is a bug in the dynamo work, more a surfacing of the fact that those internal metrics, while they helped test what was going on, turn out to be brittle to change. That is what Allen Wittenauer warned me about, so he can be happy that the first people to experience it is ourselves. we can cut the test. see: https://steveloughran.blogspot.co.uk/2016/04/distributed-testing-making-use-of.html
          Hide
          fabbri Aaron Fabbri added a comment -

          Thanks Steve Loughran. Those tests that make assertions about internal operation counts have been useful. I have disabled many of them with a check against S3AFileSystem#isMetadataStoreConfigured() or S3ATestUtils#isMetadataStoreAuthoritative(). They are a bit brittle to change but do end up catching issues, so I think ultimately useful.

          Show
          fabbri Aaron Fabbri added a comment - Thanks Steve Loughran . Those tests that make assertions about internal operation counts have been useful. I have disabled many of them with a check against S3AFileSystem#isMetadataStoreConfigured() or S3ATestUtils#isMetadataStoreAuthoritative() . They are a bit brittle to change but do end up catching issues, so I think ultimately useful.

            People

            • Assignee:
              liuml07 Mingliang Liu
              Reporter:
              cnauroth Chris Nauroth
            • Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development