[SQOOP-3345] Make Sqoop work with S3: test and document current capabilities

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Done
• Affects Version/s: 1.4.7
• Fix Version/s: no-release
• Assignee: Boglarka Egyed
• Reporter: Boglarka Egyed
• Created: 19/Jul/18 08:23
• Updated: 25/Oct/18 15:50
• Resolved: 25/Oct/18 15:50

Description

The hadoop-aws module provides support for AWS integration which could be used by Sqoop to work with S3.

The scope of this task is to explore and document the capabilities of the current Sqoop implementation regarding the "from RDBMS to S3" use case by adding the hadoop-aws dependency.

This is an umbrella ticket to collect all the related tasks and also the first step to enable further improvements to support all the required use cases.

Sub-Tasks

• Upgrade Hadoop version to 2.8.0: Resolved
• Make verify() more generic in AvroTestUtils: Resolved
• Add hadoop-aws dependency and S3 credential generator logic for tests: Resolved
• Add hadoop-aws dependency in Gradle: Resolved
• Test incremental import with S3: Resolved
• Add fail-fast scenarios to S3 incremental import use cases without --temporary-rootdir option: Resolved
• Document simple and incremental import into S3: Resolved
• Test import into external Hive table backed by S3: Resolved
• Disable FileSystem static cache in S3 tests: Resolved
• Document import into external Hive table backed by S3: Resolved
• Document S3Guard usage with Sqoop: Resolved
• Test storing AWS credentials in Hadoop CredentialProvider during import: Resolved
• External Hive table tests should use unique external dir names: Resolved
• Document Hadoop CredentialProvider usage in case of import into S3: Resolved