Uploaded image for project: 'Oozie'
  1. Oozie
  2. OOZIE-426

OOZIE-9: Support (Amazon) s3/s3n schemes

    Details

    • Type: New Feature
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Currently Oozie supports only the 'hdfs' scheme. For people having their hadoop cluster in Amazon EC2 and using S3 for storage, it would be very useful if oozie supports the 's3n/s3' scheme(s). The use case I am talking about is as follows

      Hadoop cluster in Amazon EC2
      Uses hdfs for intermediate storage
      Uses s3 for getting input for, storing output of map-reduce jobs.

      More details on the above use-case and the exceptions/failures I have seen is documented here (http://tech.groups.yahoo.com/group/Oozie-users/message/1138).

      There can be other use cases as well - say use s3 as the DFS instead of HDFS.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                rvs Roman Shaposhnik
                Reporter:
                hadoopqa Hadoop QA
              • Votes:
                4 Vote for this issue
                Watchers:
                12 Start watching this issue

                Dates

                • Created:
                  Updated: