Details

    • Type: Sub-task Sub-task
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: native
    • Labels:
      None

      Activity

      Hide
      Steve Loughran added a comment -

      Regarding the org.apache.hadoop.fs.azurenative classes

      1. keys like "fs.azure.buffer.dir" need to be pulled out and made constants; the embedding of strings is something the main codebase is slowly moving away from. Some of the code does this, but not all.
      2. The code depends on microsoft-windowsazure-api 1.2.0 , which is in the maven repository. There's also a 0.2.0 version in there -any particular reason for not using the latest release?
      3. Testing? How is anyone working with this code going to use the FS? Is there S3-style remote access, or do you have to bring up a VM in the cluster?
      4. The catch of Exception and wrapping with AzureException is best set up so that IOException exceptions aren't caught and wrapped, as they match the signature. I don't know if the native API throws these, but adding an extra layer of nesting never helps with troubleshooting live systems.

      It may be cleaner to keep the azure FS source tree outside the main hadoop code, and host it in a parallel hadoop-azurefs project with the extra dependency, and the extra output artifacts. Anyone who added a mvn or ivy dependency on hadoop-azurefs would get the -api JAR, and testing could be isolated. This could also be a good opportunity to do the same for KFS, which is under-tested in the current release process, and for any other DFS clients that people want in the codebase. Maybe the policy should be: if it is testable by anyone, put it in the hadoop source tree, but if not, the FS vendor has to do it. (I'm thinking of things like GPFS here and others, not just AzureFS)

      Show
      Steve Loughran added a comment - Regarding the org.apache.hadoop.fs.azurenative classes keys like "fs.azure.buffer.dir" need to be pulled out and made constants; the embedding of strings is something the main codebase is slowly moving away from. Some of the code does this, but not all. The code depends on microsoft-windowsazure-api 1.2.0 , which is in the maven repository. There's also a 0.2.0 version in there -any particular reason for not using the latest release? Testing? How is anyone working with this code going to use the FS? Is there S3-style remote access, or do you have to bring up a VM in the cluster? The catch of Exception and wrapping with AzureException is best set up so that IOException exceptions aren't caught and wrapped, as they match the signature. I don't know if the native API throws these, but adding an extra layer of nesting never helps with troubleshooting live systems. It may be cleaner to keep the azure FS source tree outside the main hadoop code, and host it in a parallel hadoop-azurefs project with the extra dependency, and the extra output artifacts. Anyone who added a mvn or ivy dependency on hadoop-azurefs would get the -api JAR, and testing could be isolated. This could also be a good opportunity to do the same for KFS, which is under-tested in the current release process, and for any other DFS clients that people want in the codebase. Maybe the policy should be: if it is testable by anyone, put it in the hadoop source tree, but if not, the FS vendor has to do it. (I'm thinking of things like GPFS here and others, not just AzureFS)
      Hide
      Lance Norskog added a comment -

      Mahout has a separate sub-project called 'integrations' for these kinds of external connectors. It has all of the external dependencies.

      Show
      Lance Norskog added a comment - Mahout has a separate sub-project called 'integrations' for these kinds of external connectors. It has all of the external dependencies.
      Hide
      Matt Foley added a comment -

      Changed Target Version to 1.3.0 upon release of 1.2.0. Please change to 1.2.1 if you intend to submit a fix for branch-1.2.

      Show
      Matt Foley added a comment - Changed Target Version to 1.3.0 upon release of 1.2.0. Please change to 1.2.1 if you intend to submit a fix for branch-1.2.

        People

        • Assignee:
          Unassigned
          Reporter:
          Sanjay Radia
        • Votes:
          0 Vote for this issue
          Watchers:
          8 Start watching this issue

          Dates

          • Created:
            Updated:

            Development