• Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 3.2.0
    • fs/azure
    • None


      Configuration for WASB and ABFS is too complex.  The current approach is to use four files for test configuration. 

      Both WASB and ABFS have basic test configuration which is committed to the repo (azure-test.xml and azure-bfs-test.xml).  Currently these contain the fs.AbstractFileSystem.[scheme].impl configuration, but otherwise are empty except for an include reference to a file containing the endpoint credentials. 

      Both WASB and ABFS have endpoint credential configuration files (azure-auth-keys.xml and azure-bfs-auth-keys.xml).  These have been added to .gitignore to prevent them from accidentally being submitted in a patch, which would leak the developers storage account credentials.  These files contain account names, storage account keys, and service endpoints.

      There is some overlap of the configuration for WASB and ABFS, where they use the same property name but use different values.  

      1) Let's reduce the number of test configuration files to one, if possible.

      2) Let's simplify the account name, key, and endpoint configuration for WASB and ABFS if possible, but still support the legacy way of doing it, which is very error prone.

      3) Let's improve error handling, so that typos or misconfiguration are not so difficult to troubleshoot.


        1. HADOOP-15663-HADOOP-15407-001.patch
          60 kB
          Da Zhou
        2. HADOOP-15663-HADOOP-15407-002.patch
          60 kB
          Da Zhou
        3. HADOOP-15663-HADOOP-15407-003.patch
          50 kB
          Da Zhou
        4. HADOOP-15663-HADOOP-15407-004.patch
          60 kB
          Da Zhou
        5. HADOOP-15663-HADOOP-15407-005.patch
          60 kB
          Da Zhou



            DanielZhou Da Zhou
            tmarquardt Thomas Marqardt
            0 Vote for this issue
            5 Start watching this issue