Details

    • Type: Sub-task
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: fs/azure
    • Labels:
      None

      Description

      When interacting with Azure Blob Storage via the Hadoop FileSystem client, it's currently (as of a8bbd81) not possible to set custom metadata attributes.

      Here is a snippet that demonstrates the missing behavior (throws an UnsupportedOperationException):

      val blobAccount = "SET ME"
      val blobKey = "SET ME"
      val blobContainer = "SET ME"
      val blobFile = "SET ME"
      
      import org.apache.hadoop.conf.Configuration
      import org.apache.hadoop.fs.{FileSystem, Path}
      
      val conf = new Configuration()
      conf.set("fs.wasbs.impl", "org.apache.hadoop.fs.azure.NativeAzureFileSystem")
      conf.set(s"fs.azure.account.key.$blobAccount.blob.core.windows.net", blobKey)
      
      val path = new Path(s"wasbs://$blobContainer@$blobAccount.blob.core.windows.net/$blobFile")
      
      val fs = FileSystem.get(path, conf)
      fs.setXAttr(path, "somekey", "somevalue".getBytes)
      

      Looking at the code in hadoop-tools/hadoop-azure, NativeAzureFileSystem inherits the default setXAttr from FileSystem which throws the UnsupportedOperationException.

      The underlying Azure Blob Storage service does support custom metadata (service docs) as does the azure-storage SDK that's being used by NativeAzureFileSystem (SDK docs).

      Is there another way that I should be setting custom metadata on Azure Blob Storage files? Is there a specific reason why setXAttr hasn't been implemented on NativeAzureFileSystem? If not, I can take a shot at implementing it.

        Attachments

          Activity

            People

            • Assignee:
              c-w Clemens Wolff
              Reporter:
              c-w Clemens Wolff
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated: