Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-15763 Über-JIRA: abfs phase II: Hadoop 3.3 features & fixes
  3. HADOOP-17086

ABFS: Fix the parsing errors in ABFS Driver with creation Time (being returned in ListPath)

VotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.3.0
    • Fix Version/s: 3.3.1
    • Component/s: fs/azure
    • Labels:
      None

      Description

      I am seeing errors while running ABFS Driver against stg75 build in canary. This is related to parsing errors as we receive creationTIme in the ListPath API. Here are the errors:

      RestVersion: 2020-02-10

       mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify -Dit.test=ITestAzureBlobFileSystemRenameUnicode

      [ERROR] testRenameFileUsingUnicode[0](org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemRenameUnicode)  Time elapsed: 852.083 s  <<< ERROR!

      Status code: -1 error code: null error message: InvalidAbfsRestOperationExceptionorg.codehaus.jackson.map.exc.UnrecognizedPropertyException: Unrecognized field "creationTime" (Class org.apache.hadoop.

      .azurebfs.contracts.services.ListResultEntrySchema), not marked as ignorable

       at Source: [sun.net.www.protocol.http.HttpURLConnection$HttpInputStream@49e30796 (through reference chain: org.apache.hadoop.fs.azurebfs.contracts.services.ListResultSchema["pat

      "]->org.apache.hadoop.fs.azurebfs.contracts.services.ListResultEntrySchema["creationTime"])

              at org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.executeHttpOperation(AbfsRestOperation.java:273)

              at org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.execute(AbfsRestOperation.java:188)

              at org.apache.hadoop.fs.azurebfs.services.AbfsClient.listPath(AbfsClient.java:237)

              at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.listStatus(AzureBlobFileSystemStore.java:773)

              at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.listStatus(AzureBlobFileSystemStore.java:735)

              at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.listStatus(AzureBlobFileSystem.java:373)

              at org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemRenameUnicode.testRenameFileUsingUnicode(ITestAzureBlobFileSystemRenameUnicode.java:92)

              at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

              at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

              at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

              at java.base/java.lang.reflect.Method.invoke(Method.java:566)

              at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)

              at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)

              at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)

              at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)

              at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)

              at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)

              at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)

              at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)

              at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)

              at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)

              at java.base/java.lang.Thread.run(Thread.java:834)

      Caused by: org.codehaus.jackson.map.exc.UnrecognizedPropertyException: Unrecognized field "creationTime" (Class org.apache.hadoop.fs.azurebfs.contracts.services.ListResultEntrySchema), not marked as i

      orable

       at Source: [sun.net.www.protocol.http.HttpURLConnection$HttpInputStream@49e30796 (through reference chain: org.apache.hadoop.fs.azurebfs.contracts.services.ListResultSchema["pat

      "]->org.apache.hadoop.fs.azurebfs.contracts.services.ListResultEntrySchema["creationTime"])

              at org.codehaus.jackson.map.exc.UnrecognizedPropertyException.from(UnrecognizedPropertyException.java:53)

              at org.codehaus.jackson.map.deser.StdDeserializationContext.unknownFieldException(StdDeserializationContext.java:267)

              at org.codehaus.jackson.map.deser.std.StdDeserializer.reportUnknownProperty(StdDeserializer.java:673)

              at org.codehaus.jackson.map.deser.std.StdDeserializer.handleUnknownProperty(StdDeserializer.java:659)

              at org.codehaus.jackson.map.deser.BeanDeserializer.handleUnknownProperty(BeanDeserializer.java:1365)

              at org.codehaus.jackson.map.deser.BeanDeserializer._handleUnknown(BeanDeserializer.java:725)

              at org.codehaus.jackson.map.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:703)

              at org.codehaus.jackson.map.deser.BeanDeserializer.deserialize(BeanDeserializer.java:580)

              at org.codehaus.jackson.map.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:217)

              at org.codehaus.jackson.map.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:194)

              at org.codehaus.jackson.map.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:30)

              at org.codehaus.jackson.map.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:299)

              at org.codehaus.jackson.map.deser.SettableBeanProperty$FieldProperty.deserializeAndSet(SettableBeanProperty.java:579)

              at org.codehaus.jackson.map.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:697)

              at org.codehaus.jackson.map.deser.BeanDeserializer.deserialize(BeanDeserializer.java:580)

              at org.codehaus.jackson.map.ObjectMapper._readMapAndClose(ObjectMapper.java:2732)

              at org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1909)

              at org.apache.hadoop.fs.azurebfs.services.AbfsHttpOperation.parseListFilesResponse(AbfsHttpOperation.java:501)

              at org.apache.hadoop.fs.azurebfs.services.AbfsHttpOperation.processResponse(AbfsHttpOperation.java:366)

              at org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.executeHttpOperation(AbfsRestOperation.java:255)

        Attachments

          Activity

            People

            • Assignee:
              bilahari.th Bilahari T H
              Reporter:
              ishaniahuja Ishani

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment