Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-23453

ToolBox compiled Spark UDAF causes java.lang.InternalError: Malformed class name

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Incomplete
    • 2.2.1
    • None
    • Spark Core

    • Spark 2.2.1
      Scala 2.11.11
      JDK 1.8

    Description

      Here is a weird problem I just ran into... My scenario is that I need to compile UDAF dynamically at runtime but it never worked.

      I am using Scala 2.11.11 and Spark 2.2.1, please refer to my StackOverflow post for detailed information and minimal examples. The problem itself is very similar to other Malformed class name tickets (for example  https://github.com/apache/spark/pull/9568)  which were caused by calling getSimpleName of nested class/object but actually it is different in this case and the problem is still there. The getSimpleName issue has been fixed in Java 9 which Spark doesn't support it yet...so...any solution/workaround is appreciated.

      Attachments

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            Eric Lo Eric Lo
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment