Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.2.1
    • Fix Version/s: 2.4.0
    • Component/s: Build
    • Labels:
      None

      Description

      Spark tried to upgrade json4s several times, but failed and still use a few years old version 3.2.11. I hope the situation will improve soon.

      Migration from 3.2.11

      Changes that affect Spark are as follows.

      scalap dependency

      Since 3.3, json4s has its own scalap fork and doesn't depend on scala-lang's scalap.
      https://github.com/json4s/json4s/issues/108

      Whereas Spark has added a scala-lang's scalap dependency in POM by the following issue.
      https://github.com/apache/spark/pull/480

      We need to remove scala-lang's scalap dependency for json4s 3.5.3 (same solution as in json4s).

      The behavior of \ function has changed

      Please see https://github.com/json4s/json4s/pull/309

      a single element
      val json =
            """[ {
              |  "jobId" : 0,
              |  "status" : "SUCCEEDED"
              |} ]""".stripMargin
      val ast = JsonMethods.parse(json)
      
      • version 3.2.11
        ast \ "status"  // => JString(SUCCEEDED)
        ast \\ "status" // => JString(SUCCEEDED)
        
      • version 3.5.3
        ast \ "status"  // => JArray(List(JString(SUCCEEDED)))
        ast \\ "status" // => JString(SUCCEEDED)
        
      more than a single elements
      val json =
        """[ {
          |  "jobId" : 1,
          |  "status" : "FAILED"
          |}, {
          |  "jobId" : 0,
          |  "status" : "SUCCEEDED"
          |} ]""".stripMargin
      val ast = JsonMethods.parse(json)
      
      • version 3.2.11
        ast \ "status"  // => JArray(List(JString(FAILED), JString(SUCCEEDED)))
        ast \\ "status" // => JObject(List((status,JString(FAILED)), (status,JString(SUCCEEDED))))
        
      • version 3.5.3
        ast \ "status"  // => JArray(List(JString(FAILED), JString(SUCCEEDED)))
        ast \\ "status" // => JObject(List((status,JString(FAILED)), (status,JString(SUCCEEDED))))
        

        Attachments

          Activity

            People

            • Assignee:
              shimamoto Takako Shimamoto
              Reporter:
              shimamoto Takako Shimamoto
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: