Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-9232

Duplicate code in JSONRelation

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 1.4.0
    • Fix Version/s: 1.5.0
    • Component/s: SQL
    • Labels:
      None
    • Target Version/s:

      Description

      The following block appears identically in two places:

      var success: Boolean = false
      try {
        success = fs.delete(filesystemPath, true)
      } catch {
        case e: IOException =>
          throw new IOException(
            s"Unable to clear output directory ${filesystemPath.toString} prior"
              + s" to writing to JSON table:\n${e.toString}")
      }
      if (!success) {
        throw new IOException(
          s"Unable to clear output directory ${filesystemPath.toString} prior"
            + s" to writing to JSON table.")
        }
      }
      

      https://github.com/apache/spark/blob/e5d2c37c68ac00a57c2542e62d1c5b4ca267c89e/sql/core/src/main/scala/org/apache/spark/sql/json/JSONRelation.scala#L72

      https://github.com/apache/spark/blob/e5d2c37c68ac00a57c2542e62d1c5b4ca267c89e/sql/core/src/main/scala/org/apache/spark/sql/json/JSONRelation.scala#L131

        Attachments

          Activity

            People

            • Assignee:
              andrewor14 Andrew Or
              Reporter:
              andrewor14 Andrew Or
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: