Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-22940

Test suite HiveExternalCatalogVersionsSuite fails on platforms that don't have wget installed

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.2.1
    • Fix Version/s: 2.3.0
    • Component/s: Tests
    • Labels:
      None
    • Environment:

      MacOS Sierra 10.12.6

      Description

      On platforms that don't have wget installed (e.g., Mac OS X), test suite org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite throws an exception and aborts:

      java.io.IOException: Cannot run program "wget": error=2, No such file or directory

      HiveExternalCatalogVersionsSuite uses wget to download older versions of Spark for compatibility testing. First it uses wget to find a suitable mirror, and then it uses wget to download a tar file from the mirror.

      There are several ways to fix this (in reverse order of difficulty of implementation)

      1. Require Mac OS X users to install wget if they wish to run unit tests (or at the very least if they wish to run HiveExternalCatalogVersionsSuite). Also, update documentation to make this requirement explicit.
      2. Fall back on curl when wget is not available.
      3. Use an HTTP library to query for a suitable mirror and download the tar file.

      Number 2 is easy to implement, and I did so to get the unit test to run. But it relies on another external program if wget is not installed.

      Number 3 is probably slightly more complex to implement and requires more corner-case checking (e.g, redirects, etc.).

        Attachments

          Activity

            People

            • Assignee:
              bersprockets Bruce Robbins
              Reporter:
              bersprockets Bruce Robbins
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: