[SPARK-11130] TestHive fails on machines with few cores

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Duplicate
• Affects Version/s: 1.5.0, 1.6.0
• Component/s: SQL
• Assignee: Unassigned
• Reporter: Marcelo Masiero Vanzin
• Created: 15/Oct/15 16:54
• Updated: 20/Nov/15 01:21
• Resolved: 20/Nov/15 01:21

Description

Filing so it doesn't get lost (again).

TestHive.scala has this code:

    new SparkContext(
      System.getProperty("spark.sql.test.master", "local[32]"),

On machines with less cores, that causes many tests to fail with "unable to allocate memory" errors, because the default page size calculation seems to be based on the machine's core count, and not on the core count specified for the SparkContext.

Issue Links

• duplicates: SPARK-11251 Page size calculation is wrong in local mode