[SPARK-2135] InMemoryColumnarScan does not get planned correctly

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0.0
• Fix Version/s: 1.0.1, 1.1.0
• Component/s: SQL
• Target Version/s: 1.0.1, 1.1.0
• Assignee: Michael Armbrust
• Reporter: Michael Armbrust
• Created: 13/Jun/14 00:13
• Updated: 13/Jun/14 06:54
• Resolved: 13/Jun/14 06:54

Description

Right now we do not have a logical concept for in memory data. This breaks planning in several ways:

This sort of query fails to analyze:

    TestSQLContext.sql("SELECT * FROM testData").registerAsTable("selectStar")
    TestSQLContext.cacheTable("selectStar")
    TestSQLContext.sql("SELECT * FROM selectStar WHERE key = 1").collect()

Also, we read all of the columns even when we don't need to.