[SPARK-13947] The error message from using an invalid table reference is not clear

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 2.3.0
• Component/s: SQL
• Assignee: Ruben Berenguel
• Reporter: Wes McKinney
• Created: 16/Mar/16 20:25
• Updated: 25/Oct/17 06:02
• Resolved: 25/Oct/17 06:02

Description

import numpy as np
import pandas as pd

df = pd.DataFrame({'foo': np.random.randn(1000),
                   'bar': np.random.randn(1000)})

df2 = pd.DataFrame({'foo': np.random.randn(1000),
                    'bar': np.random.randn(1000)})


sdf = sqlContext.createDataFrame(df)
sdf2 = sqlContext.createDataFrame(df2)

sdf[sdf2.foo > 0]

Produces this error message:

AnalysisException: u'resolved attribute(s) foo#91 missing from bar#87,foo#88 in operator !Filter (foo#91 > cast(0 as double));'

It may be possible to make it more clear what the user did wrong.

Issue Links

• links to: [Github] Pull Request #17100 (rberenguel)