Affects Version/s: 1.5.2, 1.6.3, 2.1.2, 2.2.0
Fix Version/s: None
Hive 1.2.1, Spark 1.5.2
spark-sql seems not to see data stored as delta files in an ACID Hive table.
Actually I encountered the same problem as describe here : http://stackoverflow.com/questions/35955666/spark-sql-is-not-returning-records-for-hive-transactional-tables-on-hdp
For example, create an ACID table with HiveCLI and insert a row :
Then make a query with spark-sql CLI :
That query gets no result and there are no errors in logs.
If you go to HDFS to inspect table files, you find only deltas
Then if you run compaction on that table (in HiveCLI) :
As a result, the delta will be compute into a base file :
Go back to spark-sql and the same query gets a result :
But next time you make an insert into Hive table :
spark-sql will immediately see changes :
Yet there was no other compaction, but spark-sql "sees" the base AND the delta file :