[CARBONDATA-3698] Drop table will iterate all databases and tables

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.6.1
• Component/s: sql
• Assignee: Unassigned
• Reporter: ShuMing Li
• Created: 13/Feb/20 06:59
• Updated: 13/Feb/20 07:04

Description

When executes `drop table carbon_table_xxx`, it costs huge time if there are a lot of tables in the warehouse.

Here is TraceStack:

 

- locked <0x00000006dd91f450> (a org.apache.spark.sql.hive.HiveExternalCatalog) at org.apache.spark.sql.hive.HiveExternalCatalog.listTables(HiveExternalCatalog.scala:869) at org.apache.spark.sql.catalyst.catalog.SessionCatalog.listTables(SessionCatalog.scala:811) at org.apache.spark.sql.catalyst.catalog.SessionCatalog.listTables(SessionCatalog.scala:795) at org.apache.spark.sql.hive.CarbonFileMetastore$$anonfun$4.apply(CarbonFileMetastore.scala:585) at org.apache.spark.sql.hive.CarbonFileMetastore$$anonfun$4.apply(CarbonFileMetastore.scala:584) at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241) at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241) at scala.collection.Iterator$class.foreach(Iterator.scala:893) at scala.collection.AbstractIterator.foreach(Iterator.scala:1336) at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) at scala.collection.AbstractIterable.foreach(Iterable.scala:54) at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241) at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104) at org.apache.spark.sql.hive.CarbonFileMetastore.removeStaleTimeStampEntries(CarbonFileMetastore.scala:583) at org.apache.spark.sql.execution.command.table.CarbonDropTableCommand.processMetadata(CarbonDropTableCommand.scala:192) at org.apache.spark.sql.execution.command.AtomicRunnableCommand$$anonfun$run$3.apply(package.scala:146) at org.apache.spark.sql.execution.command.AtomicRunnableCommand$$anonfun$run$3.apply(package.scala:145) at org.apache.spark.sql.execution.command.Auditable$class.runWithAudit(package.scala:104)

 

 

Here is code which iterates tables:

 

 

def removeStaleTimeStampEntries(sparkSession: SparkSession): Unit = {
   val tablesList = sparkSession.sessionState.catalog.listDatabases().flatMap
{     database =>       sparkSession.sessionState.catalog.listTables(database)       .map(table => CarbonTable.buildUniqueName(database, table.table))   }
 ...
 }