[CALCITE-1959] Reduce the amount of Metadata and table name calls in Druid

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.14.0
• Component/s: druid-adapter
• Assignee: Zain Humayun
• Reporter: Zain Humayun
• Created: 18/Aug/17 22:05
• Updated: 27/Feb/24 22:23
• Resolved: 23/Aug/17 17:46

Description

Currently in the Druid adapter, when a model definition file is not provided, query times can be quite slow due to excessive metadata and table names calls. Initial investigation reveals that a simple query like

SELECT * from table

produces at least 30 http calls for fetching the table names, and another 30 calls for meta data. This slows down the query considerably, even when the individual calls themselves are relatively quick.

The source of the issue seems to be DruidSchema#getTableMap, as this method dispatches calls to both DruidConnectionImpl#metadata and DruidConnectionImpl#tableNames. In addition, DruidTable#create makes another call to DruidConnectionImpl#metadata even if it's given the populated fields as arguments.

getTableMap is called fairly often indirectly from many places, including the validator, so as the query becomes larger, the problems gets a lot worse. Ideally, one would provide a model definition to prevent this, but doing so is not always possible/preferred for users.