Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-5089

Skip initializing all enabled storage plugins for every query

    XMLWordPrintableJSON

Details

    Description

      In a query's lifecycle, at attempt is made to initialize each enabled storage plugin, while building the schema tree. This is done regardless of the actual plugins involved within a query.

      Sometimes, when one or more of the enabled storage plugins have issues - either due to misconfiguration or the underlying datasource being slow or being down, the overall query time taken increases drastically. Most likely due the attempt being made to register schemas from a faulty plugin.

      For example, when a jdbc plugin is configured with SQL Server, and at one point the underlying SQL Server db goes down, any Drill query starting to execute at that point and beyond begin to slow down drastically.

      We must skip registering unrelated schemas (& workspaces) for a query.

      Attachments

        Issue Links

          breaks

          Bug - A problem which impairs or prevents the functions of the product. DRILL-6732 Queries are runnable on disable plugins

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved
          contains

          Bug - A problem which impairs or prevents the functions of the product. DRILL-4990 Use new HDFS API access instead of listStatus to check if users have permissions to access workspace.

          • Major - Major loss of function.
          • Resolved
          relates to

          Bug - A problem which impairs or prevents the functions of the product. DRILL-4207 Queries appear to hang in scenarios when hive plugin is enabled

          • Critical - Crashes, loss of data, severe memory leak.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. DRILL-4643 Queries hang when S3 storage plugin is enabled and config is incomplete

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. DRILL-1427 Failure to initialize storage plug-in results in complete failure of drill-bit initialization

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. CALCITE-1748 Make CalciteCatalogReader.getSchema extendable to support dynamically load schema tree - getSchema need to be set to protected to allow overriding

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. DRILL-4613 Skip the plugin if it throws errors when registering schemas

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open
          links to

          Web Link GitHub Pull Request #795

          Web Link GitHub Pull Request #1032

          Activity

            People

              cshi Chunhui Shi
              agirish Abhishek Girish
              Arina Ielchiieva Arina Ielchiieva
              Votes:
              1 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: