Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-3827

Empty metadata file causes queries on the table to fail

    Details

      Description

      I ran into a situation where drill created an empty metadata file (which is a separate issue and I will try to narrow it down. Suspicion is that this happens when "refresh table metada x" fails with "permission denied" error).

      However, we need to guard against situation where metadata file is empty or corrupted. We probably should skip reading it if we encounter unexpected result and continue with query planning without that information. In the same fashion as partition pruning failure. It's also important to log this information somewhere, drillbit.log as a start. It would be really nice to have a flag in the query profile that tells a user if we used metadata file for planning or not. Will help in debugging performance issues.

      Very confusing exception is thrown if you have zero length meta data file in the directory:

      [Wed Sep 23 07:45:28] # ls -la
      total 2
      drwxr-xr-x  2 root root   2 Sep 10 14:55 .
      drwxr-xr-x 16 root root  35 Sep 15 12:54 ..
      -rwxr-xr-x  1 root root 483 Jul  1 11:29 0_0_0.parquet
      -rwxr-xr-x  1 root root   0 Sep 10 14:55 .drill.parquet_metadata
      
      0: jdbc:drill:schema=dfs> select * from t1;
      Error: SYSTEM ERROR: JsonMappingException: No content to map due to end-of-input
       at [Source: com.mapr.fs.MapRFsDataInputStream@342bd88d; line: 1, column: 1]
      [Error Id: c97574f6-b3e8-4183-8557-c30df6ca675f on atsqa4-133.qa.lab:31010] (state=,code=0)
      

      Workaround is trivial, remove the file. Marking it as critical, since we don't have any concurrency control in place and this file can get corrupted as well.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                vitalii Vitalii Diravka
                Reporter:
                vicky Victoria Markman
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: