[DRILL-7125] REFRESH TABLE METADATA fails after upgrade from Drill 1.13.0 to Drill 1.15.0 - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Bug
Status: Resolved
Priority: Major
Resolution: Fixed
Affects Version/s: 1.14.0, 1.15.0
Fix Version/s: 1.16.0
Component/s: Metadata
Labels:
- ready-to-commit

Description

REFRESH TABLE METADATA command worked successfully on Drill 1.13.0, however after upgrade Drill 1.15.0 there are errors sometime.

In sqlline logging in as regular user "alice" or Drill process user "admin" gives the same error (permission denied)
If this helps, here's also what I am seeing on sqlline

Error message contains random but valid user's names other than the user (Alice) that logged in to refresh the metadata. Looks like during the refresh metadata drillbits seems to incorrectly try the metadata generation as some random user which obviously does not have write access

2019-03-12 15:27:20,564 [2377cdd9-dd6e-d213-de1a-70b50d3641d7:frag:0:0] INFO  o.a.d.e.w.fragment.FragmentExecutor - 2377cdd9-dd6e-d213-de1a-70b50d3641d7:0:0: State change requested RUNNING --> FINISHED
2019-03-12 15:27:20,564 [2377cdd9-dd6e-d213-de1a-70b50d3641d7:frag:0:0] INFO  o.a.d.e.w.f.FragmentStatusReporter - 2377cdd9-dd6e-d213-de1a-70b50d3641d7:0:0: State to report: FINISHED
2019-03-12 15:27:23,032 [2377cdb3-86cc-438d-8ada-787d2a84df9a:foreman] INFO  o.a.drill.exec.work.foreman.Foreman - Query text for query with id 2377cdb3-86cc-438d-8ada-787d2a84df9a issued by alice: REFRESH TABLE METADATA dfs.root.`/user/alice/logs/hive/warehouse/detail`
2019-03-12 15:27:23,350 [2377cdb3-86cc-438d-8ada-787d2a84df9a:foreman] ERROR o.a.d.e.s.parquet.metadata.Metadata - Failed to read 'file://user/alice/logs/hive/warehouse/detail/.drill.parquet_metadata_directories' metadata file
java.io.IOException: 2879.5854742.1036302960 /user/alice/logs/hive/warehouse/detail/file1/.drill.parquet_metadata (Permission denied)
        at com.mapr.fs.Inode.throwIfFailed(Inode.java:390) ~[maprfs-6.1.0-mapr.jar:na]
        at com.mapr.fs.Inode.flushPages(Inode.java:505) ~[maprfs-6.1.0-mapr.jar:na]
        at com.mapr.fs.Inode.releaseDirty(Inode.java:583) ~[maprfs-6.1.0-mapr.jar:na]
        at com.mapr.fs.MapRFsOutStream.dropCurrentPage(MapRFsOutStream.java:73) ~[maprfs-6.1.0-mapr.jar:na]
        at com.mapr.fs.MapRFsOutStream.write(MapRFsOutStream.java:85) ~[maprfs-6.1.0-mapr.jar:na]
        at com.mapr.fs.MapRFsDataOutputStream.write(MapRFsDataOutputStream.java:39) ~[maprfs-6.1.0-mapr.jar:na]
        at com.fasterxml.jackson.core.json.UTF8JsonGenerator._flushBuffer(UTF8JsonGenerator.java:2085) ~[jackson-core-2.9.5.jar:2.9.5]
        at com.fasterxml.jackson.core.json.UTF8JsonGenerator.flush(UTF8JsonGenerator.java:1097) ~[jackson-core-2.9.5.jar:2.9.5]
        at com.fasterxml.jackson.databind.ObjectMapper.writeValue(ObjectMapper.java:2645) ~[jackson-databind-2.9.5.jar:2.9.5]
        at com.fasterxml.jackson.core.base.GeneratorBase.writeObject(GeneratorBase.java:381) ~[jackson-core-2.9.5.jar:2.9.5]
        at com.fasterxml.jackson.core.JsonGenerator.writeObjectField(JsonGenerator.java:1726) ~[jackson-core-2.9.5.jar:2.9.5]
        at org.apache.drill.exec.store.parquet.metadata.Metadata_V3$ColumnMetadata_v3$Serializer.serialize(Metadata_V3.java:448) ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
        at org.apache.drill.exec.store.parquet.metadata.Metadata_V3$ColumnMetadata_v3$Serializer.serialize(Metadata_V3.java:417) ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
        at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:119) ~[jackson-databind-2.9.5.jar:2.9.5]
        at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:79) ~[jackson-databind-2.9.5.jar:2.9.5]
        at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:18) ~[jackson-databind-2.9.5.jar:2.9.5]
        at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:727) ~[jackson-databind-2.9.5.jar:2.9.5]
        at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:719) ~[jackson-databind-2.9.5.jar:2.9.5]
        at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155) ~[jackson-databind-2.9.5.jar:2.9.5]
        at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:119) ~[jackson-databind-2.9.5.jar:2.9.5]
        at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:79) ~[jackson-databind-2.9.5.jar:2.9.5]
        at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:18) ~[jackson-databind-2.9.5.jar:2.9.5]
        at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:727) ~[jackson-databind-2.9.5.jar:2.9.5]
        at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:719) ~[jackson-databind-2.9.5.jar:2.9.5]
        at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155) ~[jackson-databind-2.9.5.jar:2.9.5]
        at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:119) ~[jackson-databind-2.9.5.jar:2.9.5]
        at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:79) ~[jackson-databind-2.9.5.jar:2.9.5]
        at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:18) ~[jackson-databind-2.9.5.jar:2.9.5]
        at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:727) ~[jackson-databind-2.9.5.jar:2.9.5]
        at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:719) ~[jackson-databind-2.9.5.jar:2.9.5]
        at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeWithType(BeanSerializerBase.java:604) ~[jackson-databind-2.9.5.jar:2.9.5]
        at com.fasterxml.jackson.databind.ser.impl.TypeWrappedSerializer.serialize(TypeWrappedSerializer.java:32) ~[jackson-databind-2.9.5.jar:2.9.5]
        at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize(DefaultSerializerProvider.java:480) ~[jackson-databind-2.9.5.jar:2.9.5]
        at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:319) ~[jackson-databind-2.9.5.jar:2.9.5]
        at com.fasterxml.jackson.databind.ObjectWriter$Prefetch.serialize(ObjectWriter.java:1396) ~[jackson-databind-2.9.5.jar:2.9.5]
        at com.fasterxml.jackson.databind.ObjectWriter._configAndWriteValue(ObjectWriter.java:1120) ~[jackson-databind-2.9.5.jar:2.9.5]
        at com.fasterxml.jackson.databind.ObjectWriter.writeValue(ObjectWriter.java:950) ~[jackson-databind-2.9.5.jar:2.9.5]
        at org.apache.drill.exec.store.parquet.metadata.Metadata.writeFile(Metadata.java:548) [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
        at org.apache.drill.exec.store.parquet.metadata.Metadata.createMetaFilesRecursively(Metadata.java:259) [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
        at org.apache.drill.exec.store.parquet.metadata.Metadata.createMetaFilesRecursively(Metadata.java:225) [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
        at org.apache.drill.exec.store.parquet.metadata.Metadata.readBlockMeta(Metadata.java:605) [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
        at org.apache.drill.exec.store.parquet.metadata.Metadata.readMetadataDirs(Metadata.java:179) [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
        at org.apache.drill.exec.store.parquet.ParquetFormatPlugin$ParquetFormatMatcher.isReadable(ParquetFormatPlugin.java:251) [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
        at org.apache.drill.exec.store.dfs.WorkspaceSchemaFactory$WorkspaceSchema.create(WorkspaceSchemaFactory.java:615) [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
        at org.apache.drill.exec.store.dfs.WorkspaceSchemaFactory$WorkspaceSchema.create(WorkspaceSchemaFactory.java:407) [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
        at org.apache.drill.exec.planner.sql.ExpandingConcurrentMap.getNewEntry(ExpandingConcurrentMap.java:96) [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
        at org.apache.drill.exec.planner.sql.ExpandingConcurrentMap.get(ExpandingConcurrentMap.java:90) [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
        at org.apache.drill.exec.store.dfs.WorkspaceSchemaFactory$WorkspaceSchema.getTable(WorkspaceSchemaFactory.java:554) [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
        at org.apache.calcite.jdbc.SimpleCalciteSchema.getImplicitTable(SimpleCalciteSchema.java:83) [calcite-core-1.17.0-drill-r2.jar:1.17.0-drill-r2]
        at org.apache.calcite.jdbc.CalciteSchema.getTable(CalciteSchema.java:288) [calcite-core-1.17.0-drill-r2.jar:1.17.0-drill-r2]
        at org.apache.calcite.jdbc.CalciteSchema$SchemaPlusImpl.getTable(CalciteSchema.java:648) [calcite-core-1.17.0-drill-r2.jar:1.17.0-drill-r2]
        at org.apache.drill.exec.planner.sql.handlers.RefreshMetadataHandler.getPlan(RefreshMetadataHandler.java:77) [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
        at org.apache.drill.exec.planner.sql.DrillSqlWorker.getQueryPlan(DrillSqlWorker.java:184) [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
        at org.apache.drill.exec.planner.sql.DrillSqlWorker.convertPlan(DrillSqlWorker.java:110) [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
        at org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:76) [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
        at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:584) [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
        at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:272) [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_152]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_152]
        at java.lang.Thread.run(Thread.java:748) [na:1.8.0_152]
2019-03-12 15:27:23,446 [2377cdb3-86cc-438d-8ada-787d2a84df9a:foreman] ERROR o.a.d.e.p.s.h.RefreshMetadataHandler - Failed to update metadata for table '/user/alice/logs/hive/warehouse/detail'
java.io.IOException: 2879.5854742.1036302960 /user/alice/logs/hive/warehouse/detail/file1/.drill.parquet_metadata (Permission denied)
        at com.mapr.fs.Inode.throwIfFailed(Inode.java:390) ~[maprfs-6.1.0-mapr.jar:na]
        at com.mapr.fs.Inode.flushPages(Inode.java:505) ~[maprfs-6.1.0-mapr.jar:na]
        at com.mapr.fs.Inode.releaseDirty(Inode.java:583) ~[maprfs-6.1.0-mapr.jar:na]
        at com.mapr.fs.MapRFsOutStream.dropCurrentPage(MapRFsOutStream.java:73) ~[maprfs-6.1.0-mapr.jar:na]
        at com.mapr.fs.MapRFsOutStream.write(MapRFsOutStream.java:85) ~[maprfs-6.1.0-mapr.jar:na]
        at com.mapr.fs.MapRFsDataOutputStream.write(MapRFsDataOutputStream.java:39) ~[maprfs-6.1.0-mapr.jar:na]
        at com.fasterxml.jackson.core.json.UTF8JsonGenerator._flushBuffer(UTF8JsonGenerator.java:2085) ~[jackson-core-2.9.5.jar:2.9.5]
        at com.fasterxml.jackson.core.json.UTF8JsonGenerator.flush(UTF8JsonGenerator.java:1097) ~[jackson-core-2.9.5.jar:2.9.5]
        at com.fasterxml.jackson.databind.ObjectMapper.writeValue(ObjectMapper.java:2645) ~[jackson-databind-2.9.5.jar:2.9.5]
        at com.fasterxml.jackson.core.base.GeneratorBase.writeObject(GeneratorBase.java:381) ~[jackson-core-2.9.5.jar:2.9.5]
        at com.fasterxml.jackson.core.JsonGenerator.writeObjectField(JsonGenerator.java:1726) ~[jackson-core-2.9.5.jar:2.9.5]
        at org.apache.drill.exec.store.parquet.metadata.Metadata_V3$ColumnMetadata_v3$Serializer.serialize(Metadata_V3.java:448) ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
        at org.apache.drill.exec.store.parquet.metadata.Metadata_V3$ColumnMetadata_v3$Serializer.serialize(Metadata_V3.java:417) ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
        at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:119) ~[jackson-databind-2.9.5.jar:2.9.5]
        at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:79) ~[jackson-databind-2.9.5.jar:2.9.5]
        at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:18) ~[jackson-databind-2.9.5.jar:2.9.5]
        at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:727) ~[jackson-databind-2.9.5.jar:2.9.5]
        at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:719) ~[jackson-databind-2.9.5.jar:2.9.5]
        at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155) ~[jackson-databind-2.9.5.jar:2.9.5]
        at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:119) ~[jackson-databind-2.9.5.jar:2.9.5]
        at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:79) ~[jackson-databind-2.9.5.jar:2.9.5]
        at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:18) ~[jackson-databind-2.9.5.jar:2.9.5]
        at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:727) ~[jackson-databind-2.9.5.jar:2.9.5]
        at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:719) ~[jackson-databind-2.9.5.jar:2.9.5]
        at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155) ~[jackson-databind-2.9.5.jar:2.9.5]
        at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:119) ~[jackson-databind-2.9.5.jar:2.9.5]
        at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:79) ~[jackson-databind-2.9.5.jar:2.9.5]
        at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:18) ~[jackson-databind-2.9.5.jar:2.9.5]
        at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:727) ~[jackson-databind-2.9.5.jar:2.9.5]
        at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:719) ~[jackson-databind-2.9.5.jar:2.9.5]
        at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeWithType(BeanSerializerBase.java:604) ~[jackson-databind-2.9.5.jar:2.9.5]
        at com.fasterxml.jackson.databind.ser.impl.TypeWrappedSerializer.serialize(TypeWrappedSerializer.java:32) ~[jackson-databind-2.9.5.jar:2.9.5]
        at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize(DefaultSerializerProvider.java:480) ~[jackson-databind-2.9.5.jar:2.9.5]
        at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:319) ~[jackson-databind-2.9.5.jar:2.9.5]
        at com.fasterxml.jackson.databind.ObjectWriter$Prefetch.serialize(ObjectWriter.java:1396) ~[jackson-databind-2.9.5.jar:2.9.5]
        at com.fasterxml.jackson.databind.ObjectWriter._configAndWriteValue(ObjectWriter.java:1120) ~[jackson-databind-2.9.5.jar:2.9.5]
        at com.fasterxml.jackson.databind.ObjectWriter.writeValue(ObjectWriter.java:950) ~[jackson-databind-2.9.5.jar:2.9.5]
        at org.apache.drill.exec.store.parquet.metadata.Metadata.writeFile(Metadata.java:548) ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
        at org.apache.drill.exec.store.parquet.metadata.Metadata.createMetaFilesRecursively(Metadata.java:259) ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
        at org.apache.drill.exec.store.parquet.metadata.Metadata.createMetaFilesRecursively(Metadata.java:225) ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
        at org.apache.drill.exec.store.parquet.metadata.Metadata.createMeta(Metadata.java:111) ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
        at org.apache.drill.exec.planner.sql.handlers.RefreshMetadataHandler.getPlan(RefreshMetadataHandler.java:124) ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
        at org.apache.drill.exec.planner.sql.DrillSqlWorker.getQueryPlan(DrillSqlWorker.java:184) [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
        at org.apache.drill.exec.planner.sql.DrillSqlWorker.convertPlan(DrillSqlWorker.java:110) [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
        at org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:76) [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
        at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:584) [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
        at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:272) [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_152]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_152]
        at java.lang.Thread.run(Thread.java:748) [na:1.8.0_152]
2019-03-12 15:27:23,459 [2377cdb3-86cc-438d-8ada-787d2a84df9a:frag:0:0] INFO  o.a.d.e.w.fragment.FragmentExecutor - 2377cdb3-86cc-438d-8ada-787d2a84df9a:0:0: State change requested AWAITING_ALLOCATION --> RUNNING

Attachments

Issue Links

links to

GitHub Pull Request #1707

Activity

People

Assignee:: Sorabh Hamirwasia

Reporter:: Sorabh Hamirwasia

Reviewer:: Aman Sinha

Votes:: 0 Vote for this issue

Watchers:: 2 Start watching this issue

Dates

Created:: 20/Mar/19 20:25

Updated:: 01/Apr/19 23:33

Resolved:: 21/Mar/19 19:13