[IMPALA-1794] IoMgr infinite loop opening/closing file when shorter than cached metadata size

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: Impala 1.4
• Fix Version/s: Impala 2.2
• Assignee: Matthew Jacobs
• Reporter: Matthew Jacobs
• Created: 17/Feb/15 23:56
• Updated: 20/Dec/15 00:05
• Resolved: 25/Feb/15 06:01

Description

1. In impala

   > create table ids(id string);
   

2. In hive

   hive> insert overwrite ids select id from ...;
   

3. Impala - refresh and make sure the read works.

   > refresh ids;
   > select count(*) from ids;
   Query: select count(*) from ids
   +----------+
   | count(*) |
   +----------+
   | 19899    |
   +----------+
   Returned 1 row(s) in 0.80s
   

4. In hive, replace again with a query that returns no results, i.e. overwrite existing file with empty file.

   hive> insert overwrite table ids select id from ... where id < 0;
   

5. In impala, reading from ids will now never return

   > select count(*) from ids;
   Query: select count(*) from ids
   

The scanner thread repeatedly enqueues scan ranges because it thinks the file is longer than it is. We can see the same file being opened and closed repeatedly:

...
I0217 12:10:46.580857  7373 disk-io-mgr-scan-range.cc:311] hdfsCloseFile() file=hdfs://nameservice1/user/hive/warehouse/alanj.db/fds/000000_0
I0217 12:10:46.583314 18907 disk-io-mgr-scan-range.cc:257] hdfsOpenFile() file=hdfs://nameservice1/user/hive/warehouse/alanj.db/fds/000000_0
I0217 12:10:46.583454  7373 disk-io-mgr-scan-range.cc:311] hdfsCloseFile() file=hdfs://nameservice1/user/hive/warehouse/alanj.db/fds/000000_0
I0217 12:10:46.585736 18907 disk-io-mgr-scan-range.cc:257] hdfsOpenFile() file=hdfs://nameservice1/user/hive/warehouse/alanj.db/fds/000000_0
I0217 12:10:46.585836  7373 disk-io-mgr-scan-range.cc:311] hdfsCloseFile() file=hdfs://nameservice1/user/hive/warehouse/alanj.db/fds/000000_0
I0217 12:10:46.587977 18907 disk-io-mgr-scan-range.cc:257] hdfsOpenFile() file=hdfs://nameservice1/user/hive/warehouse/alanj.db/fds/000000_0
I0217 12:10:46.588083  7373 disk-io-mgr-scan-range.cc:311] hdfsCloseFile() file=hdfs://nameservice1/user/hive/warehouse/alanj.db/fds/000000_0
I0217 12:10:46.590450 18907 disk-io-mgr-scan-range.cc:257] hdfsOpenFile() file=hdfs://nameservice1/user/hive/warehouse/alanj.db/fds/000000_0
...

In this scenario, FDs are not leaked, but there may be another scenario in which close is not called and then FDs would be leaked.