Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-21546

hiveserver2 - “mapred.FileInputFormat: Total input files to process” - why single threaded?

    XMLWordPrintableJSON

Details

    Description

      I have setup Hive (v2.3.4) on Spark (exec engine, but MR gets same issue), hadoop 2.7.6 (or hadoop 2.8.5). My external hive table is Parquet format on s3 across 100s of partitions. Below settings are set to 20:

      {{hive.exec.input.listing.max.threads mapred.dfsclient.parallelism.max mapreduce.input.fileinputformat.list-status.num-threads }}

      Run a simple query:

      {{select * from s.there h_code = 'KGD78' and h_no = '265' }}

      I can see the below in HiveServer2 logs (the logs continue for more than 1000 lines listing all the different partitions). Why is the listing of files not being done in parallel? It takes more than 5mins just in the listing.

      {{2019-03-29T11:29:26,866 INFO [3fa82455-7853-4c4b-8964-847c00bec708 HiveServer2-Handler-Pool: Thread-53] compress.CodecPool: Got brand-new decompressor [.snappy] 2019-03-29T11:29:27,283 INFO [3fa82455-7853-4c4b-8964-847c00bec708 HiveServer2-Handler-Pool: Thread-53] mapred.FileInputFormat: Total input files to process : 1 2019-03-29T11:29:27,797 INFO [3fa82455-7853-4c4b-8964-847c00bec708 HiveServer2-Handler-Pool: Thread-53] mapred.FileInputFormat: Total input files to process : 1 2019-03-29T11:29:28,374 INFO [3fa82455-7853-4c4b-8964-847c00bec708 HiveServer2-Handler-Pool: Thread-53] mapred.FileInputFormat: Total input files to process : 1 2019-03-29T11:29:28,919 INFO [3fa82455-7853-4c4b-8964-847c00bec708 HiveServer2-Handler-Pool: Thread-53] mapred.FileInputFormat: Total input files to process : 1 2019-03-29T11:29:29,483 INFO [3fa82455-7853-4c4b-8964-847c00bec708 HiveServer2-Handler-Pool: Thread-53] mapred.FileInputFormat: Total input files to process : 1 2019-03-29T11:29:30,003 INFO [3fa82455-7853-4c4b-8964-847c00bec708 HiveServer2-Handler-Pool: Thread-53] mapred.FileInputFormat: Total input files to process : 1 2019-03-29T11:29:30,518 INFO [3fa82455-7853-4c4b-8964-847c00bec708 HiveServer2-Handler-Pool: Thread-53] mapred.FileInputFormat: Total input files to process : 1 2019-03-29T11:29:31,001 INFO [3fa82455-7853-4c4b-8964-847c00bec708 HiveServer2-Handler-Pool: Thread-53] mapred.FileInputFormat: Total input files to process : 1 2019-03-29T11:29:31,549 INFO [3fa82455-7853-4c4b-8964-847c00bec708 HiveServer2-Handler-Pool: Thread-53] mapred.FileInputFormat: Total input files to process : 1 2019-03-29T11:29:32,048 INFO [3fa82455-7853-4c4b-8964-847c00bec708 HiveServer2-Handler-Pool: Thread-53] mapred.FileInputFormat: Total input files to process : 1 2019-03-29T11:29:32,574 INFO [3fa82455-7853-4c4b-8964-847c00bec708 HiveServer2-Handler-Pool: Thread-53] mapred.FileInputFormat: Total input files to process : 1 2019-03-29T11:29:33,130 INFO [3fa82455-7853-4c4b-8964-847c00bec708 HiveServer2-Handler-Pool: Thread-53] mapred.FileInputFormat: Total input files to process : 1 2019-03-29T11:29:33,639 INFO [3fa82455-7853-4c4b-8964-847c00bec708 HiveServer2-Handler-Pool: Thread-53] mapred.FileInputFormat: Total input files to process : 1 2019-03-29T11:29:34,189 INFO [3fa82455-7853-4c4b-8964-847c00bec708 HiveServer2-Handler-Pool: Thread-53] mapred.FileInputFormat: Total input files to process : 1 2019-03-29T11:29:34,743 INFO [3fa82455-7853-4c4b-8964-847c00bec708 HiveServer2-Handler-Pool: Thread-53] mapred.FileInputFormat: Total input files to process : 1 2019-03-29T11:29:35,208 INFO [3fa82455-7853-4c4b-8964-847c00bec708 HiveServer2-Handler-Pool: Thread-53] mapred.FileInputFormat: Total input files to process : 1 2019-03-29T11:29:35,701 INFO [3fa82455-7853-4c4b-8964-847c00bec708 HiveServer2-Handler-Pool: Thread-53] mapred.FileInputFormat: Total input files to process : 1 2019-03-29T11:29:36,183 INFO [3fa82455-7853-4c4b-8964-847c00bec708 HiveServer2-Handler-Pool: Thread-53] mapred.FileInputFormat: Total input files to process : 1 2019-03-29T11:29:36,662 INFO [3fa82455-7853-4c4b-8964-847c00bec708 HiveServer2-Handler-Pool: Thread-53] mapred.FileInputFormat: Total input files to process : 1 2019-03-29T11:29:37,154 INFO [3fa82455-7853-4c4b-8964-847c00bec708 HiveServer2-Handler-Pool: Thread-53] mapred.FileInputFormat: Total input files to process : 1 2019-03-29T11:29:37,645 INFO [3fa82455-7853-4c4b-8964-847c00bec708 HiveServer2-Handler-Pool: Thread-53] mapred.FileInputFormat: Total input files to process : 1 }}

      I've tried

      {{hive.exec.input.listing.max.threads mapred.dfsclient.parallelism.max mapreduce.input.fileinputformat.list-status.num-threads }}

      with defaults, 1, 50...still same result

       

       

       

      Hive 3.1.1/hadoop3.1.2 also has the issue:

       

      2019-03-29T18:10:15,451 INFO [16b32706-3490-432d-b49e-67279ea88e15 HiveServer2-Handler-Pool: Thread-30] hadoop.InternalParquetRecordReader: at row 0. reading next block
      2019-03-29T18:10:15,461 INFO [16b32706-3490-432d-b49e-67279ea88e15 HiveServer2-Handler-Pool: Thread-30] hadoop.InternalParquetRecordReader: block read in memory in 10 ms. row count = 4584
      2019-03-29T18:10:15,620 INFO [16b32706-3490-432d-b49e-67279ea88e15 HiveServer2-Handler-Pool: Thread-30] mapred.FileInputFormat: Total input files to process : 1
      2019-03-29T18:10:15,714 INFO [16b32706-3490-432d-b49e-67279ea88e15 HiveServer2-Handler-Pool: Thread-30] s3a.S3AInputStream: Switching to Random IO seek policy
      2019-03-29T18:10:15,757 INFO [16b32706-3490-432d-b49e-67279ea88e15 HiveServer2-Handler-Pool: Thread-30] s3a.S3AInputStream: Switching to Random IO seek policy
      2019-03-29T18:10:15,767 INFO [16b32706-3490-432d-b49e-67279ea88e15 HiveServer2-Handler-Pool: Thread-30] hadoop.InternalParquetRecordReader: RecordReader initialized will read a total of 4584 records.
      2019-03-29T18:10:15,767 INFO [16b32706-3490-432d-b49e-67279ea88e15 HiveServer2-Handler-Pool: Thread-30] hadoop.InternalParquetRecordReader: at row 0. reading next block
      2019-03-29T18:10:15,777 INFO [16b32706-3490-432d-b49e-67279ea88e15 HiveServer2-Handler-Pool: Thread-30] hadoop.InternalParquetRecordReader: block read in memory in 10 ms. row count = 4584
      2019-03-29T18:10:15,984 INFO [16b32706-3490-432d-b49e-67279ea88e15 HiveServer2-Handler-Pool: Thread-30] mapred.FileInputFormat: Total input files to process : 1
      2019-03-29T18:10:16,033 INFO [16b32706-3490-432d-b49e-67279ea88e15 HiveServer2-Handler-Pool: Thread-30] s3a.S3AInputStream: Switching to Random IO seek policy
      2019-03-29T18:10:16,070 INFO [16b32706-3490-432d-b49e-67279ea88e15 HiveServer2-Handler-Pool: Thread-30] s3a.S3AInputStream: Switching to Random IO seek policy
      2019-03-29T18:10:16,080 INFO [16b32706-3490-432d-b49e-67279ea88e15 HiveServer2-Handler-Pool: Thread-30] hadoop.InternalParquetRecordReader: RecordReader initialized will read a total of 4584 records.
      2019-03-29T18:10:16,080 INFO [16b32706-3490-432d-b49e-67279ea88e15 HiveServer2-Handler-Pool: Thread-30] hadoop.InternalParquetRecordReader: at row 0. reading next block
      2019-03-29T18:10:16,089 INFO [16b32706-3490-432d-b49e-67279ea88e15 HiveServer2-Handler-Pool: Thread-30] hadoop.InternalParquetRecordReader: block read in memory in 9 ms. row c ount = 4584
      2019-03-29T18:10:16,287 INFO [16b32706-3490-432d-b49e-67279ea88e15 HiveServer2-Handler-Pool: Thread-30] mapred.FileInputFormat: Total input files to process : 1
      2019-03-29T18:10:16,356 INFO [16b32706-3490-432d-b49e-67279ea88e15 HiveServer2-Handler-Pool: Thread-30] s3a.S3AInputStream: Switching to Random IO seek policy
      2019-03-29T18:10:16,404 INFO [16b32706-3490-432d-b49e-67279ea88e15 HiveServer2-Handler-Pool: Thread-30] s3a.S3AInputStream: Switching to Random IO seek policy
      2019-03-29T18:10:16,415 INFO [16b32706-3490-432d-b49e-67279ea88e15 HiveServer2-Handler-Pool: Thread-30] hadoop.InternalParquetRecordReader: RecordReader initialized will read a total of 4584 records.
      2019-03-29T18:10:16,415 INFO [16b32706-3490-432d-b49e-67279ea88e15 HiveServer2-Handler-Pool: Thread-30] hadoop.InternalParquetRecordReader: at row 0. reading next block
      2019-03-29T18:10:16,426 INFO [16b32706-3490-432d-b49e-67279ea88e15 HiveServer2-Handler-Pool: Thread-30] hadoop.InternalParquetRecordReader: block read in memory in 11 ms. row count = 4584
      2019-03-29T18:10:16,613 INFO [16b32706-3490-432d-b49e-67279ea88e15 HiveServer2-Handler-Pool: Thread-30] mapred.FileInputFormat: Total input files to process : 1
      2019-03-29T18:10:16,654 INFO [16b32706-3490-432d-b49e-67279ea88e15 HiveServer2-Handler-Pool: Thread-30] s3a.S3AInputStream: Switching to Random IO seek policy
      2019-03-29T18:10:16,700 INFO [16b32706-3490-432d-b49e-67279ea88e15 HiveServer2-Handler-Pool: Thread-30] s3a.S3AInputStream: Switching to Random IO seek policy
      2019-03-29T18:10:16,712 INFO [16b32706-3490-432d-b49e-67279ea88e15 HiveServer2-Handler-Pool: Thread-30] hadoop.InternalParquetRecordReader: RecordReader initialized will read a total of 240 records.
      2019-03-29T18:10:16,712 INFO [16b32706-3490-432d-b49e-67279ea88e15 HiveServer2-Handler-Pool: Thread-30] hadoop.InternalParquetRecordReader: at row 0. reading next block
      2019-03-29T18:10:16,722 INFO [16b32706-3490-432d-b49e-67279ea88e15 HiveServer2-Handler-Pool: Thread-30] hadoop.InternalParquetRecordReader: block read in memory in 10 ms. row count = 240
      2019-03-29T18:10:16,895 INFO [16b32706-3490-432d-b49e-67279ea88e15 HiveServer2-Handler-Pool: Thread-30] mapred.FileInputFormat: Total input files to process : 1
      2019-03-29T18:10:16,934 INFO [16b32706-3490-432d-b49e-67279ea88e15 HiveServer2-Handler-Pool: Thread-30] s3a.S3AInputStream: Switching to Random IO seek policy
      2019-03-29T18:10:17,004 INFO [16b32706-3490-432d-b49e-67279ea88e15 HiveServer2-Handler-Pool: Thread-30] s3a.S3AInputStream: Switching to Random IO seek policy
      2019-03-29T18:10:17,015 INFO [16b32706-3490-432d-b49e-67279ea88e15 HiveServer2-Handler-Pool: Thread-30] hadoop.InternalParquetRecordReader: RecordReader initialized will read a total of 240 records.
      2019-03-29T18:10:17,015 INFO [16b32706-3490-432d-b49e-67279ea88e15 HiveServer2-Handler-Pool: Thread-30] hadoop.InternalParquetRecordReader: at row 0. reading next block
      2019-03-29T18:10:17,024 INFO [16b32706-3490-432d-b49e-67279ea88e15 HiveServer2-Handler-Pool: Thread-30] hadoop.InternalParquetRecordReader: block read in memory in 9 ms. row c ount = 240
      2019-03-29T18:10:17,217 INFO [16b32706-3490-432d-b49e-67279ea88e15 HiveServer2-Handler-Pool: Thread-30] mapred.FileInputFormat: Total input files to process : 1
      2019-03-29T18:10:17,269 INFO [16b32706-3490-432d-b49e-67279ea88e15 HiveServer2-Handler-Pool: Thread-30] s3a.S3AInputStream: Switching to Random IO seek policy
      2019-03-29T18:10:17,306 INFO [16b32706-3490-432d-b49e-67279ea88e15 HiveServer2-Handler-Pool: Thread-30] s3a.S3AInputStream: Switching to Random IO seek policy
      2019-03-29T18:10:17,315 INFO [16b32706-3490-432d-b49e-67279ea88e15 HiveServer2-Handler-Pool: Thread-30] hadoop.InternalParquetRecordReader: RecordReader initialized will read a total of 240 records.
      2019-03-29T18:10:17,315 INFO [16b32706-3490-432d-b49e-67279ea88e15 HiveServer2-Handler-Pool: Thread-30] hadoop.InternalParquetRecordReader: at row 0. reading next block
      2019-03-29T18:10:17,325 INFO [16b32706-3490-432d-b49e-67279ea88e15 HiveServer2-Handler-Pool: Thread-30] hadoop.InternalParquetRecordReader: block read in memory in 10 ms. row count = 240
      2019-03-29T18:10:17,478 INFO [16b32706-3490-432d-b49e-67279ea88e15 HiveServer2-Handler-Pool: Thread-30] mapred.FileInputFormat: Total input files to process : 1
      2019-03-29T18:10:17,513 INFO [16b32706-3490-432d-b49e-67279ea88e15 HiveServer2-Handler-Pool: Thread-30] s3a.S3AInputStream: Switching to Random IO seek policy
      2019-03-29T18:10:17,548 INFO [16b32706-3490-432d-b49e-67279ea88e15 HiveServer2-Handler-Pool: Thread-30] s3a.S3AInputStream: Switching to Random IO seek policy
      2019-03-29T18:10:17,559 INFO [16b32706-3490-432d-b49e-67279ea88e15 HiveServer2-Handler-Pool: Thread-30] hadoop.InternalParquetRecordReader: RecordReader initialized will read a total of 240 records.
      2019-03-29T18:10:17,559 INFO [16b32706-3490-432d-b49e-67279ea88e15 HiveServer2-Handler-Pool: Thread-30] hadoop.InternalParquetRecordReader: at row 0. reading next block
      2019-03-29T18:10:17,568 INFO [16b32706-3490-432d-b49e-67279ea88e15 HiveServer2-Handler-Pool: Thread-30] hadoop.InternalParquetRecordReader: block read in memory in 9 ms. row c ount = 240
      2019-03-29T18:10:17,729 INFO [16b32706-3490-432d-b49e-67279ea88e15 HiveServer2-Handler-Pool: Thread-30] mapred.FileInputFormat: Total input files to process : 1
      2019-03-29T18:10:17,805 INFO [16b32706-3490-432d-b49e-67279ea88e15 HiveServer2-Handler-Pool: Thread-30] s3a.S3AInputStream: Switching to Random IO seek policy
      2019-03-29T18:10:17,845 INFO [16b32706-3490-432d-b49e-67279ea88e15 HiveServer2-Handler-Pool: Thread-30] s3a.S3AInputStream: Switching to Random IO seek policy
      2019-03-29T18:10:17,854 INFO [16b32706-3490-432d-b49e-67279ea88e15 HiveServer2-Handler-Pool: Thread-30] hadoop.InternalParquetRecordReader: RecordReader initialized will read a total of 4584 records.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              toopt4 t oo
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated: