[HIVE-15415] Random "java.util.ConcurrentModificationException"

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.1.0
• Component/s: Beeline

Environment

Hadoop 2.7.3, Hive 2.1.0

• Assignee: Unassigned
• Reporter: Alexandre Linte
• Created: 12/Dec/16 10:47
• Updated: 20/Jun/17 08:38

Description

I'm regularly facing Hive job failures through Oozie or through the beeline CLI. The jobs exit with an error "FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask. java.util.ConcurrentModificationException (state=08S01,code=1)" but not 100% of the time.
it's also important to underline that only one user is working on the table when the jobs are running.

• stderr

  Connecting to jdbc:hive2://hiveserver2.bigdata.fr:10000/default
  Connected to: Apache Hive (version 2.1.0)
  Driver: Hive JDBC (version 2.1.0)
  Transaction isolation: TRANSACTION_REPEATABLE_READ
  No rows affected (1.475 seconds)
  No rows affected (0.004 seconds)
  No rows affected (0.004 seconds)
  No rows affected (58.977 seconds)
  No rows affected (5.524 seconds)
  No rows affected (5.235 seconds)
  Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask. java.util.ConcurrentModificationException (state=08S01,code=1)
  Closing: 0: jdbc:hive2://hiveserver2.bigdata.fr:10000/default
  Intercepting System.exit(2)
  

• stdout

  Beeline command arguments :
               -u
               jdbc:hive2://hiveserver2.bigdata.fr:10000/default
               -n
               my_user
               -p
               DUMMY
               -d
               org.apache.hive.jdbc.HiveDriver
               -f
               full_job
               -a
               delegationToken
               --hiveconf
               mapreduce.job.tags=oozie-75b060aacd7ec48c4ed637855e413280
  
  Fetching child yarn jobs
  tag id : oozie-75b060aacd7ec48c4ed637855e413280
  Child yarn jobs are found -
  =================================================================
  
  >>> Invoking Beeline command line now >>>
  
  0: jdbc:hive2://hiveserver2.bigdata.fr> use my_db;
  0: jdbc:hive2://hiveserver2.bigdata.fr> set hive.execution.engine=tez;
  0: jdbc:hive2://hiveserver2.bigdata.fr> set tez.queue.name=tez_queue;
  0: jdbc:hive2://hiveserver2.bigdata.fr>
  0: jdbc:hive2://hiveserver2.bigdata.fr> insert overwrite table main_table ^M_fd_livcfm
  . . . . . . . . . . . . . . . . . . . . . . .> select
  . . . . . . . . . . . . . . . . . . . . . . .> col.co_cd as co_cd,
  . . . . . . . . . . . . . . . . . . . . . . .> col.line_co_cd as line_co_cd,
  . . . . . . . . . . . . . . . . . . . . . . .> unix_timestamp(min(tt.statut_dt)) ^M as statut_dt
  . . . . . . . . . . . . . . . . . . . . . . .> from dlk_scf_rn_customer_order_li ^Mne col
  . . . . . . . . . . . . . . . . . . . . . . .> join dlk_scf_rn_shipment_handling ^M_utility shu
  . . . . . . . . . . . . . . . . . . . . . . .> on shu.co_cd =col.co_cd
  . . . . . . . . . . . . . . . . . . . . . . .> and shu.line_co_cd = col.line_co_ ^Mcd
  . . . . . . . . . . . . . . . . . . . . . . .> join ( select scaler_internal_ref ^M, statut_dt,recep_number,state,reason
  . . . . . . . . . . . . . . . . . . . . . . .> from dlk_scf_rn_transport_trackin ^Mg where state='LIV' and reason='CFM' ) tt
  . . . . . . . . . . . . . . . . . . . . . . .> on concat('CAL',shu.c_waybill_no) ^M =tt.scaler_internal_ref group by col.co_cd,col.line_co_cd;
  Heart beat
  Heart beat
  0: jdbc:hive2://hiveserver2.bigdata.fr>
  0: jdbc:hive2://hiveserver2.bigdata.fr> insert overwrite table main_table ^M_fd_cae
  . . . . . . . . . . . . . . . . . . . . . . .> select
  . . . . . . . . . . . . . . . . . . . . . . .> po_cd as cae, line_po_cd as lcae, ^M origin_co_cd, origin_line_co_cd
  . . . . . . . . . . . . . . . . . . . . . . .> from dlk_scf_rn_purchase_order_li ^Mne
  . . . . . . . . . . . . . . . . . . . . . . .> where instr(po_cd,"7")=1;
  0: jdbc:hive2://hiveserver2.bigdata.fr>
  0: jdbc:hive2://hiveserver2.bigdata.fr> insert overwrite table main_table ^M_fd_cai
  . . . . . . . . . . . . . . . . . . . . . . .> select
  . . . . . . . . . . . . . . . . . . . . . . .> po_cd as cai, line_po_cd as lcai, ^M origin_co_cd, origin_line_co_cd
  . . . . . . . . . . . . . . . . . . . . . . .> from dlk_scf_rn_purchase_order_li ^Mne
  . . . . . . . . . . . . . . . . . . . . . . .> where instr(po_cd,"L")=1;
  0: jdbc:hive2://hiveserver2.bigdata.fr>
  0: jdbc:hive2://hiveserver2.bigdata.fr> insert overwrite table main_table ^M_fd_cce
  . . . . . . . . . . . . . . . . . . . . . . .> select co_cd as cce,line_co_cd as ^M lcce,demand_order_ref1,
  . . . . . . . . . . . . . . . . . . . . . . .> case when demand_order_ref2=''
  . . . . . . . . . . . . . . . . . . . . . . .> then 1
  . . . . . . . . . . . . . . . . . . . . . . .> else demand_order_ref2
  . . . . . . . . . . . . . . . . . . . . . . .> end as demand_order_ref2
  . . . . . . . . . . . . . . . . . . . . . . .> from dlk_scf_rn_customer_order_li ^Mne
  . . . . . . . . . . . . . . . . . . . . . . .> where instr(co_cd,"I")=1;
  Heart beat
  
  
  <<< Invocation of Beeline command completed <<<
  
   Hadoop Job IDs executed by Beeline:
  
  Intercepting System.exit(2)
  
  Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.Hive2Main], exit code [2]
  
  Oozie Launcher failed, finishing Hadoop job gracefully
  

This error was present with Hadoop 2.7.2 and is still visible with Hadoop 2.7.3.