[ORC-547] ORC write on Map Reduce fwk is extremely slow

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.3.3
• Component/s: MapReduce
• Environment: Map Reduce FWK
• Assignee: Unassigned
• Reporter: Thrinath Dosapati
• Created: 23/Aug/19 11:13
• Updated: 26/Aug/19 08:50

Description

Recently, we have encountered cases where the ORC write is extremely slow for certain workloads. 

I tried to run this on Spark env, but the behaviour remains same

What could be the reason for the slowness?

Schema : 

struct<rc:struct<cc:struct<appv:string,cht:string>,pc,ac:array<struct<layer:string,abid:string>>,mp:string,rsc:bigint,pt:string,ai:struct<supercat:string,subcat:string,v:string,cat:string>,prid:string,pid:array<string>,rid:string,uc:struct<abid:string,aid:string>,p:array<struct<productid:string,meta:array<struct<mv:string,mk:string>>,nid:string,lid:string>>,sc:array<struct<score:double,sid:string>>,ui:struct<ss:string,dg:struct<mds:string,fds:string>,ps:string,bg:struct<ms:string,fs:string>,ms:string,ul:array<struct<c:string,s:string,p:string>>,iscc:boolean,ic:boolean,rfmb:struct<rb:string,fb:string,mb:string,rfmsg:string,imlb:boolean>>,pck:string,pi:string,dc:struct<os:string,ip:string,did:string>>,rws:array<struct<rccs:array<struct<eid:string,bc:string,mp:string,lid:string,nid:string,cm:array<struct<mv:string,mk:string>>,mtomlfs:array<struct<rv:string,lid:string,ms:string,mid:string,mv:string,mlfs:array<struct<fw:string,f:string>>>>,rpid:string,et:string,dt:string,cs:string,ct:string,t:string,cid:string>>,wm:array<struct<mv:string,mk:string>>,mtomlfs:array<struct<rv:string,lid:string,ms:string,mid:string,mv:string,mlfs:array<struct<fw:string,f:string>>>>,wc:struct<murl:string,rt:string,djct:string,wimpid:string,va:string,title:string,ws:string,wc:string,mtext:string,wt:string,vt:string,urms:array<struct<rk:string,dc:bigint>>,mrcc:bigint,sc:bigint>>>>

 

Logs and sample records are attached for reference.