[SPOT-32] DNS Operational Analytics error at ingest_summary "file name too long"

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Assignee: Unassigned
• Reporter: Gustavo Rodrigues Silva
• Created: 19/Jan/17 18:40
• Updated: 08/Feb/17 15:19
• Resolved: 08/Feb/17 15:19

Description

Running the Operational Analytics module generates an error at the output of Impala saying that the generated file name was too long. The contents of this non existing file are used by df which generates the Index

[spot@ip-10-0-0-53 oa]$ python2.7 start_oa.py -d 20160707 -t dns -l 3000
2017-01-18 10:44:56,080 - OA - INFO - -------------------- STARTING OA ---------------------
2017-01-18 10:44:56,080 - OA - INFO - Validating input parameter values
2017-01-18 10:44:56,339 - OA.DATA - INFO - Reading data component configuration: /home/spot/spot-oa/oa/components/data/engine.json
2017-01-18 10:44:56,339 - OA.DATA - INFO - Initializating impala instance
Starting Impala Shell without Kerberos authentication
Starting Impala Shell without Kerberos authentication
2017-01-18 10:44:57,031 - OA.DNS - INFO - Creating folder structure for OA (data and ipynb)
2017-01-18 10:44:57,032 - OA.DNS - INFO - Adding edge investigation IPython Notebook
2017-01-18 10:44:57,032 - OA.DNS - INFO - Adding threat investigation IPython Notebook
2017-01-18 10:44:57,032 - OA.DNS - INFO - Getting 20160707 Machine Learning Results from HDFS
get: `/home/spot/spot-oa/data/dns/20160707/./dns_results.csv': File exists
2017-01-18 10:44:59,122 - OA.DNS - INFO - hadoop fs -get /user/spot/dns/scored_results/20160707/scores/dns_results.csv /home/spot/spot-oa/data/dns/20160707/.
2017-01-18 10:44:59,123 - OA.DNS - INFO - Reading 20160707 dns results file: /home/spot/spot-oa/data/dns/20160707/dns_results.csv
2017-01-18 10:44:59,132 - OA.DNS - INFO - Adding headers
('--- NC CONF FILE', '/home/spot/spot-oa/oa/components/nc/nc_config.json')
('--- NC CONF',

{'network_context': '/home/spot/spot-oa/context/networkcontext.csv'}

)
('---- CONFIG NETWORK CONTEXT',

{'network_context': '/home/spot/spot-oa/context/networkcontext.csv'}

)
('---- PRINT CONFIG[NC] TEST', '/home/spot/spot-oa/context/networkcontext.csv')
('---- _nc_file_path', '/home/spot/spot-oa/context/networkcontext.csv')
/home/spot/spot-oa/context/networkcontext.csv
('---- NC ROWS', [['10.0.0.53', 'ip-10-0-0-53.ec2.internal'], ['10.0.0.222', 'ip-10-0-0-222.ec2.internal']])
2017-01-18 10:44:59,253 - OA.DNS - INFO - Getting OA DNS suspicious details/chord diagram
Starting Impala Shell without Kerberos authentication
Error opening output file for writing: [Errno 36] File name too long: '/home/spot/spot-oa/data/dns/20160707/edge-z1azmwypaab1al0ag0st8ti7pdcq0nnp74hbt62lvf5cdkkeks5msi7nn4dj0ns.moqnv6sk2voc8cjerpf55f67mqa2a5k5qefpvdl106kn4qbq1hf6u9oc4aaio6m.odedu7nnke2i49bn92omhjkqmv4nf2s9e891g8l1srg1sr2bcek8gpu1pmnoca7.ql1eul1o3a74h5akatvf1teg32pvoarhpaq64aw.a.r.ipass.com_0_00.tmp'
Process Process-1:
Traceback (most recent call last):
File "/usr/local/lib/python2.7/multiprocessing/process.py", line 232, in _bootstrap
self.run()
File "/usr/local/lib/python2.7/multiprocessing/process.py", line 88, in run
self._target(*self._args, **self._kwargs)
File "/home/spot/spot-oa/oa/dns/dns_oa.py", line 264, in _get_suspicious_details
self._get_dns_details(dns_qry_name,year,month,day,hh,dns_iana)
File "/home/spot/spot-oa/oa/dns/dns_oa.py", line 277, in _get_dns_details
self._engine.query(dns_qry,edge_tmp)
File "/home/spot/spot-oa/oa/components/data/data.py", line 34, in query
self._engine.query(query,output_file,delimiter)
File "/home/spot/spot-oa/oa/components/data/impala.py", line 23, in query
check_output(impala_cmd,shell=True)
File "/usr/local/lib/python2.7/subprocess.py", line 537, in check_output
raise CalledProcessError(retcode, cmd, output=output)
CalledProcessError: Command 'impala-shell -i ip-10-0-0-61.ec2.internal --quiet --print_header -B --output_delimiter=',' -q "SELECT frame_time,frame_len,ip_dst,ip_src,dns_qry_name,dns_qry_class,dns_qry_type,dns_qry_rcode,dns_a FROM spot.dns WHERE y=2016 AND m=07 AND d=7 AND dns_qry_name LIKE '%z1azmwypaab1al0ag0st8ti7pdcq0nnp74hbt62lvf5cdkkeks5msi7nn4dj0ns.moqnv6sk2voc8cjerpf55f67mqa2a5k5qefpvdl106kn4qbq1hf6u9oc4aaio6m.odedu7nnke2i49bn92omhjkqmv4nf2s9e891g8l1srg1sr2bcek8gpu1pmnoca7.ql1eul1o3a74h5akatvf1teg32pvoarhpaq64aw.a.r.ipass.com%' AND h=0 LIMIT 250;" -o /home/spot/spot-oa/data/dns/20160707/edge-z1azmwypaab1al0ag0st8ti7pdcq0nnp74hbt62lvf5cdkkeks5msi7nn4dj0ns.moqnv6sk2voc8cjerpf55f67mqa2a5k5qefpvdl106kn4qbq1hf6u9oc4aaio6m.odedu7nnke2i49bn92omhjkqmv4nf2s9e891g8l1srg1sr2bcek8gpu1pmnoca7.ql1eul1o3a74h5akatvf1teg32pvoarhpaq64aw.a.r.ipass.com_0_00.tmp' returned non-zero exit status 1
2017-01-18 10:44:59,558 - OA.DNS - INFO - Getting ingest summary data for the day
Starting Impala Shell without Kerberos authentication
Traceback (most recent call last):
File "start_oa.py", line 64, in <module>
main()
File "start_oa.py", line 22, in main
start_oa(args)
File "start_oa.py", line 37, in start_oa
oa_process.start()
File "/home/spot/spot-oa/oa/dns/dns_oa.py", line 73, in start
self._ingest_summary()
File "/home/spot/spot-oa/oa/dns/dns_oa.py", line 377, in _ingest_summary
df_new = pd.DataFrame([["

{0}

-

{1}

-

{2}

{3}

:

{4}

".format(yr, mn, dy,val['frame_time'].split(" ")[3].split(":")[0].zfill(2),val['frame_time'].split(" ")[3].split(":")[1].zfill(2)), int(val['total']) if not math.isnan(val['total']) else 0 ] for key,val in df_results.iterrows()],columns = ingest_summary_cols)
IndexError: list index out of range

Issue Links

• links to: GitHub Pull Request #16