[METRON-2088] Support HDP 3.1.0

• Type: New Feature
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Assignee: Nick Allen
• Reporter: Nick Allen
• Created: 23/Apr/19 14:16
• Updated: 26/Nov/19 08:39

Description

Metron currently supports HDP 2.6.5. We need to upgrade Metron to run against the stack provided by HDP 3.1.0.

HDP Component Versions

• HDP 3.1.0
• HDP 2.6.5
• HDP 2.5.6

See initial discuss thread here.

Issue Links

• depends upon: METRON-2217 Migrate current HBase client from HTableInterface to Table
• links to: GitHub Pull Request #1572
• links to: HDP versions comparison spreadsheet

Sub-Tasks

• Create CentOS 7 Development Environment: Done
• Install Metron MPack in Ambari 2.7.3.0: Done
• Ambari client exception occurred: No JSON object could be decoded: Done
• Introduce HBase Connection Abstractions for HBase 2.0.2: Done
• Upgrade REST for HBase 2.0.2: Done
• Upgrade Profiler for HBase 2.0.2: Done
• Upgrade to Kafka 2.0.0 and Storm 1.2.1: Done
• Upgrade to Solr 7.4.0: Done
• Upgrade to Zeppelin 0.8.0: Done
• Profiler Integration Test Fails After Storm 1.2.1 Upgrade: Done
• Revert METRON-2175, METRON-2176, METRON-2177 in HDP 3.1 upgrade feature branch: Done
• Upgrade Core Enrichments for HBase 2.0.2: Done
• Upgrade Data Management for HBase 2.0.2: Done
• Remove Legacy HBase Client: Done
• Upgrade Streaming Enrichments for HBase 2.0.2: Done
• Merge Master into Feature Branch: Done
• Missing services in HDP 3.1 metron mpack and installer stuck: Done
• PcapTopologyIntegrationTest Intermittent Failures: Done
• Upgrade to HBase 2.0.2: Done
• Isolate Curator Dependencies: Done
• Upgrade metron-hbase-client to HBase 2.0.2: Done
• Solr Indexing Topology Fails to Start on Secure Cluster with HDP 3.1: Done
• Unable to Index to Solr with Kerberos: Done
• Enrichment Topology Unable to Load Geo IP Data from HDFS: Done
• Update Kerberos settings: Done
• Upgrade to Hadoop 3.1.1: Done
• Remove Overrides for Storm 1.0.x: Done
• Deprecate CentOS 6 Development Environment: Done
• Building Against Wrong Storm Flux Version: Done
• Reconcile Versions in Root POM with HDP 3.1: In Progress