[FALCON-2163] Feed replication based on partition does not produce expected output

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: trunk
• Fix Version/s: trunk, 0.11
• Component/s: feed, prism
• Assignee: Pallavi Rao
• Reporter: Pragya Mittal
• Created: 28/Sep/16 11:02
• Updated: 08/Jun/17 12:18
• Resolved: 25/Oct/16 11:02

Description

Feed :

<?xml version="1.0" encoding="UTF-8"?><feed xmlns="uri:falcon:feed:0.1" name="PrismFeedReplicationPartitionExpTest-InputFeed-021eefe6" description="Input File">
    <partitions>
        <partition name="colo"/>
        <partition name="eventTime"/>
        <partition name="impressionHour"/>
        <partition name="pricingModel"/>
    </partitions>
    <frequency>minutes(5)</frequency>
    <late-arrival cut-off="days(100000)"/>
    <clusters>
        <cluster name="PrismFeedReplicationPartitionExpTest-falcon2-stg4-5698a59c" type="source" partition="${cluster.colo}">
            <validity start="2012-10-01T12:00Z" end="2099-10-01T12:10Z"/>
            <retention limit="days(1000000)" action="delete"/>
            <locations>
                <location type="data" path="/tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/source1/${YEAR}/${MONTH}/${DAY}/${HOUR}/${MINUTE}"/>
            </locations>
        </cluster>
        <cluster name="PrismFeedReplicationPartitionExpTest-falcon2-stg4-7a96289d" type="target">
            <validity start="2012-10-01T12:00Z" end="2099-10-01T12:25Z"/>
            <retention limit="days(1000000)" action="delete"/>
            <locations>
                <location type="data" path="/tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/clusterPath/localDC/rc/billing/replicated/${YEAR}/${MONTH}/${DAY}/${HOUR}/${MINUTE}"/>
            </locations>
        </cluster>
        <cluster name="PrismFeedReplicationPartitionExpTest-falcon2-stg4-d4fbf2d9" type="source" partition="${cluster.colo}">
            <validity start="2012-10-01T12:00Z" end="2099-01-01T00:00Z"/>
            <retention limit="days(1000000)" action="delete"/>
            <locations>
                <location type="data" path="/tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/localDC/rc/billing/${YEAR}/${MONTH}/${DAY}/${HOUR}/${MINUTE}"/>
            </locations>
        </cluster>
    </clusters>
    <locations>
        <location type="data" path="/data/regression/fetlrc/billing/${YEAR}/${MONTH}/${DAY}/${HOUR}/${MINUTE}"/>
        <location type="stats" path="/data/regression/fetlrc/billing/stats"/>
        <location type="meta" path="/data/regression/fetlrc/billing/metadata"/>
    </locations>
    <ACL owner="pragya" group="dataqa" permission="*"/>
    <schema location="/databus/streams_local/click_rr/schema/" provider="protobuf"/>
    <properties>
        <property name="field1" value="value1"/>
        <property name="field2" value="value2"/>
    </properties>
</feed>

Cluster 1 data :

dataqa@falcon1001:~$ hlr /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/source1/2012/10/01/12/*/*
16/09/28 10:53:02 INFO util.Shell: ===Loading custom Shell class JIRA:GRIDOPS-4518 ==
lsr: DEPRECATED: Please use 'ls -R' instead.
-rw-r--r--   3 pragya hdfs       2298 2016-09-28 10:33 /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/source1/2012/10/01/12/00/ua1/dataFile.xml
-rw-r--r--   3 pragya hdfs       2298 2016-09-28 10:33 /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/source1/2012/10/01/12/00/ua2/dataFile.xml
-rw-r--r--   3 pragya hdfs       2298 2016-09-28 10:33 /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/source1/2012/10/01/12/00/ua3/dataFile.xml
-rw-r--r--   3 pragya hdfs        922 2016-09-28 10:33 /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/source1/2012/10/01/12/05/ua1/id.pig
-rw-r--r--   3 pragya hdfs        922 2016-09-28 10:33 /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/source1/2012/10/01/12/05/ua2/id.pig
-rw-r--r--   3 pragya hdfs        922 2016-09-28 10:33 /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/source1/2012/10/01/12/05/ua3/id.pig

Cluster 3 data :

dataqa@hydra:~$ hlr /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/dataBillingRC/fetlrc/billing/2012/10/01/12/*/*
lsr: DEPRECATED: Please use 'ls -R' instead.
-rw-r--r--   3 pragya hdfs       2298 2016-09-28 10:33 /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/dataBillingRC/fetlrc/billing/2012/10/01/12/00/ua1/dataFile.xml
-rw-r--r--   3 pragya hdfs       2298 2016-09-28 10:33 /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/dataBillingRC/fetlrc/billing/2012/10/01/12/00/ua2/dataFile.xml
-rw-r--r--   3 pragya hdfs       2298 2016-09-28 10:33 /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/dataBillingRC/fetlrc/billing/2012/10/01/12/00/ua3/dataFile.xml
-rw-r--r--   3 pragya hdfs        922 2016-09-28 10:33 /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/dataBillingRC/fetlrc/billing/2012/10/01/12/05/ua1/id.pig
-rw-r--r--   3 pragya hdfs        922 2016-09-28 10:33 /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/dataBillingRC/fetlrc/billing/2012/10/01/12/05/ua2/id.pig
-rw-r--r--   3 pragya hdfs        922 2016-09-28 10:33 /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/dataBillingRC/fetlrc/billing/2012/10/01/12/05/ua3/id.pig
-rw-r--r--   3 pragya hdfs       1711 2016-09-28 10:33 /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/dataBillingRC/fetlrc/billing/2012/10/01/12/10/ua1/cluster-0.1.xml
-rw-r--r--   3 pragya hdfs       1711 2016-09-28 10:33 /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/dataBillingRC/fetlrc/billing/2012/10/01/12/10/ua2/cluster-0.1.xml
-rw-r--r--   3 pragya hdfs       1711 2016-09-28 10:33 /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/dataBillingRC/fetlrc/billing/2012/10/01/12/10/ua3/cluster-0.1.xml
-rw-r--r--   3 pragya hdfs       1031 2016-09-28 10:33 /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/dataBillingRC/fetlrc/billing/2012/10/01/12/15/ua1/dataFile.properties
-rw-r--r--   3 pragya hdfs       1031 2016-09-28 10:33 /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/dataBillingRC/fetlrc/billing/2012/10/01/12/15/ua2/dataFile.properties
-rw-r--r--   3 pragya hdfs       1031 2016-09-28 10:33 /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/dataBillingRC/fetlrc/billing/2012/10/01/12/15/ua3/dataFile.properties
-rw-r--r--   3 pragya hdfs       1031 2016-09-28 10:33 /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/dataBillingRC/fetlrc/billing/2012/10/01/12/20/ua1/dataFile.properties
-rw-r--r--   3 pragya hdfs       1031 2016-09-28 10:33 /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/dataBillingRC/fetlrc/billing/2012/10/01/12/20/ua2/dataFile.properties
-rw-r--r--   3 pragya hdfs       1031 2016-09-28 10:33 /tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/dataBillingRC/fetlrc/billing/2012/10/01/12/20/ua3/dataFile.properties

Target data :

/tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/clusterPath/localDC/rc/billing/replicated/2012/10/01/12/00/ua1/_SUCCESS
/tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/clusterPath/localDC/rc/billing/replicated/2012/10/01/12/00/ua1/dataFile.xml
/tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/clusterPath/localDC/rc/billing/replicated/2012/10/01/12/05/ua1/_SUCCESS
/tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/clusterPath/localDC/rc/billing/replicated/2012/10/01/12/05/ua1/id.pig

/tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/clusterPath/localDC/rc/billing/replicated/2012/10/01/12/10/ua3/_SUCCESS
/tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/clusterPath/localDC/rc/billing/replicated/2012/10/01/12/10/ua3/cluster-0.1.xml
/tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/clusterPath/localDC/rc/billing/replicated/2012/10/01/12/15/ua3/_SUCCESS
/tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/clusterPath/localDC/rc/billing/replicated/2012/10/01/12/15/ua3/dataFile.properties
/tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/clusterPath/localDC/rc/billing/replicated/2012/10/01/12/20/ua3/_SUCCESS
/tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/clusterPath/localDC/rc/billing/replicated/2012/10/01/12/20/ua3/dataFile.properties

Expected output :

/tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/clusterPath/localDC/rc/billing/replicated/2012/10/01/12/00/ua1/_SUCCESS
/tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/clusterPath/localDC/rc/billing/replicated/2012/10/01/12/00/ua1/dataFile.xml
/tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/clusterPath/localDC/rc/billing/replicated/2012/10/01/12/05/ua1/_SUCCESS
/tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/clusterPath/localDC/rc/billing/replicated/2012/10/01/12/05/ua1/id.pig

/tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/clusterPath/localDC/rc/billing/replicated/2012/10/01/12/00/ua3/_SUCCESS
/tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/clusterPath/localDC/rc/billing/replicated/2012/10/01/12/00/ua3/dataFile.xml
/tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/clusterPath/localDC/rc/billing/replicated/2012/10/01/12/05/ua3/_SUCCESS
/tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/clusterPath/localDC/rc/billing/replicated/2012/10/01/12/05/ua3/id.pig

/tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/clusterPath/localDC/rc/billing/replicated/2012/10/01/12/10/ua3/_SUCCESS
/tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/clusterPath/localDC/rc/billing/replicated/2012/10/01/12/10/ua3/cluster-0.1.xml
/tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/clusterPath/localDC/rc/billing/replicated/2012/10/01/12/15/ua3/_SUCCESS
/tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/clusterPath/localDC/rc/billing/replicated/2012/10/01/12/15/ua3/dataFile.properties
/tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/clusterPath/localDC/rc/billing/replicated/2012/10/01/12/20/ua3/_SUCCESS
/tmp/falcon-regression/PrismFeedReplicationPartitionExpTest/clusterPath/localDC/rc/billing/replicated/2012/10/01/12/20/ua3/dataFile.properties

Issue Links

• is duplicated by: FALCON-2171 When feed from multiple colos are replicated, the colo folders get overwritten