[HDDS-10675] [Hsync] OverWrite using "sh key cp" shouldn't be allowed on hsynced files.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Component/s: OM
• Assignee: Unassigned
• Reporter: Pratyush Bhatt
• Created: 10/Apr/24 08:56
• Updated: 29/Apr/24 20:09
• Resolved: 29/Apr/24 20:09

Description

Similar Scenario on the lines of HDDS-10670

Scenario: Do a sh key cp on the same key that has been hsynced and is open.

Observations:
Open a file, write data and do hsync

Data written on iteration : 1
Data written on iteration : 2
Data written on iteration : 3
Data written on iteration : 4
Data written on iteration : 5
Data written on iteration : 6
Data written on iteration : 7
Data written on iteration : 8
Data written on iteration : 9
Data written on iteration : 10
Hsync completed on file, counter: 10 

-Here the code sleeps of 60 seconds-

Checking the file is Open and hsynced

 ozone admin om lof --service-id=ozone1712556312 --prefix=/hsyncvol/hsyncbuck/
1809 total open files (est.). Showing 1 open files (limit 100) under path prefix:
  /hsyncvol/hsyncbuck/Client ID        Creation time    Hsync'ed    Open File Path
112246067997904292    1712739074675    Yes        /hsyncvol/hsyncbuck/-9223372036845049599/File_new_12.txtReached the end of the list.

Doing a key info till now:

ozone sh key info hsyncvol/hsyncbuck/hsync/File_new_12.txt
24/04/10 08:51:18 INFO protocolPB.OmTransportFactory: Loading OM transport implementation org.apache.hadoop.ozone.om.protocolPB.Hadoop3OmTransportFactory as specified by configuration.
24/04/10 08:51:18 INFO client.ClientTrustManager: Loading certificates for client.
{
  "volumeName" : "hsyncvol",
  "bucketName" : "hsyncbuck",
  "name" : "hsync/File_new_12.txt",
  "dataSize" : 51200,
  "creationTime" : "2024-04-10T08:51:14.675Z",
  "modificationTime" : "2024-04-10T08:51:16.008Z",
  "replicationConfig" : {
    "replicationFactor" : "THREE",
    "requiredNodes" : 3,
    "replicationType" : "RATIS"
  },
  "metadata" : {
    "hsyncClientId" : "112246067997904292"
  },
  "ozoneKeyLocations" : [ {
    "containerID" : 6,
    "localID" : 113750153625606497,
    "length" : 51200,
    "offset" : 0,
    "keyOffset" : 0
  } ],
  "file" : true
}

Now OverWrite the same key using sh key cp

>> ozone sh key put hsyncvol/hsyncbuck/dummy_file.txt /etc/passwd
24/04/10 08:51:32 INFO protocolPB.OmTransportFactory: Loading OM transport implementation org.apache.hadoop.ozone.om.protocolPB.Hadoop3OmTransportFactory as specified by configuration.
24/04/10 08:51:32 INFO client.ClientTrustManager: Loading certificates for client.
24/04/10 08:51:33 WARN impl.MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-xceiverclientmetrics.properties,hadoop-metrics2.properties
24/04/10 08:51:33 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).
24/04/10 08:51:33 INFO impl.MetricsSystemImpl: XceiverClientMetrics metrics system started
24/04/10 08:51:34 INFO metrics.MetricRegistries: Loaded MetricRegistries class org.apache.ratis.metrics.dropwizard3.Dm3MetricRegistriesImpl

>> ozone sh key cp /hsyncvol/hsyncbuck/ dummy_file.txt hsync/File_new_12.txt
24/04/10 08:51:48 INFO protocolPB.OmTransportFactory: Loading OM transport implementation org.apache.hadoop.ozone.om.protocolPB.Hadoop3OmTransportFactory as specified by configuration.
24/04/10 08:51:49 INFO client.ClientTrustManager: Loading certificates for client.
24/04/10 08:51:50 WARN impl.MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-xceiverclientmetrics.properties,hadoop-metrics2.properties
24/04/10 08:51:50 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).
24/04/10 08:51:50 INFO impl.MetricsSystemImpl: XceiverClientMetrics metrics system started
24/04/10 08:51:51 INFO metrics.MetricRegistries: Loaded MetricRegistries class org.apache.ratis.metrics.dropwizard3.Dm3MetricRegistriesImpl

Key info, successfully overwritten:

ozone sh key info hsyncvol/hsyncbuck/hsync/File_new_12.txt
24/04/10 08:51:56 INFO protocolPB.OmTransportFactory: Loading OM transport implementation org.apache.hadoop.ozone.om.protocolPB.Hadoop3OmTransportFactory as specified by configuration.
24/04/10 08:51:57 INFO client.ClientTrustManager: Loading certificates for client.
{
  "volumeName" : "hsyncvol",
  "bucketName" : "hsyncbuck",
  "name" : "hsync/File_new_12.txt",
  "dataSize" : 6644,
  "creationTime" : "2024-04-10T08:51:14.675Z",
  "modificationTime" : "2024-04-10T08:51:51.673Z",
  "replicationConfig" : {
    "replicationFactor" : "THREE",
    "requiredNodes" : 3,
    "replicationType" : "RATIS"
  },
  "metadata" : { },
  "ozoneKeyLocations" : [ {
    "containerID" : 3,
    "localID" : 113750153625606500,
    "length" : 6644,
    "offset" : 0,
    "keyOffset" : 0
  } ],
  "file" : true
}

But key is still open in the OpenKeyTable:

 ozone admin om lof --service-id=ozone1712556312 --prefix=/hsyncvol/hsyncbuck/
1809 total open files (est.). Showing 1 open files (limit 100) under path prefix:
  /hsyncvol/hsyncbuck/Client ID        Creation time    Hsync'ed    Open File Path
112246067997904292    1712739074675    Yes        /hsyncvol/hsyncbuck/-9223372036845049599/File_new_12.txtReached the end of the list.

Perform LeaseRecovery:

ozone debug recover --path=ofs://ozone1712556312//hsyncvol/hsyncbuck/hsync/File_new_12.txt
24/04/10 08:52:13 INFO protocolPB.OmTransportFactory: Loading OM transport implementation org.apache.hadoop.ozone.om.protocolPB.Hadoop3OmTransportFactory as specified by configuration.
24/04/10 08:52:13 INFO client.ClientTrustManager: Loading certificates for client.
Lease recovery SUCCEEDED on ofs://ozone1712556312//hsyncvol/hsyncbuck/hsync/File_new_12.txt

Still the key is open:

ozone admin om lof --service-id=ozone1712556312 --prefix=/hsyncvol/hsyncbuck/
1809 total open files (est.). Showing 1 open files (limit 100) under path prefix:
  /hsyncvol/hsyncbuck/Client ID        Creation time    Hsync'ed    Open File Path
112246067997904292    1712739074675    Yes        /hsyncvol/hsyncbuck/-9223372036845049599/File_new_12.txtReached the end of the list.

– 60 seconds of code pause is resumed now--
Still able to write the data into the file and hsync

Data written on iteration : 11
Data written on iteration : 12
Data written on iteration : 13
Data written on iteration : 14
Data written on iteration : 15
Data written on iteration : 16
Data written on iteration : 17
Data written on iteration : 18
Data written on iteration : 19
Data written on iteration : 20
Hsync completed on file, counter: 20 

Basically looks like lease recovery gets broken if we allow such overwriting.

Issue Links

• is duplicated by: HDDS-10670 [Hsync] OverWrite using "sh key put" shouldn't be allowed on hsynced files.
• relates to: HDDS-10670 [Hsync] OverWrite using "sh key put" shouldn't be allowed on hsynced files.
• relates to: HDDS-10770 [Hsync] Allow overwrite hsynced file