Description
The Federation Membership State encapsulates the information about the Namenodes of each sub-cluster that are participating in Federation. The information includes addresses for RPC, Web. This information is stored in the State Store and later used by the Router to find data in the federation.
Attachments
Attachments
- HDFS-10687-HDFS-10467-008.patch
- 203 kB
- Íñigo Goiri
- HDFS-10687-HDFS-10467-007.patch
- 207 kB
- Íñigo Goiri
- HDFS-10687-HDFS-10467-006.patch
- 207 kB
- Íñigo Goiri
- HDFS-10687-HDFS-10467-005.patch
- 204 kB
- Íñigo Goiri
- HDFS-10687-HDFS-10467-004.patch
- 172 kB
- Íñigo Goiri
- HDFS-10687-HDFS-10467-003.patch
- 127 kB
- Íñigo Goiri
- HDFS-10687-HDFS-10467-002.patch
- 130 kB
- Íñigo Goiri
- HDFS-10467-HDFS-10687-001.patch
- 257 kB
- Jason Kace
Issue Links
- depends upon
-
HDFS-10630 Federation State Store FS Implementation
- Resolved
-
HDFS-10882 Federation State Store Interface API
- Resolved
- is depended upon by
-
HDFS-11826 Federation Namenode Heartbeat
- Resolved
- is related to
-
YARN-3662 Federation Membership State Store internal APIs
- Resolved
-
YARN-5300 Exclude generated federation protobuf sources from YARN Javadoc/findbugs build
- Resolved
- relates to
-
HDFS-12772 RBF: Federation Router State State Store internal API
- Resolved
Activity
To do this I will need something like YARN-5300. To simplify, I will include this in the patch unless somebody prefers a separated JIRA.
Attaching first patch version for review.
This patch contains some overlap code (common dependencies) with HDFS-10630 (Federation State Store). The code compiles as is, but is not functional until HDFS-10630 is committed.
The Federation membership state API functions are defined in org.apache.hadoop.hdfs.server.federation.store.FederationMembershipStateStore.java and are implemented in org.apache.hadoop.hdfs.server.federation.store.impl.FederationMembershipStateStoreImpl.java.
The primary function of the membership state API is to persist the data required to identify the most active, or recently active, NN for each nameservice. This is a key feature for HDFS federation. Individual routers monitor 1-N namenodes and heartbeat the namenode status into the state store. The NN heartbeat code is available for review in HDFS-10629. This patch contains the APIs to query, update and cache the reported state of all NNs in the cluster.
The additional files pertain to request/response API objects and the protobuf backing implementation of them.
All comments and feedback are welcome!
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 14s | Docker mode activated. |
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
-1 | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
0 | mvndep | 0m 27s | Maven dependency ordering for branch |
+1 | mvninstall | 7m 7s | trunk passed |
+1 | compile | 7m 20s | trunk passed |
+1 | checkstyle | 1m 29s | trunk passed |
+1 | mvnsite | 1m 3s | trunk passed |
+1 | mvneclipse | 0m 21s | trunk passed |
0 | findbugs | 0m 0s | Skipped patched modules with no Java source: hadoop-build-tools |
+1 | findbugs | 1m 45s | trunk passed |
+1 | javadoc | 1m 6s | trunk passed |
0 | mvndep | 0m 14s | Maven dependency ordering for patch |
+1 | mvninstall | 1m 0s | the patch passed |
+1 | compile | 7m 19s | the patch passed |
+1 | cc | 7m 19s | the patch passed |
+1 | javac | 7m 19s | the patch passed |
-0 | checkstyle | 1m 50s | root: The patch generated 78 new + 0 unchanged - 0 fixed = 78 total (was 0) |
+1 | mvnsite | 1m 5s | the patch passed |
+1 | mvneclipse | 0m 21s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | xml | 0m 1s | The patch has no ill-formed XML file. |
0 | findbugs | 0m 0s | Skipped patched modules with no Java source: hadoop-build-tools |
-1 | findbugs | 1m 58s | hadoop-hdfs-project/hadoop-hdfs generated 11 new + 0 unchanged - 0 fixed = 11 total (was 0) |
-1 | javadoc | 1m 2s | hadoop-hdfs-project_hadoop-hdfs generated 11 new + 7 unchanged - 0 fixed = 18 total (was 7) |
+1 | unit | 0m 10s | hadoop-build-tools in the patch passed. |
-1 | unit | 60m 40s | hadoop-hdfs in the patch failed. |
-1 | asflicense | 0m 23s | The patch generated 4 ASF License warnings. |
98m 19s |
Reason | Tests |
---|---|
FindBugs | module:hadoop-hdfs-project/hadoop-hdfs |
Useless control flow in org.apache.hadoop.hdfs.federation.protocol.proto.HdfsServerFederationProtos$FederationNamespaceInfoProto$Builder.maybeForceBuilderInitialization() At HdfsServerFederationProtos.java: At HdfsServerFederationProtos.java:[line 3248] | |
Useless control flow in org.apache.hadoop.hdfs.federation.protocol.proto.HdfsServerFederationProtos$GetExpiredRegistrationsRequestProto$Builder.maybeForceBuilderInitialization() At HdfsServerFederationProtos.java: At HdfsServerFederationProtos.java:[line 5937] | |
Useless control flow in org.apache.hadoop.hdfs.federation.protocol.proto.HdfsServerFederationProtos$GetNamespaceInfoRequestProto$Builder.maybeForceBuilderInitialization() At HdfsServerFederationProtos.java: At HdfsServerFederationProtos.java:[line 6275] | |
Useless control flow in org.apache.hadoop.hdfs.federation.protocol.proto.HdfsServerFederationProtos$NamenodeHeartbeatResponseProto$Builder.maybeForceBuilderInitialization() At HdfsServerFederationProtos.java: At HdfsServerFederationProtos.java:[line 9190] | |
Useless control flow in org.apache.hadoop.hdfs.federation.protocol.proto.HdfsServerFederationProtos$NamenodeMembershipEntityProto$Builder.maybeForceBuilderInitialization() At HdfsServerFederationProtos.java: At HdfsServerFederationProtos.java:[line 1480] | |
Useless control flow in org.apache.hadoop.hdfs.federation.protocol.proto.HdfsServerFederationProtos$OverrideNamenodeRegistrationRequestProto$Builder.maybeForceBuilderInitialization() At HdfsServerFederationProtos.java: At HdfsServerFederationProtos.java:[line 7564] | |
Useless control flow in org.apache.hadoop.hdfs.federation.protocol.proto.HdfsServerFederationProtos$OverrideNamenodeRegistrationResponseProto$Builder.maybeForceBuilderInitialization() At HdfsServerFederationProtos.java: At HdfsServerFederationProtos.java:[line 8211] | |
Useless control flow in org.apache.hadoop.hdfs.federation.protocol.proto.HdfsServerFederationProtos$QueryPairProto$Builder.maybeForceBuilderInitialization() At HdfsServerFederationProtos.java: At HdfsServerFederationProtos.java:[line 4007] | |
Exception is caught when Exception is not thrown in org.apache.hadoop.hdfs.server.federation.store.FederationMembershipStateStoreUtils.aggregateFederationNamespaceInfo(FederationMembershipStateStore, String) At FederationMembershipStateStoreUtils.java:is not thrown in org.apache.hadoop.hdfs.server.federation.store.FederationMembershipStateStoreUtils.aggregateFederationNamespaceInfo(FederationMembershipStateStore, String) At FederationMembershipStateStoreUtils.java:[line 65] | |
org.apache.hadoop.hdfs.server.federation.store.FederationNamespaceInfo defines compareTo(FederationNamespaceInfo) and uses Object.equals() At FederationNamespaceInfo.java:Object.equals() At FederationNamespaceInfo.java:[line 70] | |
Exception is caught when Exception is not thrown in org.apache.hadoop.hdfs.server.federation.store.protocol.impl.pb.FederationProtocolPBTranslator.getBuilder() At FederationProtocolPBTranslator.java:is not thrown in org.apache.hadoop.hdfs.server.federation.store.protocol.impl.pb.FederationProtocolPBTranslator.getBuilder() At FederationProtocolPBTranslator.java:[line 71] | |
Failed junit tests | hadoop.hdfs.server.blockmanagement.TestReplicationPolicy |
This message was automatically generated.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 0s | Docker mode activated. |
-1 | patch | 0m 8s | |
Subsystem | Report/Notes |
---|---|
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12862513/HDFS-10467-HDFS-10687-002.patch |
Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/19011/console |
Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 1m 4s | Docker mode activated. |
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
+1 | mvninstall | 16m 1s | |
+1 | compile | 0m 49s | |
+1 | checkstyle | 0m 35s | |
+1 | mvnsite | 0m 54s | |
+1 | mvneclipse | 0m 13s | |
+1 | findbugs | 1m 45s | |
+1 | javadoc | 0m 40s | |
+1 | mvninstall | 0m 48s | the patch passed |
+1 | compile | 0m 45s | the patch passed |
+1 | cc | 0m 45s | the patch passed |
+1 | javac | 0m 45s | the patch passed |
-0 | checkstyle | 0m 33s | hadoop-hdfs-project/hadoop-hdfs: The patch generated 30 new + 0 unchanged - 0 fixed = 30 total (was 0) |
+1 | mvnsite | 0m 50s | the patch passed |
+1 | mvneclipse | 0m 11s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | xml | 0m 1s | The patch has no ill-formed XML file. |
-1 | findbugs | 1m 56s | hadoop-hdfs-project/hadoop-hdfs generated 9 new + 0 unchanged - 0 fixed = 9 total (was 0) |
+1 | javadoc | 0m 41s | the patch passed |
-1 | unit | 89m 30s | hadoop-hdfs in the patch failed. |
+1 | asflicense | 0m 19s | The patch does not generate ASF License warnings. |
118m 59s |
Reason | Tests |
---|---|
FindBugs | module:hadoop-hdfs-project/hadoop-hdfs |
Useless control flow in org.apache.hadoop.hdfs.federation.protocol.proto.HdfsServerFederationProtos$FederationNamespaceInfoProto$Builder.maybeForceBuilderInitialization() At HdfsServerFederationProtos.java: At HdfsServerFederationProtos.java:[line 5067] | |
Useless control flow in org.apache.hadoop.hdfs.federation.protocol.proto.HdfsServerFederationProtos$GetExpiredRegistrationsRequestProto$Builder.maybeForceBuilderInitialization() At HdfsServerFederationProtos.java: At HdfsServerFederationProtos.java:[line 7082] | |
Useless control flow in org.apache.hadoop.hdfs.federation.protocol.proto.HdfsServerFederationProtos$GetNamespaceInfoRequestProto$Builder.maybeForceBuilderInitialization() At HdfsServerFederationProtos.java: At HdfsServerFederationProtos.java:[line 7420] | |
Useless control flow in org.apache.hadoop.hdfs.federation.protocol.proto.HdfsServerFederationProtos$NamenodeHeartbeatResponseProto$Builder.maybeForceBuilderInitialization() At HdfsServerFederationProtos.java: At HdfsServerFederationProtos.java:[line 10335] | |
Useless control flow in org.apache.hadoop.hdfs.federation.protocol.proto.HdfsServerFederationProtos$NamenodeMembershipStatsRecordProto$Builder.maybeForceBuilderInitialization() At HdfsServerFederationProtos.java: At HdfsServerFederationProtos.java:[line 1466] | |
Useless control flow in org.apache.hadoop.hdfs.federation.protocol.proto.HdfsServerFederationProtos$QueryPairProto$Builder.maybeForceBuilderInitialization() At HdfsServerFederationProtos.java: At HdfsServerFederationProtos.java:[line 424] | |
Useless control flow in org.apache.hadoop.hdfs.federation.protocol.proto.HdfsServerFederationProtos$UpdateNamenodeRegistrationRequestProto$Builder.maybeForceBuilderInitialization() At HdfsServerFederationProtos.java: At HdfsServerFederationProtos.java:[line 8709] | |
Useless control flow in org.apache.hadoop.hdfs.federation.protocol.proto.HdfsServerFederationProtos$UpdateNamenodeRegistrationResponseProto$Builder.maybeForceBuilderInitialization() At HdfsServerFederationProtos.java: At HdfsServerFederationProtos.java:[line 9356] | |
Redundant nullcheck of records, which is known to be non-null in org.apache.hadoop.hdfs.server.federation.store.CachedRecordStore.loadCache(boolean) Redundant null check at CachedRecordStore.java:is known to be non-null in org.apache.hadoop.hdfs.server.federation.store.CachedRecordStore.loadCache(boolean) Redundant null check at CachedRecordStore.java:[line 109] | |
Failed junit tests | hadoop.hdfs.TestMaintenanceState |
hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl |
This message was automatically generated.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 38s | Docker mode activated. |
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
+1 | mvninstall | 14m 10s | |
+1 | compile | 0m 56s | |
+1 | checkstyle | 0m 39s | |
+1 | mvnsite | 1m 6s | |
+1 | mvneclipse | 0m 15s | |
-1 | findbugs | 1m 52s | hadoop-hdfs-project/hadoop-hdfs in |
+1 | javadoc | 0m 42s | |
+1 | mvninstall | 0m 50s | the patch passed |
+1 | compile | 0m 47s | the patch passed |
+1 | cc | 0m 47s | the patch passed |
+1 | javac | 0m 47s | the patch passed |
-0 | checkstyle | 0m 35s | hadoop-hdfs-project/hadoop-hdfs: The patch generated 29 new + 0 unchanged - 0 fixed = 29 total (was 0) |
+1 | mvnsite | 0m 55s | the patch passed |
+1 | mvneclipse | 0m 12s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | xml | 0m 2s | The patch has no ill-formed XML file. |
-1 | findbugs | 1m 54s | hadoop-hdfs-project/hadoop-hdfs generated 7 new + 10 unchanged - 0 fixed = 17 total (was 10) |
+1 | javadoc | 0m 46s | the patch passed |
-1 | unit | 93m 59s | hadoop-hdfs in the patch failed. |
+1 | asflicense | 0m 22s | The patch does not generate ASF License warnings. |
122m 14s |
Reason | Tests |
---|---|
FindBugs | module:hadoop-hdfs-project/hadoop-hdfs |
Useless control flow in org.apache.hadoop.hdfs.federation.protocol.proto.HdfsServerFederationProtos$FederationNamespaceInfoProto$Builder.maybeForceBuilderInitialization() At HdfsServerFederationProtos.java: At HdfsServerFederationProtos.java:[line 4573] | |
Useless control flow in org.apache.hadoop.hdfs.federation.protocol.proto.HdfsServerFederationProtos$GetExpiredRegistrationsRequestProto$Builder.maybeForceBuilderInitialization() At HdfsServerFederationProtos.java: At HdfsServerFederationProtos.java:[line 6424] | |
Useless control flow in org.apache.hadoop.hdfs.federation.protocol.proto.HdfsServerFederationProtos$GetNamespaceInfoRequestProto$Builder.maybeForceBuilderInitialization() At HdfsServerFederationProtos.java: At HdfsServerFederationProtos.java:[line 6762] | |
Useless control flow in org.apache.hadoop.hdfs.federation.protocol.proto.HdfsServerFederationProtos$NamenodeHeartbeatResponseProto$Builder.maybeForceBuilderInitialization() At HdfsServerFederationProtos.java: At HdfsServerFederationProtos.java:[line 9677] | |
Useless control flow in org.apache.hadoop.hdfs.federation.protocol.proto.HdfsServerFederationProtos$NamenodeMembershipStatsRecordProto$Builder.maybeForceBuilderInitialization() At HdfsServerFederationProtos.java: At HdfsServerFederationProtos.java:[line 888] | |
Useless control flow in org.apache.hadoop.hdfs.federation.protocol.proto.HdfsServerFederationProtos$UpdateNamenodeRegistrationRequestProto$Builder.maybeForceBuilderInitialization() At HdfsServerFederationProtos.java: At HdfsServerFederationProtos.java:[line 8051] | |
Useless control flow in org.apache.hadoop.hdfs.federation.protocol.proto.HdfsServerFederationProtos$UpdateNamenodeRegistrationResponseProto$Builder.maybeForceBuilderInitialization() At HdfsServerFederationProtos.java: At HdfsServerFederationProtos.java:[line 8698] | |
Failed junit tests | hadoop.hdfs.server.federation.store.driver.TestStateStoreFileSystem |
hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010 | |
hadoop.hdfs.server.datanode.fsdataset.impl.TestSpaceReservation | |
hadoop.hdfs.server.namenode.TestReconstructStripedBlocks | |
hadoop.hdfs.server.federation.store.driver.TestStateStoreFile |
This message was automatically generated.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 15s | Docker mode activated. |
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 5 new or modified test files. |
+1 | mvninstall | 16m 56s | |
+1 | compile | 0m 49s | |
+1 | checkstyle | 0m 42s | |
+1 | mvnsite | 0m 54s | |
+1 | mvneclipse | 0m 16s | |
-1 | findbugs | 1m 44s | hadoop-hdfs-project/hadoop-hdfs in |
+1 | javadoc | 0m 41s | |
+1 | mvninstall | 0m 49s | the patch passed |
+1 | compile | 0m 46s | the patch passed |
+1 | cc | 0m 46s | the patch passed |
+1 | javac | 0m 47s | the patch passed |
-0 | checkstyle | 0m 38s | hadoop-hdfs-project/hadoop-hdfs: The patch generated 30 new + 402 unchanged - 0 fixed = 432 total (was 402) |
+1 | mvnsite | 0m 51s | the patch passed |
+1 | mvneclipse | 0m 11s | the patch passed |
-1 | whitespace | 0m 0s | The patch has 5 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
+1 | xml | 0m 2s | The patch has no ill-formed XML file. |
-1 | findbugs | 1m 52s | hadoop-hdfs-project/hadoop-hdfs generated 7 new + 10 unchanged - 0 fixed = 17 total (was 10) |
+1 | javadoc | 0m 41s | the patch passed |
-1 | unit | 67m 55s | hadoop-hdfs in the patch failed. |
+1 | asflicense | 0m 21s | The patch does not generate ASF License warnings. |
97m 52s |
Reason | Tests |
---|---|
FindBugs | module:hadoop-hdfs-project/hadoop-hdfs |
Useless control flow in org.apache.hadoop.hdfs.federation.protocol.proto.HdfsServerFederationProtos$FederationNamespaceInfoProto$Builder.maybeForceBuilderInitialization() At HdfsServerFederationProtos.java: At HdfsServerFederationProtos.java:[line 4573] | |
Useless control flow in org.apache.hadoop.hdfs.federation.protocol.proto.HdfsServerFederationProtos$GetExpiredRegistrationsRequestProto$Builder.maybeForceBuilderInitialization() At HdfsServerFederationProtos.java: At HdfsServerFederationProtos.java:[line 6424] | |
Useless control flow in org.apache.hadoop.hdfs.federation.protocol.proto.HdfsServerFederationProtos$GetNamespaceInfoRequestProto$Builder.maybeForceBuilderInitialization() At HdfsServerFederationProtos.java: At HdfsServerFederationProtos.java:[line 6762] | |
Useless control flow in org.apache.hadoop.hdfs.federation.protocol.proto.HdfsServerFederationProtos$NamenodeHeartbeatResponseProto$Builder.maybeForceBuilderInitialization() At HdfsServerFederationProtos.java: At HdfsServerFederationProtos.java:[line 9677] | |
Useless control flow in org.apache.hadoop.hdfs.federation.protocol.proto.HdfsServerFederationProtos$NamenodeMembershipStatsRecordProto$Builder.maybeForceBuilderInitialization() At HdfsServerFederationProtos.java: At HdfsServerFederationProtos.java:[line 888] | |
Useless control flow in org.apache.hadoop.hdfs.federation.protocol.proto.HdfsServerFederationProtos$UpdateNamenodeRegistrationRequestProto$Builder.maybeForceBuilderInitialization() At HdfsServerFederationProtos.java: At HdfsServerFederationProtos.java:[line 8051] | |
Useless control flow in org.apache.hadoop.hdfs.federation.protocol.proto.HdfsServerFederationProtos$UpdateNamenodeRegistrationResponseProto$Builder.maybeForceBuilderInitialization() At HdfsServerFederationProtos.java: At HdfsServerFederationProtos.java:[line 8698] | |
Failed junit tests | hadoop.hdfs.TestMaintenanceState |
hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped | |
hadoop.hdfs.server.namenode.ha.TestPipelinesFailover | |
hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 | |
hadoop.hdfs.TestDFSRSDefault10x4StripedOutputStreamWithFailure | |
hadoop.tools.TestHdfsConfigFields |
This message was automatically generated.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 19s | Docker mode activated. |
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 6 new or modified test files. |
+1 | mvninstall | 15m 0s | |
+1 | compile | 0m 48s | |
+1 | checkstyle | 0m 41s | |
+1 | mvnsite | 0m 53s | |
+1 | mvneclipse | 0m 15s | |
-1 | findbugs | 1m 44s | hadoop-hdfs-project/hadoop-hdfs in |
+1 | javadoc | 0m 40s | |
+1 | mvninstall | 0m 50s | the patch passed |
+1 | compile | 0m 46s | the patch passed |
+1 | cc | 0m 46s | the patch passed |
+1 | javac | 0m 46s | the patch passed |
-0 | checkstyle | 0m 38s | hadoop-hdfs-project/hadoop-hdfs: The patch generated 14 new + 402 unchanged - 0 fixed = 416 total (was 402) |
+1 | mvnsite | 0m 51s | the patch passed |
+1 | mvneclipse | 0m 11s | the patch passed |
-1 | whitespace | 0m 0s | The patch has 3 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
+1 | xml | 0m 3s | The patch has no ill-formed XML file. |
-1 | findbugs | 1m 49s | hadoop-hdfs-project/hadoop-hdfs generated 7 new + 10 unchanged - 0 fixed = 17 total (was 10) |
+1 | javadoc | 0m 40s | the patch passed |
-1 | unit | 71m 7s | hadoop-hdfs in the patch failed. |
+1 | asflicense | 0m 19s | The patch does not generate ASF License warnings. |
99m 1s |
Reason | Tests |
---|---|
FindBugs | module:hadoop-hdfs-project/hadoop-hdfs |
Useless control flow in org.apache.hadoop.hdfs.federation.protocol.proto.HdfsServerFederationProtos$FederationNamespaceInfoProto$Builder.maybeForceBuilderInitialization() At HdfsServerFederationProtos.java: At HdfsServerFederationProtos.java:[line 4573] | |
Useless control flow in org.apache.hadoop.hdfs.federation.protocol.proto.HdfsServerFederationProtos$GetExpiredRegistrationsRequestProto$Builder.maybeForceBuilderInitialization() At HdfsServerFederationProtos.java: At HdfsServerFederationProtos.java:[line 6424] | |
Useless control flow in org.apache.hadoop.hdfs.federation.protocol.proto.HdfsServerFederationProtos$GetNamespaceInfoRequestProto$Builder.maybeForceBuilderInitialization() At HdfsServerFederationProtos.java: At HdfsServerFederationProtos.java:[line 6762] | |
Useless control flow in org.apache.hadoop.hdfs.federation.protocol.proto.HdfsServerFederationProtos$NamenodeHeartbeatResponseProto$Builder.maybeForceBuilderInitialization() At HdfsServerFederationProtos.java: At HdfsServerFederationProtos.java:[line 9677] | |
Useless control flow in org.apache.hadoop.hdfs.federation.protocol.proto.HdfsServerFederationProtos$NamenodeMembershipStatsRecordProto$Builder.maybeForceBuilderInitialization() At HdfsServerFederationProtos.java: At HdfsServerFederationProtos.java:[line 888] | |
Useless control flow in org.apache.hadoop.hdfs.federation.protocol.proto.HdfsServerFederationProtos$UpdateNamenodeRegistrationRequestProto$Builder.maybeForceBuilderInitialization() At HdfsServerFederationProtos.java: At HdfsServerFederationProtos.java:[line 8051] | |
Useless control flow in org.apache.hadoop.hdfs.federation.protocol.proto.HdfsServerFederationProtos$UpdateNamenodeRegistrationResponseProto$Builder.maybeForceBuilderInitialization() At HdfsServerFederationProtos.java: At HdfsServerFederationProtos.java:[line 8698] | |
Failed junit tests | hadoop.hdfs.server.federation.resolver.TestNamenodeResolver |
hadoop.hdfs.TestDFSRSDefault10x4StripedOutputStreamWithFailure | |
Timed out junit tests | org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean |
This message was automatically generated.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 16s | Docker mode activated. |
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 6 new or modified test files. |
+1 | mvninstall | 13m 36s | |
+1 | compile | 0m 47s | |
+1 | checkstyle | 0m 38s | |
+1 | mvnsite | 0m 50s | |
+1 | mvneclipse | 0m 13s | |
-1 | findbugs | 1m 42s | hadoop-hdfs-project/hadoop-hdfs in |
+1 | javadoc | 0m 39s | |
+1 | mvninstall | 0m 46s | the patch passed |
+1 | compile | 0m 44s | the patch passed |
+1 | cc | 0m 44s | the patch passed |
+1 | javac | 0m 44s | the patch passed |
-0 | checkstyle | 0m 36s | hadoop-hdfs-project/hadoop-hdfs: The patch generated 4 new + 402 unchanged - 0 fixed = 406 total (was 402) |
+1 | mvnsite | 0m 48s | the patch passed |
+1 | mvneclipse | 0m 11s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | xml | 0m 2s | The patch has no ill-formed XML file. |
-1 | findbugs | 1m 59s | hadoop-hdfs-project/hadoop-hdfs generated 7 new + 10 unchanged - 0 fixed = 17 total (was 10) |
+1 | javadoc | 0m 40s | the patch passed |
-1 | unit | 69m 52s | hadoop-hdfs in the patch failed. |
-1 | asflicense | 0m 18s | The patch generated 1 ASF License warnings. |
96m 2s |
Reason | Tests |
---|---|
FindBugs | module:hadoop-hdfs-project/hadoop-hdfs |
Useless control flow in org.apache.hadoop.hdfs.federation.protocol.proto.HdfsServerFederationProtos$FederationNamespaceInfoProto$Builder.maybeForceBuilderInitialization() At HdfsServerFederationProtos.java: At HdfsServerFederationProtos.java:[line 4573] | |
Useless control flow in org.apache.hadoop.hdfs.federation.protocol.proto.HdfsServerFederationProtos$GetExpiredRegistrationsRequestProto$Builder.maybeForceBuilderInitialization() At HdfsServerFederationProtos.java: At HdfsServerFederationProtos.java:[line 6424] | |
Useless control flow in org.apache.hadoop.hdfs.federation.protocol.proto.HdfsServerFederationProtos$GetNamespaceInfoRequestProto$Builder.maybeForceBuilderInitialization() At HdfsServerFederationProtos.java: At HdfsServerFederationProtos.java:[line 6762] | |
Useless control flow in org.apache.hadoop.hdfs.federation.protocol.proto.HdfsServerFederationProtos$NamenodeHeartbeatResponseProto$Builder.maybeForceBuilderInitialization() At HdfsServerFederationProtos.java: At HdfsServerFederationProtos.java:[line 9677] | |
Useless control flow in org.apache.hadoop.hdfs.federation.protocol.proto.HdfsServerFederationProtos$NamenodeMembershipStatsRecordProto$Builder.maybeForceBuilderInitialization() At HdfsServerFederationProtos.java: At HdfsServerFederationProtos.java:[line 888] | |
Useless control flow in org.apache.hadoop.hdfs.federation.protocol.proto.HdfsServerFederationProtos$UpdateNamenodeRegistrationRequestProto$Builder.maybeForceBuilderInitialization() At HdfsServerFederationProtos.java: At HdfsServerFederationProtos.java:[line 8051] | |
Useless control flow in org.apache.hadoop.hdfs.federation.protocol.proto.HdfsServerFederationProtos$UpdateNamenodeRegistrationResponseProto$Builder.maybeForceBuilderInitialization() At HdfsServerFederationProtos.java: At HdfsServerFederationProtos.java:[line 8698] | |
Failed junit tests | hadoop.hdfs.server.blockmanagement.TestBlockManager |
hadoop.hdfs.server.namenode.ha.TestPipelinesFailover | |
hadoop.hdfs.server.balancer.TestBalancer | |
hadoop.hdfs.web.TestWebHdfsTimeouts | |
hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160 |
This message was automatically generated.
I only skimmed the PB translation classes, assuming those are correct. Only a few minor questions:
- MembershipNamenodeResolver#getMembershipStore can't return a partially-constructed object accessed by multiple threads, because the service init guarantees that it only receives the instance created by the state store? Through getRegisteredRecordStores, this holds no locks. addRecordStore could include a check that the service is still in the init state and/or the recordStores map could be unmodifiable at the end of init.
- In MembershipStoreImpl, the activeRegistrations and expiredRegistrations fields are protected by a r/w lock, except in the heartbeat handling when it's used for logging. Do either of these need to be a ConcurrentHashMap?
- EphemeralBaseRecord is kind of a confusing name for that type, but I don't have a better suggestion...
The patch could update findbugsExcludeFile.xml to prevent it from flagging generated code (under dev-support).
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 16s | Docker mode activated. |
Prechecks | |||
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 6 new or modified test files. |
|
|||
-1 | mvninstall | 7m 5s | root in |
-1 | compile | 0m 29s | hadoop-hdfs in |
+1 | checkstyle | 0m 41s | |
-1 | mvnsite | 0m 30s | hadoop-hdfs in |
-1 | findbugs | 0m 32s | hadoop-hdfs in |
+1 | javadoc | 0m 44s | |
Patch Compile Tests | |||
-1 | mvninstall | 0m 32s | hadoop-hdfs in the patch failed. |
-1 | compile | 0m 29s | hadoop-hdfs in the patch failed. |
-1 | cc | 0m 29s | hadoop-hdfs in the patch failed. |
-1 | javac | 0m 29s | hadoop-hdfs in the patch failed. |
-0 | checkstyle | 0m 42s | hadoop-hdfs-project/hadoop-hdfs: The patch generated 3 new + 403 unchanged - 0 fixed = 406 total (was 403) |
-1 | mvnsite | 0m 29s | hadoop-hdfs in the patch failed. |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | xml | 0m 4s | The patch has no ill-formed XML file. |
-1 | findbugs | 0m 32s | hadoop-hdfs in the patch failed. |
-1 | javadoc | 0m 42s | hadoop-hdfs-project_hadoop-hdfs generated 1 new + 9 unchanged - 2 fixed = 10 total (was 11) |
Other Tests | |||
-1 | unit | 0m 31s | hadoop-hdfs in the patch failed. |
+1 | asflicense | 0m 16s | The patch does not generate ASF License warnings. |
15m 57s |
This message was automatically generated.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 16s | Docker mode activated. |
Prechecks | |||
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 6 new or modified test files. |
|
|||
+1 | mvninstall | 15m 10s | |
+1 | compile | 0m 55s | |
+1 | checkstyle | 0m 42s | |
+1 | mvnsite | 1m 3s | |
-1 | findbugs | 1m 58s | hadoop-hdfs-project/hadoop-hdfs in |
+1 | javadoc | 0m 46s | |
Patch Compile Tests | |||
-1 | mvninstall | 0m 32s | hadoop-hdfs in the patch failed. |
-1 | compile | 0m 31s | hadoop-hdfs in the patch failed. |
-1 | cc | 0m 31s | hadoop-hdfs in the patch failed. |
-1 | javac | 0m 31s | hadoop-hdfs in the patch failed. |
-0 | checkstyle | 0m 46s | hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 403 unchanged - 0 fixed = 404 total (was 403) |
-1 | mvnsite | 0m 34s | hadoop-hdfs in the patch failed. |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | xml | 0m 5s | The patch has no ill-formed XML file. |
-1 | findbugs | 0m 30s | hadoop-hdfs in the patch failed. |
-1 | javadoc | 0m 46s | hadoop-hdfs-project_hadoop-hdfs generated 1 new + 9 unchanged - 0 fixed = 10 total (was 9) |
Other Tests | |||
-1 | unit | 0m 32s | hadoop-hdfs in the patch failed. |
+1 | asflicense | 0m 15s | The patch does not generate ASF License warnings. |
26m 52s |
This message was automatically generated.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 18s | Docker mode activated. |
Prechecks | |||
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 6 new or modified test files. |
|
|||
+1 | mvninstall | 17m 35s | |
+1 | compile | 0m 56s | |
+1 | checkstyle | 0m 46s | |
+1 | mvnsite | 1m 5s | |
-1 | findbugs | 1m 57s | hadoop-hdfs-project/hadoop-hdfs in |
+1 | javadoc | 0m 47s | |
Patch Compile Tests | |||
+1 | mvninstall | 0m 59s | the patch passed |
+1 | compile | 0m 54s | the patch passed |
+1 | cc | 0m 54s | the patch passed |
+1 | javac | 0m 54s | the patch passed |
-0 | checkstyle | 0m 40s | hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 403 unchanged - 0 fixed = 404 total (was 403) |
+1 | mvnsite | 0m 57s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | xml | 0m 4s | The patch has no ill-formed XML file. |
+1 | findbugs | 1m 56s | the patch passed |
+1 | javadoc | 0m 43s | the patch passed |
Other Tests | |||
-1 | unit | 89m 35s | hadoop-hdfs in the patch failed. |
+1 | asflicense | 0m 19s | The patch does not generate ASF License warnings. |
121m 2s |
Reason | Tests |
---|---|
Failed junit tests | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 |
hadoop.hdfs.server.namenode.TestReconstructStripedBlocks | |
hadoop.hdfs.TestRollingUpgrade | |
hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy |
This message was automatically generated.
- The FindBugs don't seem related to the patch.
- The checkstyle is for number of parameters, I would ignore it.
- I don't think the failed unit tests are related to this patch?
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13045 (See https://builds.apache.org/job/Hadoop-trunk-Commit/13045/)
HDFS-10687. Federation Membership State Store internal API. Contributed (inigoiri: rev 55da7fd7ebe2f3fa1c1c828dda727fddc75a1b81)
- (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/store/records/impl/pb/MembershipStatsPBImpl.java
- (edit) hadoop-hdfs-project/hadoop-hdfs/pom.xml
- (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/store/protocol/impl/pb/GetNamenodeRegistrationsResponsePBImpl.java
- (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/store/impl/MembershipStoreImpl.java
- (add) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/federation/store/TestStateStoreBase.java
- (edit) hadoop-hdfs-project/hadoop-hdfs/dev-support/findbugsExcludeFile.xml
- (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/federation/store/FederationStateStoreTestUtils.java
- (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/store/records/MembershipStats.java
- (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/store/protocol/GetNamenodeRegistrationsRequest.java
- (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/store/protocol/impl/pb/GetNamenodeRegistrationsRequestPBImpl.java
- (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/store/StateStoreCacheUpdateService.java
- (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/store/protocol/GetNamespaceInfoRequest.java
- (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
- (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
- (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/resolver/MembershipNamenodeResolver.java
- (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/store/records/impl/pb/MembershipStatePBImpl.java
- (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/store/impl/package-info.java
- (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/store/records/MembershipState.java
- (add) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/federation/store/TestStateStoreMembershipState.java
- (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/store/protocol/impl/pb/GetNamespaceInfoRequestPBImpl.java
- (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/store/protocol/impl/pb/GetNamespaceInfoResponsePBImpl.java
- (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/store/protocol/impl/pb/UpdateNamenodeRegistrationRequestPBImpl.java
- (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/store/protocol/impl/pb/UpdateNamenodeRegistrationResponsePBImpl.java
- (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/federation/store/driver/TestStateStoreDriverBase.java
- (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/store/StateStoreCache.java
- (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/store/protocol/impl/pb/NamenodeHeartbeatResponsePBImpl.java
- (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/store/protocol/impl/pb/NamenodeHeartbeatRequestPBImpl.java
- (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/store/protocol/NamenodeHeartbeatRequest.java
- (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/store/protocol/NamenodeHeartbeatResponse.java
- (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/store/MembershipStore.java
- (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/store/StateStoreService.java
- (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/store/CachedRecordStore.java
- (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/store/protocol/UpdateNamenodeRegistrationRequest.java
- (add) hadoop-hdfs-project/hadoop-hdfs/src/main/proto/FederationProtocol.proto
- (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/store/protocol/GetNamenodeRegistrationsResponse.java
- (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/store/protocol/impl/pb/FederationProtocolPBTranslator.java
- (add) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/federation/store/records/TestMembershipState.java
- (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/router/FederationUtil.java
- (add) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/federation/resolver/TestNamenodeResolver.java
- (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/store/protocol/GetNamespaceInfoResponse.java
- (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/store/protocol/UpdateNamenodeRegistrationResponse.java
- (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/store/protocol/impl/pb/package-info.java
jakace elgoiri I am trying to understand the EXPIRE process in MembershipState in router and please correct me if I am understanding wrong.
- MembershipState and RouterState seem always using caching through CachedRecordStore and loading their data from the actual StateStoreDriver.
- StateStoreDriver doesn't actually run threads directly to update the state of membership to EXPIRE, instead, it relies on when the loadCache is called, the caching will check the expiration and update the state store.
Thanks!
fengnanli, you are correct, that's the approach.
Do you see a need to have an updater?
After discussion with subru, we thought that it was best to match the code of YARN federation which uses protobuf for serialization. This was also the first patch in
YARN-2915so I will try to push first for this one.