List of settings that default to ephemeral ports:

  public static final String  DFS_NAMENODE_SECONDARY_HTTP_ADDRESS_DEFAULT = "0.0.0.0:50090";
  public static final String  DFS_NAMENODE_SECONDARY_HTTPS_ADDRESS_DEFAULT = "0.0.0.0:50091";
  public static final int     DFS_DATANODE_DEFAULT_PORT = 50010;
  public static final int     DFS_DATANODE_HTTP_DEFAULT_PORT = 50075;
  public static final int     DFS_DATANODE_HTTPS_DEFAULT_PORT = 50475;
  public static final int     DFS_DATANODE_IPC_DEFAULT_PORT = 50020;

Changing these values for 2.x will be incompatible since we'd break deployments on upgrade that rely on defaults. Thanks to Josh Elser for reporting this.

This issue has been a pet peeve of mine for years. But every time this topic comes up, someone complains that this makes testing on a single node hard. I really, really, really hope we're over that hump and we can not only remove the default of being ephemeral but also set the ports to be a contiguous range.

+1, let's make this happen. Seems long overdue to me.

Following are also falls in same range

  int     DFS_NAMENODE_HTTP_PORT_DEFAULT = 50070;
  String  DFS_NAMENODE_HTTP_ADDRESS_KEY = "dfs.namenode.http-address";
  int     DFS_NAMENODE_HTTPS_PORT_DEFAULT = 50470;
  String  DFS_NAMENODE_HTTPS_ADDRESS_KEY = "dfs.namenode.https-address";

How about changing into something like below? Just replace 50 with 9

Namenode ports
----------------
50070 --> 9070
50470 --> 9470

Datanode ports
---------------
50010 --> 9010
50020 --> 9020
50075 --> 9075
50475 --> 9475

Secondary NN ports
---------------
50090 --> 9090
50091 --> 9091

HBASE-10123 could be useful as reference.

Thanks Masatake Iwasaki for the pointer. The numbers started with 9000 suggested by Vinayakumar B are safe on various OS according to Jonathan Hsieh HBASE-10123 comment.

There are two more

final String  DFS_NAMENODE_BACKUP_ADDRESS_DEFAULT = "localhost:50100";
final String  DFS_NAMENODE_BACKUP_HTTP_ADDRESS_DEFAULT = "0.0.0.0:50105";

Posted initial patch for review, thanks. Will make chances to keep comments/docs consistent in upcoming patches.

Here's another to consider: the hadoop-kms port. It was brought up on HBASE-10123 that it uses 16000 as default (and clashes with hbase currently)

this line indicates that 16000 is the default – likely a search for that ENV variable will find where it is set by default in the code or confs.
https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-kms/src/main/conf/kms-env.sh#L25

How about changing into something like below? Just replace 50 with 9

No. Let's actually move the ports to be contiguous instead of all over the place. It's extremely sloppy and highlights the community's inability to plan to have these giant gaps all over the place.

001 changed comments/docs/sources that were inconsistent.

-1 overall

This message was automatically generated.

-1 overall

This message was automatically generated.

Thanks Allen Wittenauer, do you think if it's good to start the contiguous range from 9000 like I did in the patches posted?

V002 changed ports into contiguous range as:

Namenode ports
----------------
50070 --> 9070
50470 --> 9071

Secondary NN ports
---------------
50090 --> 9072
50091 --> 9073

backup NN ports
---------------
50100 --> 9074
50105 --> 9075

Datanode ports
---------------
50010 --> 9076
50020 --> 9077
50075 --> 9078
50475 --> 9079

-1 overall

This message was automatically generated.

The v002 patch lgtm. I'll postpone committing at least 3-4 days in case there are additional comments.

9070-9079 are unassigned so this looks like a good range to use.

I know nothing about KMS so not sure if it is safe to change the port number as Jonathan Hsieh proposed. I'll file a separate Jira for that and we can address the ephemeral ports here.

Great idea, but can we please not use port 9075? This port is the default HTrace RPC port.

I know nothing about KMS so not sure if it is safe to change the port number as Jonathan Hsieh proposed. I'll file a separate Jira for that and we can address the ephemeral ports here.

Looks like there's already HADOOP-12811 created for this. I can follow up on there and fix the kms default port as well.
I don't think there's extra problems on changing the port on KMS, except for compatibility.

Thanks for the pointer Xiao Chen.

Xiaobing Zhou, do you want to update the patch to free up port 9075?

Thanks everyone for the feedback.

I posted V003 with the mapping:

Namenode ports 
---------------- 
50070 --> 9803
50470 --> 9804 
Secondary NN ports 
--------------- 
50090 --> 9805
50091 --> 9806 
backup NN ports 
--------------- 
50100 --> 9807
50105 --> 9808 
Datanode ports 
--------------- 
50010 --> 9809
50020 --> 9810
50075 --> 9811
50475 --> 9812	

9803-9874 is a wider range unassigned, any upcoming ports can be fit into this range.

Thanks, Xiaobing Zhou. Looks great.

-1 overall

This message was automatically generated.

So at the risk of bike-shedding this - I think the 907x port range is easier to remember.

Xiaobing Zhou - I'd just eliminate the backup node changes form your v2 patch which frees up port 9075 (BN is as good as unsupported).

Arpit Agarwal here is the new mappings to keep 9870 as being easily remembered. Thanks all for review.

Namenode ports 
---------------- 
50470 --> 9871
50070 --> 9870

Secondary NN ports 
--------------- 
50091 --> 9869
50090 --> 9868

Datanode ports 
--------------- 
50020 --> 9867
50010 --> 9866
50475 --> 9865
50075 --> 9864

I also removed changes of backup NN ports.

-1 overall

This message was automatically generated.

I haven't looked at the patch in detail yet but I am +1 on the assignments (thanks for the nice summary).

I will hold off committing until late next week at least to give the Jira watchers a chance to comment.

Thanks Xiaobing Zhou for working on this.

Have we considered about moving NN's RPC port to the same range? Now NN HTTP/HTTPS are on 9870/9871, while RPC on 8020. DN has HTTP/HTTPS on 9864/9865 while IPC on 9867. I think this jira is a good chance to bring the NN RPC to the same range. Please let me know your thoughts.

Nit: I see one reference of 50070 in CommandsManual.md in an example, could you please update that?

I posted patch v005 your comments, thanks Xiao Chen for review. Here's mapping in the following.

I intentionally have 9820 being not strictly contiguous with other NN ports in a sense people easily remember 9820 as they are more familiar with 8020 thanks to Arpit Agarwal pointed me based on such a spirit.

I didn't see any 50070 reference in CommandsManual.md, which's already fixed in the patches.

Namenode ports
----------------
50470 --> 9871
50070 --> 9870
8020 --> 9820

Secondary NN ports
---------------
50091 --> 9869
50090 --> 9868

Datanode ports
---------------
50020 --> 9867
50010 --> 9866
50475 --> 9865
50075 --> 9864

Thanks Xiaobing Zhou for the new rev, LGTM. +1 (non-binding).

-1 overall

This message was automatically generated.

I verified the test failures are not related to the patch. Thank you everyone for review.

Could anyone help to commit this? Thanks.

Thanks Xiaobing for pushing this forward. We need a committer +1 as well.

+1. I will commit the patch shortly.

I've committed this to trunk. Thanks Xiaobing Zhou for the contribution! Thanks all for the discussion and review.

Thank you Jing Zhao for +1ing and committing it. Thank you all for review.

FAILURE: Integrated in Hadoop-trunk-Commit #9659 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9659/)
HDFS-9427. HDFS should not default to ephemeral ports. Contributed by (jing9: rev 63e5412f1aef5e17e15da69d646181c629fc8682)

• hadoop-common-project/hadoop-common/src/site/markdown/SingleCluster.md.vm
• hadoop-tools/hadoop-openstack/src/site/markdown/index.md
• hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSHighAvailabilityWithNFS.md
• hadoop-common-project/hadoop-common/src/site/markdown/ClusterSetup.md
• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/main/java/org/apache/hadoop/registry/client/types/AddressTypes.java
• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/api/protocolrecords/impl/pb/TestPBRecordImpl.java
• hadoop-tools/hadoop-distcp/src/site/markdown/DistCp.md.vm
• hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestFileInputFormat.java
• hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestDelegateToFileSystem.java
• hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsUserGuide.md
• hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestOptionsParser.java
• hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java
• hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestQuota.java
• hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/resources/job_1329348432655_0001_conf.xml
• hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsDesign.md
• hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/GenericOptionsParser.java
• hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestPendingDataNodeMessages.java
• hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/HdfsClientConfigKeys.java
• hadoop-common-project/hadoop-common/src/site/markdown/SecureMode.md
• hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ViewFs.md
• hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAllowFormat.java
• hadoop-common-project/hadoop-common/src/site/markdown/filesystem/filesystem.md
• hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestFileInputFormat.java
• hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReplaceDatanodeOnFailure.java
• hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUtil.java
• hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/TestSshFenceByTcpPort.java
• hadoop-common-project/hadoop-common/src/site/markdown/Tracing.md
• hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestRequestHedgingProxyProvider.java
• hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java
• hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockPoolManager.java
• hadoop-common-project/hadoop-common/src/site/markdown/CommandsManual.md
• hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSHighAvailabilityWithQJM.md
• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/test/java/org/apache/hadoop/registry/RegistryTestHelper.java
• hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/JMXGet.java
• hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
• hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/resources/job_0.23.9-FAILED.jhist
• hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientFailover.java
• hadoop-hdfs-project/hadoop-hdfs/src/test/resources/hdfs-site.malformed.xml
• hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
• hadoop-tools/hadoop-sls/src/main/data/2jobs2min-rumen-jh.json