Details
Minor Description
While I do appreciate that the HA and federated setups do take care of some of the issues running in a multi node distributed fault tolerant way I believe that a singe name node is a bit of a pain point for me.
My wish, and your mileage may vary, is that here could be a name node on each data node system so that each system could act as a gateway and worker for the cluster.
In my tests right now I can push nearly 2 million records / second through a single node instance, but when I bring up 4 more nodes it doesn't go 4x faster as it has to retrieve from the name node and go through it as a gateway to the cluster. For speed and resiliency if this could be spread out among n number of nodes and put behind a load balancer it would be the ideal solution for distributed resilient and high throughput shared storage.