Uploaded image for project: 'Nutch'
  1. Nutch
  2. NUTCH-1499

Usage of multiple ipv4 addresses and network cards on fetcher machines

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Minor
    • Resolution: Won't Fix
    • 1.5.1
    • 1.7
    • fetcher
    • None
    • Patch Available

    Description

      Adds for the fetcher threads the ability to use multiple configured ipv4 addresses.

      On some cluster machines there are several ipv4 addresses configured where each ip address is associated with its own network interface.

      This patch enables to configure the protocol-http and the protocol-httpclient to use these network interfaces in a round robin style.

      If the feature is enabled, a helper class reads at startup the network configuration. In each http network connection the next ip address is taken. This method is synchronized, but this should be no bottleneck for the overall performance of the fetcher threads.

      This feature is tested on our cluster for the protocol-http and the protocol-httpclient protocol.

      Attachments

        Activity

          People

            Unassigned Unassigned
            wal Walter Tietze
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: