Uploaded image for project: 'Infrastructure'
  1. Infrastructure
  2. INFRA-14163

Networking issue from sa-vm1 to sa-update.secnap.net

    Details

    • Type: Task
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Fix Version/s: None
    • Component/s: Other/Misc
    • Labels:
      None
    • Project:
      SpamAssassin

      Description

      This is one of the SA mirrors that we monitor with a local script on sa-vm1 to make sure we can connect to port 80 and do basic verification of content to know if it's down.

      The odd thing is our rsyncd logs show that this server is able to connect to sa-vm1 so this may be a routing problem in one direction.

      Several of the SA sysadmins are able to connect to port 80 from other Internet locations so I think we are going to need to get a traceroute from the other end which I will request from security@secnap.net and paste into this ticket shortly.

      root@sa-vm1:/var/log# traceroute sa-update.secnap.net
      traceroute to sa-update.secnap.net (204.89.241.6), 30 hops max, 60 byte packets
       1 garl.apache.org (163.172.22.164) 0.100 ms 0.091 ms 0.084 ms
       2 163-172-22-1.rev.poneytelecom.eu (163.172.22.1) 0.315 ms 0.410 ms 0.455 ms
       3 195.154.1.226 (195.154.1.226) 0.732 ms 195.154.1.228 (195.154.1.228) 0.709 ms 195.154.1.226 (195.154.1.226) 0.887 ms
       4 * lag-110.ear3.Paris1.Level3.net (212.3.235.197) 1.321 ms 1.510 ms
       5 NTT-level3-100G.Paris1.Level3.net (4.68.73.66) 1.713 ms 1.616 ms 1.762 ms
       6 NTT-level3-100G.Paris1.Level3.net (4.68.73.66) 1.754 ms ae-2.r25.londen12.uk.bb.gin.ntt.net (129.250.6.13) 8.731 ms 8.075 ms
       7 ae-2.r25.londen12.uk.bb.gin.ntt.net (129.250.6.13) 8.724 ms ae-1.r24.londen12.uk.bb.gin.ntt.net (129.250.2.26) 8.422 ms 8.389 ms
       8 ae-5.r24.nycmny01.us.bb.gin.ntt.net (129.250.2.18) 86.796 ms 86.861 ms ae-1.r24.londen12.uk.bb.gin.ntt.net (129.250.2.26) 8.486 ms
       9 ae-5.r24.nycmny01.us.bb.gin.ntt.net (129.250.2.18) 86.834 ms ae-1.r25.nycmny01.us.bb.gin.ntt.net (129.250.3.207) 86.074 ms 86.353 ms
      10 ae-9.r22.asbnva02.us.bb.gin.ntt.net (129.250.2.149) 86.263 ms ae-1.r25.nycmny01.us.bb.gin.ntt.net (129.250.3.207) 83.881 ms 79.049 ms
      11 ae-9.r22.asbnva02.us.bb.gin.ntt.net (129.250.2.149) 92.510 ms 91.865 ms 86.011 ms
      12 ae-1.r20.miamfl02.us.bb.gin.ntt.net (129.250.2.87) 117.062 ms 112.294 ms ae-0.r23.asbnva02.us.bb.gin.ntt.net (129.250.3.85) 86.724 ms
      13 ae-1.r20.miamfl02.us.bb.gin.ntt.net (129.250.2.87) 113.779 ms 118.493 ms ae-1.r05.miamfl02.us.bb.gin.ntt.net (129.250.2.185) 112.849 ms
      14 ae-2.a01.miamfl02.us.bb.gin.ntt.net (129.250.3.167) 113.289 ms 113.269 ms ae-1.r05.miamfl02.us.bb.gin.ntt.net (129.250.2.185) 117.155 ms
      15 ae-2.a01.miamfl02.us.bb.gin.ntt.net (129.250.3.167) 116.922 ms xe-0-0-24-0.a01.miamfl02.us.ce.gin.ntt.net (157.238.179.66) 117.944 ms ae-2.a01.miamfl02.us.bb.gin.ntt.net (129.250.3.167) 113.677 ms
      16 xe-0-0-24-0.a01.miamfl02.us.ce.gin.ntt.net (157.238.179.66) 113.994 ms te2-4.dist02.fll.peak10.net (96.46.240.62) 104.961 ms xe-0-0-24-0.a01.miamfl02.us.ce.gin.ntt.net (157.238.179.66) 118.743 ms
      17 te2-5.dist01.fll.peak10.net (96.46.240.54) 113.752 ms 114.435 ms *
      18 * * *
      19 * * *
      20 * * *
      21 * * *
      22 * * *
      23 * * *
      24 * * *
      25 * * *
      26 * * *
      27 * * *
      28 * * *
      29 * * *
      30 * * *

        Activity

        Hide
        pono Daniel Takamori added a comment -
        This looks like it's getting past our side of the network, have you emailed the NOC on the otherside?
        Show
        pono Daniel Takamori added a comment - This looks like it's getting past our side of the network, have you emailed the NOC on the otherside?
        Hide
        davej Dave Jones added a comment - Reporter
        I have emailed security@secnap.net to get a traceroute from their side. I have seen this before when there is asymmetrical routing in one direction. It might not be an issue with the Apache Infra networking but one or two hops out.
        Show
        davej Dave Jones added a comment - Reporter I have emailed security@secnap.net to get a traceroute from their side. I have seen this before when there is asymmetrical routing in one direction. It might not be an issue with the Apache Infra networking but one or two hops out.
        Hide
        davej Dave Jones added a comment - Reporter
        Here is the requested traceroute:
        root@sa-update:~# traceroute sa-vm1.apache.org
        traceroute to sa-vm1.apache.org (62.210.60.231), 30 hops max, 60 byte packets
         1 10.70.0.1 (10.70.0.1) 0.447 ms 0.419 ms 0.415 ms
         2 204.89.241.1 (204.89.241.1) 1.313 ms 1.301 ms 1.287 ms
         3 te0-0-0-1.edge02.fll.peak10.net (96.46.240.61) 1.766 ms 1.852 ms 1.732 ms
         4 xe-0-0-24-0.a01.miamfl02.us.bb.gin.ntt.net (157.238.179.65) 2.723 ms 2.708 ms 2.688 ms
         5 ae-5.r04.miamfl02.us.bb.gin.ntt.net (129.250.3.209) 2.728 ms 2.710 ms 2.644 ms
         6 mai-b1-link.telia.net (213.248.81.62) 2.676 ms 2.577 ms 2.507 ms
         7 ash-bb4-link.telia.net (62.115.141.80) 27.015 ms ash-bb3-link.telia.net (62.115.143.64) 28.288 ms ash-bb3-link.telia.net (62.115.143.68) 27.767 ms
         8 prs-bb3-link.telia.net (80.91.252.37) 129.363 ms prs-bb2-link.telia.net (80.91.251.103) 118.946 ms prs-bb2-link.telia.net (62.115.124.163) 191.637 ms
         9 prs-b8-link.telia.net (62.115.118.79) 121.069 ms prs-b8-link.telia.net (62.115.118.97) 119.000 ms prs-b8-link.telia.net (62.115.118.55) 121.047 ms
        10 online-ic-315748-prs-b8.c.telia.net (62.115.63.94) 113.887 ms 113.970 ms 113.470 ms
        11 195.154.1.229 (195.154.1.229) 113.694 ms 106.467 ms 113.484 ms
        12 * * *
        13 * * *
        14 * * *
        15 * * *
        16 * * *
        17 * * *
        18 * * *
        19 * * *
        20 * * *
        21 * * *
        22 * * *
        23 * * *
        24 * * *
        25 * * *
        26 * * *
        27 * * *
        28 * * *
        29 * * *
        30 * * *
        root@sa-update:~#
        Show
        davej Dave Jones added a comment - Reporter Here is the requested traceroute: root@sa-update :~# traceroute sa-vm1.apache.org traceroute to sa-vm1.apache.org (62.210.60.231), 30 hops max, 60 byte packets  1 10.70.0.1 (10.70.0.1) 0.447 ms 0.419 ms 0.415 ms  2 204.89.241.1 (204.89.241.1) 1.313 ms 1.301 ms 1.287 ms  3 te0-0-0-1.edge02.fll.peak10.net (96.46.240.61) 1.766 ms 1.852 ms 1.732 ms  4 xe-0-0-24-0.a01.miamfl02.us.bb.gin.ntt.net (157.238.179.65) 2.723 ms 2.708 ms 2.688 ms  5 ae-5.r04.miamfl02.us.bb.gin.ntt.net (129.250.3.209) 2.728 ms 2.710 ms 2.644 ms  6 mai-b1-link.telia.net (213.248.81.62) 2.676 ms 2.577 ms 2.507 ms  7 ash-bb4-link.telia.net (62.115.141.80) 27.015 ms ash-bb3-link.telia.net (62.115.143.64) 28.288 ms ash-bb3-link.telia.net (62.115.143.68) 27.767 ms  8 prs-bb3-link.telia.net (80.91.252.37) 129.363 ms prs-bb2-link.telia.net (80.91.251.103) 118.946 ms prs-bb2-link.telia.net (62.115.124.163) 191.637 ms  9 prs-b8-link.telia.net (62.115.118.79) 121.069 ms prs-b8-link.telia.net (62.115.118.97) 119.000 ms prs-b8-link.telia.net (62.115.118.55) 121.047 ms 10 online-ic-315748-prs-b8.c.telia.net (62.115.63.94) 113.887 ms 113.970 ms 113.470 ms 11 195.154.1.229 (195.154.1.229) 113.694 ms 106.467 ms 113.484 ms 12 * * * 13 * * * 14 * * * 15 * * * 16 * * * 17 * * * 18 * * * 19 * * * 20 * * * 21 * * * 22 * * * 23 * * * 24 * * * 25 * * * 26 * * * 27 * * * 28 * * * 29 * * * 30 * * * root@sa-update :~#
        Hide
        davej Dave Jones added a comment - Reporter
        I don't think this is an issue on the ASF side. We should be able to close this ticket soon.

        To: SECNAP Network Security <support@secnap.com>

        More information. I ran this past one of our networking gurus at my day job and he noticed something. I am able to ping 204.89.241.1.

        root@sa-vm1:# ping 204.89.241.1
        PING 204.89.241.1 (204.89.241.1) 56(84) bytes of data.
        64 bytes from 204.89.241.1: icmp_seq=1 ttl=246 time=105 ms
        64 bytes from 204.89.241.1: icmp_seq=2 ttl=246 time=105 ms
        64 bytes from 204.89.241.1: icmp_seq=3 ttl=246 time=105 ms
        64 bytes from 204.89.241.1: icmp_seq=4 ttl=246 time=105 ms

        Since the 204.89.241.0/24 network is yours, this means the traffic is getting into your network and this is not a routing problem.

        A working connection goes through 204.89.241.175 just before the final destination of 204.89.241.6. My guess is there is some filtering happening on the 204.89.241.175 device.

        The purpose of this sa-update.secnap.net server should be a world-wide mirror for SpamAssassin updates.

        Thanks,
        Dave
        Show
        davej Dave Jones added a comment - Reporter I don't think this is an issue on the ASF side. We should be able to close this ticket soon. To: SECNAP Network Security < support@secnap.com > More information. I ran this past one of our networking gurus at my day job and he noticed something. I am able to ping 204.89.241.1. root@sa-vm1 :# ping 204.89.241.1 PING 204.89.241.1 (204.89.241.1) 56(84) bytes of data. 64 bytes from 204.89.241.1: icmp_seq=1 ttl=246 time=105 ms 64 bytes from 204.89.241.1: icmp_seq=2 ttl=246 time=105 ms 64 bytes from 204.89.241.1: icmp_seq=3 ttl=246 time=105 ms 64 bytes from 204.89.241.1: icmp_seq=4 ttl=246 time=105 ms Since the 204.89.241.0/24 network is yours, this means the traffic is getting into your network and this is not a routing problem. A working connection goes through 204.89.241.175 just before the final destination of 204.89.241.6. My guess is there is some filtering happening on the 204.89.241.175 device. The purpose of this sa-update.secnap.net server should be a world-wide mirror for SpamAssassin updates. Thanks, Dave
        Hide
        davej Dave Jones added a comment - Reporter
        This issue is resolved. It was a networking issue on the other end. I think they found an ACL that they didn't expect and was inadvertently blocking some sources.
        Show
        davej Dave Jones added a comment - Reporter This issue is resolved. It was a networking issue on the other end. I think they found an ACL that they didn't expect and was inadvertently blocking some sources.

          People

          • Assignee:
            Unassigned
            Reporter:
            davej Dave Jones
            Request participants:
            None
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: