[NBLUG/talk] Strange ethernet issue

Steve Johnson fratm at adnd.com
Tue Apr 10 16:01:43 PDT 2012


Update :

I shared the strange MAC address with our router tech, and he
confirmed it is the address of the firewalls DMZ interface (These
boxes are on the DMZ) and when I told him the time line that this
started happening he confirmed that they did a firmware update around
that time, but did not change any configs.  He thinks a config option
may have been added or changed, or there is a bug in the firmware.

Thankfully no rogue machines..  So it's in the network techs hands now.

Thank you all for your help on this, I was starting to feel like I was
going to go crazy ha.   Thanks again, Kyle, David and Michael.  You
all rock.

I've been maintaining Linux boxes since the early 90s and I have never
had this happen before.  It's always fun (after the fact) when you
discover new things :)

-Steve


On Tue, Apr 10, 2012 at 3:55 PM, Steve Johnson <fratm at adnd.com> wrote:
> Okay, that makes sense.. Using arping, I am getting two repsonses back
> with different Mac Addresses, I just called our router tech and he is
> going to look up the Mac address of the router.
>
> I think we may be onto something here.
>
> Thanks a tone for your input.
>
> Oh and there was no arm twisting, I questioned it to understand it,
> not to doubt it :)
>
> I'll post what I find out when I hear from the tech.
>
>
> -Steve
>
>
> On Tue, Apr 10, 2012 at 3:33 PM, Kyle Rankin <kyle at nblug.org> wrote:
>> On Tue, Apr 10, 2012 at 03:20:26PM -0700, Steve Johnson wrote:
>>> I just don't see the point in the arp query when I am sitting in the
>>> same room as all the gear and I can see what is plugged into the
>>> switch.  Is there a point that I am missing?
>>>
>>> -Steve
>>
>> Your server has trouble allocating its IPs when connected to the network
>> because it does an ARP check beforehand and gets a reply back that another
>> MAC address already has those IPs. When you unplug the host, those ARP
>> queries never go out or come back, so it goes ahead and assigns the IPs.
>> This makes it seem quite likely there is another machine on the network
>> replying back to those ARP queries that it has those IPs.
>>
>> What you are testing is not what's plugged in or not or what you can see
>> physically, but whether there is a /different/ device on your network that
>> claims it owns those IPs. I suspect your switch (and hopefully not a rogue
>> server) is misconfigured and claiming to own those IPs to anyone else that
>> asks. An arp query from a second machine on the same subnet /might/ reveal
>> if this is the case because the MAC address you get back won't match the
>> MAC address for the first machine. Alternatively, it might be a race
>> condition where your host /and/ the other host both send ARP replies
>> back (that's something tcpdump would reveal).
>>
>> I'm just bringing this up because I've seen a misconfigured switch do this
>> before. I mean I won't twist your arm, but it's a quick and safe test.
>>
>> -Kyle
>>
>>>
>>>
>>> On Tue, Apr 10, 2012 at 3:17 PM, Kyle Rankin <kyle at nblug.org> wrote:
>>> > On Tue, Apr 10, 2012 at 09:59:46AM -0700, Steve Johnson wrote:
>>> >> Yes, I physically checked the switch (Cisco switch.. not sure on
>>> >> model).. Also just to be sure I ran mmap on the IP of one the boxes
>>> >> when it was down and nothing came back.  I know pings are unreliable,
>>> >> but nmap isn't supposed to use just IMCP, so it should have detected
>>> >> something if someone got on my network.
>>> >>
>>> >> I will try the arp queries after tonights reboot.. These machines are
>>> >> production machines, so can't be down long in the middle of the day..
>>> >> :)
>>> >>
>>> >> -Steve
>>> >
>>> > Even if the machine is up, you might get interesting information from an
>>> > arp query from a different host on the same subnet. Perform the arp query
>>> > from a different host and confirm that you get back the MAC address you
>>> > expect.
>>> >
>>> > -Kyle
>>> >
>>> >>
>>> >>
>>> >> On Tue, Apr 10, 2012 at 9:54 AM, Kyle Rankin <kyle at nblug.org> wrote:
>>> >> > On Tue, Apr 10, 2012 at 09:40:31AM -0700, Steve Johnson wrote:
>>> >> >> Hi Guys,
>>> >> >>
>>> >> >> I am running 3 linux boxes all on the same network, running static 10
>>> >> >> net addresses, each on their own IP address..  A strange thing has
>>> >> >> started happening about a month ago, if I reboot the box when the
>>> >> >> system comes up at the point when it tries to bring up the eth0
>>> >> >> interface I get an error "IP Address in use by another host" and then
>>> >> >> the interface does not come up.  Loggin in from the console as root
>>> >> >> and running ifup eth0 gives me the same error.  The only way I can get
>>> >> >> the interface to come up is to physically unplug the ethernet, then
>>> >> >> run ifup eth0, that brings up the eth0 correctly, and then plug the
>>> >> >> ethernet cable back in.. Then it runs fine until another reboot (Or if
>>> >> >> I ifdown eth0 I will have the same problem)..
>>> >> >>
>>> >> > <snip>
>>> >> >>
>>> >> >> Ay ideas, or clues would be greatly appreciated.. I've been trying to
>>> >> >> trouble shoot this for over a month now with now luck.
>>> >> >>
>>> >> >> -Steve
>>> >> >>
>>> >> >
>>> >> > Are you absolutely sure that only one host truly has those IP addresses on
>>> >> > that subnet? When the host comes up and tries to assign the IP addresses to
>>> >> > itself, it will first perform an ARP and see if another MAC address on the
>>> >> > network claims to have that IP. What I would do is take down one of the
>>> >> > hosts, then from a different machine run ARP queries for those 10 IPs
>>> >> > belonging to the first host and see if the MAC address you get back is the
>>> >> > correct one. If your networking guys are trying to do anything fancy with
>>> >> > NAT and misconfigured something, it could be that your switch is claiming
>>> >> > to have those IPs (it's easy to check, an arp query against one of the IPs
>>> >> > will return back a MAC belonging to a Cisco, HP, or whatever switch you
>>> >> > have).
>>> >> >
>>> >> > --
>>> >> > Kyle Rankin
>>> >> > NBLUG President
>>> >> > The North Bay Linux Users' Group
>>> >> > http://nblug.org
>>> >> > IRC: greenfly at irc.freenode.net #nblug
>>> >> > kyle at nblug.org
>>> >> >
>>> >
>>
>> _______________________________________________
>> talk mailing list
>> talk at nblug.org
>> http://nblug.org/cgi-bin/mailman/listinfo/talk



More information about the talk mailing list