Troubleshooting High CPU usage on a Catalyst 6500 due to ARP Input.

Faced with a Cat 6500 with dual Sup 2T's that responded to a 

show running-config

with

Unable to read Configuration.Try Again later

and general cli slowness, it became aparrent that the CPU load on the supervisors was massive:

cat-6500-sw1#sh proc cpu sorted
CPU utilization for five seconds: 99%/83%; one minute: 99%; five minutes: 99%
PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process 
 11   595538792 824019705        722  9.09%  8.54%  9.10%   0 ARP Input    

​The next step is to isolate the source of the arp requests.  This is a fairly manual process dependant on the number of interfaces you have – but here's how I went about it:

cat-6500-sw1#clear counters
Clear "show interface" counters on all interfaces [confirm]
cat-6500-sw1#

cat-6500-sw1#show int stat       
  
Vlan101
         Switching path    Pkts In   Chars In   Pkts Out  Chars Out
              Processor          0          0        110       7040
            Route cache          0          0          0          0
      Distributed cache      90801   19878330     112007   79166896
                  Total      90801   19878330     112117   79173936
Vlan102
         Switching path    Pkts In   Chars In   Pkts Out  Chars Out
              Processor         13        832         21       1344
            Route cache          0          0          0          0
      Distributed cache      14980    5568056      21858   19621765
                  Total      14993    5568888      21879   19623109
Vlan103
         Switching path    Pkts In   Chars In   Pkts Out  Chars Out
              Processor          0          0          0          0
            Route cache          0          0          0          0
      Distributed cache          0          0          0          0
                  Total          0          0          0          0
Vlan104
         Switching path    Pkts In   Chars In   Pkts Out  Chars Out
              Processor         14        896         14        896
            Route cache          0          0          0          0
      Distributed cache       1126     493152       2449     387180
                  Total       1140     494048       2463     388076
Vlan105
         Switching path    Pkts In   Chars In   Pkts Out  Chars Out
              Processor     411216   34901876      79195    5068480
            Route cache          0          0          0          0
      Distributed cache         45       3904        105      10566
                  Total     411261   34905780      79300    5079046

Seems strange that Vlan105 is sending such a large amount of packets to the general purpose CPU.

cat-6500-sw1#sh int accounting   Vlan105
Vlan105  Test Network
               Protocol    Pkts In   Chars In   Pkts Out  Chars Out
                  Other       7049     422940          0          0
                     IP     231708   21413675        120      12268
                    ARP     142076    8524560      87541    9804592

There's no way that interface should be sending that many ARP requests.  There's either some nasty in there, or a misconfiguration.

cat-6500-sw1#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
cat-6500-sw1(config)#int vlan 105
cat-6500-sw1(config-if)#shutdown
cat-6500-sw1(config-if)#end

And to verify, after a few moments the CPU utilisation had dropped back to normal.

cat-6500-sw1#sh proc cpu
CPU utilization for five seconds: 4%/2%; one minute: 85%; five minutes: 96%