Project

General

Profile

The NodeLocator Handbook
January 2014
D. Wohlt
Internal Use Only

Introduction
NodeLocator is a network management tool developed by the Fermilab Core Computing Network Services Group for identifying the switch port that a host is connected to, in response to a web-based query by hostname, IP address or MAC address. The resulting web page reports the current network location of the host and includes links to the MRTG graphs for the port and a button for requesting further port information from the switch. The Network Management-level NodeLocator screen also features a tool for listing all nodes on a switch or switch port.

Working from a stored list of devices, NodeLocator polls routers for their ARP tables and switches for their MAC forwarding tables, roughly every 5 minutes. In addition, switches are surveyed semi-hourly for any static MAC address entries that have been applied by security applications.

NodeLocator pollers and a number of related utilities run on nodelocator.fnal.gov, which resides on the Network Management LAN and is not accessible outside of this LAN. Its MAC and IP databases are regularly copied to mrtg.fnal.gov which hosts the web page mentioned above.

NodeLocator is tightly coupled with MRTG management software and uses the same device list file.

NodeLocator keeps logs of its activities, and the MRTG systems provide graphs of runtimes of the polling programs. Many of nodelocator’s internal logs are pruned automatically to help conserve disk space. Its MAC and IP address files are managed by removing invalid or stale records; these removals are logged for historical use. Changes in MAC or IP address disposition are also journalized. An experimental vigil feature makes log entries when specified MAC addresses are detected.

Quick Info
nodelocator performs these basic functions:
1. Poll switches for MAC tables
2. Poll routers for ARP tables
3. Poll switches for filtered MAC info

If nodelocator is rebooted, the polling programs will automatically start. The MAC and ARP data will pick up from where it left off.

If nodelocator is down or disconnected, no updates will be made to the database copies on mrtg. There is no alternate polling machine.

All NodeLocator software is kept under the ‘netadmin’ account. The operating system is RedHat Linux, and the only user interface is by command line. nodelocator resides on the Network Management LAN. Access is by SSH only, and requires connection to the network management LAN (e.g., via netvpn). Network Services does not have root access.

All production files and directories are under /usr/local/netadmin/v1/. On nodelocator there is a corresponding symbolic link with the name /home/netadmin/current/, so when you logon as netadmin , your first command should be

cd current

All communications with network devices from this machine use SNMPv2 (read-only) and ICMP-Ping. No device configurations are altered. No off-site devices (e.g., Starlight) are routinely polled. Soudan Mine and Ash River devices are polled through a pseudo-wire IP connection.

Stopping everything manually:
1. Login as netadmin
2. Type nlstop
This stops only the nextgen-db daemon. It will restart if you reboot or type nlstart.
3. Type EDITOR=vi; export EDITOR
4. Type crontab –e
5. Go into Insert mode by pressing lower-case i
6. Comment out all lines with a # character, navigating with the arrow keys
7. Press escape to leave Insert mode
8. Type :wq and press Enter
This prevents all cron-scheduled programs from being restarted
9. Check the nextgen log periodically to watch for the ending message
tail ~/current/nl/logs/nextgen.log
“Final switches.db written, Nextgen-db ending.” means nextgen-db is stopped.
10. Check if any lock files are still there, which would indicate that a recurring program is still running.
ls –l ~/current/nl/db/*.lock

a. If you see testrtr.lock, get the PID with a ps aux | grep testrtr.pl command (the PID is the second field from the left). If you cannot wait for it to complete, kill it with the kill xxxx command (xxxx is the PID).
b. If you see genperm.lock, get the PID with a ps aux | grep genperm-db.pl command (the PID is the second field from the left). If you cannot wait for it to complete, kill it with the kill xxxx command (xxxx is the PID).

Starting everything manually after having stopped everything as above:
1. If you rebooted, nextgen-db will start automatically.
Otherwise, type nlstart
2. type EDITOR=vi; export EDITOR
3. Type crontab –e
4. Use the x character and arrow keys to delete the # character from the beginning of each line
5. Type :wq and press Enter

Changing the SNMP read-only community string: do this on mrtg1 by editing the ~/current/mrtg/in/cs file there. It will propagate to nodelocator within 30 minutes. After that, the ARP collector and MAC collector programs will see it within 5-10 minutes. To send the cs file immediately to nodelocator:
scp cs netadmin@nodelocator:~netadmin/current/nl/in/

Required input: Most of the software on this system requires the device information in ~/current/nl/in/inventory.in. This file originates on mrtg1 and is automatically copied to nodelocator every 30 minutes.

Polling the Switches: A daemon called nextgen-db.pl polls switches several times an hour. It maintains an in-memory hash array containing MACaddress:SwitchPort info and writes the array to ~/current/nl/db/switches.db every 5 minutes. The individual records in switches.db are expired and removed by nextgen-db after 14 days (defined by ~/current/nl/in/threshold). Nextgen-db updates its in-memory list of devices to poll from ~/current/mrtg/in/inventory.in, every 5 minutes.

Nextgen-db is started at boot time or by command line using the /etc/init.d/nextgen program with a ‘start’ argument (which calls launcher.pl). Launcher and nextgen-db contain mechanisms to prevent multiple copies from being started. Under the netadmin account you can type the nlstart command.

Nextgen-db can be stopped with the /etc/init.d/nextgen program and a ‘stop’ argument. Another way is to remove ~/current/nl/db/nextgen.lock. Under the netadmin account you can type the nlstop command.

The nextgen-db process can be safely killed (by a root user) at any time if necessary.

Polling the Routers: Routers and Native-IOS switches are polled by testrtr.pl for their ARP tables. This program is called by ~/current/cron/Every5m, and reads ~/current/nl/in/inventory.in for a list of devices. Testrtr reads and updates ~/current/nl/db/routersarchive.db, and overwrites ~/current/nl/db/routers.db which is used by the NodeLocator ‘Search’ application on mrtg. These files contain IPaddress:MACaddress info. The records in routersarchive.db are never expired, but the records in routers.db are very short-term (a few minutes).

Feeding NIMI: Switches.db and routersarchive.db are still pulled (using scp) every 5 minutes by the NIMI server. These files used to serve as raw data for updating the NIMI databases, but are no longer needed as the new NCIS system performs its own polling. Routersarchive.db is called routers.db on the NIMI host.

Polling for Filtered MACs: The genperm-db.pl program is called by ~/current/cron/Every30m to poll the switches in inventory.in for any MAC addresses that have been set to a “static” condition, typically by a security application for the purpose of blocking (filtering) that node. Genperm-db rewrites ~/current/nl/db/cam_perm.db which contains MACaddress:Switch info for each filtered MAC address. Then it rewrites ~/current/nl/db/exportcp.db which is sent to mrtg so it can make the web page called “LocalSwitchBlocks”.

Web Interface: Here are the links to the web pages mentioned above.
a. http://mrtg.fnal.gov/NodeLocator/search.html
b. http://mrtg.fnal.gov/MRTG-SiteMapNetMgmt/search.html
c. http://mrtg.fnal.gov/NodeLocator/LocalSwitchBlocks

The NodeLocator (a) and LocalSwitchBlocks (c) links are publicly known and are included on the “Utilities” page (https://sharepoint.fnal.gov/cd/sites/ns/SitePages/Home.aspx). The full-featured NodeLocator link, which includes the Switch Tool that can list all nodes on a switch, is given in (b).

Components
The NodeLocator system contains these main polling programs:
• Nextgen-db, a daemon that continuously polls the switches in the device inventory list for their MAC forwarding tables and updates a file containing MAC addresses with their associated switch and port location, time of detection, VLAN, and root cost of the port (deprecated).
• Testrtr, a program called by ~/current/cron/Every5m every 5 minutes to poll routers in the device inventory list for their ARP tables and update a file containing IP addresses with their associated MAC address, router interface, router name, and time of detection.
• Genperm-db, a program called by ~/current/cron/Every30m every 30 minutes to poll switches for their static MAC entries and update a file containing MAC addresses with their associated switch name, time of detection, type of static entry (filter type), and VLAN. permcamlist and wi-blacklist are external programs called by genperm-db to extract static MAC entries from switches, and blacklisted MAC’s from WISM devices, respectively.

These are the main operational programs:
• Launcher is called by /etc/init.d/nextgen at boot time. It checks to make sure there are no ‘nextgen-db.pl’ processes running, then it removes /current/nl/db/nextgen.lock if it’s there, then it starts nextgen-db.pl. Launcher is a convenient way to manually start nextgen-db when needed. In early deployments, it was executed every 5 minutes to ensure nextgen-db was running, but this is no longer necessary.
• Routers-db-compare is called by ~/current/cron/Every30m to compare the current IP address file (
/current/nl/db/routers.db) with the one saved 30 minutes earlier, categorically logging changes to several log files in ~/current/nl/logs/. If no changes are detected, testrtr is declared “stalled”, and its stale lock file is automatically removed to allow testrtr to start again within 5 minutes.
• Perm-db-compare is called by ~/current/cron/Every30m to compare the current list of static MAC entries with the one saved 30 minutes earlier, logging any changes to a log file in ~/current/nl/logs/.
• Log-analyzer is called by ~/current/cron/EveryHour every 60 minutes. It reads about the last hour’s worth of nextgen-db’s polling logs and creates a report showing how many times each switch was polled, how many MAC’s were seen, how long the poll took, and how much time passed between polls. It appends this report to ~/current/nl/logs/loganalysis.log.
• Pageme is called by ~/current/cron/Every5m to check nextgen-db’s device status files and issue an email if any devices have become unreachable, i.e., not responding to Ping. The email recipient list could include a pager number, hence the “pageme” name, but ‘’ is the current setting.
• Watchpollers is called by ~/current/cron/Every30m to check for stalled pollers in the nextgen-db program. If the last entry in any poller log is more than 30 minutes old, a stalled poller is declared and nextgen-db is gracefully brought down and restarted.
• Cpustat is called by ~/current/cron/Every5m to sample the 1-minute and 5-minute CPU load values on nodelocator, saving them in a small file that is sent to mrtg1 for MRTG processing.
• Diskstat is called by ~/current/cron/Every5m to determine the amount of disk space (blocks) being used in the home directory, saving that value along with the total allocated value in a small file for MRTG processing.
• Ethstat is called by ~/current/cron/Every5m to get the current RX and TX byte counts from nodelocator’s active network interface (eth0), saving them in a small file for MRTG processing.
• Memstat is called by ~/current/cron/Every5m to check the amount of free memory and total installed memory, saving the values in a small file for MRTG processing.

Data Files
Input and output data used and produced by NodeLocator are contained in subdirectories of ~/current/nl/. All configuration input to the programs is contained in the ~/current/nl/in/ directory. Collected data is contained in the ~/current/nl/db/ directory, and activity logs are kept in ~/current/nl/logs/.

~/current/nl/in/ contains these files:

cs The default read-only SNMP community string for switches and routers. This file is read by many programs on this host, and contains one line with one word, no blanks. The authoritative copy of this file is on mrtg1 in ~/current/mrtg/in/, and is copied over to nodelocator every 30 minutes.
inventory.in This is the device inventory list used by many of the programs on this host, particularly nextgen-db, testrtr, and genperm-db. This file controls which switches and routers get polled. The authoritative copy of this file is on mrtg1 in ~/current/mrtg/in/, and is copied over to nodelocator every 30 minutes.
perps.in nextgen-db will write log entries whenever one of its pollers “sees” a MAC address in this list. You may edit this list anytime without stopping nextgen-db.
threshold Contains the days,hours,minutes that MAC records in switches.db are retained. You must stop and start nextgen-db if you change this file (the current contents are 14,0,0).

~/current/nl/db/ contains these files:

cam_perm.db A list of MAC addresses that are being filtered on one or more devices. The timestamp in each record reflects the latest poll time by genperm-db, not the time that the MAC was initially set to ‘static’ on the device. This file is rewritten every 30 minutes.
cam_perm.save A copy of cam_perm made by ~/current/cron/Every30m. This is used by perm-db-compare.pl for comparing with the current cam_perm.db in order to log any changes to ~/current/nl/logs/perm-macchanges.log. Look there for a rough idea of the time that a MAC was blocked or unblocked.
devicesdown.db Maintained by pageme.pl as a scratchpad containing the names of devices deemed unreachable, so that the alert email or page goes out only once per device.
devstat.db Updated every 15 seconds by nextgen-db, this is a list of devices currently on nextgen-db’s polling list, along with their status (numbers 1 through 5): Reachable No SNMP Response No Ping response Unknown Hostname Not Yet Polled.
exportcp.db A sorted and slightly formatted version of cam_perm.db, used by makeLSBreport.pl to make the LocalSwitchBlocks web page.
Monitors.db Contains a list of URL “segments” that help the “Search” web application build links to MRTG graphs of the switch ports it finds.
nextgen.lock Contains the PID of the currently running nextgen-db process. Removing or altering this file will cause nextgen-db to shut itself down, as if you issued the nlstop command. The purpose of this file is to prevent multiple nextgen-db programs from being started.
routers.db Updated by testrtr.pl every polling cycle, this is a list of IP address / MAC address pairings along with router name and interface info. It is essentially a composite ARP table from all the routers. It is keyed on an ‘IP address MAC address’ combination, which allows NodeLocator to accommodate hosts with multiple network connections. IP:MAC pairs can and do appear on multiple routers, and each record in this file reflects the last router polled that contained the pair. Records are retained in this file only if they are not older than twice the time it last took to poll all of the routers. The NodeLocator “Search” application on mrtg uses this file.
routers.save A copy of routers.db made by ~/current/cron/Every30m. This is used by routers-db-compare.pl for comparing with the current routers.db in order to log any changes to the ~/current/nl/logs/arpchanges-*.log files.
routersarchive.db This is the same as routers.db, except that its records are retained indefinitely.
switches.db Updated by nextgen-db every 5 minutes, this is a list of MAC address / switch port pairings. It is keyed on MAC address. MAC:port pairings can and do appear on several switches, each pair indicating the local port of the respective switch (which may be a link port), but the polling code in nextgen-db performs deep inspection in order to extract only the locally-resident MACs on each switch. The NodeLocator “Search” application on mrtg uses this file.
Testrtr.lock Created by testrtr.pl when it starts up, to prevent a subsequent testrtr.pl from starting, should this one take longer than 5 minutes (the configured interval for running the program).

~/current/nl/logs/ contains these files:

arpchanges-diffmac.log Entries made by routers-db-compare.pl every 30 minutes for IP addresses that are now paired with a different MAC address. This file is cumulative and not automatically pruned. Each year the contents are manually moved to the ~/current/nl/logs/archive/ directory.
arpchanges-diffrtr.log Entries made by routers-db-compare.pl every 30 minutes for IP addresses that are now associated with a different router. This file is cumulative and not automatically pruned. Each year the contents are manually moved to the ~/current/nl/logs/archive/ directory.
arpchanges-newandexpired.log Entries made by routers-db-compare.pl every 30 minutes for IP addresses that have been either removed from or reinstated to routers.db due to age. This file is cumulative and not automatically pruned. Each year the contents are manually moved to the ~/current/nl/logs/archive/ directory.
genperm_jobtime.log The end time of each genperm-db job, along with the elapsed time in seconds. This file is pruned automatically by ~/current/cron/EveryEve.
macchanges.log Updated in real time by the nextgen-db pollers when a new MAC address is discovered, or a MAC “moves” from one switchport to another. This now includes activity on the wireless access points.
neogen_error.log Nextgen-db makes an entry whenever one of its pollers cannot get a response from a device. This file is pruned automatically by ~/current/cron/EveryEve.
nextgen.log Records starting and stopping of nextgen-db and any devices added to or removed from its device list while it is running.
perm-all.dat Contains the number of seconds it took to run genperm-db, and the number of seconds of the scheduled interval for starting the program. This file is copied to mrtg1 every 5 minutes for graphing.
perm-all.log Results of the most recent run of genperm-db. One line for each device polled, showing number of static MAC entries.
perm-macchanges.log Entries made by perm-db-compare.pl every 30 minutes for MAC addresses that have been added to or removed from cam_perm.db. This file is cumulative and not automatically pruned. Each year the contents are manually moved to the ~/current/nl/logs/archive/ directory.
perp.log Entries made by nextgen-db whenever a MAC in the perps.in list is detected by a poller. This file is cumulative and not automatically pruned.
poller*.log Each nextgen-db poller process makes log entries for each device it polls, showing the number of MACs detected and the elapsed polling time. These files are pruned automatically by ~/current/cron/EveryEve. They are copied to mrtg1 every 5 minutes for graphing the polling time and interval for each device.
rejects.log Before saving switches.db every 5 minutes, nextgen-db sweeps through its in-memory database and removes records that are too old or contain invalid data, and copies those records to this file. This file is cumulative and not automatically pruned. Each year the contents are manually moved to the ~/current/nl/logs/archive/ directory.
Testrtr.dat Contains the number of seconds it took to run testrtr, and the number of seconds of the scheduled interval for starting the program. This file is copied to mrtg1 every 5 minutes for graphing.
testrtr.log The testrtr.pl program makes log entries for each device it polls, showing the number of ARP entries detected and the elapsed polling time. This file is pruned automatically by ~/current/cron/EveryEve.
testrtr_error.log Testrtr makes an entry whenever it cannot get a response from a device. This file is pruned automatically by ~/current/cron/EveryEve.
testrtr_jobtime.log The end time of each testrtr job, along with the total run time in seconds. This file is pruned automatically by ~/current/cron/EveryEve.

Troubleshooting
The NodeLocator facility is embodied in three virtual machines: nodelocator, mrtg, and mrtg1. Nodelocator polls the routers and switches, using an inventory list provided by mrtg1. The web interface resides on mrtg, using data transferred to it from nodelocator.

The nodelocator machine maintains 3 files: switches.db, routers.db, and routersarchive.db as it polls. The mrtg machine receives the latest copies of these files every 5 minutes. Switches.db contains MAC-Switch-Port info, routersarchive.db contains IP-MAC info, and routers.db contains short-term IP-MAC info. In the NodeLocator Search form, the locator function uses switches.db and routers.db. The Name/IP/Mac Tool uses switches.db and routersarchive.db.

If nodelocator alone fails:
• ARP and MAC data will not be collected
• Mrtg’s web interface will continue, using the last received data
• Mrtg1 will continue to poll devices

If mrtg alone fails:
• NodeLocator and MRTG users will lose access to the web pages
• Nodelocator will continue to poll devices
• Mrtg1 will continue to poll devices

If mrtg1 alone fails:
• Port utilization and other data will not be collected
• Nodelocator will continue to collect ARP and MAC data from devices, using the last inventory list it received from mrtg1
• Mrtg will still operate normally for NodeLocator searches, but MRTG graphs will not work

The following items presume that all three machines are up and running normally.

If a node is “not located”:
• First double-check the IP address, nodename, or MAC that was entered
• Try using the Name/IP/MAC tool on the NodeLocator Search Form to confirm if nodelocator has IP and MAC entries for the node.
• If IP and MAC entries are both present, it may be that the IP info is not “fresh” enough to be considered current by the Search program. Try clicking on the MAC address returned by this tool.
• Try grepping the IP address in the routers.db and routersarchive.db file, or the MAC in the switches.db file.

If “IP address not found”:
• May not be polling the router or firewall that supports this subnet. Check inventory.in on mrtg1. “cisco_router” and “cisco_native” devices will be polled for ARP info by nodelocator unless “nopoll” is set.
• Try grepping the router name in ~current/nl/logs/testrtr.log and testrtr_error.log
• Try grepping the IP address in the arp*.log files in ~current/nl/logs/ on nodelocator to see if there is any historical information. Also look in the ~current/nl/logs/archive/ directory for past years’ logs.

If “MAC address not found”:
• May not be polling the switch where it resides. Check inventory.in on mrtg1. “cisco_switch”, “cisco_ios”, “cisco_native”, and “generic” devices will be polled for MAC info by nodelocator unless “nopoll” is set.
• Try grepping the switch name in ~current/nl/logs/poller*.log and neogen_error.log.
• Try grepping the MAC address in the macchanges.log, rejects.log, and arp*.log files in ~current/nl/logs/ on nodelocator to see if there is any historical info. Also look in the ~current/nl/logs/archive/ directory for past years’ logs.

If “Last detected” time stamps seem stale:
• Mrtg may not be receiving .db updates every 5 minutes from nodelocator.
• Nodelocator may have stalled pollers. It should have sent email with instructions, but you can also check this by logging on to nodelocator and looking at the last line of each poller.log file, with “tail –n1 ~current/nl/logs/poller
.log”.

For a Nexus switch (e.g. s-s-access-fcc3-2) that doesn’t respond to polls for MAC info but you need a list of hosts connected to it:
• Logon to s-s-access-fcc3-2 (or whatever device)
• show cdp neighbors to determine uplink ports
• show mac address-table | excl <those uplink ports>
• Edit the output, removing the extraneous lines to leave only the MAC address lines
• Edit further to position the VLAN, MAC, and Port fields into the following format:
MAC<b>Switch<b>Port<b>Timestamp<b>VLAN<b>0
• MACS should be exactly 12 hex characters, no dots (d4856477e4ae)
• <b> means 1 blank character ( )
• Timestamp can be made up, and can be the same for each line (2012/03/09/17:38)
• The last field is always “0” (zero)
• Copy the file you made to your clipboard, ready to paste
• Now login to mrtg (not nodelocator) as netadmin
• cd current/nl/db
• cp switches.db test.db
• cat >> test.db
• Paste your file
• When it’s done, press control-D to end the operation
• cd ..
• perl testsearchports.pl s-s-access-fcc3-2
• Voila, your list.

How-To’s

Notifications and how to respond
NodeLocator: One or more devices are unreachable
While being polled by NodeLocator, if a switch doesn’t respond to pings or is not resolved by a DNS query, NodeLocator sends email to and a syslog message to nsrelay.
NodeLocator: error(s) while attempting to collect data
These are informational messages sent when a switch doesn’t return a response to an SNMP query. It doesn’t mean the switch is down, more likely that it’s CPU or the network path is busy, or that there’s a discrepancy in the SNMP community string or the associated ACL.
NodeLocator: One or more pollers are stalled
When a major network disturbance occurs, some of NodeLocator’s polling processes may hang, which requires a restart of the nextgen-db program. The restart is normally automatically done by the watchpollers program, but if it fails:
1. Connect to netvpn
2. Logon to nodelocator as netadmin
3. cd current/nl/logs
4. nlstop
5. tail –f nextgen.log
and wait until you see "Final switches.db written, Nextgen-db ending."
6. Control-c to break out of the tail command
7. nlstart
8. cd ../db
9. ls –l
10. If testrtr.lock is there but no testrtr.pl is running:
rm testrtr.lock
11. If genperm.lock is there but no genperm-db.pl is running:
rm genperm.lock
NodeLocator: router poller stalled
When a major network disturbance occurs, the testrtr program may take longer than its 5-minute allotted interval to get through all the routers, and its lock file prevents the next run from starting up. Normally it will eventually resume, but if the routers-db-compare program detects no ARP data changes over the past 30 minutes, it will remove the lock file. Should the routers-db-compare program fail:
1. Connect to netvpn
2. Logon to nodelocator as netadmin
3. cd current/nl/db
4. rm testrtr.lock
How to use the archives
The NodeLocator “archives” is a collection of old log files going back a few years. You can sometimes glean a sort of activity history of a node by searching the log files for its MAC or IP addresses.
1. Connect to netvpn
2. Logon to nodelocator as netadmin
3. cd current/nl/logs/archives
4. ls –l
macchanges
rejects
arpchanges

Interpreting the Loganalysis.log
The log-analyzer.pl program is executed by cron every hour to examine all of the poller*.logs and create a report about the average polling performance of the network devices over the past hour (approximately). The output is appended to ~/current/nl/logs/loganalysis.log. Unlike many other log files, this file is not pruned every night.

You can look at the report for the past hour (the number varies with the number of devices polled):
cat loganalysis.log | tail –n332

You can watch the polling performance trend, or the number of MACs detected for a device by grepping it:
cat loganalysis.log | grep <device-name>

You can identify devices with the longest polling times in the past hour:
cat loganalysis.log | tail –n332 | sort –k 7,7

Maybe a better way to grep the last hour’s device info is to first tail the log file and copy the date and hour from the “Last Polled” column, and use that in a grep:
grep "2012-05-02 12" loganalysis.log | sort -k 7,7

Listing the Nodes on a Switch or Wireless Access Point
You can get a list of active IP nodes on a switch or AP from the NodeLocator web application for network management users.

1. Connect to netvpn
2. Open http://mrtg.fnal.gov/MRTG-SiteMapNetMgmt/search.html
3. Under Switch tool, enter the name of the switch or AP, then click the Look Up button.
4. Any of the five columns of data can be sorted by clicking on that column’s header. Click once for ascending order, click again to reverse it.
5. You can copy the table to clipboard and paste it to another window – but it’s recommended that you first paste it to NotePad to clear out any formatting, then copy/paste it elsewhere.
6. Note that the Switch Tool also lets you enter a port name (e.g. Gi8/22), or the first part of the port name (e.g. Gi8) to get the entire module’s occupants.

You can also get a list of nodes on a switch using a command line program on mrtg.
1. Connect to netvpn
2. Logon to mrtg as netadmin
3. cd current/nl
4. perl searchports.pl <switchname>
5. The output can be sorted by a column number. For example, to list nodes on s-d0-dob-1 and sort them by nodename, enter:
perl searchports.pl s-d0-dob-1 | sort -k 3

Listing the Nodes in a Range of IP Addresses
You can get a list of IP nodes and their locations generically.

1. Connect to netvpn
2. Open http://mrtg.fnal.gov/MRTG-SiteMapNetMgmt/search.html
3. In the “Enter a hostname, IP Address or MAC Address” field, enter an IP address with a ‘*’ as a wildcard character (e.g. 131.225.179.*, or even 131.225.*).
4. Click Search Database

Listing the Nodes Having a Particular MAC Vendor Code (OUI)
You can locate all the active IP nodes that have a certain OUI.
1. Connect to netvpn
2. Open http://mrtg.fnal.gov/MRTG-SiteMapNetMgmt/search.html
3. In the “Enter a hostname, IP Address or MAC Address” field, enter the first 3 bytes of a MAC address followed by ‘*’ (e.g. 003048*, no dashes or colons).
4. Click “Check here if what you entered is a MAC Address”
5. Click Search Database

Identifying Access Points on the Wireless Controllers
Sometimes you need to know which controller is handling a particular AP, or you can’t remember the AP names for an area. Use the Device Info web application to list the MAC addresses on a controller. This list will show the AP that each MAC is associated to.

1. Connect to netvpn
2. Open http://mrtg.fnal.gov/MRTG-SiteMapNetMgmt/dsu.html
3. Under Device Info, enter the name of the wireless controller, select MAC List, then click the Query Device button.
4. Any of the five columns of data can be sorted by clicking on that column’s header. Click once for ascending order, click again to reverse it.
5. You can copy the table to clipboard and paste it to another window – but if you send it to NotePad it won’t retain the column formatting. Send it to Word instead. This program will be changed in the future to produce a non-formatted report, as Switch Tool does.

Some Pages Have Sortable Columns
If you see the Sortable Columns icon at the upper right corner of a web page created by NodeLocator, it means you can click a column heading to sort the table by that column. Click it again to reverse the order.

Ping All Nodes in a Subnet
There’s a utility on nodelocator that can ping a list of IP addresses. In this example we’ll ping all the nodes detected by NodeLocator in subnet 111.

1. Connect to netvpn
2. Logon to nodelocator as netadmin
3. cd current/nl/util
4. perl searchgroup.pl 131.225.111.* > vl111
5. perl pingalist.pl vl111

You can create your own file of IP addresses and adjust pingalist.pl’s arguments accordingly.

$perl pingalist.pl
Usage: perl pingalist.pl file-of-ip-addrs [field number (0-n)] ['delimiter character']

The file name is required. The default key field number is 0 (the first field in the lines), and the default delimiter is the blank character.