User documentation

White Board Design

10/19/2011 09:44 AM

Upper left:
List of utilities for FBI.
Left center:
Thinking about CIA design.
Light bulb on Event<->Connection design pattern.

CIA Design: Analysis of NodeLocator

11/21/2011 03:44 PM

The current NodeLocator application polls all switches/routers on site and gather operational information
about IP<->MAC addresses, switch port/VLANs, router ports, etc. This information is important for
Computer Security investigations and for network device blocking.

Current NIMI view

09/01/2011 11:22 AM

The current NIMI uses a design that associated a specific table with a type of network connection.
For example, the DHCP leases managed by the InfoBlox appliance are stored in the 'dhcplease' table.
The location where MAC addresses are connected to the network are stored in the 'switch_table_intervals' table. The data (rows) in any table is not associated with other tables....

Proposed NCIS

09/01/2011 02:41 PM

The NCIS database will contain data similar to the existing NIMI. However, the internal structure
of the NCIS database will allow data to be associated with network-attached devices.

A Network-Connected Device View

09/01/2011 11:30 AM

Computer Security is interested in devices connected to the Fermilab network.
This diagram is a "view" of network-connected devices which show how a network
attribute such as IP address can be "traced" through network-connected devices.

The intent is to show how the new NCIS will capture relationships such as IP address

Tissue/FBI/NCIS Big Picture

08/05/2011 02:43 PM

NCIS (Network Core InfraStructure) is the application which contains:

- Tissue - Issue Tracking
- FBI - Fermi Blocking Interface
- NCIS - network configuration database which contains:
- NIMI - "What systems are currently connected to the Fermi network"

NCIS Simple picture

06/03/2014 01:09 PM

  • Gathers data from network devices (SNMP polling)
  • Analyzes the polling data, builds end-hosts (MAC<->IP)
  • Keeps historical record of all end-hosts
Security Detector:
  • Scans network IPs looking for vulnerabilities or configuration problems.

NCIS Test Fixture

08/27/2019 03:50 PM

A list of activities for setting up and subsequent configuration of a Raspberry Pi 3.

Changes made for newer python and newer modules

10/10/2016 02:08 PM

High level listing of the major changes to tissue/fbi/ncis to support moving to python 2.7 (on RHEL7).

NCIS Polling Holes

08/31/2018 04:37 PM

Slide presentation of current and addressed missed/ignored polling NCIS data.

Technical documentation

CIA Design: Polling vs SNMP Traps

12/20/2011 11:15 AM

The current NodeLocator software uses periodic polling to gather the network device
operational data. Using SNMP Traps may be an alternative method. However,
SNMP Traps have the following issues:
  • Difficult to configure if a MIB doesn't define the trap.

NCIS Big Picture

05/08/2012 03:35 PM

The attached diagram show the major components of NCIS:
- The Database of End-Hosts: the IP-MAC pairs and time range.
- The Database of Connections: for each IP-MAC pair, a list of connections to the network
(i.e. swtches).
- The Database of Snapshots: the raw data collected by either the NodeLocator application

VLAN Paradigm Shift: design of polling

09/07/2012 02:48 PM

We have proven that we are technically able to use pySNMP and twisted to speak with network equipment in order to obtain connection information. Now that we have a better idea of what is going on, we need to figure out how to do it in a maintainable and supportable way. Michael and I had long design discussions (some of which also involved Jim as Management Who Can Make The Hard Political Decisions). This is what came out of our discussions, September 2012....

Some snapshots of performance as we tweak and tune

05/10/2013 12:58 PM

Various code branches and tweaks:
1) Michael's unmaintainable but fast NcisSnmpAsync code branch (cleaned up as far as possible)(the _zasync version tree, used by setting USE_MIKE_TWISTED_POLLER=True in the ncis_core/NcisSpy/ file and recompiling

NCIS Database Design: ER Diagram

08/15/2017 02:27 PM

The heart of the new NCIS is the organization around "EndHosts", defined to be a mac/ip pair seen on a network over a particular timespan. EndHosts have:

  • connections -- these are where the endHost is "connected" to the network (switch, router, etc.) An endHost may migrate from one connection point to another, may have multiple connections at a time (due to caching within a switch/router/etc).

Reference documentation

NCIS Processes

11/06/2012 11:10 AM

NCIS processes for gathering live network status use the Twisted Python package.

Twisted is an event-driven networking engine written in Python and licensed under the open source MIT license.

The NCIS processes are packaged as a suite of robots. Robots are Linux daemons

gui snapshot from 2013-03-11

03/11/2013 09:27 AM

Started the poller running in SYNCHRONOUS (not newer asynchronous) mode on Friday afternoon, ran it over the weekend. Statistics from the new dashboard (with the two extended polling plots of throughput per past day and throughput per past week taking 15 minutes to generate). Stashing these here for comparison later when we go to the asynchronous mode....