Welcome To suyashjain.blogspot.com

For Latest and more contents visit http://www.i3w.in

Friday, July 25, 2008

Netflow Tools

Pointer Collections

Cisco NetFlow Ecosystem Solutions
Contains descriptions of many applications that integrate NetFlow support, including a few "freeware" ones.
Cisco now lists Netflow applications on their Web pages, specifically applications from Cisco, commercial, and "freeware" applications.
Network Uptime list of Free NetFlow Tools
Nice overview of Netflow tools with screenshots.
FreshMeat NetFlow projects
A list of pointers to open-source projects related to NetFlow.

NetFlow

NFDUMP and NfSen
NFDUMP is a set of tools to capture/record, dump, filter, and replay NetFlow (v5/v7/9) data. Can filter flows according to multiple user-defined profiles. NfSen is a Graphical Web-based front-end for the NFDUMP tools. Plots aggregate statistics over time, supports filtering and drilling down up to the individual flow level.
CoMo
Traffic monitoring toolkit from Intel Research. Supports both continuous real-time processing and retrospective processing. Supports Netflow and many other traffic capture sources.
YAF - Yet Another Flow sensor
YAF snoops packets from pcap dump files or live capture, and produces bidirectional flows. These flows can be sent to IPFIX collectors, or be stored in an IPFIX-derived file format.
VERMONT (VERsatile MONitoring Toolkit)
A reference implementation of the IPFIX and PSAMP protocols developed as part of the HISTORY project at the German universities of Erlangen and Tübingen, and of the European DIADEM Firewall project.
libipfix
A C library that implements the IPFIX protocol.
libfixbuf
Aims to be a compliant implementation of the IPFIX protocol message format, from which fully compliant IPFIX Collecting Processes and IPFIX Exporting Processes may be built. In addition of the IPFIX Protocol, libfixbuf supports efficient persistent storage of IPFIX data using the method outlined in draft-trammell-ipfix-file-NN.
NetSA Aggregated Flow (NAF) toolchain
Tools for creating and analyzing timeslice-organized bidirectional flow files in the IPFIX-inspired NAF format.
FlowScan
A Perl-based system to analyze and report on flows collected by flow-tools, lfapd or cflowd, by Dave Plonka. Sample output graphs are available too, as well as Majordomo-driven mailing lists for announcements and general discussion (archive). It is currently built on Cflow.pm. User-contributed tools based on FlowScan include:
CarrierIn from Stanislav Sinyagin
which claims to be more suitable for larger ISP/Carriers
CUFlow from Matt Selsky and Johan M. Andersen at Columbia University
which is an alternative graphing tool "designed to combine the features of CampusIO and SubNetIO". Robert S. Galloway has contributed a nice howto-style document describing how it can be used.
FlowMonitor from Johan M. Andersen at Columbia University
monitors individual users' network usage against a bandwidth usage policy.
JKFlow by Jurgen Kobierczynski
A new reporting module which is highly configurable using an XML configuration file.
FlowScan+
An extension to FlowScan developed by KISTI/KAIST. Adds servlet-based visualization and support for queries for top user, AS, port, protocol, etc. This is supposed to be available under http://flowscan.kreonet2.net/, but that site doesn't seem to be responsive.
flow-tools
Similar to cflowd but implemented as a set of smaller tools, with the addition of compression of the recorded data, thus capable of recording many more flows in a given amount of disk space. See paper about its application for Intrusion Detection. There is also a mailing list for the package.
There is a short presentation called Ohio Gigapop Traffic Measurements that shows some examples on how flow-tools can be used.
The package is widely used, and there are quite a few user contributions, such as
FlowViewer
Web-interface to flow-tools. Consists of three tools: FlowViewer provides the user with web access to many of the textual and statistical flow-tools reports. FlowGrapher provides a web page with a graph of the selected flow data. These web pages can be saved. FlowTracker (introduced in FlowViewer 3.0, released in July 2006) allows the user to maintain this information long-term by creating four MRTG-like graphs. Filtered flow data is collected every five minutes and the graphs are updated. FlowTracker requires Tobi Oetiker's RRDtool package. Screenshots are available.
flow-extract
which can be used to filter flow-tools-recorded flows through user-specified tests
a set of "Inter.netPH contribs"
by Horatio B. Bogbindero
some patches and a Python module
by Robin Sommer.
flow-pairs
A script that extracts lists of the highest bandwidth consumers by host and by port. Installed at UCB. Seems to have similar uses as the older MATHE system.
Net::Flow NEW:
Perl module for de- and encoding Netflow (v5/v9) and IPFIX packets.
jflow
A set of Java classes for collecting and analyzing NetFlow data. Supports Netflow versions 5 and 6, multithreaded implementation to facilitate real-time traffic accounting and analysis.
Autofocus
A traffic analysis and visualization tool that describes the traffic mix of a link through textual reports and time series plots. The underlying research is documented in a SIGCOMM 2003 paper, Automatically Inferring Patterns of Resource Consumption in Network Traffic, C. Estan, S. Savage, G. Varghese (PDF paper, PPT slides).
Wisconsin Netpy
Netpy is a network traffic analysis and visualization package developed at University of Wisconsin-Madison. This application is intended for the use of network administrators and it can help understand usage trends in your network as well as support interactive analysis of specific network events of interest. Netpy is distributed under GPL and a BDS-like license. Netpy stores NetFlow records in a local database after applying some sampling to reduce the size of the data. The analysis engine supports interactive analyses on this data where the user chooses the time interval of interest, the filtering rules to apply to the traffic and the type of analysis. The netpy console allows the user to manage the database, and perform analyses interactively or through scripts. The graphical user interface visualizes the results of the analyses accessing the database locally or remotely through a netpy server that is also part of the package.
Stager
Stager is a system for aggregation and presentation of network statistics from the flow-tools package. Includes PostgreSQL storage of aggregated statistics, as well as a Web frontend. A public demo is available.
nfstat
Developed to analyze (sampled) Netflow data from the Internet2 Abilene backbone. This is used to generate the Internet2 NetFlow Weekly Reports, which contain interesting statistics not easily found elsewhere, such as distribution of bulk flow throughput. There are two mailing lists for announcements and for user discussions, respectively.
CAIDA cflowd
Rather complex system with distributed log servers. Released in 1998, this was the first open-source software system to work on NetFlow data, but doesn't seem to be maintained anymore. CAIDA have prepared a nice FAQ which contains interesting information both on Cflowd and on NetFlow in general. CAIDA has announced that they no longer support cflowd, and recommend that people move to flow-tools instead.
Aflow
Small Netflow monitoring tool developed by ARIN, available under GPL. Features include easy configuration, maintenance of and graph generation from RRDtool files, pf/tcpdump-style filter rules. There is a mailing list for announcements and discussion.
ASFLOW (already missing in action?)
Tool to analyze traffic to "would-be" BGP neighbors. Presented by Richard Steenbergen and Nathan Patrick at NANOG 35, October 2005. There is supposed to be both an easy-to-use Perl version and a high-performance (but somewhat complex) C version.
Fluxoscope
Software used for charging, monitoring, and traffic analysis at SWITCH. Includes its own NetFlow v5 accounting receiver which aggregates traffic into multidimensional matrices (AS/site/application). Most of the software is written in Common Lisp.
UDP Samplicator
A small program that receives UDP datagrams and redistributes them to a set of receivers. Useful to distribute NetFlow accounting streams to multiple post-processing programs. Is able to distribute only a specified percentage of all packets to each receiver. Note that recent versions added the possibility of ``spoofing'' the original sender's IP address.
Anonymization Application Programming Interface (AAPI)/AnonTool
An open-source implementation of Anonymization API. Includes a set of ready-to-use applications for anonymization of Netflow (v5 and v9), as well as PCAP traces.
CANINE
"A NetFlows Conversion/Anonymization Tool for Format Interoperability and Secure Sharing". Converts NetFlow data between various formats including NetFlow v5 and v7, NFDUMP, CiscoNCSA and ArgusNCSA, and is able to apply various methods of anonymization based on user configuration. See also the FlowCon 2005 paper by K. Luo, Y. Li, A. Slagell, and W. Yurick.
Panoptis
An open-source project started in 2001 by Costas Kotsokalis of GRNET. Uses NetFlow accounting data to detect (Distributed) Denial of Service attacks. Status as of November 2006: Supports NetFlow v1, v5 and v8 (router-aggregated) (with v8 untested for its biggest part). The system supports proof-of-concept attack trace-back using a mesh of detectors. Updates have been introduced so that the project compiles on newer systems.
Flamingo
Real-time 3D traffic visualization system developed at Merit. This client/server system based on Netflow and OpenGL plots traffic patterns by IP address, AS, or port numbers, and allows interactive exploration of this data. Sample graphics and a paper are available from the Website.
MHTG (Multi Host Traffic Grapher)
Uses NetFlow to generate per-host graphs of traffic for a campus network. Nice user interface implemented as a Java applet which allows interaction with traffic plots. The software consists of a C++ program to process NetFlow data, a Mysql backend, and Perl frontend and the Java grapher.
Matt's Quick & Dirty CFLOWD tutorial and scripts...
Postprocessing scripts for cflowd data by Matthew Petach
flow2rrd.pl
Converts a cisco NetFlow stream into set of RRDtool files, based on set of IP netmasks. By Alex Pilosov.
bmpcount
A library of bitmap counting algorithms that count the number of active flows in a network traffic trace. To be able to use it, you should be familiar with the paper that describes the algorithms it implements: _Bitmap algorithms for counting active flows on high speed links_, C. Estan, G. Varghese, M. Fisk, Internet Measurement Conference 2003 (PDF paper, PPT slides)
Slate
An application that converts LFAP data into NetFlow records - see http://www.nmops.org/.
Ntop
This well-known libpcap-based network usage monitor has been extended to produce NetFlow v5 accounting data. It also supports sFlow.
SiLK
SiLK, the System for Internet-Level Knowledge, is a collection of netflow tools developed by the CERT/NetSA (Network Situational Awareness) Team to facilitate security analysis in large networks. The toolset includes programs such as rwfilter, rwcount, rwuniq. There are plans to develop this further into an "Analyst's Desktop", described in a FloCon'05 paper, R: A Proposed Analysis and Visualization Environment for Network Security Data, J. McNutt (PDF). (Ed.: Should this be "RAVE: A Proposed..."?) The idea is to base this on the R statistical programming language (see www.r-project.org), which supports exploratory data analysis well.
Java Netflow Collect-Analyzer
Collects Netflow v1/v5/v7/v8/v9 packets from Cisco/Juniper routers or nProbe. It can store both raw data or analyzed contents to a database using JDBC.
UPFrame
This UDP/Netflow Processing Framework is a system for real-time processing of UDP packet streams such as Netflow export data. It features a general infrastructure for dynamically configurable plugin modules.
nProbe
A small self-contained program that generates NetFlow accounting data for a traffic stream sniffed off one or several interfaces. Works under Unix and Windows environments. It can be used to build inexpensive NetFlow probes.
fprobe (I)
Traffic probe that can generate NetFlow data. Based on the libpcap library. Fairly small implementation in C.
fprobe (II)
Another NetFlow-generating software traffic probe.
Softflowd
Traffic probe that can generate NetFlow data. Based on libpcap. Comes with a NetFlow collector in Perl. Both the server (probe) and client (collector) support export/import over IPv6. Very lean (as of June 2004) implementation in C.
The pfflowd variant is based on OpenBSD's PF interface.
The flowd companion NetFlow collector includes features such as multicast, IPv6 and NetFlow v9 support, as well as fast upfront filtering.
Argus from QoSient
This network Audit Record Generation and Utilization System can be used for intrusion detection and QoS monitoring. It is also mentioned in the reference section of these pages.
RENETCOL (RENATER Network Collector)
GPL'ed Netflow collector with support for Netflow v9, IPv6, Multicast, and MPLS.
Flowc
"a tool for gathering, storing and analyzing traffic accounting for Cisco routers with NetFlow enabled switching (version 5). This package could be used by ISP for planning, analysis and billing procedures."
CESNET NetFlow Monitor
by Jan Nejman.
RUS-CERT tools
The CERT of the Stuttgart University computing center (RUS-CERT) has published some tools that they use internally to analyze Netflow data. Some of the documentation is in German.
pmacct
A set of tools to account and aggregate IP traffic. Supports libpcap, Netflow v1/v5/v7/v8/v9, and sFlow v2/v4/v5 for both IPv4 and IPv6 traffic.
NEye
NEye is a Netflow V5 collector. It logs incoming Netflow V5 data to ASCII, MySQL, or SQLite databases, and it makes full use of POSIX threads if available. It works on most major platforms (Linux, Solaris, AIX, Irix, HP/UX, Mac OS X, Digital Unix, etc.) and older ones too (Ultrix, Nextstep, etc.).
NetFlow2MySQL, NetFlow2XML, and pcNetFlow
Three products from a research project at the NARA Institute of Science and Technology.
F.L.A.V.I.O. (see also the FreshMeat page)
A Perl-based NetFlow collector that stores flow data "into a MySQL database and gets it back to graph daily, weekly, monthly and yearly charts."
NetFlowMet
Starting with release 4.2, Nevil Brownlee's NeTraMet package includes NetFlowMet, which implements an RTFM meter fed on Netflow accounting data.
NetFlow Accounting software from ABPSoft
A self-contained NetFlow processing system written in C. Writes captured flows to file. Postprocessor breaks up this data over peers according to a definition file.
EHNT (Extreme Happy NetFlow Tool) by Nik Weidenbacher
Another self-contained NetFlow accounting packet processor. The receiving process also functions as a server to which various kinds of clients can connect. Also written in C.
Hendrik Visage's NetFlow tools
FTP site with various tools for NetFlow postprocessing. In particular, you will find:
  1. a UDP duplicator (hack of samplicator to preserve the source router IP)
  2. a couple of hacks to cflowd for dumping the flows every %n seconds as well as a "flhh" to output flowdump stuff aggregated, ready for a `grep|sed "s/../update  /"|rrdtool -`
netMET - Network's METrology
Network measurement solution for the French regional academic networking community, developed at the C.I.R.I.L in Nancy. Includes an HTML interface and support for accounting and security monitoring.
MATHE
An article (in French) about a Netflow accounting and visualization system used at EPFL. Uses an Oracle database and Perl DBI/GD scripts to generate a nice breakdown of external traffic to departments/institutes.
JANET Traffic Accounting Site
An impressive application of Netflow which is used for volume-based charging for JANET's U.S. connection. Other statistics at JANET were done using NeTraMet.
InMon sFlow Toolkit
Open source tools for analyzing sFlow data. Allows sFlow data to be used with a number of open source tools, including: tcpdump, snort and MRTG or rrdtool. Also able to convert sFlow packets to NetFlow packets.
Net::sFlow
Perl module to parse sFlow messages. Written by Elisa Jasinska from AMS-IX as a basis of the sFlow-based traffic analysis service for AMS-IX members. The use of this at AMS-IX has been described in presentations and a paper, links to which can be found in the references section.
Webview Netflow Reporter
Webview Netflow Reporter is an enterprise-focused Netflow reporter/analyzer tool featuring clickable graphs, powerful categorization that goes beyond simple TCP/UDP port names, automatic exporter discovery, and full access to all aspects of the raw flow data (millisecond accuracy, QoS settings, TCP flags, etc). It uses flow-tools and/or flowd as a collector.

Commercial Applications

Andrisoft WANGuard
The Andrisoft WANGuard Platform relies on NetFlow v.5 or Port Mirroring / SPAN to provide in-depth network traffic analysis and DDoS detection and mitigation. It can be used to generate traffic graphs and traffic accounting reports per IP, per subnet, per IP Zone or per router interface / switch port.
Watch4net APG (Automated Performance Grapher)
APG is a reporting tool that provides performance and capacity reports on network, servers, applications and Netflow data.
Apogee Networks
The NetCountant network usage-based billing system and the NetScope real-time network monitoring and performance analysis solution support NetFlow, RMON2, RADIUS, other SNMP MIBs, and ``Layer 7'' application/content switches.
Arbor Networks
Peakflow DOS detects denial-of-service attacks, and Peakflow Traffic analyzes traffic and routing history. Both can process NetFlow accounting data. As of November 2003, Arbor is said to support Netflow v9.
Network Signature BENTO
BENTO stands for ``BGP Enabled Network Traffic Organizer'' and is a high-performance NetFlow data processor with an integrated BGP-4 implementation to facilitate traffic analysis based on complex external routing relationships. Product offerings include a software/support package and an ``appliance'' consisting of a preconfigured rack-mount server.
Caligare Flow Inspector and NetImonitor
Analyzes NetFlow data for network monitoring as well as attack detection and response. Works with NetFlow data export version 1,5,6,7 and 9. NetImonitor is primarily designed for use in the United States.
Cisco
NetFlow FlowCollector/Network Data Analyzer
Similar to cflowd but productized, with a (Java-based) GUI and possibly better possibilities of defining filters and aggregation schemes.
Cisco NAM (Network Analyzer Module)
This is a "NetFlow collector on a linecard" for the Catalyst 6500/7600 OSR platform.
Concord
Network Health uses NetFlow and RMON2 accounting information ``to determine application, bandwitdth and server usage.''
FlowFe NEW:
FlowFe is a Netflow v5 and v9 collector and front-end with an SQL backend for accurate real-time and historical reporting. It also has the ability to save reports as PDFs for archival purposes.
Crannog Software's (now Fluke Networks) Netflow Monitor
LAN and WAN bandwidth analysis based on NetFlow data. Includes a Web interface including Java applets to display traffic graphs and to enable drill-down. Runs on Microsoft Windows NT4/2000/XP and on Unix. Evaluation version of NetFlow Live available. Note that Crannog has been acquired by Fluke Networks in January 2007, and rebranded this product as NetFlow Tracker.
Cyclades-nQuirer
A network traffic monitoring appliance that can generate data in both Netflow and nTop formats.
Digiquant
IMS accounting and billing system based on Oracle 9i under Unix.
Gadgets Software & Professional Services Ltd.
Network Intelligence traffic measurement and visualisation software for GNU/Linux and Windows (client only) platforms. Free trial available. Includes 3D visualization using OpenGL.
The author also wrote bbnfc, a ``bare-bones Netflow collector tool'' that simply receives and displayes Netflow v5 packets.
Hewlett-Packard
The Smart Internet Billing Solution usage management system and well as OpenView Performance Insight for Networks (OVPI) use NetFlow accounting data as possible input.
Infosim StableNet - Performance Management Engine
StableNet PME provides End-to-End (E2E) Service Level Management (SLM) by monitoring and reporting on the systems, networks and applications. StableNet supports the following flow technologies out of the box: Netflow, cFlow, sFlow, Netstream.
InfoVista Corporation
InfoVista Service Level Management (SLM) and conformance solution.
InMon Traffic Sentinel
is a commercial, web-based application running on Linux that provides real-time and historical analysis of flow information from NetFlow, sFlow, LFAP or HP Extended RMON sources. Web queries provide easy access to historical traffic matrices. Real-time top talker charts identify sources of congestion. Includes network-wide threshold and alert features as well as anomaly detection.
IsarFlow from IsarNet
IsarFlow is a traffic analysis tool for accounting, capacity planning, QoS monitoring, and application distribution within Citrix sessions based on Netflow.
Ixia
IxTraffic integrates NetFlow accounting data with topology information from a live BGP-4 feed to allow analysis of inter-domain traffic patterns.
Lancope StealthWatch
Flow-based Network Behavior Analysis appliance with advanced user identity tracking. Can handle Netflow and sFlow data, or capture packets from mirrored ports.
LoriotPro
A network monitoring ("supervision" in franglais) system that includes a Netflow plugin. Stores flow data in a MySQL database.
ManageEngine NetFlow Analyzer
Netflow-based bandwidth monitoring tool from AdventNet. Supports location of bottlenecks and allows drilling down to find traffic that is causing them. Thirty-day evaluation license available free of charge. Versions for Windows and Linux (x86).
Mazu Networks
Mazu Profiler analyzes and models enterprise network traffic. It provides visibility into network behavior, protects against worms and other malware, and supports auditing and policy enforcement. It supports Netflow v1/5/7/9 as well as other data collection mechanisms.
Micromuse
Cisco Info Center USM ``acquires, analyzes, displays and exports Internet usage data.'' Note that Micromuse was integrated into IBM under the "IBM Tivoli Netcool" brand.
NARUS
OSS Mediation solutions. They also do anomaly detection.
Nazca.Billing
Integrated billing software for "Telephony, Internet and Networks". Contains interfaces to many accounting systems including NetFlow.
NetQoS ReporterAnalyzer
Scalable solution for network capacity planning, troubleshooting, and traffic analysis, including traffic visualization capabilities.
NetReflex by Guavus NEW:
Network-wide analytics and anomaly detection platform. The system fuses traffic and routing data, builds traffix matrices, and performs anomaly detection and classification.
NetUp Products
UTM is a billing system for ISPs. It can use Netflow (v5) and several other accounting methods. It supports a rich variety of charging and payment schemes.
NDSAD Traffic Collector is an open-source (GPL'ed) tool that captures packets and generates a Netflow (v5) accounting stream.
NetUsage from Apoapsis (formerly called WANBUS)
The NetUsage suite strives to provide visibility of network traffic, producing meaningful reports not only for network professionals, but for IT management, business managers and accounts departments. Supports network traffic monitoring, capacity planning, business justification and cost control.
SolarWinds Orion NetFlow Traffic Analyzer
Windows-based commercial system that stores NetFlow data, generates various types of charts, and provides "drill-down" capabilities.
PRTG Traffic Manager
Windows-based bandwidth management software from Paessler. Uses SNMP, Netflow, and packet capture for monitoring and classifying bandwidth usage. Besides the commercial license, there is also a (restricted) "freeware" license.
QRadar from Q1 Labs
The system can use Netflow data, but also includes its own payload-aware flow collector which produces bi-directional flow information in a format called QFlow. Includes anomaly detection.
Plixer Scrutinizer NetFlow Analyzer
NetFlow-based Enterprise-level traffic analysis tool with GUI-based reporting (topN hosts/applications etc.) and zoom/drill-down. Uses MySQL back-end. Free (as in free beer) edition available.
I-ABA and M-NTM from Tek Yazilim
Windows-based software to analyze NetFlow (and Cisco IP Accounting) statistics. I-ABA specifically analyzes AS-to-AS traffic streams. Trial versions can be downloaded.
Quallaby
Has a Netflow Application Pack for its PROVISO system for network performance monitoring and service assurance. Quallaby was acquired by Micromuse, which was itself acquired by IBM. The Netflow Application Pack is maintained in the 4.4.1 release and supports Netflow versions up to v8.
NetScout
nGenius Performance Manager ``is a complete solution for proactive monitoring, troubleshooting, capacity planning, and Voice over IP (VoIP) monitoring''.
Portal Software
Infranet real-time customer management and billing software.
RODOPI
Billing software for ISPs.
XACCT
Commercial vendor of accounting and billing solutions with the ability to process (among others) Netflow accounting data

No comments: