Friday, October 27, 2006

How does MARS use NetFlow information?

Netflow is a Cisco technology, that supports monitoring network and is supported on all basic IOS images.

Netflow is an efficient method of monitoring high volumes of traffic compared to traditional SYSLOG and SNMP methods. To receive comparable syslog data from a firewall device, the syslog logging level on the firewall would need to be set to DEBUG.

CS-MARS uses NetFlow Version 5 or Version 7. Ideally Netflow should be collected from the Core and Distribution switches in your network.

Netflow data enables MARS to identify anomalies by profiling typical data flows across your network. This allows MARS to detect day-zero attacks, including worm outbreaks.

MARS uses NetFlow data and Firewall traffic logs to

1. Profile the network usage - Statistical profiling takes between four days and two weeks for a MARS appliance to complete.
2. Detect statistically significant anomalous behavior and traffic flows (from computed baseline) and creates incidents in response to them.
3. Correlate anomalous behavior to attacks and other events reported by Network IDS systems.

The NetFlow data and Firewall traffic logs are treated uniformly since they both represent traffic in an enterprise network.

After getting inserted into a network, MARS learns the network usage for a few days. Then it switches to a detection mode where it looks for statistically significant behavior, e.g. the current value exceeds the mean by 2 to 3 times the standard deviation.

CS-MARS profiles the network at a detailed level and keeps a profile of top combinations of flows learned and packets exchanged for every hour, every day of the week.

By default MARS does not store the NetFlow records in the database because of the high data volume, although MARS can be configured to do so.

Upon detecting an anomalous behavior, MARS starts to dynamically store the full NetFlow records for the anomalous (host, port) - thus the full context of a security incident, e.g. the infected source,destination etc. is easily available to the administrator.

CS-MARS can be configured to profile hosts that belong to a set of known valid networks. This reduces MARS memory usage.

Upon detecting an anomalous behavior, CS-MARS starts to dynamically store the full NetFlow records for the anomalous activity. This intelligent collection system provides all the information that a security analyst needs and does not overburden CPU and disk resources.

MARS provides built-in rules for automatically correlating anomalous behavior with attacks reported by Network IDS systems - such a rule can easily capture a successful attack since a compromised host is likely to exhibit anomalous Network DoS behavior of the form
• Sending too many e-mails containing a worm
• Sending excessive traffic on a particular port looking for additional systems to compromise

IE, say that CS-MARS learns that most connections to and from a server in your environment are made over ports 137, 138, and 139. Then suddenly that server makes multiple outbound port 25 connections through the default gateway. CS-MARS would trigger a "mass-mailing worm" incident to notify the operators of the change in behavior.

Even without a Network IDS system in place, a host compromised by an unknown attack is highly likely to exhibit anomalous behavior of the form described above and hence will be easily detected by the MARS Appliance.

CS-MARS uses NetFlow and Firewall Log information to graphically depict port-level activity on your network using a graph that measures protocol connections over time.

CS-MARS Activity: All Events and NetFlow Chart

And you can run queries too.

As can be seen in the picture MARS has reported huge deviations from typical port 80 activity. This would obviously be more interesting with a day zero worm or virus , using a non statndard port, but this is an example of behaviorally profiling port activity.

Often when these malicious programs are released, they do not contain names or have detection signatures for them. Your organization's only defense against detecting and mitigating day-zero attacks at a network level is to use CS-MARS. Firewall data and NetFlow information give you this unique capability.

A word of warning though about Netflow. Too much will increase the CPU requirements of your MARS Appliance. MARS does though have a couple of warning alerts, can can give you an indication you have undersized the right appliance for your network. See below.

Thursday, October 26, 2006

Analysis and Identification in Action using Cisco MARS

The guys over at Priveon have a good article here on Cisco MARS in action in the Analysis and Identification of the Blackworm email worm. (CME-24)

If you dont know Blackworm, also known as BlackWorm/Nyxem/Blackmal/Blueworm/Grew was scheduled to delete certain file types on Feb 3, 2006, a very nasty piece of work.

More info on Blackworm here and more info on CME (Common Malware Enumeration) Identifiers here.

Friday, October 20, 2006

CS-MARS NAC Reporting and Functionality

MARS supports the Cisco NAC Framework by storing and reporting on NAC based events generated by the various reporting devices on your network.

I`m not going to go into how to set NAC up, or how to configure your NAC implementation in MARS, since this can be found in the MARS Userguides.

What i will try to demonstrate here, is how MARS can aid you in reporting on your Network Admission Control setup.

Consider MARS has started to alert us to P2P traffic in the network.

Drilling down into the incident, will give us the Source and Destination IP Addresses.

And clicking on an internal Source Address will give us more information about the host. Now we nearly always have some Static Info available to us, but in a NAC environment we have Dynamic Info, which could be continually changing.

Static vs Dynamic Info for a 802.1X Reported Host.

And the same again if we choose to try and mitigate the host, since we are mitigating an 802.1X host, we get more info.

The information you get on these screens will obviously depend on what authentications you are performing in your 802.1X setup, with the above performing anonymous machine authentication.

You get similar dynamic info for a host performing L2/L3_IP NAC assessment.

Now onto REPORTING, this is where MARS overcomes some of the short falls with Cisco ACS.

Out of the box, MARS comes with some ready to go NAC reports as shown below...

I have run a couple of the reports below, for 802.1X and L2IP Top Tokens and Sessions.

802.1X NAC Report - Number of Healthy/Unhealthy Tokens over the last hr

802.1X NAC Report - Healthy/Unhealthy Sessions Listed by Time over the last hr

L2IP NAC Report - Number of Healthy/Unhealthy Tokens over the last hr

L2IP NAC Report - Healthy/Unhealthy Sessions Listed by Time over the last hr

There is also a Detailed NAC Report that can be run over time, that gives far more information including Posture States, NAS Port and CTA Versions reported by Hosts.

There will be a demo of this on the demolabs website shortly, so you can see this on a live network.

I think in the later versions of MARS, this reporting functionality will become far better, and a few of the niggly MAC reporting issues will be fixed.

Thursday, October 19, 2006

MAC Address Reporting

Sometimes i get asked whether MARS can give me the MAC address of host ABC.

Well yes it can, via one of its built in Queries.

In the example above, i have selected the source address as a single host. This could obviously of been a whole subnet, or select number of hosts.

We need to select the Result Format as MAC Address Report, and submit, over a given time period.

The result given below, will list the known MAC Address for the host, and which device it has queried for it, and more importantly the last time it checked the data.

Now you will notice an "i" next to each IP Address. Clicking on this will give more information that MARS holds for the given host.

Usually MARS would report IP Address, NetBIOS name and Domain if known, but if you are integrating with a 3rd Party Vulnerability Scanner like Qualys or Foundstone or eeye, you will get the host OS, application and vunerability data to.

Normal or Italic Text?

Well this one may seem really obvious, but it confused me for a few minutes.

If when you look at an Incident or Session, you see some Source or Destination addresses in Italic Font, where as the rest are in Normal Font. What does this mean?

Well its dead simple! It means you have already looked at that Particular Source or Destination IP`s Static/Dynamic Info for that particular Session. Like a visited link on a webpage. Doh!!

Try it out, if you haven`t noticed it already.

Monday, October 16, 2006

CS-MARS Database Info

CS-MARS Database Info

The CS-MARS appliance uses an Oracle 9.2i Enterprise database. The database is fully licensed for operation on the appliance and requires no administration whatsoever; therefore, it is completely self-sustaining.

CS-MARS Database Structure

You will not find much info about the database anywhere, as it looks like a Cisco secret, but searching blogs, and reading the Cisco Press book below, here is what we have come up with.

Each CS-MARS appliance has its own database storage requirements.

The actual event-data storage is alot smaller than the Total Storage available to each appliance. The remainder of the storage is used for other database on the box, that comprise configuration files, reports, vunerability data etc..

There were also a couple of vunerabilities in the Oracle database earlier this year, in how to modifiy the "expert" username and password to get root access to the appliance. Thank fully these have been fixed now, if you are running the later releases.

Mars Appliances Events Storage and Total Storage
(reference Cisco Security MARS - Cisco Press)

When storing event data in its database, CS-MARS stores it in its raw format, uncompressed, using a first-in, first-out (FIFO) approach. When the 77 GB of storage is reached in a M20, it wipes out the oldest day (database) of event data. This data is lost if you are not archiving the data. This process allows CS-MARS to have room for event data in case a network infection or an attack happens.

How do we view the amount of storage being used on our MARS appliance?

SSH into the box and run the command "pndbusage" This will give a result similar to below...

[pnadmin]$ pndbusage
Current partition started on Tue Aug 8 00:41:54 BST 2006 and uses 30.7% of its available capacity.
Switching to next partition is estimated for Thu Mar 22 18:23:29 GMT 2007.
9 empty partitions are available for storage

This command displays the percentage used within the current partition, as well as specifies whether additional partitions are available. If no unused partitions exist, the command identifies which partition will be purged, provides an approximate schedule for when that purge will occur, and specifies the date range and total number of events scheduled to be purged.

If the database was full, then you would get an output similar to this...

Current partition started on and uses % of its available capacity.
Switching to next partition is estimated for events, received between and will be purged.

A word of warning if you are running CS-MARS 4.2.1, that there was an open caveat, for this, as detailed below...

Reference Number: CSCse54808

Issue: The time stamp shown by the pndbusage command is incorrect

Description: Two consecutive uses of the pndbusage command display a different current partition starting time.

Workaround: None.

So if you have a very busy network, please make sure you size your CS-MARS box accordingly (ask your Cisco Account Rep, to size the box), and also think about using the Archive Feature.

There is a new event since v4.2.1, CS-MARS DB partition filling up causing the next partition to be purged soon, notifies the administrators when the current partition is 75% full and switching to the next partition will result in data being purged from a previously used partition.

The system inspection rule and report allow you to monitor when this event fires. The inspection rule is System Rule: CS-MARS Database Partition Usage, and the report is Resource Utilization: CS-MARS-All Events.

CS-MARS Number 1 in Security Event Management?

I am refering to John Katsaros blog entry below, posted recently.

Quote "We recently checked in with the Cisco MARS team and almost fell off our collective chairs when they told us that, since Cisco started marketing MARS (after getting the product line when it acquired security startup Protego in December, 2004), Cisco now has 2,200 MARS customers (MARS appliance shipments are higher since customers may buy several). Considering that Cisco has been seriously marketing the product for about a year, the run rate of new customer acquisitions must be in the 200 to 300 range. That's incredible since it probably means that MARS has jumped to the number one position in Security Event Management (SEM or SIM, whichever you prefer) market share based on units sold (our guess is that if you add up all the customers that the other SEM vendors have sold, Cisco would now have more than half of the total). That would probably make MARS the fastest growing part in Cisco's security product group. It goes to show the strength of Cisco's marketing prowess -- as a standalone company, Protego had produced an interesting product, but as a startup in the security space, was only getting a minor amount of traction in the SEM space. But in the hands of Cisco, customers view both the products and the category differently."

Thursday, October 12, 2006

Implementing Monitoring, Analysis and Response System (MARS) v2.0 Course

For those interested, there are a few training partners running the Cisco MARS Course, this is entitled "Implementing Monitoring, Analysis and Response System (MARS)", and your materials should be V2.0 of the course.

I know these training partners are running the course, but are are surely more as well..


Fundamental knowledge of implementing network security / CCSP or Security CQS and working knowledge of routing and switching / CCNA

Course Objectives

After completing this course the delegate will be able to:
Describe the MARS solution, features and functions in context to the issues of security incidents and security information in an enterprise network.
Cover the basic physical installation process.
Add Cisco security and network devices into MARS appliance.
Add Non-Cisco security and network devices into MARS appliance.
Configure security devices to generate interesting events that constitute an attack scenario and have MARS collect the interesting events for incident investigation.
Discuss attack mitigation and false positive confirmation in context to MARS appliance. Configure appliance to perform Incident Investigation and attack mitigation.
Explain how to create, view and save a long-duration query and reports on the MARS appliance.
Configure the MARS appliance to send an alert.
Describe and configure rules that detect interesting patterns of network activity.
Use management features in the MARS appliance to assign event, addressing, service, and user information.
Configure hardware maintenance chores like viewing audit trail, data archiving, hot swapping hard drives, upgrading software on MARS appliance.
Provide overview of MARS Global Controller.
Provide overview of Log Parser Templates.

Wednesday, October 11, 2006

Unable to send request for processing. Try again later.

On the Hotspot Graphs,/Attack Diagram or Mitigation Screens, if you get the error...

"Unable to send request for processing. Try again later."

Its a good idea to SSH into your CS-MARS box and check all the services are running.

How do we do that? Use the command "pnstatus"

For the above, examples, you can see from the "pnstatus" command, that something isnt right.

[pnadmin]$ pnstatus
Module State Uptime
DbIncidentLoaderSrv STOPPED
csiosips STOPPED
device_monitor STOPPED
discover STOPPED
graphgen STOPPED
pnarchiver STOPPED
pndbpurger STOPPED
pnesloader STOPPED
pnparser STOPPED
process_event_srv STOPPED
process_inlinerep_srv STOPPED
process_postfire_srv STOPPED
process_query_srv STOPPED
superV RUNNING 1-22:51:47

In this case, running the command "pnstart", started all the necessary services, and this box was back up and running.

More CLI commands, coming soon..

Tuesday, October 10, 2006

Cisco Security Manager - Policy Table Lookups

MARS can be configured to do Policy Table Lookups from Cisco Security Manager.

What does this mean? Well, when your MARS box receives a syslog from a Cisco PIX/ASA/FWSW/IOS device, and this event is sessionized, you will get a new "Security Manager Policy Table Lookup Icon" in the Reporting Device column..

If we click on this icon, then we can invoke a query to our Cisco Security Manager installation, and this will identify the access rule of the device, which created the incident.

Obviously looking at the raw event messages from the device would show the access-list, but not the particular entry in that list.

Prerequisities for Policy Table Lookup

MARS Local Controller running 4.2.1 or above
Cisco Security Manager v3.0.1 or above

See the example below...

We have a Cisco PIX being managed via the Security Manager, and have created some noddy access rules.

We have defined the CSM as a reporting device in MARS, and we are ready to go!

1) Log onto MARS as the Admin or Security Analyst Role

2) Identify the Incident

3) Click the Security Manager Policy Query Icon in the Reporting Field, to invoke the Cisco Security Manager Policy Table Lookup.

It is important to note here, that there are 3 pop-ups that can now appear, depending if there are mulitple events/devices that match the criteria.

a) Multiple Events Window (Shown) - Lists all the Security Manager device events, in the session

b) Multiple Devices Window - Lists all the matching SM devices that meet the criteria available to MARS

c) The Policy Table Window, shown when there is only one event, and a unique SM device indentifed.

4) Finally the Lookup Table is shown, and as can be seen the Access Rule on the device has been identified.

5) Finally we are offered advice on Mitigation, if we click the button, but remember the devices in CSM are Layer3 devices, so we cannot PUSH the configs. Any changes would need to be done MANUALLY on the device or via CSM.

Change Management i think its called!

Monday, October 02, 2006

MARS patch 4.2.2 (2303) available.

If you are having problems with the new release, and you have upgraded, rather than done a fresh build, a patch is available.


Running queries based on a pre-defined report with query type "Custom Columns ranked by time" will result in a pink error box being generated with message "java.lang.NullPointerException" and "System Error Please contact technical support".


This affects pre-defined report and custom queries with "Custom Columns ranked by time" [ex: IOS IPS DTM - All Events (Total View)]. There are ~70 of these types of reports. This also only affects users that have upgraded from 4.2(1) to 4.2(2) and NOT users that have installed directly from the 4.2(2) ISO CD.