CS-MARS and Archiving - Part 1

The MARS appliance stores data on-box in raw format, ie, uncompressed. It breaks event data into ten different partitions.

Remember from an earlier post, the command "pndbusage"

[pnadmin]$ pndbusage
Current partition started on Tue Aug 8 00:41:54 BST 2006 and uses 30.7% of its available capacity.
Switching to next partition is estimated for Thu Mar 22 18:23:29 GMT 2007.
9 empty partitions are available for storage

So in the above example we have plenty of space available, but what if this was a busy MARS box? Planning your Archiving and Space Requirements becomes very important.

The MARS appliance uses a FIFO approach, and when the internal storage reaches its capacity, it automatically purges the data in the oldest of the ten partitions, roughly 10% of the stored event and session data. Ouch!

Some rough estimations (1st example via Cisco Press Book)....

1) Assume we are receiving 100 events per second, and an average of 200 bytes/event. This gives for 1 day (100x200x86400 secs)=1,728,000,000 bytes of data or 1.72GB of storage required a day.

Remember from the "earlier post" the M20 has 77GB of storage onbox available, but minus 20% of this for tables/indexing etc, with 61.6GB remaining. Now 61.6/1.72 gives approx 36 days worth of storage.

2) Assume we are receiving 50 events per second, and an average of 200 bytes/event. This gives for 1 day (50x200x86400 secs)= 864,000,000 bytes of data or 864MB of storage required a day.

Remember from the "earlier post" the M20 has 77GB of storage onbox available, but minus 20% of this for tables/indexing etc, with 61.6GB remaining. Now 61.6/0.864 gives approx 72 days worth of storage.

It is obviously apparent that we need some form of Archiving the data, especially if you have a busy network, and are following compliance regulations.

The good news, is that we can archive data from a MARS appliance, and use this data to restore the OS, system configuration settings, dynamic data (event data) or the complete system.

This is done via NFS, but you cannot schedule when the data backup occurs. The MARS appliance performs a configuration backup every morning at 2am, and event data is archived every hour.

The not so good news, is that dynamic data (event data) archived, includes only the data that is received or generated after you enabled archiving.

Therefore, it is recommended that you enable archiving and plan your storage requirements before you configure your MARS appliance to receive events from reporting devices!

Once you have enabled archiving via NFS, dynamic data is then written by MARS once to its local database, and once to the NFS Archive (and this is compressed with the Ziv-Lempel algorithm, also used in GZIP).

It is recommended that a reliable network connection of at least 10MB/s exists between the NFS Server and MARS Appliance. You should consider using the eth1 interface on the MARS appliance, to ensure that the backup operation is not competiting with other operations on the MARS box.

Another thing to note, is that you can use the same NFS Server for archiving multiple MARS appliances, BUT, you must specific unique directory's in the NFS path for each appliance, or you will corrupt the archives.

Now when we configure the MARS box to archive, we need to set a "Remote Storage Capacity" in days. ie, to keep a years worth of data on the NFS server, we need to specify 365 days!

In planning the NFS space requirements, the same formula used for on-box storage is used, but the result is divided by the compression ratio.

Using the previous example, of receiving 100 events per second, and an average of 200 bytes/event. This gives for 1 day (100x200x86400 secs)=1,728,000,000 bytes of data uncompressed, or (100x200x86400/10) = 172,800,000 bytes compressed = 172MB per day.

Or for a year (172MB x 365) = 63072000000 bytes of data = approx 63GB.

In Part 2 of this document, i`ll go into setting up a NFS Server in Windows, and the format of the Archived Data, and how we can restore via the pnrestore command.