SmarterStats and how to batch add IP address filters

by Ted Jardine Monday, April 22, 2013 3:25 PM

If you're using SmarterStats as one means of tracking site traffic (if site stats mean anything to you you should be using at least two means of tracking) that is also monitored by Pingdom (or any other monitoring service, but we highly recommend Pingdom), you'll find your stats are *slightly* increased by Pingdom's bots (which come from around 30 monitoring locations around the world with unique IP addresses for each). For example, if you're pinging your site once per minute that adds up to somewhere around 43,000 visits per month - a not so lovely way to pad your stats.

Fortunately, SmarterStats provides a means to filter out these requests. Unfortunately, SmarterStats naively only allows for one-by-one entry via a relatively clunky interface and can only filter on IP address, IP address range, host header inclusion/exclusion, file, and directory. We have contacted SmarterStats multiple times in the past for a means to batch import IP addresses or better yet, filter requests based on a specific user-agent request header (which you can set in Pingdom), but no solution has been forthcoming. We've added Pingdom-specific host headers where feasible, but for the most part we've had to make do with a very manual (and time consuming) process.

This state of affairs is so wrong on so many levels so here I am to say "no more!" Bring out some scripting and enough already. 

Four caveats:

  1. Tested only with SmarterStats v6. Future or past versions may not operate similarily. 
  2. You'll need direct access to your server's SmarterStats config files. 
  3. The script is for *nix-based systems and is only tested on a Mac. Therefore, Linux (whatever flavour as long as it's got uuidgen available) should work fine. Windows will be a no go. Sorry. Feel free to write a Powershell equivalent.
  4. If this busts something or kills your hamster, we're not responsible.

As the Lorax would say, "You have been warned!"

The old way

For fun and jollies, review the usual, and until now only way, to manually add an IP address filter in SmarterStats:

  1. Log in as a site admin in SmarterStats.
  2. Go to "Settings".
  3. Go to "General Settings".
  4. Go to "Import Filtering".
  5. Click "Add > Import Filter".
  6. Select your import filter type, add your IP address, check "Make exclusions permanent" and click "Okay".
  7. Look at your list of remaining IP addresses.
  8. Look at the SmarterStats interface.
  9. Add another IP address.
  10. Forget to click save before navigating somewhere else. Lose all your entries.
  11. Look at your list of remaining IP addresses.
  12. Squint and try to determine which IP addresses you added and which ones you missed.
  13. Count them again.
  14. Cry.

And now for enlightenment

The first step is getting the goodies in the script. Relatively simple, but it does the trick:

# DESCRIPTION: Script to generate a list of all the latest
# Pingdom monitoring servers and export into an XML format 
# recognized by SmarterStats in a site's configuration file 
# (typically located in C:\Program Files (x86)\SmarterTools\SmarterStats\MRS\App_Data\Config\Sites\Site##.xml)
# USAGE: ./ | pbcopy to copy to clipboard # or ./ to simply output to console.

ips=`wget --quiet -O- | \
        grep "pingdom:ip" | \
        sed -e 's|</.*||' -e 's|.*>||'`

for ip in $ips
        guid=$(uuidgen | tr '[:upper:]' '[:lower:]' | tr -d '-')

        echo "<ImportFilter>
All the above does is get the current list of Pingdom's monitoring servers, parses out the IP addresses, and loops through each while generating a unique UUID (GUID for Windows speakers) for each along with the surrounding necessary bits and pieces. Once you've got the above script located somewhere nice on your local machine, do the following:

  1. Contact SmarterStats and ask them why they don't have a filter based on a request header user agent already.
  2. Make the above script file executable. For example, assuming the script file is at ~/, simply:

    chmod +x ~/
  3. Generate the necessary XML and stick it in your clipboard by running the following (adjusting file name and location as necessary):

    ./ | pbcopy
  4. Determine the applicable site ID/s. The easiest way is in SmarterStats site listing.
  5. Make sure the site in question is not currently running an import. You may wish to stop your SmarterStats service. YMMV.
  6. Locate the site's respective config file on the site's host server. This file is typically in C:\Program Files (x86)\SmarterTools\SmarterStats\MRS\App_Data\Config\Sites\Site##.xml where the ## is the site ID determined in the previous steps. 
  7. If it will be a lot of work to recreate the site from scratch, make a backup of the config file before proceeding any further.
  8. Open the config file in your text editor and right after the <TimeZoneIndex> entry, add in your multiple <ImportFilter> entries which should be in your clipboard. If you can't find this, make a dummy filter request entry via the Web GUI and then located it in the config file afterwards.
  9. Save after making sure you didn't muff anything up.
  10. Reload your filter settings for the site and see all 30+ entries. Rejoice. Re-index if you need to filter out bad data from before. Rejoice again.
  11. Repeat for all sites that use SmarterStats and are monitored by Pingdom. Note that you'll want to regenerate the XML in order to have unique UUID's for each entry across all sites.

Tags: analytics

blog comments powered by Disqus

Month List