SmarterStats and how to batch add IP address filters

by Ted Jardine Monday, April 22, 2013 3:25 PM

If you're using SmarterStats as one means of tracking site traffic (if site stats mean anything to you you should be using at least two means of tracking) that is also monitored by Pingdom (or any other monitoring service, but we highly recommend Pingdom), you'll find your stats are *slightly* increased by Pingdom's bots (which come from around 30 monitoring locations around the world with unique IP addresses for each). For example, if you're pinging your site once per minute that adds up to somewhere around 43,000 visits per month - a not so lovely way to pad your stats.

Fortunately, SmarterStats provides a means to filter out these requests. Unfortunately, SmarterStats naively only allows for one-by-one entry via a relatively clunky interface and can only filter on IP address, IP address range, host header inclusion/exclusion, file, and directory. We have contacted SmarterStats multiple times in the past for a means to batch import IP addresses or better yet, filter requests based on a specific user-agent request header (which you can set in Pingdom), but no solution has been forthcoming. We've added Pingdom-specific host headers where feasible, but for the most part we've had to make do with a very manual (and time consuming) process.

This state of affairs is so wrong on so many levels so here I am to say "no more!" Bring out some scripting and enough already. 

Four caveats:

  1. Tested only with SmarterStats v6. Future or past versions may not operate similarily. 
  2. You'll need direct access to your server's SmarterStats config files. 
  3. The script is for *nix-based systems and is only tested on a Mac. Therefore, Linux (whatever flavour as long as it's got uuidgen available) should work fine. Windows will be a no go. Sorry. Feel free to write a Powershell equivalent.
  4. If this busts something or kills your hamster, we're not responsible.

As the Lorax would say, "You have been warned!"

The old way

For fun and jollies, review the usual, and until now only way, to manually add an IP address filter in SmarterStats:

  1. Log in as a site admin in SmarterStats.
  2. Go to "Settings".
  3. Go to "General Settings".
  4. Go to "Import Filtering".
  5. Click "Add > Import Filter".
  6. Select your import filter type, add your IP address, check "Make exclusions permanent" and click "Okay".
  7. Look at your list of remaining IP addresses.
  8. Look at the SmarterStats interface.
  9. Add another IP address.
  10. Forget to click save before navigating somewhere else. Lose all your entries.
  11. Look at your list of remaining IP addresses.
  12. Squint and try to determine which IP addresses you added and which ones you missed.
  13. Count them again.
  14. Cry.

And now for enlightenment

The first step is getting the goodies in the script. Relatively simple, but it does the trick:

# DESCRIPTION: Script to generate a list of all the latest
# Pingdom monitoring servers and export into an XML format 
# recognized by SmarterStats in a site's configuration file 
# (typically located in C:\Program Files (x86)\SmarterTools\SmarterStats\MRS\App_Data\Config\Sites\Site##.xml)
#
# USAGE: ./pingdom.sh | pbcopy to copy to clipboard # or ./pingdom.sh to simply output to console.

ips=`wget --quiet -O- https://my.pingdom.com/probes/feed | \
        grep "pingdom:ip" | \
        sed -e 's|</.*||' -e 's|.*>||'`

for ip in $ips
do
        guid=$(uuidgen | tr '[:upper:]' '[:lower:]' | tr -d '-')

        echo "<ImportFilter>
        <Guid>"$guid"</Guid>
        <FilterValue>$ip</FilterValue>
        <MatchBehavior>Exclude</MatchBehavior>
        <Type>Ip</Type>
        <Permanent>True</Permanent>
</ImportFilter>"
done
All the above does is get the current list of Pingdom's monitoring servers, parses out the IP addresses, and loops through each while generating a unique UUID (GUID for Windows speakers) for each along with the surrounding necessary bits and pieces. Once you've got the above script located somewhere nice on your local machine, do the following:

  1. Contact SmarterStats and ask them why they don't have a filter based on a request header user agent already.
  2. Make the above script file executable. For example, assuming the script file is at ~/pingdom.sh, simply:

    chmod +x ~/pingdom.sh
  3. Generate the necessary XML and stick it in your clipboard by running the following (adjusting file name and location as necessary):

    ./pingdom.sh | pbcopy
  4. Determine the applicable site ID/s. The easiest way is in SmarterStats site listing.
  5. Make sure the site in question is not currently running an import. You may wish to stop your SmarterStats service. YMMV.
  6. Locate the site's respective config file on the site's host server. This file is typically in C:\Program Files (x86)\SmarterTools\SmarterStats\MRS\App_Data\Config\Sites\Site##.xml where the ## is the site ID determined in the previous steps. 
  7. If it will be a lot of work to recreate the site from scratch, make a backup of the config file before proceeding any further.
  8. Open the config file in your text editor and right after the <TimeZoneIndex> entry, add in your multiple <ImportFilter> entries which should be in your clipboard. If you can't find this, make a dummy filter request entry via the Web GUI and then located it in the config file afterwards.
  9. Save after making sure you didn't muff anything up.
  10. Reload your filter settings for the site and see all 30+ entries. Rejoice. Re-index if you need to filter out bad data from before. Rejoice again.
  11. Repeat for all sites that use SmarterStats and are monitored by Pingdom. Note that you'll want to regenerate the XML in order to have unique UUID's for each entry across all sites.

Tags: analytics

Generating Additional Statistics from Inmagic Genie

by Jonathan Jacobsen Friday, January 04, 2013 1:49 PM

Inmagic Genie includes a statistics page (usually found under Other in the left-side navigation menu) that provides a brief summary of important information about Genie. Out of the box, these statistics include:

  • Overdue Loans (total and by age: 1-7 days overdue, 8-14 days overdue, and more than 14 days overdue)
  • Overdue Serials 
  • Serials with Routing Lists
  • Catalog Records with Reserves

This is a good starting point, but we've found our clients want to know more about their collection and its usage. Some of the additional statistics we add to this page include:

  • Total Number of Catalog Records
  • Total Number of Item Records
  • Total Active Journal Subscriptions
  • Total Journal Issues (and other totals of specific types of materials)
  • Total Number of Borrowers
  • Total Loans per Calendar Year
  • Serials Not Received (differentiating between those marked Missing and those expected but not yet arrived)

It takes very little time to set these up, but they can then be accessed any time, by any authorized staff person.

These statistics are created by edited the MyStatistics.config Genie configuration file. Previously this required access to the server to edit this file. However, as of Genie version 3.5, you can enable access to all the configuration files through the Genie web interface. With this enabled, you would download the configuration file to your PC, edit it, and upload it again. 

Other statistical information can be obtained from Genie by searching the individual databases and selecting a pre-created statistical report. For example, while you can configure the Statistics page to give you a count of total loans for a calendar year, to get a breakdown by type of record (book, DVD, report, etc.) or by department, or by how long the item was kept out, you would search in Loans and select one of the statistical reports provided, such as Loan Statistics by Department. If the existing reports don't provide the statistic you need, it's easy to create a new report by editing the MyReports.config file. You can then include links to these reports in the Statistics page.

Andornot keeps a set of configuration files with many additional reports, search screens, pre-created statistics and similar "best practice" changes to the default configuration that ships with Genie. A list of many of this is available here. We can combine our set of configuration files with your existing ones to keep what you most like about yours, while adding new features and functionality from our experience with many Genie systems.

Contact us to review changes that might help you take better advantage of all the features in Genie.

 

Month List