My Custom Stats Package
Tuesday, 29th March 2011
For a long time, I relied on web statistics from packages that trawl the HTTP access logs on my web server. They analyse the access logs and convert all of the requests into statistics. This is alright, but it isn't a very good way of analysing your traffic.
The first problem with your HTTP access logs is that they get overrun with information from web crawlers. If you have Google, Bing, Yahoo (especially Slurp) and countless other robots crawling your website (sometimes in disguise) they dramatically skew your web statistics. This is reason number 1 for HTTP access logs over-inflating your numbers.
The third problem is speed. Parsing a busy HTTP access log is just plain slow.
It was for all of these reasons that I originally wrote a simple statistics package, which was included in the Swift Point Content Management System. The package solves all of these problems.
Secondly, it counts page-views and unique visits rather than "hits". This means you can see how many people came to your website and how many pages they all viewed.
Thirdly, because it processes the numbers quietly in the background, it doesn't need to perform a big parsing operation to display the numbers - so its fast. So as a simple statistics package it is rather useful for seeing how much traffic you really have. It has the following breakdowns...
- Monthly Pages
- Monthly Visits
- Top Content (Most popular pages)
- Bottom Content (Least popular pages)
- Hourly Usage (Popular times of day)
- Weekday Usage (Popular days of the week)
Back in September last year, I decided to take things up a notch and wrote an extended version of the statistics package, with a view to handling larger traffic volumes and also with the intention of finding out more information about what people are interested in.
I wanted to find out more about the pages people were interested in and where people came from. In particular, if I spotted a spike in my traffic, I wanted to find out what caused the sudden increase and what pages were in the traffic party. To this end, I wrote an industrial version of the statistics package and plonked it on this website to try it out and see what I might learn. The new statistics view had some additional features:
- Daily pages
- Monthly pages
- Daily visits
- Monthly visits
- Popular pages (all visited pages from most to least popular)
- Page referrers (where traffic came from for a specific page)
- Referrers (general top referrers)
- plus some information on browsers and screen resolutions
The first lesson I learned from this was that browsers lie. A lot. Not a single browser wants to tell you who it is - so browser statistics are largely hit and miss. I wrote an article about this, so you can read "Why Do Browsers Have To Be Such Big Fat Liars" for more on that. The bottom line was that I decided to ditch browser statistics. Suffice to say, a lot of people are using Internet Explorer (6, 7 and 8), Firefox (3) and Chrome and a few people are using Safari and Opera.
The next lesson I took from this project was that knowing about who refers you traffic is a great idea. Instead of being told by a friend that you've been featured in an article on Smashing Magazine, you can see it appear in your list as a big source of traffic. You can also see more general information on the root domains that send you people as well as more specific information on the exact pages that send people to a particular page on your website.
The search engines came into their own for some of the blogs I've posted on specific subjects, such as technical articles. The search engines were finding specific answers for specific searches and give a steady stream of traffic.
And finally, occasional mentions in online magazines give sudden spikes of traffic, easily doubling the normal daily traffic for the first day or two and then falling away over time to just a gradual flow.
The best bit about all of this is that I decided to install this package on The Mag to help analyse the traffic and it took all of 5 minutes to install. So not only is this the best web statistics tool I've ever installed, it's also one of the easiest! I don't know what the future holds for the "Swift Stats Pro" project, maybe I'll open-source it once it has been running for a year or so!