gitphp 0.2.9.1 :: ckanext-ga-report.git/commitdiff

Feature #162: Added sparkline graphs to the overview of analytics. Could be query optimised.

23 files changed: (show all)
.gitignore
README.rst
ckanext/ga_report/command.py
ckanext/ga_report/controller.py
ckanext/ga_report/download_analytics.py
ckanext/ga_report/ga_auth.py
ckanext/ga_report/ga_model.py
ckanext/ga_report/helpers.py (new)
ckanext/ga_report/plugin.py
ckanext/ga_report/public/scripts/vendor/jquery.sparkline.modified.js (new)
ckanext/ga_report/report_model.py (deleted)
ckanext/ga_report/templates/ga_report/ga_popular_datasets.html (new)
ckanext/ga_report/templates/ga_report/ga_popular_single.html (new)
ckanext/ga_report/templates/ga_report/ga_util.html (new)
ckanext/ga_report/templates/ga_report/notes.html (new)
ckanext/ga_report/templates/ga_report/publisher/index.html
ckanext/ga_report/templates/ga_report/publisher/popular.html (new)
ckanext/ga_report/templates/ga_report/publisher/read.html
ckanext/ga_report/templates/ga_report/site/downloads.html (new)
ckanext/ga_report/templates/ga_report/site/index.html
ckanext/ga_report/tests/test_download.py (new)
ckanext/ga_report/tests/test_model.py (new)
setup.py

file:a/.gitignore -> file:b/.gitignore

*.py[co]	*.py[co]
*.py~	*.py~
.gitignore	.gitignore
	ckan.log

# Packages	# Packages
*.egg	*.egg
*.egg-info	*.egg-info
dist	dist
build	build
eggs	eggs
parts	parts
bin	bin
var	var
sdist	sdist
develop-eggs	develop-eggs
.installed.cfg	.installed.cfg

# Private info	# Private info
credentials.json	credentials.json
token.dat	token.dat

# Installer logs	# Installer logs
pip-log.txt	pip-log.txt

# Unit test / coverage reports	# Unit test / coverage reports
.coverage	.coverage
.tox	.tox

#Translations	#Translations
*.mo	*.mo

#Mr Developer	#Mr Developer
.mr.developer.cfg	.mr.developer.cfg

file:a/README.rst -> file:b/README.rst

ckanext-ga-report	ckanext-ga-report
=================	=================

Status: Development	Status: Development

CKAN Version: 1.7.1+	CKAN Version: 1.7.1+


Overview	Overview
--------	--------

For creating detailed reports of CKAN analytics, including totals per group.	For creating detailed reports of CKAN analytics, including totals per group.

Whereas ckanext-googleanalytics focusses on providing page view stats a recent period and for all time (aimed at end users), ckanext-ga-report is more interested in building regular periodic reports (more for site managers to monitor).	Whereas ckanext-googleanalytics focusses on providing page view stats a recent period and for all time (aimed at end users), ckanext-ga-report is more interested in building regular periodic reports (more for site managers to monitor).

Contents of this extension:	Contents of this extension:

* Use the CLI tool to download Google Analytics data for each time period into this extension's database tables	* Use the CLI tool to download Google Analytics data for each time period into this extension's database tables

* Users can view the data as web page reports	* Users can view the data as web page reports


Installation	Installation
------------	------------

1. Activate you CKAN python environment and install this extension's software::	1. Activate you CKAN python environment and install this extension's software::

$ pyenv/bin/activate	$ pyenv/bin/activate
$ pip install -e git+https://github.com/datagovuk/ckanext-ga-report.git#egg=ckanext-ga-report	$ pip install -e git+https://github.com/datagovuk/ckanext-ga-report.git#egg=ckanext-ga-report

2. Ensure you development.ini (or similar) contains the info about your Google Analytics account and configuration::	2. Ensure you development.ini (or similar) contains the info about your Google Analytics account and configuration::

googleanalytics.id = UA-1010101-1	googleanalytics.id = UA-1010101-1
googleanalytics.account = Account name (i.e. data.gov.uk, see top level item at https://www.google.com/analytics)	googleanalytics.account = Account name (e.g. data.gov.uk, see top level item at https://www.google.com/analytics)
	googleanalytics.token.filepath = ~/pyenv/token.dat
ga-report.period = monthly	ga-report.period = monthly
	ga-report.bounce_url = /

Note that your credentials will be readable by system administrators on your server. Rather than use sensitive account details, it is suggested you give access to the GA account to a new Google account that you create just for this purpose.	The ga-report.bounce_url specifies a particular path to record the bounce rate for. Typically it is / (the home page).

3. Set up this extension's database tables using a paster command. (Ensure your CKAN pyenv is still activated, run the command from ``src/ckanext-ga-report``, alter the ``--config`` option to point to your site config file)::	3. Set up this extension's database tables using a paster command. (Ensure your CKAN pyenv is still activated, run the command from ``src/ckanext-ga-report``, alter the ``--config`` option to point to your site config file)::

$ paster initdb --config=../ckan/development.ini	$ paster initdb --config=../ckan/development.ini

4. Enable the extension in your CKAN config file by adding it to ``ckan.plugins``::	4. Enable the extension in your CKAN config file by adding it to ``ckan.plugins``::

ckan.plugins = ga-report	ckan.plugins = ga-report

	Problem shooting
	----------------

	* ``(ProgrammingError) relation "ga_url" does not exist``
	This means that the ``paster initdb`` step has not been run successfully. Refer to the installation instructions for this extension.


Authorization	Authorization
--------------	--------------

Before you can access the data, you need to set up the OAUTH details which you can do by following the `instructions <https://developers.google.com/analytics/resources/tutorials/hello-analytics-api>`_ the outcome of which will be a file called credentials.json which should look like credentials.json.template with the relevant fields completed. These steps are below for convenience:	Before you can access the data, you need to set up the OAUTH details which you can do by following the `instructions <https://developers.google.com/analytics/resources/tutorials/hello-analytics-api>`_ the outcome of which will be a file called credentials.json which should look like credentials.json.template with the relevant fields completed. These steps are below for convenience:

1. Visit the `Google APIs Console <https://code.google.com/apis/console>`_	1. Visit the `Google APIs Console <https://code.google.com/apis/console>`_

2. Sign-in and create a project or use an existing project.	2. Sign-in and create a project or use an existing project.

3. In the `Services pane <https://code.google.com/apis/console#:services>`_ , activate Analytics API for your project. If prompted, read and accept the terms of service.	3. In the `Services pane <https://code.google.com/apis/console#:services>`_ , activate Analytics API for your project. If prompted, read and accept the terms of service.

4. Go to the `API Access pane <https://code.google.com/apis/console/#:access>`_	4. Go to the `API Access pane <https://code.google.com/apis/console/#:access>`_

5. Click Create an OAuth 2.0 client ID....	5. Click Create an OAuth 2.0 client ID....

6. Fill out the Branding Information fields and click Next.	6. Fill out the Branding Information fields and click Next.

7. In Client ID Settings, set Application type to Installed application.	7. In Client ID Settings, set Application type to Installed application.

8. Click Create client ID	8. Click Create client ID

9. The details you need below are Client ID, Client secret, and Redirect URIs	9. The details you need below are Client ID, Client secret, and Redirect URIs


Once you have set up your credentials.json file you can generate an oauth token file by using the	Once you have set up your credentials.json file you can generate an oauth token file by using the
following command, which will store your oauth token in a file called token.dat once you have finished	following command, which will store your oauth token in a file called token.dat once you have finished
giving permission in the browser::	giving permission in the browser::

$ paster getauthtoken --config=../ckan/development.ini	$ paster getauthtoken --config=../ckan/development.ini

	Now ensure you reference the correct path to your token.dat in your CKAN config file (e.g. development.ini)::

	googleanalytics.token.filepath = ~/pyenv/token.dat


Tutorial	Tutorial
--------	--------

Download some GA data and store it in CKAN's db. (Ensure your CKAN pyenv is still activated, run the command from ``src/ckanext-ga-report``, alter the ``--config`` option to point to your site config file) and specifying the name of your auth file (token.dat by default) from the previous step::	Download some GA data and store it in CKAN's database. (Ensure your CKAN pyenv is still activated, run the command from ``src/ckanext-ga-report``, alter the ``--config`` option to point to your site config file) and specifying the name of your auth file (token.dat by default) from the previous step::

$ paster loadanalytics token.dat latest --config=../ckan/development.ini	$ paster loadanalytics latest --config=../ckan/development.ini

The value after the token file is how much data you want to retrieve, this can be	The value after the token file is how much data you want to retrieve, this can be

* all - data for all time (since 2010)	* all - data for all time (since 2010)

* latest - (default) just the 'latest' data	* latest - (default) just the 'latest' data

* YYYY-MM-DD - just data for all time periods going back to (and including) this date	* YYYY-MM-DD - just data for all time periods going back to (and including) this date



Software Licence	Software Licence
================	================

This software is developed by Cabinet Office. It is Crown Copyright and opened up under the Open Government Licence (OGL) (which is compatible with Creative Commons Attibution License).	This software is developed by Cabinet Office. It is Crown Copyright and opened up under the Open Government Licence (OGL) (which is compatible with Creative Commons Attibution License).

OGL terms: http://www.nationalarchives.gov.uk/doc/open-government-licence/	OGL terms: http://www.nationalarchives.gov.uk/doc/open-government-licence/

file:a/ckanext/ga_report/command.py -> file:b/ckanext/ga_report/command.py

import logging	import logging
import datetime	import datetime
	import os

	from pylons import config

from ckan.lib.cli import CkanCommand	from ckan.lib.cli import CkanCommand
# No other CKAN imports allowed until _load_config is run,	# No other CKAN imports allowed until _load_config is run,
# or logging is disabled	# or logging is disabled


class InitDB(CkanCommand):	class InitDB(CkanCommand):
"""Initialise the extension's database tables	"""Initialise the extension's database tables
"""	"""
summary = __doc__.split('\n')[0]	summary = __doc__.split('\n')[0]
usage = __doc__	usage = __doc__
max_args = 0	max_args = 0
min_args = 0	min_args = 0

def command(self):	def command(self):
self._load_config()	self._load_config()

import ckan.model as model	import ckan.model as model
model.Session.remove()	model.Session.remove()
model.Session.configure(bind=model.meta.engine)	model.Session.configure(bind=model.meta.engine)
log = logging.getLogger('ckanext.ga-report')	log = logging.getLogger('ckanext.ga-report')

import ga_model	import ga_model
ga_model.init_tables()	ga_model.init_tables()
log.info("DB tables are setup")	log.info("DB tables are setup")


class GetAuthToken(CkanCommand):	class GetAuthToken(CkanCommand):
""" Get's the Google auth token	""" Get's the Google auth token

Usage: paster getauthtoken <credentials_file>	Usage: paster getauthtoken <credentials_file>

Where <credentials_file> is the file name containing the details	Where <credentials_file> is the file name containing the details
for the service (obtained from https://code.google.com/apis/console).	for the service (obtained from https://code.google.com/apis/console).
By default this is set to credentials.json	By default this is set to credentials.json
"""	"""
summary = __doc__.split('\n')[0]	summary = __doc__.split('\n')[0]
usage = __doc__	usage = __doc__
max_args = 0	max_args = 0
min_args = 0	min_args = 0

def command(self):	def command(self):
"""	"""
In this case we don't want a valid service, but rather just to	In this case we don't want a valid service, but rather just to
force the user through the auth flow. We allow this to complete to	force the user through the auth flow. We allow this to complete to
act as a form of verification instead of just getting the token and	act as a form of verification instead of just getting the token and
assuming it is correct.	assuming it is correct.
"""	"""
from ga_auth import init_service	from ga_auth import init_service
init_service('token.dat',	init_service('token.dat',
self.args[0] if self.args	self.args[0] if self.args
else 'credentials.json')	else 'credentials.json')

	class FixTimePeriods(CkanCommand):
	"""
	Fixes the 'All' records for GA_Urls

	It is possible that older urls that haven't recently been visited
	do not have All records. This command will traverse through those
	records and generate valid All records for them.
	"""
	summary = __doc__.split('\n')[0]
	usage = __doc__
	max_args = 0
	min_args = 0

	def __init__(self, name):
	super(FixTimePeriods, self).__init__(name)

	def command(self):
	import ckan.model as model
	from ga_model import post_update_url_stats
	self._load_config()
	model.Session.remove()
	model.Session.configure(bind=model.meta.engine)

	log = logging.getLogger('ckanext.ga_report')

	log.info("Updating 'All' records for old URLs")
	post_update_url_stats()
	log.info("Processing complete")



class LoadAnalytics(CkanCommand):	class LoadAnalytics(CkanCommand):
"""Get data from Google Analytics API and save it	"""Get data from Google Analytics API and save it
in the ga_model	in the ga_model

Usage: paster loadanalytics <tokenfile> <time-period>	Usage: paster loadanalytics <time-period>

Where <tokenfile> is the name of the auth token file from	Where <time-period> is:
the getauthtoken step.

And where <time-period> is:
all - data for all time	all - data for all time
latest - (default) just the 'latest' data	latest - (default) just the 'latest' data
YYYY-MM - just data for the specific month	YYYY-MM - just data for the specific month
"""	"""
summary = __doc__.split('\n')[0]	summary = __doc__.split('\n')[0]
usage = __doc__	usage = __doc__
max_args = 2	max_args = 1
min_args = 1	min_args = 0

	def __init__(self, name):
	super(LoadAnalytics, self).__init__(name)
	self.parser.add_option('-d', '--delete-first',
	action='store_true',
	default=False,
	dest='delete_first',
	help='Delete data for the period first')
	self.parser.add_option('-s', '--skip_url_stats',
	action='store_true',
	default=False,
	dest='skip_url_stats',
	help='Skip the download of URL data - just do site-wide stats')

def command(self):	def command(self):
self._load_config()	self._load_config()

from download_analytics import DownloadAnalytics	from download_analytics import DownloadAnalytics
from ga_auth import (init_service, get_profile_id)	from ga_auth import (init_service, get_profile_id)

	ga_token_filepath = os.path.expanduser(config.get('googleanalytics.token.filepath', ''))
	if not ga_token_filepath:
	print 'ERROR: In the CKAN config you need to specify the filepath of the ' \
	'Google Analytics token file under key: googleanalytics.token.filepath'
	return

try:	try:
svc = init_service(self.args[0], None)	svc = init_service(ga_token_filepath, None)
except TypeError:	except TypeError:
print ('Have you correctly run the getauthtoken task and '	print ('Have you correctly run the getauthtoken task and '
'specified the correct file here')	'specified the correct token file in the CKAN config under '
	'"googleanalytics.token.filepath"?')
return	return

downloader = DownloadAnalytics(svc, profile_id=get_profile_id(svc))	downloader = DownloadAnalytics(svc, profile_id=get_profile_id(svc),
	delete_first=self.options.delete_first,
	skip_url_stats=self.options.skip_url_stats)

time_period = self.args[1] if self.args and len(self.args) > 1 \	time_period = self.args[0] if self.args else 'latest'
else 'latest'
if time_period == 'all':	if time_period == 'all':
downloader.all_()	downloader.all_()
elif time_period == 'latest':	elif time_period == 'latest':
downloader.latest()	downloader.latest()
else:	else:
# The month to use	# The month to use
for_date = datetime.datetime.strptime(time_period, '%Y-%m')	for_date = datetime.datetime.strptime(time_period, '%Y-%m')
downloader.specific_month(for_date)	downloader.specific_month(for_date)

file:a/ckanext/ga_report/controller.py -> file:b/ckanext/ga_report/controller.py

	import re
	import csv
	import sys
import logging	import logging
import operator	import operator
from ckan.lib.base import BaseController, c, render, request, response	import collections
	from ckan.lib.base import (BaseController, c, g, render, request, response, abort)

import sqlalchemy	import sqlalchemy
from sqlalchemy import func, cast, Integer	from sqlalchemy import func, cast, Integer
import ckan.model as model	import ckan.model as model
from ga_model import GA_Url, GA_Stat	from ga_model import GA_Url, GA_Stat, GA_ReferralStat, GA_Publisher

log = logging.getLogger('ckanext.ga-report')	log = logging.getLogger('ckanext.ga-report')

	DOWNLOADS_AVAILABLE_FROM = '2012-12'

def _get_month_name(strdate):	def _get_month_name(strdate):
import calendar	import calendar
from time import strptime	from time import strptime
d = strptime(strdate, '%Y-%m')	d = strptime(strdate, '%Y-%m')
return '%s %s' % (calendar.month_name[d.tm_mon], d.tm_year)	return '%s %s' % (calendar.month_name[d.tm_mon], d.tm_year)


def _month_details(cls):	def _month_details(cls, stat_key=None):
	'''
	Returns a list of all the periods for which we have data, unfortunately
	knows too much about the type of the cls being passed as GA_Url has a
	more complex query

	This may need extending if we add a period_name to the stats
	'''
months = []	months = []
vals = model.Session.query(cls.period_name).distinct().all()	day = None

	q = model.Session.query(cls.period_name,cls.period_complete_day)\
	.filter(cls.period_name!='All').distinct(cls.period_name)
	if stat_key:
	q= q.filter(cls.stat_name==stat_key)

	vals = q.order_by("period_name desc").all()

	if vals and vals[0][1]:
	day = int(vals[0][1])
	ordinal = 'th' if 11 <= day <= 13 \
	else {1:'st',2:'nd',3:'rd'}.get(day % 10, 'th')
	day = "{day}{ordinal}".format(day=day, ordinal=ordinal)

for m in vals:	for m in vals:
months.append( (m[0], _get_month_name(m[0])))	months.append( (m[0], _get_month_name(m[0])))
return sorted(months, key=operator.itemgetter(0), reverse=True)
	return months, day


class GaReport(BaseController):	class GaReport(BaseController):

def csv(self, month):	def csv(self, month):
import csv	import csv

entries = model.Session.query(GA_Stat).\	q = model.Session.query(GA_Stat).filter(GA_Stat.stat_name!='Downloads')
filter(GA_Stat.period_name==month).\	if month != 'all':
order_by('GA_Stat.stat_name, GA_Stat.key').all()	q = q.filter(GA_Stat.period_name==month)
	entries = q.order_by('GA_Stat.period_name, GA_Stat.stat_name, GA_Stat.key').all()

response.headers['Content-Type'] = "text/csv; charset=utf-8"	response.headers['Content-Type'] = "text/csv; charset=utf-8"
	response.headers['Content-Disposition'] = str('attachment; filename=stats_%s.csv' % (month,))

writer = csv.writer(response)	writer = csv.writer(response)
writer.writerow(["Period", "Statistic", "Key", "Value"])	writer.writerow(["Period", "Statistic", "Key", "Value"])

for entry in entries:	for entry in entries:
writer.writerow([entry.period_name.encode('utf-8'),	writer.writerow([entry.period_name.encode('utf-8'),
entry.stat_name.encode('utf-8'),	entry.stat_name.encode('utf-8'),
entry.key.encode('utf-8'),	entry.key.encode('utf-8'),
entry.value.encode('utf-8')])	entry.value.encode('utf-8')])


def index(self):	def index(self):

# Get the month details by fetching distinct values and determining the	# Get the month details by fetching distinct values and determining the
# month names from the values.	# month names from the values.
c.months = _month_details(GA_Stat)	c.months, c.day = _month_details(GA_Stat)

# Work out which month to show, based on query params of the first item	# Work out which month to show, based on query params of the first item
c.month = request.params.get('month', c.months[0][0] if c.months else '')	c.month_desc = 'all months'
c.month_desc = ''.join([m[1] for m in c.months if m[0]==c.month])	c.month = request.params.get('month', '')
	if c.month:
entries = model.Session.query(GA_Stat).\	c.month_desc = ''.join([m[1] for m in c.months if m[0]==c.month])
filter(GA_Stat.stat_name=='Totals').\
filter(GA_Stat.period_name==c.month).\	q = model.Session.query(GA_Stat).\
order_by('ga_stat.key').all()	filter(GA_Stat.stat_name=='Totals')
c.global_totals = [(s.key, s.value) for s in entries ]	if c.month:
	q = q.filter(GA_Stat.period_name==c.month)
	entries = q.order_by('ga_stat.key').all()

	def clean_key(key, val):
	if key in ['Average time on site', 'Pages per visit', 'New visits', 'Bounce rate (home page)']:
	val = "%.2f" % round(float(val), 2)
	if key == 'Average time on site':
	mins, secs = divmod(float(val), 60)
	hours, mins = divmod(mins, 60)
	val = '%02d:%02d:%02d (%s seconds) ' % (hours, mins, secs, val)
	if key in ['New visits','Bounce rate (home page)']:
	val = "%s%%" % val
	if key in ['Total page views', 'Total visits']:
	val = int(val)

	return key, val

	# Query historic values for sparkline rendering
	graph_query = model.Session.query(GA_Stat)\
	.filter(GA_Stat.stat_name=='Totals')\
	.order_by(GA_Stat.period_name)
	graph_data = {}
	for x in graph_query:
	graph_data[x.key] = graph_data.get(x.key,[])
	key, val = clean_key(x.key,float(x.value))
	tooltip = '%s: %s' % (_get_month_name(x.period_name), val)
	graph_data[x.key].append( (tooltip,x.value) )
	# Trim the latest month, as it looks like a huge dropoff
	for key in graph_data:
	graph_data[key] = graph_data[key][:-1]

	c.global_totals = []
	if c.month:
	for e in entries:
	key, val = clean_key(e.key, e.value)
	sparkline = graph_data[e.key]
	c.global_totals.append((key, val, sparkline))
	else:
	d = collections.defaultdict(list)
	for e in entries:
	d[e.key].append(float(e.value))
	for k, v in d.iteritems():
	if k in ['Total page views', 'Total visits']:
	v = sum(v)
	else:
	v = float(sum(v))/float(len(v))
	sparkline = graph_data[k]
	key, val = clean_key(k,v)

	c.global_totals.append((key, val, sparkline))
	c.global_totals = sorted(c.global_totals, key=operator.itemgetter(0))

keys = {	keys = {
'Browser versions': 'browsers',	'Browser versions': 'browser_versions',
'Operating Systems versions': 'os',	'Browsers': 'browsers',
	'Operating Systems versions': 'os_versions',
	'Operating Systems': 'os',
'Social sources': 'social_networks',	'Social sources': 'social_networks',
'Languages': 'languages',	'Languages': 'languages',
'Country': 'country'	'Country': 'country'
}	}

	def shorten_name(name, length=60):
	return (name[:length] + '..') if len(name) > 60 else name

	def fill_out_url(url):
	import urlparse
	return urlparse.urljoin(g.site_url, url)

	c.social_referrer_totals, c.social_referrers = [], []
	q = model.Session.query(GA_ReferralStat)
	q = q.filter(GA_ReferralStat.period_name==c.month) if c.month else q
	q = q.order_by('ga_referrer.count::int desc')
	for entry in q.all():
	c.social_referrers.append((shorten_name(entry.url), fill_out_url(entry.url),
	entry.source,entry.count))

	q = model.Session.query(GA_ReferralStat.url,
	func.sum(GA_ReferralStat.count).label('count'))
	q = q.filter(GA_ReferralStat.period_name==c.month) if c.month else q
	q = q.order_by('count desc').group_by(GA_ReferralStat.url)
	for entry in q.all():
	c.social_referrer_totals.append((shorten_name(entry[0]), fill_out_url(entry[0]),'',
	entry[1]))

for k, v in keys.iteritems():	for k, v in keys.iteritems():
entries = model.Session.query(GA_Stat).\	q = model.Session.query(GA_Stat).\
filter(GA_Stat.stat_name==k).\	filter(GA_Stat.stat_name==k)
filter(GA_Stat.period_name==c.month).\	if c.month:
order_by('ga_stat.value::int desc').all()	entries = []
setattr(c, v, [(s.key, s.value) for s in entries ])	q = q.filter(GA_Stat.period_name==c.month).\
	order_by('ga_stat.value::int desc')

	d = collections.defaultdict(int)
	for e in q.all():
	d[e.key] += int(e.value)
	entries = []
	for key, val in d.iteritems():
	entries.append((key,val,))
	entries = sorted(entries, key=operator.itemgetter(1), reverse=True)

	# Get the total for each set of values and then set the value as
	# a percentage of the total
	if k == 'Social sources':
	total = sum([x for n,x,graph in c.global_totals if n == 'Total visits'])
	else:
	total = sum([num for _,num in entries])
	setattr(c, v, [(k,_percent(v,total)) for k,v in entries ])

return render('ga_report/site/index.html')	return render('ga_report/site/index.html')


class GaPublisherReport(BaseController):	class GaDatasetReport(BaseController):
"""	"""
Displays the pageview and visit count for specific publishers based on	Displays the pageview and visit count for datasets
the datasets associated with the publisher.	with options to filter by publisher and time period.
"""	"""
	def publisher_csv(self, month):
def index(self):	'''
	Returns a CSV of each publisher with the total number of dataset
	views & visits.
	'''
	c.month = month if not month == 'all' else ''
	response.headers['Content-Type'] = "text/csv; charset=utf-8"
	response.headers['Content-Disposition'] = str('attachment; filename=publishers_%s.csv' % (month,))

	writer = csv.writer(response)
	writer.writerow(["Publisher Title", "Publisher Name", "Views", "Visits", "Period Name"])

	for publisher,view,visit in _get_top_publishers(None):
	writer.writerow([publisher.title.encode('utf-8'),
	publisher.name.encode('utf-8'),
	view,
	visit,
	month])

	def dataset_csv(self, id='all', month='all'):
	'''
	Returns a CSV with the number of views & visits for each dataset.

	:param id: A Publisher ID or None if you want for all
	:param month: The time period, or 'all'
	'''
	c.month = month if not month == 'all' else ''
	if id != 'all':
	c.publisher = model.Group.get(id)
	if not c.publisher:
	abort(404, 'A publisher with that name could not be found')

	packages = self._get_packages(c.publisher)
	response.headers['Content-Type'] = "text/csv; charset=utf-8"
	response.headers['Content-Disposition'] = \
	str('attachment; filename=datasets_%s_%s.csv' % (c.publisher_name, month,))

	writer = csv.writer(response)
	writer.writerow(["Dataset Title", "Dataset Name", "Views", "Visits", "Resource downloads", "Period Name"])

	for package,view,visit,downloads in packages:
	writer.writerow([package.title.encode('utf-8'),
	package.name.encode('utf-8'),
	view,
	visit,
	downloads,
	month])

	def publishers(self):
	'''A list of publishers and the number of views/visits for each'''

# Get the month details by fetching distinct values and determining the	# Get the month details by fetching distinct values and determining the
# month names from the values.	# month names from the values.
c.months = _month_details(GA_Url)	c.months, c.day = _month_details(GA_Url)

# Work out which month to show, based on query params of the first item	# Work out which month to show, based on query params of the first item
c.month = request.params.get('month', c.months[0][0] if c.months else '')	c.month = request.params.get('month', '')
c.month_desc = ''.join([m[1] for m in c.months if m[0]==c.month])	c.month_desc = 'all months'
# and not url like '/publisher/%%'	if c.month:
connection = model.Session.connection()	c.month_desc = ''.join([m[1] for m in c.months if m[0]==c.month])
q = """
select department_id, sum(pageviews::int) views, sum(visitors::int) visits	c.top_publishers = _get_top_publishers()
from ga_url
where department_id <> ''
and period_name=%s
group by department_id order by views desc limit 20;
"""
c.top_publishers = []
res = connection.execute(q, c.month)
for row in res:
c.top_publishers.append((model.Group.get(row[0]), row[1], row[2]))

return render('ga_report/publisher/index.html')	return render('ga_report/publisher/index.html')

	def _get_packages(self, publisher=None, count=-1):
def read(self, id):	'''Returns the datasets in order of views'''
c.publisher = model.Group.get(id)	have_download_data = True
	month = c.month or 'All'
	if month != 'All':
	have_download_data = month >= DOWNLOADS_AVAILABLE_FROM

	q = model.Session.query(GA_Url,model.Package)\
	.filter(model.Package.name==GA_Url.package_id)\
	.filter(GA_Url.url.like('/dataset/%'))
	if publisher:
	q = q.filter(GA_Url.department_id==publisher.name)
	q = q.filter(GA_Url.period_name==month)
	q = q.order_by('ga_url.pageviews::int desc')
	top_packages = []
	if count == -1:
	entries = q.all()
	else:
	entries = q.limit(count)

	for entry,package in entries:
	if package:
	# Downloads ....
	if have_download_data:
	dls = model.Session.query(GA_Stat).\
	filter(GA_Stat.stat_name=='Downloads').\
	filter(GA_Stat.key==package.name)
	if month != 'All': # Fetch everything unless the month is specific
	dls = dls.filter(GA_Stat.period_name==month)

	downloads = sum(int(d.value) for d in dls.all())
	else:
	downloads = 'No data'
	top_packages.append((package, entry.pageviews, entry.visits, downloads))
	else:
	log.warning('Could not find package associated package')

	return top_packages

	def read(self):
	'''
	Lists the most popular datasets across all publishers
	'''
	return self.read_publisher(None)

	def read_publisher(self, id):
	'''
	Lists the most popular datasets for a publisher (or across all publishers)
	'''
	count = 20

	c.publishers = _get_publishers()

	id = request.params.get('publisher', id)
	if id and id != 'all':
	c.publisher = model.Group.get(id)
	if not c.publisher:
	abort(404, 'A publisher with that name could not be found')
	c.publisher_name = c.publisher.name
c.top_packages = [] # package, dataset_views in c.top_packages	c.top_packages = [] # package, dataset_views in c.top_packages

# Get the month details by fetching distinct values and determining the	# Get the month details by fetching distinct values and determining the
# month names from the values.	# month names from the values.
c.months = _month_details(GA_Url)	c.months, c.day = _month_details(GA_Url)

# Work out which month to show, based on query params of the first item	# Work out which month to show, based on query params of the first item
c.month = request.params.get('month', c.months[0][0] if c.months else '')	c.month = request.params.get('month', '')
c.month_desc = ''.join([m[1] for m in c.months if m[0]==c.month])	if not c.month:
	c.month_desc = 'all months'
entry = model.Session.query(GA_Url).\	else:
filter(GA_Url.url=='/publisher/%s' % c.publisher.name).\	c.month_desc = ''.join([m[1] for m in c.months if m[0]==c.month])
filter(GA_Url.period_name==c.month).first()
	month = c.month or 'All'
	c.publisher_page_views = 0
	q = model.Session.query(GA_Url).\
	filter(GA_Url.url=='/publisher/%s' % c.publisher_name)
	entry = q.filter(GA_Url.period_name==c.month).first()
c.publisher_page_views = entry.pageviews if entry else 0	c.publisher_page_views = entry.pageviews if entry else 0

entries = model.Session.query(GA_Url).\	c.top_packages = self._get_packages(c.publisher, 20)
filter(GA_Url.department_id==c.publisher.name).\
filter(GA_Url.period_name==c.month).\
order_by('ga_url.pageviews::int desc')[:20]
for entry in entries:
if entry.url.startswith('/dataset/'):
p = model.Package.get(entry.url[len('/dataset/'):])
c.top_packages.append((p,entry.pageviews,entry.visitors))

return render('ga_report/publisher/read.html')	return render('ga_report/publisher/read.html')

	def _get_top_publishers(limit=20):
	'''
	Returns a list of the top 20 publishers by dataset visits.
	(The number to show can be varied with 'limit')
	'''
	month = c.month or 'All'
	connection = model.Session.connection()
	q = """
	select department_id, sum(pageviews::int) views, sum(visits::int) visits
	from ga_url
	where department_id <> ''
	and package_id <> ''
	and url like '/dataset/%%'
	and period_name=%s
	group by department_id order by views desc
	"""
	if limit:
	q = q + " limit %s;" % (limit)

	top_publishers = []
	res = connection.execute(q, month)
	for row in res:
	g = model.Group.get(row[0])
	if g:
	top_publishers.append((g, row[1], row[2]))
	return top_publishers


	def _get_publishers():
	'''
	Returns a list of all publishers. Each item is a tuple:
	(name, title)
	'''
	publishers = []
	for pub in model.Session.query(model.Group).\
	filter(model.Group.type=='publisher').\
	filter(model.Group.state=='active').\
	order_by(model.Group.name):
	publishers.append((pub.name, pub.title))
	return publishers

	def _percent(num, total):
	p = 100 * float(num)/float(total)
	return "%.2f%%" % round(p, 2)

file:a/ckanext/ga_report/download_analytics.py -> file:b/ckanext/ga_report/download_analytics.py

import os	import os
import logging	import logging
import datetime	import datetime
	import collections
from pylons import config	from pylons import config
	from ga_model import _normalize_url
import ga_model	import ga_model

#from ga_client import GA	#from ga_client import GA

log = logging.getLogger('ckanext.ga-report')	log = logging.getLogger('ckanext.ga-report')

FORMAT_MONTH = '%Y-%m'	FORMAT_MONTH = '%Y-%m'
	MIN_VIEWS = 50
	MIN_VISITS = 20
	MIN_DOWNLOADS = 10

class DownloadAnalytics(object):	class DownloadAnalytics(object):
'''Downloads and stores analytics info'''	'''Downloads and stores analytics info'''

def __init__(self, service=None, profile_id=None):	def __init__(self, service=None, profile_id=None, delete_first=False,
	skip_url_stats=False):
self.period = config['ga-report.period']	self.period = config['ga-report.period']
self.service = service	self.service = service
self.profile_id = profile_id	self.profile_id = profile_id
	self.delete_first = delete_first
	self.skip_url_stats = skip_url_stats

def specific_month(self, date):	def specific_month(self, date):
import calendar	import calendar

first_of_this_month = datetime.datetime(date.year, date.month, 1)	first_of_this_month = datetime.datetime(date.year, date.month, 1)
_, last_day_of_month = calendar.monthrange(int(date.year), int(date.month))	_, last_day_of_month = calendar.monthrange(int(date.year), int(date.month))
last_of_this_month = datetime.datetime(date.year, date.month, last_day_of_month)	last_of_this_month = datetime.datetime(date.year, date.month, last_day_of_month)
periods = ((date.strftime(FORMAT_MONTH),	periods = ((date.strftime(FORMAT_MONTH),
last_day_of_month,	last_day_of_month,
first_of_this_month, last_of_this_month),)	first_of_this_month, last_of_this_month),)
self.download_and_store(periods)	self.download_and_store(periods)


def latest(self):	def latest(self):
if self.period == 'monthly':	if self.period == 'monthly':
# from first of this month to today	# from first of this month to today
now = datetime.datetime.now()	now = datetime.datetime.now()
first_of_this_month = datetime.datetime(now.year, now.month, 1)	first_of_this_month = datetime.datetime(now.year, now.month, 1)
periods = ((now.strftime(FORMAT_MONTH),	periods = ((now.strftime(FORMAT_MONTH),
now.day,	now.day,
first_of_this_month, now),)	first_of_this_month, now),)
else:	else:
raise NotImplementedError	raise NotImplementedError
self.download_and_store(periods)	self.download_and_store(periods)


def for_date(self, for_date):	def for_date(self, for_date):
assert isinstance(since_date, datetime.datetime)	assert isinstance(since_date, datetime.datetime)
periods = [] # (period_name, period_complete_day, start_date, end_date)	periods = [] # (period_name, period_complete_day, start_date, end_date)
if self.period == 'monthly':	if self.period == 'monthly':
first_of_the_months_until_now = []	first_of_the_months_until_now = []
year = for_date.year	year = for_date.year
month = for_date.month	month = for_date.month
now = datetime.datetime.now()	now = datetime.datetime.now()
first_of_this_month = datetime.datetime(now.year, now.month, 1)	first_of_this_month = datetime.datetime(now.year, now.month, 1)
while True:	while True:
first_of_the_month = datetime.datetime(year, month, 1)	first_of_the_month = datetime.datetime(year, month, 1)
if first_of_the_month == first_of_this_month:	if first_of_the_month == first_of_this_month:
periods.append((now.strftime(FORMAT_MONTH),	periods.append((now.strftime(FORMAT_MONTH),
now.day,	now.day,
first_of_this_month, now))	first_of_this_month, now))
break	break
elif first_of_the_month < first_of_this_month:	elif first_of_the_month < first_of_this_month:
in_the_next_month = first_of_the_month + datetime.timedelta(40)	in_the_next_month = first_of_the_month + datetime.timedelta(40)
last_of_the_month = datetime.datetime(in_the_next_month.year,	last_of_the_month = datetime.datetime(in_the_next_month.year,
in_the_next_month.month, 1)\	in_the_next_month.month, 1)\
- datetime.timedelta(1)	- datetime.timedelta(1)
periods.append((now.strftime(FORMAT_MONTH), 0,	periods.append((now.strftime(FORMAT_MONTH), 0,
first_of_the_month, last_of_the_month))	first_of_the_month, last_of_the_month))
else:	else:
# first_of_the_month has got to the future somehow	# first_of_the_month has got to the future somehow
break	break
month += 1	month += 1
if month > 12:	if month > 12:
year += 1	year += 1
month = 1	month = 1
else:	else:
raise NotImplementedError	raise NotImplementedError
self.download_and_store(periods)	self.download_and_store(periods)

@staticmethod	@staticmethod
def get_full_period_name(period_name, period_complete_day):	def get_full_period_name(period_name, period_complete_day):
if period_complete_day:	if period_complete_day:
return period_name + ' (up to %ith)' % period_complete_day	return period_name + ' (up to %ith)' % period_complete_day
else:	else:
return period_name	return period_name


def download_and_store(self, periods):	def download_and_store(self, periods):
for period_name, period_complete_day, start_date, end_date in periods:	for period_name, period_complete_day, start_date, end_date in periods:
log.info('Downloading Analytics for period "%s" (%s - %s)',	log.info('Period "%s" (%s - %s)',
self.get_full_period_name(period_name, period_complete_day),	self.get_full_period_name(period_name, period_complete_day),
start_date.strftime('%Y %m %d'),	start_date.strftime('%Y-%m-%d'),
end_date.strftime('%Y %m %d'))	end_date.strftime('%Y-%m-%d'))

data = self.download(start_date, end_date, '~/dataset/[a-z0-9-_]+')	if self.delete_first:
log.info('Storing Dataset Analytics for period "%s"',	log.info('Deleting existing Analytics for this period "%s"',
self.get_full_period_name(period_name, period_complete_day))	period_name)
self.store(period_name, period_complete_day, data, )	ga_model.delete(period_name)

data = self.download(start_date, end_date, '~/publisher/[a-z0-9-_]+')	if not self.skip_url_stats:
log.info('Storing Publisher Analytics for period "%s"',	# Clean out old url data before storing the new
self.get_full_period_name(period_name, period_complete_day))	ga_model.pre_update_url_stats(period_name)
self.store(period_name, period_complete_day, data,)
	accountName = config.get('googleanalytics.account')
ga_model.update_publisher_stats(period_name) # about 30 seconds.
self.sitewide_stats( period_name )	log.info('Downloading analytics for dataset views')
	data = self.download(start_date, end_date, '~/%s/dataset/[a-z0-9-_]+' % accountName)

def download(self, start_date, end_date, path='~/dataset/[a-z0-9-_]+'):	log.info('Storing dataset views (%i rows)', len(data.get('url')))
	self.store(period_name, period_complete_day, data, )

	log.info('Downloading analytics for publisher views')
	data = self.download(start_date, end_date, '~/%s/publisher/[a-z0-9-_]+' % accountName)

	log.info('Storing publisher views (%i rows)', len(data.get('url')))
	self.store(period_name, period_complete_day, data,)

	# Make sure the All records are correct.
	ga_model.post_update_url_stats()

	log.info('Aggregating datasets by publisher')
	ga_model.update_publisher_stats(period_name) # about 30 seconds.


	log.info('Downloading and storing analytics for site-wide stats')
	self.sitewide_stats( period_name, period_complete_day )

	log.info('Downloading and storing analytics for social networks')
	self.update_social_info(period_name, start_date, end_date)


	def update_social_info(self, period_name, start_date, end_date):
	start_date = start_date.strftime('%Y-%m-%d')
	end_date = end_date.strftime('%Y-%m-%d')
	query = 'ga:hasSocialSourceReferral=~Yes$'
	metrics = 'ga:entrances'
	sort = '-ga:entrances'

	# Supported query params at
	# https://developers.google.com/analytics/devguides/reporting/core/v3/reference
	results = self.service.data().ga().get(
	ids='ga:' + self.profile_id,
	filters=query,
	start_date=start_date,
	metrics=metrics,
	sort=sort,
	dimensions="ga:landingPagePath,ga:socialNetwork",
	max_results=10000,
	end_date=end_date).execute()
	data = collections.defaultdict(list)
	rows = results.get('rows',[])
	for row in rows:
	url = _normalize_url('http:/' + row[0])
	data[url].append( (row[1], int(row[2]),) )
	ga_model.update_social(period_name, data)


	def download(self, start_date, end_date, path=None):
'''Get data from GA for a given time period'''	'''Get data from GA for a given time period'''
start_date = start_date.strftime('%Y-%m-%d')	start_date = start_date.strftime('%Y-%m-%d')
end_date = end_date.strftime('%Y-%m-%d')	end_date = end_date.strftime('%Y-%m-%d')
query = 'ga:pagePath=%s$' % path	query = 'ga:pagePath=%s$' % path
metrics = 'ga:uniquePageviews, ga:visitors'	metrics = 'ga:pageviews, ga:visits'
sort = '-ga:uniquePageviews'	sort = '-ga:pageviews'

# Supported query params at	# Supported query params at
# https://developers.google.com/analytics/devguides/reporting/core/v3/reference	# https://developers.google.com/analytics/devguides/reporting/core/v3/reference
results = self.service.data().ga().get(	results = self.service.data().ga().get(
ids='ga:' + self.profile_id,	ids='ga:' + self.profile_id,
filters=query,	filters=query,
start_date=start_date,	start_date=start_date,
metrics=metrics,	metrics=metrics,
sort=sort,	sort=sort,
dimensions="ga:pagePath",	dimensions="ga:pagePath",
max_results=10000,	max_results=10000,
end_date=end_date).execute()	end_date=end_date).execute()

if os.getenv('DEBUG'):
import pprint
pprint.pprint(results)
print 'Total results: %s' % results.get('totalResults')

packages = []	packages = []
	log.info("There are %d results" % results['totalResults'])
for entry in results.get('rows'):	for entry in results.get('rows'):
(loc,pageviews,visits) = entry	(loc,pageviews,visits) = entry
packages.append( ('http:/' + loc, pageviews, visits,) ) # Temporary hack	url = _normalize_url('http:/' + loc) # strips off domain e.g. www.data.gov.uk or data.gov.uk

	if not url.startswith('/dataset/') and not url.startswith('/publisher/'):
	# filter out strays like:
	# /data/user/login?came_from=http://data.gov.uk/dataset/os-code-point-open
	# /403.html?page=/about&from=http://data.gov.uk/publisher/planning-inspectorate
	continue
	packages.append( (url, pageviews, visits,) ) # Temporary hack
return dict(url=packages)	return dict(url=packages)

def store(self, period_name, period_complete_day, data):	def store(self, period_name, period_complete_day, data):
if 'url' in data:	if 'url' in data:
ga_model.update_url_stats(period_name, period_complete_day, data['url'])	ga_model.update_url_stats(period_name, period_complete_day, data['url'])

def sitewide_stats(self, period_name):	def sitewide_stats(self, period_name, period_complete_day):
import calendar	import calendar
year, month = period_name.split('-')	year, month = period_name.split('-')
_, last_day_of_month = calendar.monthrange(int(year), int(month))	_, last_day_of_month = calendar.monthrange(int(year), int(month))

start_date = '%s-01' % period_name	start_date = '%s-01' % period_name
end_date = '%s-%s' % (period_name, last_day_of_month)	end_date = '%s-%s' % (period_name, last_day_of_month)
print 'Sitewide_stats for %s (%s -> %s)' % (period_name, start_date, end_date)

funcs = ['_totals_stats', '_social_stats', '_os_stats',	funcs = ['_totals_stats', '_social_stats', '_os_stats',
'_locale_stats', '_browser_stats', '_mobile_stats']	'_locale_stats', '_browser_stats', '_mobile_stats', '_download_stats']
for f in funcs:	for f in funcs:
print ' + Fetching %s stats' % f.split('_')[1]	log.info('Downloading analytics for %s' % f.split('_')[1])
getattr(self, f)(start_date, end_date, period_name)	getattr(self, f)(start_date, end_date, period_name, period_complete_day)

def _get_results(result_data, f):	def _get_results(result_data, f):
data = {}	data = {}
for result in result_data:	for result in result_data:
key = f(result)	key = f(result)
data[key] = data.get(key,0) + result[1]	data[key] = data.get(key,0) + result[1]
return data	return data

def _totals_stats(self, start_date, end_date, period_name):	def _totals_stats(self, start_date, end_date, period_name, period_complete_day):
""" Fetches distinct totals, total pageviews etc """	""" Fetches distinct totals, total pageviews etc """
results = self.service.data().ga().get(	results = self.service.data().ga().get(
ids='ga:' + self.profile_id,	ids='ga:' + self.profile_id,
start_date=start_date,	start_date=start_date,
metrics='ga:uniquePageviews',	metrics='ga:pageviews',
sort='-ga:uniquePageviews',	sort='-ga:pageviews',
max_results=10000,	max_results=10000,
end_date=end_date).execute()	end_date=end_date).execute()
result_data = results.get('rows')	result_data = results.get('rows')
ga_model.update_sitewide_stats(period_name, "Totals", {'Total pageviews': result_data[0][0]})	ga_model.update_sitewide_stats(period_name, "Totals", {'Total page views': result_data[0][0]},
	period_complete_day)
results = self.service.data().ga().get(
ids='ga:' + self.profile_id,	results = self.service.data().ga().get(
start_date=start_date,	ids='ga:' + self.profile_id,
metrics='ga:pageviewsPerVisit,ga:bounces,ga:avgTimeOnSite,ga:percentNewVisits',	start_date=start_date,
	metrics='ga:pageviewsPerVisit,ga:avgTimeOnSite,ga:percentNewVisits,ga:visits',
max_results=10000,	max_results=10000,
end_date=end_date).execute()	end_date=end_date).execute()
result_data = results.get('rows')	result_data = results.get('rows')
data = {	data = {
'Pages per visit': result_data[0][0],	'Pages per visit': result_data[0][0],
'Bounces': result_data[0][1],	'Average time on site': result_data[0][1],
'Average time on site': result_data[0][2],	'New visits': result_data[0][2],
'Percent new visits': result_data[0][3],	'Total visits': result_data[0][3],
}	}
ga_model.update_sitewide_stats(period_name, "Totals", data)	ga_model.update_sitewide_stats(period_name, "Totals", data, period_complete_day)

	# Bounces from / or another configurable page.
def _locale_stats(self, start_date, end_date, period_name):	path = '/%s%s' % (config.get('googleanalytics.account'),
	config.get('ga-report.bounce_url', '/'))
	results = self.service.data().ga().get(
	ids='ga:' + self.profile_id,
	filters='ga:pagePath==%s' % (path,),
	start_date=start_date,
	metrics='ga:visitBounceRate',
	dimensions='ga:pagePath',
	max_results=10000,
	end_date=end_date).execute()
	result_data = results.get('rows')
	if not result_data or len(result_data) != 1:
	log.error('Could not pinpoint the bounces for path: %s. Got results: %r',
	path, result_data)
	return
	results = result_data[0]
	bounces = float(results[1])
	# visitBounceRate is already a %
	log.info('Google reports visitBounceRate as %s', bounces)
	ga_model.update_sitewide_stats(period_name, "Totals", {'Bounce rate (home page)': float(bounces)},
	period_complete_day)


	def _locale_stats(self, start_date, end_date, period_name, period_complete_day):
""" Fetches stats about language and country """	""" Fetches stats about language and country """
results = self.service.data().ga().get(	results = self.service.data().ga().get(
ids='ga:' + self.profile_id,	ids='ga:' + self.profile_id,
start_date=start_date,	start_date=start_date,
metrics='ga:uniquePageviews',	metrics='ga:pageviews',
sort='-ga:uniquePageviews',	sort='-ga:pageviews',
dimensions="ga:language,ga:country",	dimensions="ga:language,ga:country",
max_results=10000,	max_results=10000,
end_date=end_date).execute()	end_date=end_date).execute()
result_data = results.get('rows')	result_data = results.get('rows')
data = {}	data = {}
for result in result_data:	for result in result_data:
data[result[0]] = data.get(result[0], 0) + int(result[2])	data[result[0]] = data.get(result[0], 0) + int(result[2])
ga_model.update_sitewide_stats(period_name, "Languages", data)	self._filter_out_long_tail(data, MIN_VIEWS)
	ga_model.update_sitewide_stats(period_name, "Languages", data, period_complete_day)

data = {}	data = {}
for result in result_data:	for result in result_data:
data[result[1]] = data.get(result[1], 0) + int(result[2])	data[result[1]] = data.get(result[1], 0) + int(result[2])
ga_model.update_sitewide_stats(period_name, "Country", data)	self._filter_out_long_tail(data, MIN_VIEWS)
	ga_model.update_sitewide_stats(period_name, "Country", data, period_complete_day)

def _social_stats(self, start_date, end_date, period_name):
	def _download_stats(self, start_date, end_date, period_name, period_complete_day):
	""" Fetches stats about language and country """
	import ckan.model as model

	data = {}

	results = self.service.data().ga().get(
	ids='ga:' + self.profile_id,
	start_date=start_date,
	filters='ga:eventAction==download',
	metrics='ga:totalEvents',
	sort='-ga:totalEvents',
	dimensions="ga:eventLabel",
	max_results=10000,
	end_date=end_date).execute()
	result_data = results.get('rows')
	if not result_data:
	# We may not have data for this time period, so we need to bail
	# early.
	log.info("There is no download data for this time period")
	return

	def process_result_data(result_data, cached=False):
	for result in result_data:
	url = result[0].strip()

	# Get package id associated with the resource that has this URL.
	q = model.Session.query(model.Resource)
	if cached:
	r = q.filter(model.Resource.cache_url.like("%s%%" % url)).first()
	else:
	r = q.filter(model.Resource.url.like("%s%%" % url)).first()

	package_name = r.resource_group.package.name if r else ""
	if package_name:
	data[package_name] = data.get(package_name, 0) + int(result[1])
	else:
	log.warning(u"Could not find resource for URL: {url}".format(url=url))
	continue

	process_result_data(results.get('rows'))

	results = self.service.data().ga().get(
	ids='ga:' + self.profile_id,
	start_date=start_date,
	filters='ga:eventAction==download-cache',
	metrics='ga:totalEvents',
	sort='-ga:totalEvents',
	dimensions="ga:eventLabel",
	max_results=10000,
	end_date=end_date).execute()
	process_result_data(results.get('rows'), cached=False)

	self._filter_out_long_tail(data, MIN_DOWNLOADS)
	ga_model.update_sitewide_stats(period_name, "Downloads", data, period_complete_day)

	def _social_stats(self, start_date, end_date, period_name, period_complete_day):
""" Finds out which social sites people are referred from """	""" Finds out which social sites people are referred from """
results = self.service.data().ga().get(	results = self.service.data().ga().get(
ids='ga:' + self.profile_id,	ids='ga:' + self.profile_id,
start_date=start_date,	start_date=start_date,
metrics='ga:uniquePageviews',	metrics='ga:pageviews',
sort='-ga:uniquePageviews',	sort='-ga:pageviews',
dimensions="ga:socialNetwork,ga:referralPath",	dimensions="ga:socialNetwork,ga:referralPath",
max_results=10000,	max_results=10000,
end_date=end_date).execute()	end_date=end_date).execute()
result_data = results.get('rows')	result_data = results.get('rows')
twitter_links = []
data = {}	data = {}
for result in result_data:	for result in result_data:
if not result[0] == '(not set)':	if not result[0] == '(not set)':
data[result[0]] = data.get(result[0], 0) + int(result[2])	data[result[0]] = data.get(result[0], 0) + int(result[2])
if result[0] == 'Twitter':	self._filter_out_long_tail(data, 3)
twitter_links.append(result[1])	ga_model.update_sitewide_stats(period_name, "Social sources", data, period_complete_day)
ga_model.update_sitewide_stats(period_name, "Social sources", data)

	def _os_stats(self, start_date, end_date, period_name, period_complete_day):
def _os_stats(self, start_date, end_date, period_name):
""" Operating system stats """	""" Operating system stats """
results = self.service.data().ga().get(	results = self.service.data().ga().get(
ids='ga:' + self.profile_id,	ids='ga:' + self.profile_id,
start_date=start_date,	start_date=start_date,
metrics='ga:uniquePageviews',	metrics='ga:pageviews',
sort='-ga:uniquePageviews',	sort='-ga:pageviews',
dimensions="ga:operatingSystem,ga:operatingSystemVersion",	dimensions="ga:operatingSystem,ga:operatingSystemVersion",
max_results=10000,	max_results=10000,
end_date=end_date).execute()	end_date=end_date).execute()
result_data = results.get('rows')	result_data = results.get('rows')
data = {}	data = {}
for result in result_data:	for result in result_data:
data[result[0]] = data.get(result[0], 0) + int(result[2])	data[result[0]] = data.get(result[0], 0) + int(result[2])
ga_model.update_sitewide_stats(period_name, "Operating Systems", data)	self._filter_out_long_tail(data, MIN_VIEWS)
	ga_model.update_sitewide_stats(period_name, "Operating Systems", data, period_complete_day)
data = {}
for result in result_data:	data = {}
key = "%s (%s)" % (result[0],result[1])	for result in result_data:
data[key] = result[2]	if int(result[2]) >= MIN_VIEWS:
ga_model.update_sitewide_stats(period_name, "Operating Systems versions", data)	key = "%s %s" % (result[0],result[1])
	data[key] = result[2]
	ga_model.update_sitewide_stats(period_name, "Operating Systems versions", data, period_complete_day)
def _browser_stats(self, start_date, end_date, period_name):

	def _browser_stats(self, start_date, end_date, period_name, period_complete_day):
""" Information about browsers and browser versions """	""" Information about browsers and browser versions """
results = self.service.data().ga().get(	results = self.service.data().ga().get(
ids='ga:' + self.profile_id,	ids='ga:' + self.profile_id,
start_date=start_date,	start_date=start_date,
metrics='ga:uniquePageviews',	metrics='ga:pageviews',
sort='-ga:uniquePageviews',	sort='-ga:pageviews',
dimensions="ga:browser,ga:browserVersion",	dimensions="ga:browser,ga:browserVersion",
max_results=10000,	max_results=10000,
end_date=end_date).execute()	end_date=end_date).execute()
result_data = results.get('rows')	result_data = results.get('rows')
	# e.g. [u'Firefox', u'19.0', u'20']

data = {}	data = {}
for result in result_data:	for result in result_data:
data[result[0]] = data.get(result[0], 0) + int(result[2])	data[result[0]] = data.get(result[0], 0) + int(result[2])
ga_model.update_sitewide_stats(period_name, "Browsers", data)	self._filter_out_long_tail(data, MIN_VIEWS)
	ga_model.update_sitewide_stats(period_name, "Browsers", data, period_complete_day)
data = {}
for result in result_data:	data = {}
key = "%s (%s)" % (result[0], result[1])	for result in result_data:
data[key] = result[2]	key = "%s %s" % (result[0], self._filter_browser_version(result[0], result[1]))
ga_model.update_sitewide_stats(period_name, "Browser versions", data)	data[key] = data.get(key, 0) + int(result[2])
	self._filter_out_long_tail(data, MIN_VIEWS)
	ga_model.update_sitewide_stats(period_name, "Browser versions", data, period_complete_day)
def _mobile_stats(self, start_date, end_date, period_name):
	@classmethod
	def _filter_browser_version(cls, browser, version_str):
	'''
	Simplifies a browser version string if it is detailed.
	i.e. groups together Firefox 3.5.1 and 3.5.2 to be just 3.
	This is helpful when viewing stats and good to protect privacy.
	'''
	ver = version_str
	parts = ver.split('.')
	if len(parts) > 1:
	if parts[1][0] == '0':
	ver = parts[0]
	else:
	ver = "%s" % (parts[0])
	# Special case complex version nums
	if browser in ['Safari', 'Android Browser']:
	ver = parts[0]
	if len(ver) > 2:
	num_hidden_digits = len(ver) - 2
	ver = ver[0] + ver[1] + 'X' * num_hidden_digits
	return ver

	def _mobile_stats(self, start_date, end_date, period_name, period_complete_day):
""" Info about mobile devices """	""" Info about mobile devices """

results = self.service.data().ga().get(	results = self.service.data().ga().get(
ids='ga:' + self.profile_id,	ids='ga:' + self.profile_id,
start_date=start_date,	start_date=start_date,
metrics='ga:uniquePageviews',	metrics='ga:pageviews',
sort='-ga:uniquePageviews',	sort='-ga:pageviews',
dimensions="ga:mobileDeviceBranding, ga:mobileDeviceInfo",	dimensions="ga:mobileDeviceBranding, ga:mobileDeviceInfo",
max_results=10000,	max_results=10000,
end_date=end_date).execute()	end_date=end_date).execute()

result_data = results.get('rows')	result_data = results.get('rows')
data = {}	data = {}
for result in result_data:	for result in result_data:
data[result[0]] = data.get(result[0], 0) + int(result[2])	data[result[0]] = data.get(result[0], 0) + int(result[2])
ga_model.update_sitewide_stats(period_name, "Mobile brands", data)	self._filter_out_long_tail(data, MIN_VIEWS)
	ga_model.update_sitewide_stats(period_name, "Mobile brands", data, period_complete_day)

data = {}	data = {}
for result in result_data:	for result in result_data:
data[result[1]] = data.get(result[1], 0) + int(result[2])	data[result[1]] = data.get(result[1], 0) + int(result[2])
ga_model.update_sitewide_stats(period_name, "Mobile devices", data)	self._filter_out_long_tail(data, MIN_VIEWS)
	ga_model.update_sitewide_stats(period_name, "Mobile devices", data, period_complete_day)

	@classmethod
	def _filter_out_long_tail(cls, data, threshold=10):
	'''
	Given data which is a frequency distribution, filter out
	results which are below a threshold count. This is good to protect
	privacy.
	'''
	for key, value in data.items():
	if value < threshold:
	del data[key]

file:a/ckanext/ga_report/ga_auth.py -> file:b/ckanext/ga_report/ga_auth.py

file:a/ckanext/ga_report/ga_model.py -> file:b/ckanext/ga_report/ga_model.py

import re	import re
import uuid	import uuid

from sqlalchemy import Table, Column, MetaData	from sqlalchemy import Table, Column, MetaData, ForeignKey
from sqlalchemy import types	from sqlalchemy import types
from sqlalchemy.sql import select	from sqlalchemy.sql import select
from sqlalchemy.orm import mapper	from sqlalchemy.orm import mapper, relation
from sqlalchemy import func	from sqlalchemy import func

import ckan.model as model	import ckan.model as model
from ckan.lib.base import *	from ckan.lib.base import *

	log = __import__('logging').getLogger(__name__)

def make_uuid():	def make_uuid():
return unicode(uuid.uuid4())	return unicode(uuid.uuid4())

	metadata = MetaData()

class GA_Url(object):	class GA_Url(object):

def __init__(self, **kwargs):	def __init__(self, **kwargs):
for k,v in kwargs.items():	for k,v in kwargs.items():
setattr(self, k, v)	setattr(self, k, v)

class GA_Stat(object):

def __init__(self, **kwargs):
for k,v in kwargs.items():
setattr(self, k, v)

class GA_Publisher(object):

def __init__(self, **kwargs):
for k,v in kwargs.items():
setattr(self, k, v)


metadata = MetaData()
url_table = Table('ga_url', metadata,	url_table = Table('ga_url', metadata,
Column('id', types.UnicodeText, primary_key=True,	Column('id', types.UnicodeText, primary_key=True,
default=make_uuid),	default=make_uuid),
Column('period_name', types.UnicodeText),	Column('period_name', types.UnicodeText),
Column('period_complete_day', types.Integer),	Column('period_complete_day', types.Integer),
Column('pageviews', types.UnicodeText),	Column('pageviews', types.UnicodeText),
Column('visitors', types.UnicodeText),	Column('visits', types.UnicodeText),
Column('url', types.UnicodeText),	Column('url', types.UnicodeText),
Column('department_id', types.UnicodeText),	Column('department_id', types.UnicodeText),
	Column('package_id', types.UnicodeText),
)	)
mapper(GA_Url, url_table)	mapper(GA_Url, url_table)


	class GA_Stat(object):

	def __init__(self, **kwargs):
	for k,v in kwargs.items():
	setattr(self, k, v)

stat_table = Table('ga_stat', metadata,	stat_table = Table('ga_stat', metadata,
Column('id', types.UnicodeText, primary_key=True,	Column('id', types.UnicodeText, primary_key=True,
default=make_uuid),	default=make_uuid),
Column('period_name', types.UnicodeText),	Column('period_name', types.UnicodeText),
	Column('period_complete_day', types.UnicodeText),
Column('stat_name', types.UnicodeText),	Column('stat_name', types.UnicodeText),
Column('key', types.UnicodeText),	Column('key', types.UnicodeText),
Column('value', types.UnicodeText), )	Column('value', types.UnicodeText), )
mapper(GA_Stat, stat_table)	mapper(GA_Stat, stat_table)


	class GA_Publisher(object):

	def __init__(self, **kwargs):
	for k,v in kwargs.items():
	setattr(self, k, v)

pub_table = Table('ga_publisher', metadata,	pub_table = Table('ga_publisher', metadata,
Column('id', types.UnicodeText, primary_key=True,	Column('id', types.UnicodeText, primary_key=True,
default=make_uuid),	default=make_uuid),
Column('period_name', types.UnicodeText),	Column('period_name', types.UnicodeText),
Column('publisher_name', types.UnicodeText),	Column('publisher_name', types.UnicodeText),
Column('views', types.UnicodeText),	Column('views', types.UnicodeText),
Column('visitors', types.UnicodeText),	Column('visits', types.UnicodeText),
Column('toplevel', types.Boolean, default=False),	Column('toplevel', types.Boolean, default=False),
Column('subpublishercount', types.Integer, default=0),	Column('subpublishercount', types.Integer, default=0),
Column('parent', types.UnicodeText),	Column('parent', types.UnicodeText),
)	)
mapper(GA_Publisher, pub_table)	mapper(GA_Publisher, pub_table)


	class GA_ReferralStat(object):

	def __init__(self, **kwargs):
	for k,v in kwargs.items():
	setattr(self, k, v)

	referrer_table = Table('ga_referrer', metadata,
	Column('id', types.UnicodeText, primary_key=True,
	default=make_uuid),
	Column('period_name', types.UnicodeText),
	Column('source', types.UnicodeText),
	Column('url', types.UnicodeText),
	Column('count', types.Integer),
	)
	mapper(GA_ReferralStat, referrer_table)



def init_tables():	def init_tables():
metadata.create_all(model.meta.engine)	metadata.create_all(model.meta.engine)


cached_tables = {}	cached_tables = {}


def get_table(name):	def get_table(name):
if name not in cached_tables:	if name not in cached_tables:
meta = MetaData()	meta = MetaData()
meta.reflect(bind=model.meta.engine)	meta.reflect(bind=model.meta.engine)
table = meta.tables[name]	table = meta.tables[name]
cached_tables[name] = table	cached_tables[name] = table
return cached_tables[name]	return cached_tables[name]


def _normalize_url(url):	def _normalize_url(url):
'''Strip off the hostname etc. Do this before storing it.	'''Strip off the hostname etc. Do this before storing it.

>>> normalize_url('http://data.gov.uk/dataset/weekly_fuel_prices')	>>> normalize_url('http://data.gov.uk/dataset/weekly_fuel_prices')
'/dataset/weekly_fuel_prices'	'/dataset/weekly_fuel_prices'
'''	'''
url = re.sub('https?://(www\.)?data.gov.uk', '', url)	return '/' + '/'.join(url.split('/')[3:])
return url

	def _get_package_and_publisher(url):
def _get_department_id_of_url(url):
# e.g. /dataset/fuel_prices	# e.g. /dataset/fuel_prices
# e.g. /dataset/fuel_prices/resource/e63380d4	# e.g. /dataset/fuel_prices/resource/e63380d4
dataset_match = re.match('/dataset/([^/]+)(/.*)?', url)	dataset_match = re.match('/dataset/([^/]+)(/.*)?', url)
if dataset_match:	if dataset_match:
dataset_ref = dataset_match.groups()[0]	dataset_ref = dataset_match.groups()[0]
dataset = model.Package.get(dataset_ref)	dataset = model.Package.get(dataset_ref)
if dataset:	if dataset:
publisher_groups = dataset.get_groups('publisher')	publisher_groups = dataset.get_groups('publisher')
if publisher_groups:	if publisher_groups:
return publisher_groups[0].name	return dataset_ref,publisher_groups[0].name
	return dataset_ref, None
else:	else:
publisher_match = re.match('/publisher/([^/]+)(/.*)?', url)	publisher_match = re.match('/publisher/([^/]+)(/.*)?', url)
if publisher_match:	if publisher_match:
return publisher_match.groups()[0]	return None, publisher_match.groups()[0]
	return None, None

def update_sitewide_stats(period_name, stat_name, data):	def update_sitewide_stats(period_name, stat_name, data, period_complete_day):
for k,v in data.iteritems():	for k,v in data.iteritems():
item = model.Session.query(GA_Stat).\	item = model.Session.query(GA_Stat).\
filter(GA_Stat.period_name==period_name).\	filter(GA_Stat.period_name==period_name).\
filter(GA_Stat.key==k).\	filter(GA_Stat.key==k).\
filter(GA_Stat.stat_name==stat_name).first()	filter(GA_Stat.stat_name==stat_name).first()
if item:	if item:
item.period_name = period_name	item.period_name = period_name
item.key = k	item.key = k
item.value = v	item.value = v
	item.period_complete_day = period_complete_day
model.Session.add(item)	model.Session.add(item)
else:	else:
# create the row	# create the row
values = {'id': make_uuid(),	values = {'id': make_uuid(),
'period_name': period_name,	'period_name': period_name,
	'period_complete_day': period_complete_day,
'key': k,	'key': k,
'value': v,	'value': v,
'stat_name': stat_name	'stat_name': stat_name
}	}
model.Session.add(GA_Stat(**values))	model.Session.add(GA_Stat(**values))
model.Session.commit()	model.Session.commit()


	def pre_update_url_stats(period_name):
	log.debug("Deleting '%s' records" % period_name)
	model.Session.query(GA_Url).\
	filter(GA_Url.period_name==period_name).delete()

	count = model.Session.query(GA_Url).\
	filter(GA_Url.period_name == 'All').count()
	log.debug("Deleting %d 'All' records" % count)
	count = model.Session.query(GA_Url).\
	filter(GA_Url.period_name == 'All').delete()
	log.debug("Deleted %d 'All' records" % count)

	model.Session.flush()
	model.Session.commit()
	model.repo.commit_and_remove()

	def post_update_url_stats():

	""" Check the distinct url field in ga_url and make sure
	it has an All record. If not then create one.

	After running this then every URL should have an All
	record regardless of whether the URL has an entry for
	the month being currently processed.
	"""
	query = """select url, pageviews::int, visits::int
	from ga_url
	where url not in (select url from ga_url where period_name ='All')"""
	connection = model.Session.connection()
	res = connection.execute(query)

	views, visits = {}, {}
	# url, views, visits
	for row in res:
	views[row[0]] = views.get(row[0], 0) + row[1]
	visits[row[0]] = visits.get(row[0], 0) + row[2]

	for key in views.keys():
	package, publisher = _get_package_and_publisher(key)

	values = {'id': make_uuid(),
	'period_name': "All",
	'period_complete_day': 0,
	'url': key,
	'pageviews': views[key],
	'visits': visits[key],
	'department_id': publisher,
	'package_id': publisher
	}
	model.Session.add(GA_Url(**values))
	model.Session.commit()


def update_url_stats(period_name, period_complete_day, url_data):	def update_url_stats(period_name, period_complete_day, url_data):
for url, views, visitors in url_data:	'''
url = _normalize_url(url)	Given a list of urls and number of hits for each during a given period,
department_id = _get_department_id_of_url(url)	stores them in GA_Url under the period and recalculates the totals for
	the 'All' period.
# see if the row for this url & month is in the table already	'''
	for url, views, visits in url_data:
	package, publisher = _get_package_and_publisher(url)


item = model.Session.query(GA_Url).\	item = model.Session.query(GA_Url).\
filter(GA_Url.period_name==period_name).\	filter(GA_Url.period_name==period_name).\
filter(GA_Url.url==url).first()	filter(GA_Url.url==url).first()
if item:	if item:
item.period_name = period_name	item.pageviews = item.pageviews + views
item.pageviews = views	item.visits = item.visits + visits
item.visitors = visitors	if not item.package_id:
item.department_id = department_id	item.package_id = package
	if not item.department_id:
	item.department_id = publisher
model.Session.add(item)	model.Session.add(item)
else:	else:
# create the row
values = {'id': make_uuid(),	values = {'id': make_uuid(),
'period_name': period_name,	'period_name': period_name,
'period_complete_day': period_complete_day,	'period_complete_day': period_complete_day,
'url': url,	'url': url,
'pageviews': views,	'pageviews': views,
'visitors': visitors,	'visits': visits,
'department_id': department_id	'department_id': publisher,
	'package_id': package
}	}
model.Session.add(GA_Url(**values))	model.Session.add(GA_Url(**values))
model.Session.commit()	model.Session.commit()

	if package:
	old_pageviews, old_visits = 0, 0
	old = model.Session.query(GA_Url).\
	filter(GA_Url.period_name=='All').\
	filter(GA_Url.url==url).all()
	old_pageviews = sum([int(o.pageviews) for o in old])
	old_visits = sum([int(o.visits) for o in old])

	entries = model.Session.query(GA_Url).\
	filter(GA_Url.period_name!='All').\
	filter(GA_Url.url==url).all()
	values = {'id': make_uuid(),
	'period_name': 'All',
	'period_complete_day': 0,
	'url': url,
	'pageviews': sum([int(e.pageviews) for e in entries]) + int(old_pageviews),
	'visits': sum([int(e.visits or 0) for e in entries]) + int(old_visits),
	'department_id': publisher,
	'package_id': package
	}

	model.Session.add(GA_Url(**values))
	model.Session.commit()




	def update_social(period_name, data):
	# Clean up first.
	model.Session.query(GA_ReferralStat).\
	filter(GA_ReferralStat.period_name==period_name).delete()

	for url,data in data.iteritems():
	for entry in data:
	source = entry[0]
	count = entry[1]

	item = model.Session.query(GA_ReferralStat).\
	filter(GA_ReferralStat.period_name==period_name).\
	filter(GA_ReferralStat.source==source).\
	filter(GA_ReferralStat.url==url).first()
	if item:
	item.count = item.count + count
	model.Session.add(item)
	else:
	# create the row
	values = {'id': make_uuid(),
	'period_name': period_name,
	'source': source,
	'url': url,
	'count': count,
	}
	model.Session.add(GA_ReferralStat(**values))
	model.Session.commit()

def update_publisher_stats(period_name):	def update_publisher_stats(period_name):
"""	"""
Updates the publisher stats from the data retrieved for /dataset/*	Updates the publisher stats from the data retrieved for /dataset/*
and /publisher/*. Will run against each dataset and generates the	and /publisher/*. Will run against each dataset and generates the
totals for the entire tree beneath each publisher.	totals for the entire tree beneath each publisher.
"""	"""
toplevel = get_top_level()	toplevel = get_top_level()
publishers = model.Session.query(model.Group).\	publishers = model.Session.query(model.Group).\
filter(model.Group.type=='publisher').\	filter(model.Group.type=='publisher').\
filter(model.Group.state=='active').all()	filter(model.Group.state=='active').all()
for publisher in publishers:	for publisher in publishers:
views, visitors, subpub = update_publisher(period_name, publisher, publisher.name)	views, visits, subpub = update_publisher(period_name, publisher, publisher.name)
parent, parents = '', publisher.get_groups('publisher')	parent, parents = '', publisher.get_groups('publisher')
if parents:	if parents:
parent = parents[0].name	parent = parents[0].name
item = model.Session.query(GA_Publisher).\	item = model.Session.query(GA_Publisher).\
filter(GA_Publisher.period_name==period_name).\	filter(GA_Publisher.period_name==period_name).\
filter(GA_Publisher.publisher_name==publisher.name).first()	filter(GA_Publisher.publisher_name==publisher.name).first()
if item:	if item:
item.views = views	item.views = views
item.visitors = visitors	item.visits = visits
item.publisher_name = publisher.name	item.publisher_name = publisher.name
item.toplevel = publisher in toplevel	item.toplevel = publisher in toplevel
item.subpublishercount = subpub	item.subpublishercount = subpub
item.parent = parent	item.parent = parent
model.Session.add(item)	model.Session.add(item)
else:	else:
# create the row	# create the row
values = {'id': make_uuid(),	values = {'id': make_uuid(),
'period_name': period_name,	'period_name': period_name,
'publisher_name': publisher.name,	'publisher_name': publisher.name,
'views': views,	'views': views,
'visitors': visitors,	'visits': visits,
'toplevel': publisher in toplevel,	'toplevel': publisher in toplevel,
'subpublishercount': subpub,	'subpublishercount': subpub,
'parent': parent	'parent': parent
}	}
model.Session.add(GA_Publisher(**values))	model.Session.add(GA_Publisher(**values))
model.Session.commit()	model.Session.commit()


def update_publisher(period_name, pub, part=''):	def update_publisher(period_name, pub, part=''):
views,visitors,subpub = 0, 0, 0	views,visits,subpub = 0, 0, 0
for publisher in go_down_tree(pub):	for publisher in go_down_tree(pub):
subpub = subpub + 1	subpub = subpub + 1
items = model.Session.query(GA_Url).\	items = model.Session.query(GA_Url).\
filter(GA_Url.period_name==period_name).\	filter(GA_Url.period_name==period_name).\
filter(GA_Url.department_id==publisher.name).all()	filter(GA_Url.department_id==publisher.name).all()
for item in items:	for item in items:
views = views + int(item.pageviews)	views = views + int(item.pageviews)
visitors = visitors + int(item.visitors)	visits = visits + int(item.visits)

return views, visitors, (subpub-1)	return views, visits, (subpub-1)


def get_top_level():	def get_top_level():
'''Returns the top level publishers.'''	'''Returns the top level publishers.'''
return model.Session.query(model.Group).\	return model.Session.query(model.Group).\
outerjoin(model.Member, model.Member.table_id == model.Group.id and \	outerjoin(model.Member, model.Member.table_id == model.Group.id and \
model.Member.table_name == 'group' and \	model.Member.table_name == 'group' and \
model.Member.state == 'active').\	model.Member.state == 'active').\
filter(model.Member.id==None).\	filter(model.Member.id==None).\
filter(model.Group.type=='publisher').\	filter(model.Group.type=='publisher').\
order_by(model.Group.name).all()	order_by(model.Group.name).all()

def get_children(publisher):	def get_children(publisher):
'''Finds child publishers for the given publisher (object). (Not recursive)'''	'''Finds child publishers for the given publisher (object). (Not recursive)'''
from ckan.model.group import HIERARCHY_CTE	from ckan.model.group import HIERARCHY_CTE
return model.Session.query(model.Group).\	return model.Session.query(model.Group).\
from_statement(HIERARCHY_CTE).params(id=publisher.id, type='publisher').\	from_statement(HIERARCHY_CTE).params(id=publisher.id, type='publisher').\
all()	all()

def go_down_tree(publisher):	def go_down_tree(publisher):
'''Provided with a publisher object, it walks down the hierarchy and yields each publisher,	'''Provided with a publisher object, it walks down the hierarchy and yields each publisher,
including the one you supply.'''	including the one you supply.'''
yield publisher	yield publisher
for child in get_children(publisher):	for child in get_children(publisher):
for grandchild in go_down_tree(child):	for grandchild in go_down_tree(child):
yield grandchild	yield grandchild

	def delete(period_name):
	'''
	Deletes table data for the specified period, or specify 'all'
	for all periods.
	'''
	for object_type in (GA_Url, GA_Stat, GA_Publisher, GA_ReferralStat):
	q = model.Session.query(object_type)
	if period_name != 'All':
	q = q.filter_by(period_name=period_name)
	q.delete()
	model.repo.commit_and_remove()

	def get_score_for_dataset(dataset_name):
	'''
	Returns a "current popularity" score for a dataset,
	based on how many views it has had recently.
	'''
	import datetime
	now = datetime.datetime.now()
	last_month = now - datetime.timedelta(days=30)
	period_names = ['%s-%02d' % (last_month.year, last_month.month),
	'%s-%02d' % (now.year, now.month),
	]

	score = 0
	for period_name in period_names:
	score /= 2 # previous periods are discounted by 50%
	entry = model.Session.query(GA_Url)\
	.filter(GA_Url.period_name==period_name)\
	.filter(GA_Url.package_id==dataset_name).first()
	# score
	if entry:
	views = float(entry.pageviews)
	if entry.period_complete_day:
	views_per_day = views / entry.period_complete_day
	else:
	views_per_day = views / 15 # guess
	score += views_per_day

	score = int(score * 100)
	log.debug('Popularity %s: %s', score, dataset_name)
	return score

file:b/ckanext/ga_report/helpers.py (new)

	import logging
	import operator

	import ckan.lib.base as base
	import ckan.model as model
	from ckan.logic import get_action

	from ckanext.ga_report.ga_model import GA_Url, GA_Publisher
	from ckanext.ga_report.controller import _get_publishers
	_log = logging.getLogger(__name__)

	def popular_datasets(count=10):
	import random

	publisher = None
	publishers = _get_publishers(30)
	total = len(publishers)
	while not publisher or not datasets:
	rand = random.randrange(0, total)
	publisher = publishers[rand][0]
	if not publisher.state == 'active':
	publisher = None
	continue
	datasets = _datasets_for_publisher(publisher, 10)[:count]

	ctx = {
	'datasets': datasets,
	'publisher': publisher
	}
	return base.render_snippet('ga_report/ga_popular_datasets.html', **ctx)

	def single_popular_dataset(top=20):
	'''Returns a random dataset from the most popular ones.

	:param top: the number of top datasets to select from
	'''
	import random

	top_datasets = model.Session.query(GA_Url).\
	filter(GA_Url.url.like('/dataset/%')).\
	order_by('ga_url.pageviews::int desc')
	num_top_datasets = top_datasets.count()

	dataset = None
	if num_top_datasets:
	count = 0
	while not dataset:
	rand = random.randrange(0, min(top, num_top_datasets))
	ga_url = top_datasets[rand]
	dataset = model.Package.get(ga_url.url[len('/dataset/'):])
	if dataset and not dataset.state == 'active':
	dataset = None
	# When testing, it is possible that top datasets are not available
	# so only go round this loop a few times before falling back on
	# a random dataset.
	count += 1
	if count > 10:
	break
	if not dataset:
	# fallback
	dataset = model.Session.query(model.Package)\
	.filter_by(state='active').first()
	if not dataset:
	return None
	dataset_dict = get_action('package_show')({'model': model,
	'session': model.Session,
	'validate': False},
	{'id':dataset.id})
	return dataset_dict

	def single_popular_dataset_html(top=20):
	dataset_dict = single_popular_dataset(top)
	groups = package.get('groups', [])
	publishers = [ g for g in groups if g.get('type') == 'publisher' ]
	publisher = publishers[0] if publishers else {'name':'', 'title': ''}
	context = {
	'dataset': dataset_dict,
	'publisher': publisher_dict
	}
	return base.render_snippet('ga_report/ga_popular_single.html', **context)


	def most_popular_datasets(publisher, count=20):

	if not publisher:
	_log.error("No valid publisher passed to 'most_popular_datasets'")
	return ""

	results = _datasets_for_publisher(publisher, count)

	ctx = {
	'dataset_count': len(results),
	'datasets': results,

	'publisher': publisher
	}

	return base.render_snippet('ga_report/publisher/popular.html', **ctx)

	def _datasets_for_publisher(publisher, count):
	datasets = {}
	entries = model.Session.query(GA_Url).\
	filter(GA_Url.department_id==publisher.name).\
	filter(GA_Url.url.like('/dataset/%')).\
	order_by('ga_url.pageviews::int desc').all()
	for entry in entries:
	if len(datasets) < count:
	p = model.Package.get(entry.url[len('/dataset/'):])
	if not p in datasets:
	datasets[p] = {'views':0, 'visits': 0}
	datasets[p]['views'] = datasets[p]['views'] + int(entry.pageviews)
	datasets[p]['visits'] = datasets[p]['visits'] + int(entry.visits)

	results = []
	for k, v in datasets.iteritems():
	results.append((k,v['views'],v['visits']))

	return sorted(results, key=operator.itemgetter(1), reverse=True)

file:a/ckanext/ga_report/plugin.py -> file:b/ckanext/ga_report/plugin.py

import logging	import logging
import ckan.lib.helpers as h	import ckan.lib.helpers as h
import ckan.plugins as p	import ckan.plugins as p
from ckan.plugins import implements, toolkit	from ckan.plugins import implements, toolkit
#import gasnippet
#import commands	from ckanext.ga_report.helpers import (most_popular_datasets,
#import dbutil	popular_datasets,
	single_popular_dataset)

log = logging.getLogger('ckanext.ga-report')	log = logging.getLogger('ckanext.ga-report')

class GAReportPlugin(p.SingletonPlugin):	class GAReportPlugin(p.SingletonPlugin):
implements(p.IConfigurer, inherit=True)	implements(p.IConfigurer, inherit=True)
implements(p.IRoutes, inherit=True)	implements(p.IRoutes, inherit=True)
	implements(p.ITemplateHelpers, inherit=True)

def update_config(self, config):	def update_config(self, config):
toolkit.add_template_directory(config, 'templates')	toolkit.add_template_directory(config, 'templates')
toolkit.add_public_directory(config, 'public')	toolkit.add_public_directory(config, 'public')

	def get_helpers(self):
	"""
	A dictionary of extra helpers that will be available to provide
	ga report info to templates.
	"""
	return {
	'ga_report_installed': lambda: True,
	'popular_datasets': popular_datasets,
	'most_popular_datasets': most_popular_datasets,
	'single_popular_dataset': single_popular_dataset
	}

def after_map(self, map):	def after_map(self, map):
	# GaReport
map.connect(	map.connect(
'/data/analytics/publisher',	'/data/site-usage',
controller='ckanext.ga_report.controller:GaPublisherReport',
action='index'
)
map.connect(
'/data/analytics/publisher/{id}',
controller='ckanext.ga_report.controller:GaPublisherReport',
action='read'
)
map.connect(
'/data/analytics',
controller='ckanext.ga_report.controller:GaReport',	controller='ckanext.ga_report.controller:GaReport',
action='index'	action='index'
)	)
map.connect(	map.connect(
'/data/analytics/data_{month}.csv',	'/data/site-usage/data_{month}.csv',
controller='ckanext.ga_report.controller:GaReport',	controller='ckanext.ga_report.controller:GaReport',
action='csv'	action='csv'
	)
	map.connect(
	'/data/site-usage/downloads',
	controller='ckanext.ga_report.controller:GaReport',
	action='downloads'
	)
	map.connect(
	'/data/site-usage/downloads_{month}.csv',
	controller='ckanext.ga_report.controller:GaReport',
	action='csv_downloads'
	)

	# GaDatasetReport
	map.connect(
	'/data/site-usage/publisher',
	controller='ckanext.ga_report.controller:GaDatasetReport',
	action='publishers'
	)
	map.connect(
	'/data/site-usage/publishers_{month}.csv',
	controller='ckanext.ga_report.controller:GaDatasetReport',
	action='publisher_csv'
	)
	map.connect(
	'/data/site-usage/dataset/datasets_{id}_{month}.csv',
	controller='ckanext.ga_report.controller:GaDatasetReport',
	action='dataset_csv'
	)
	map.connect(
	'/data/site-usage/dataset',
	controller='ckanext.ga_report.controller:GaDatasetReport',
	action='read'
	)
	map.connect(
	'/data/site-usage/dataset/{id}',
	controller='ckanext.ga_report.controller:GaDatasetReport',
	action='read_publisher'
)	)
return map	return map

file:b/ckanext/ga_report/public/scripts/vendor/jquery.sparkline.modified.js (new)

	/*
	* This file has been modified!
	* I've added a static Tooltip option.
	* - Tom Rees
	* - January 2013
	*/
	/**
	*
	* jquery.sparkline.js
	*
	* v2.1
	* (c) Splunk, Inc
	* Contact: Gareth Watts (gareth@splunk.com)
	* http://omnipotent.net/jquery.sparkline/
	*
	* Generates inline sparkline charts from data supplied either to the method
	* or inline in HTML
	*
	* Compatible with Internet Explorer 6.0+ and modern browsers equipped with the canvas tag
	* (Firefox 2.0+, Safari, Opera, etc)
	*
	* License: New BSD License
	*
	* Copyright (c) 2012, Splunk Inc.
	* All rights reserved.
	*
	* Redistribution and use in source and binary forms, with or without modification,
	* are permitted provided that the following conditions are met:
	*
	* * Redistributions of source code must retain the above copyright notice,
	* this list of conditions and the following disclaimer.
	* * Redistributions in binary form must reproduce the above copyright notice,
	* this list of conditions and the following disclaimer in the documentation
	* and/or other materials provided with the distribution.
	* * Neither the name of Splunk Inc nor the names of its contributors may
	* be used to endorse or promote products derived from this software without
	* specific prior written permission.
	*
	* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY
	* EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
	* OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT
	* SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
	* SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT
	* OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
	* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
	* OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
	* SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
	*
	*
	* Usage:
	* $(selector).sparkline(values, options)
	*
	* If values is undefined or set to 'html' then the data values are read from the specified tag:
	* <p>Sparkline: <span class="sparkline">1,4,6,6,8,5,3,5</span></p>
	* $('.sparkline').sparkline();
	* There must be no spaces in the enclosed data set
	*
	* Otherwise values must be an array of numbers or null values
	* <p>Sparkline: <span id="sparkline1">This text replaced if the browser is compatible</span></p>
	* $('#sparkline1').sparkline([1,4,6,6,8,5,3,5])
	* $('#sparkline2').sparkline([1,4,6,null,null,5,3,5])
	*
	* Values can also be specified in an HTML comment, or as a values attribute:
	* <p>Sparkline: <span class="sparkline"><!--1,4,6,6,8,5,3,5 --></span></p>
	* <p>Sparkline: <span class="sparkline" values="1,4,6,6,8,5,3,5"></span></p>
	* $('.sparkline').sparkline();
	*
	* For line charts, x values can also be specified:
	* <p>Sparkline: <span class="sparkline">1:1,2.7:4,3.4:6,5:6,6:8,8.7:5,9:3,10:5</span></p>
	* $('#sparkline1').sparkline([ [1,1], [2.7,4], [3.4,6], [5,6], [6,8], [8.7,5], [9,3], [10,5] ])
	*
	* By default, options should be passed in as teh second argument to the sparkline function:
	* $('.sparkline').sparkline([1,2,3,4], {type: 'bar'})
	*
	* Options can also be set by passing them on the tag itself. This feature is disabled by default though
	* as there's a slight performance overhead:
	* $('.sparkline').sparkline([1,2,3,4], {enableTagOptions: true})
	* <p>Sparkline: <span class="sparkline" sparkType="bar" sparkBarColor="red">loading</span></p>
	* Prefix all options supplied as tag attribute with "spark" (configurable by setting tagOptionPrefix)
	*
	* Supported options:
	* lineColor - Color of the line used for the chart
	* fillColor - Color used to fill in the chart - Set to '' or false for a transparent chart
	* width - Width of the chart - Defaults to 3 times the number of values in pixels
	* height - Height of the chart - Defaults to the height of the containing element
	* chartRangeMin - Specify the minimum value to use for the Y range of the chart - Defaults to the minimum value supplied
	* chartRangeMax - Specify the maximum value to use for the Y range of the chart - Defaults to the maximum value supplied
	* chartRangeClip - Clip out of range values to the max/min specified by chartRangeMin and chartRangeMax
	* chartRangeMinX - Specify the minimum value to use for the X range of the chart - Defaults to the minimum value supplied
	* chartRangeMaxX - Specify the maximum value to use for the X range of the chart - Defaults to the maximum value supplied
	* composite - If true then don't erase any existing chart attached to the tag, but draw
	* another chart over the top - Note that width and height are ignored if an
	* existing chart is detected.
	* tagValuesAttribute - Name of tag attribute to check for data values - Defaults to 'values'
	* enableTagOptions - Whether to check tags for sparkline options
	* tagOptionPrefix - Prefix used for options supplied as tag attributes - Defaults to 'spark'
	* disableHiddenCheck - If set to true, then the plugin will assume that charts will never be drawn into a
	* hidden dom element, avoding a browser reflow
	* disableInteraction - If set to true then all mouseover/click interaction behaviour will be disabled,
	* making the plugin perform much like it did in 1.x
	* disableTooltips - If set to true then tooltips will be disabled - Defaults to false (tooltips enabled)
	* disableHighlight - If set to true then highlighting of selected chart elements on mouseover will be disabled
	* defaults to false (highlights enabled)
	* highlightLighten - Factor to lighten/darken highlighted chart values by - Defaults to 1.4 for a 40% increase
	* tooltipContainer - Specify which DOM element the tooltip should be rendered into - defaults to document.body
	* tooltipClassname - Optional CSS classname to apply to tooltips - If not specified then a default style will be applied
	* tooltipOffsetX - How many pixels away from the mouse pointer to render the tooltip on the X axis
	* tooltipOffsetY - How many pixels away from the mouse pointer to render the tooltip on the r axis
	* tooltipFormatter - Optional callback that allows you to override the HTML displayed in the tooltip
	* callback is given arguments of (sparkline, options, fields)
	* tooltipChartTitle - If specified then the tooltip uses the string specified by this setting as a title
	* tooltipFormat - A format string or SPFormat object (or an array thereof for multiple entries)
	* to control the format of the tooltip
	* tooltipPrefix - A string to prepend to each field displayed in a tooltip
	* tooltipSuffix - A string to append to each field displayed in a tooltip
	* tooltipSkipNull - If true then null values will not have a tooltip displayed (defaults to true)
	* tooltipValueLookups - An object or range map to map field values to tooltip strings
	* (eg. to map -1 to "Lost", 0 to "Draw", and 1 to "Win")
	* numberFormatter - Optional callback for formatting numbers in tooltips
	* numberDigitGroupSep - Character to use for group separator in numbers "1,234" - Defaults to ","
	* numberDecimalMark - Character to use for the decimal point when formatting numbers - Defaults to "."
	* numberDigitGroupCount - Number of digits between group separator - Defaults to 3
	*
	* There are 7 types of sparkline, selected by supplying a "type" option of 'line' (default),
	* 'bar', 'tristate', 'bullet', 'discrete', 'pie' or 'box'
	* line - Line chart. Options:
	* spotColor - Set to '' to not end each line in a circular spot
	* minSpotColor - If set, color of spot at minimum value
	* maxSpotColor - If set, color of spot at maximum value
	* spotRadius - Radius in pixels
	* lineWidth - Width of line in pixels
	* normalRangeMin
	* normalRangeMax - If set draws a filled horizontal bar between these two values marking the "normal"
	* or expected range of values
	* normalRangeColor - Color to use for the above bar
	* drawNormalOnTop - Draw the normal range above the chart fill color if true
	* defaultPixelsPerValue - Defaults to 3 pixels of width for each value in the chart
	* highlightSpotColor - The color to use for drawing a highlight spot on mouseover - Set to null to disable
	* highlightLineColor - The color to use for drawing a highlight line on mouseover - Set to null to disable
	* valueSpots - Specify which points to draw spots on, and in which color. Accepts a range map
	*
	* bar - Bar chart. Options:
	* barColor - Color of bars for postive values
	* negBarColor - Color of bars for negative values
	* zeroColor - Color of bars with zero values
	* nullColor - Color of bars with null values - Defaults to omitting the bar entirely
	* barWidth - Width of bars in pixels
	* colorMap - Optional mappnig of values to colors to override the *BarColor values above
	* can be an Array of values to control the color of individual bars or a range map
	* to specify colors for individual ranges of values
	* barSpacing - Gap between bars in pixels
	* zeroAxis - Centers the y-axis around zero if true
	*
	* tristate - Charts values of win (>0), lose (<0) or draw (=0)
	* posBarColor - Color of win values
	* negBarColor - Color of lose values
	* zeroBarColor - Color of draw values
	* barWidth - Width of bars in pixels
	* barSpacing - Gap between bars in pixels
	* colorMap - Optional mappnig of values to colors to override the *BarColor values above
	* can be an Array of values to control the color of individual bars or a range map
	* to specify colors for individual ranges of values
	*
	* discrete - Options:
	* lineHeight - Height of each line in pixels - Defaults to 30% of the graph height
	* thesholdValue - Values less than this value will be drawn using thresholdColor instead of lineColor
	* thresholdColor
	*
	* bullet - Values for bullet graphs msut be in the order: target, performance, range1, range2, range3, ...
	* options:
	* targetColor - The color of the vertical target marker
	* targetWidth - The width of the target marker in pixels
	* performanceColor - The color of the performance measure horizontal bar
	* rangeColors - Colors to use for each qualitative range background color
	*
	* pie - Pie chart. Options:
	* sliceColors - An array of colors to use for pie slices
	* offset - Angle in degrees to offset the first slice - Try -90 or +90
	* borderWidth - Width of border to draw around the pie chart, in pixels - Defaults to 0 (no border)
	* borderColor - Color to use for the pie chart border - Defaults to #000
	*
	* box - Box plot. Options:
	* raw - Set to true to supply pre-computed plot points as values
	* values should be: low_outlier, low_whisker, q1, median, q3, high_whisker, high_outlier
	* When set to false you can supply any number of values and the box plot will
	* be computed for you. Default is false.
	* showOutliers - Set to true (default) to display outliers as circles
	* outlierIQR - Interquartile range used to determine outliers. Default 1.5
	* boxLineColor - Outline color of the box
	* boxFillColor - Fill color for the box
	* whiskerColor - Line color used for whiskers
	* outlierLineColor - Outline color of outlier circles
	* outlierFillColor - Fill color of the outlier circles
	* spotRadius - Radius of outlier circles
	* medianColor - Line color of the median line
	* target - Draw a target cross hair at the supplied value (default undefined)
	*
	*
	*
	* Examples:
	* $('#sparkline1').sparkline(myvalues, { lineColor: '#f00', fillColor: false });
	* $('.barsparks').sparkline('html', { type:'bar', height:'40px', barWidth:5 });
	* $('#tristate').sparkline([1,1,-1,1,0,0,-1], { type:'tristate' }):
	* $('#discrete').sparkline([1,3,4,5,5,3,4,5], { type:'discrete' });
	* $('#bullet').sparkline([10,12,12,9,7], { type:'bullet' });
	* $('#pie').sparkline([1,1,2], { type:'pie' });
	*/

	/jslint regexp: true, browser: true, jquery: true, white: true, nomen: false, plusplus: false, maxerr: 500, indent: 4 /

	(function(factory) {
	if(typeof define === 'function' && define.amd) {
	define(['jquery'], factory);
	}
	else {
	factory(jQuery);
	}
	}
	(function($) {
	'use strict';

	var UNSET_OPTION = {},
	getDefaults, createClass, SPFormat, clipval, quartile, normalizeValue, normalizeValues,
	remove, isNumber, all, sum, addCSS, ensureArray, formatNumber, RangeMap,
	MouseHandler, Tooltip, barHighlightMixin,
	line, bar, tristate, discrete, bullet, pie, box, defaultStyles, initStyles,
	VShape, VCanvas_base, VCanvas_canvas, VCanvas_vml, pending, shapeCount = 0;

	/**
	* Default configuration settings
	*/
	getDefaults = function () {
	return {
	// Settings common to most/all chart types
	common: {
	type: 'line',
	lineColor: '#00f',
	fillColor: '#cdf',
	defaultPixelsPerValue: 3,
	width: 'auto',
	height: 'auto',
	composite: false,
	tagValuesAttribute: 'values',
	tagOptionsPrefix: 'spark',
	enableTagOptions: false,
	enableHighlight: true,
	highlightLighten: 1.4,
	tooltipSkipNull: true,
	tooltipPrefix: '',
	tooltipSuffix: '',
	disableHiddenCheck: false,
	numberFormatter: false,
	tooltips: false,
	numberDigitGroupCount: 3,
	numberDigitGroupSep: ',',
	numberDecimalMark: '.',
	disableTooltips: false,
	disableInteraction: false
	},
	// Defaults for line charts
	line: {
	spotColor: '#f80',
	highlightSpotColor: '#5f5',
	highlightLineColor: '#f22',
	spotRadius: 1.5,
	minSpotColor: '#f80',
	maxSpotColor: '#f80',
	lineWidth: 1,
	normalRangeMin: undefined,
	normalRangeMax: undefined,
	normalRangeColor: '#ccc',
	drawNormalOnTop: false,
	chartRangeMin: undefined,
	chartRangeMax: undefined,
	chartRangeMinX: undefined,
	chartRangeMaxX: undefined,
	tooltipFormat: new SPFormat('<span style="color: {{color}}">●</span> {{prefix}}{{y}}{{suffix}}')
	},
	// Defaults for bar charts
	bar: {
	barColor: '#3366cc',
	negBarColor: '#f44',
	stackedBarColor: ['#3366cc', '#dc3912', '#ff9900', '#109618', '#66aa00',
	'#dd4477', '#0099c6', '#990099'],
	zeroColor: undefined,
	nullColor: undefined,
	zeroAxis: true,
	barWidth: 4,
	barSpacing: 1,
	chartRangeMax: undefined,
	chartRangeMin: undefined,
	chartRangeClip: false,
	colorMap: undefined,
	tooltipFormat: new SPFormat('<span style="color: {{color}}">●</span> {{prefix}}{{value}}{{suffix}}')
	},
	// Defaults for tristate charts
	tristate: {
	barWidth: 4,
	barSpacing: 1,
	posBarColor: '#6f6',
	negBarColor: '#f44',
	zeroBarColor: '#999',
	colorMap: {},
	tooltipFormat: new SPFormat('<span style="color: {{color}}">●</span> {{value:map}}'),
	tooltipValueLookups: { map: { '-1': 'Loss', '0': 'Draw', '1': 'Win' } }
	},
	// Defaults for discrete charts
	discrete: {
	lineHeight: 'auto',
	thresholdColor: undefined,
	thresholdValue: 0,
	chartRangeMax: undefined,
	chartRangeMin: undefined,
	chartRangeClip: false,
	tooltipFormat: new SPFormat('{{prefix}}{{value}}{{suffix}}')
	},
	// Defaults for bullet charts
	bullet: {
	targetColor: '#f33',
	targetWidth: 3, // width of the target bar in pixels
	performanceColor: '#33f',
	rangeColors: ['#d3dafe', '#a8b6ff', '#7f94ff'],
	base: undefined, // set this to a number to change the base start number
	tooltipFormat: new SPFormat('{{fieldkey:fields}} - {{value}}'),
	tooltipValueLookups: { fields: {r: 'Range', p: 'Performance', t: 'Target'} }
	},
	// Defaults for pie charts
	pie: {
	offset: 0,
	sliceColors: ['#3366cc', '#dc3912', '#ff9900', '#109618', '#66aa00',
	'#dd4477', '#0099c6', '#990099'],
	borderWidth: 0,
	borderColor: '#000',
	tooltipFormat: new SPFormat('<span style="color: {{color}}">●</span> {{value}} ({{percent.1}}%)')
	},
	// Defaults for box plots
	box: {
	raw: false,
	boxLineColor: '#000',
	boxFillColor: '#cdf',
	whiskerColor: '#000',
	outlierLineColor: '#333',
	outlierFillColor: '#fff',
	medianColor: '#f00',
	showOutliers: true,
	outlierIQR: 1.5,
	spotRadius: 1.5,
	target: undefined,
	targetColor: '#4a2',
	chartRangeMax: undefined,
	chartRangeMin: undefined,
	tooltipFormat: new SPFormat('{{field:fields}}: {{value}}'),
	tooltipFormatFieldlistKey: 'field',
	tooltipValueLookups: { fields: { lq: 'Lower Quartile', med: 'Median',
	uq: 'Upper Quartile', lo: 'Left Outlier', ro: 'Right Outlier',
	lw: 'Left Whisker', rw: 'Right Whisker'} }
	}
	};
	};

	// You can have tooltips use a css class other than jqstooltip by specifying tooltipClassname
	defaultStyles = '.jqstooltip { ' +
	'position: absolute;' +
	'left: 0px;' +
	'top: 0px;' +
	'visibility: hidden;' +
	'background: rgb(0, 0, 0) transparent;' +
	'background-color: rgba(0,0,0,0.6);' +
	'filter:progid:DXImageTransform.Microsoft.gradient(startColorstr=#99000000, endColorstr=#99000000);' +
	'-ms-filter: "progid:DXImageTransform.Microsoft.gradient(startColorstr=#99000000, endColorstr=#99000000)";' +
	'color: white;' +
	'font: 10px arial, san serif;' +
	'text-align: left;' +
	'white-space: nowrap;' +
	'padding: 5px;' +
	'border: 1px solid white;' +
	'z-index: 10000;' +
	'}' +
	'.jqsfield { ' +
	'color: white;' +
	'font: 10px arial, san serif;' +
	'text-align: left;' +
	'}';

	/**
	* Utilities
	*/

	createClass = function (/* [baseclass, [mixin, ...]], definition */) {
	var Class, args;
	Class = function () {
	this.init.apply(this, arguments);
	};
	if (arguments.length > 1) {
	if (arguments[0]) {
	Class.prototype = $.extend(new arguments[0](), arguments[arguments.length - 1]);
	Class._super = arguments[0].prototype;
	} else {
	Class.prototype = arguments[arguments.length - 1];
	}
	if (arguments.length > 2) {
	args = Array.prototype.slice.call(arguments, 1, -1);
	args.unshift(Class.prototype);
	$.extend.apply($, args);
	}
	} else {
	Class.prototype = arguments[0];
	}
	Class.prototype.cls = Class;
	return Class;
	};

	/**
	* Wraps a format string for tooltips
	* {{x}}
	* {{x.2}
	* {{x:months}}
	*/
	$.SPFormatClass = SPFormat = createClass({
	fre: /\{\{([\w.]+?)(:(.+?))?\}\}/g,
	precre: /(\w+)\.(\d+)/,

	init: function (format, fclass) {
	this.format = format;
	this.fclass = fclass;
	},

	render: function (fieldset, lookups, options) {
	var self = this,
	fields = fieldset,
	match, token, lookupkey, fieldvalue, prec;
	return this.format.replace(this.fre, function () {
	var lookup;
	token = arguments[1];
	lookupkey = arguments[3];
	match = self.precre.exec(token);
	if (match) {
	prec = match[2];
	token = match[1];
	} else {
	prec = false;
	}
	fieldvalue = fields[token];
	if (fieldvalue === undefined) {
	return '';
	}
	if (lookupkey && lookups && lookups[lookupkey]) {
	lookup = lookups[lookupkey];
	if (lookup.get) { // RangeMap
	return lookups[lookupkey].get(fieldvalue) \|\| fieldvalue;
	} else {
	return lookups[lookupkey][fieldvalue] \|\| fieldvalue;
	}
	}
	if (isNumber(fieldvalue)) {
	if (options.get('tooltips')) {
	var tooltipArray = options.get('tooltips').split(',');
	fieldvalue = tooltipArray[ fields['x'] ];
	}
	else if (options.get('numberFormatter')) {
	fieldvalue = options.get('numberFormatter')(fieldvalue);
	} else {
	fieldvalue = formatNumber(fieldvalue, prec,
	options.get('numberDigitGroupCount'),
	options.get('numberDigitGroupSep'),
	options.get('numberDecimalMark'));
	}
	}
	return fieldvalue;
	});
	}
	});

	// convience method to avoid needing the new operator
	$.spformat = function(format, fclass) {
	return new SPFormat(format, fclass);
	};

	clipval = function (val, min, max) {
	if (val < min) {
	return min;
	}
	if (val > max) {
	return max;
	}
	return val;
	};

	quartile = function (values, q) {
	var vl;
	if (q === 2) {
	vl = Math.floor(values.length / 2);
	return values.length % 2 ? values[vl] : (values[vl-1] + values[vl]) / 2;
	} else {
	if (values.length % 2 ) { // odd
	vl = (values.length * q + q) / 4;
	return vl % 1 ? (values[Math.floor(vl)] + values[Math.floor(vl) - 1]) / 2 : values[vl-1];
	} else { //even
	vl = (values.length * q + 2) / 4;
	return vl % 1 ? (values[Math.floor(vl)] + values[Math.floor(vl) - 1]) / 2 : values[vl-1];

	}
	}
	};

	normalizeValue = function (val) {
	var nf;
	switch (val) {
	case 'undefined':
	val = undefined;
	break;
	case 'null':
	val = null;
	break;
	case 'true':
	val = true;
	break;
	case 'false':
	val = false;
	break;
	default:
	nf = parseFloat(val);
	if (val == nf) {
	val = nf;
	}
	}
	return val;
	};

	normalizeValues = function (vals) {
	var i, result = [];
	for (i = vals.length; i--;) {
	result[i] = normalizeValue(vals[i]);
	}
	return result;
	};

	remove = function (vals, filter) {
	var i, vl, result = [];
	for (i = 0, vl = vals.length; i < vl; i++) {
	if (vals[i] !== filter) {
	result.push(vals[i]);
	}
	}
	return result;
	};

	isNumber = function (num) {
	return !isNaN(parseFloat(num)) && isFinite(num);
	};

	formatNumber = function (num, prec, groupsize, groupsep, decsep) {
	var p, i;
	num = (prec === false ? parseFloat(num).toString() : num.toFixed(prec)).split('');
	p = (p = $.inArray('.', num)) < 0 ? num.length : p;
	if (p < num.length) {
	num[p] = decsep;
	}
	for (i = p - groupsize; i > 0; i -= groupsize) {
	num.splice(i, 0, groupsep);
	}
	return num.join('');
	};

	// determine if all values of an array match a value
	// returns true if the array is empty
	all = function (val, arr, ignoreNull) {
	var i;
	for (i = arr.length; i--; ) {
	if (ignoreNull && arr[i] === null) continue;
	if (arr[i] !== val) {
	return false;
	}
	}
	return true;
	};

	// sums the numeric values in an array, ignoring other values
	sum = function (vals) {
	var total = 0, i;
	for (i = vals.length; i--;) {
	total += typeof vals[i] === 'number' ? vals[i] : 0;
	}
	return total;
	};

	ensureArray = function (val) {
	return $.isArray(val) ? val : [val];
	};

	// http://paulirish.com/2008/bookmarklet-inject-new-css-rules/
	addCSS = function(css) {
	var tag;
	//if ('\v' == 'v') /* ie only */ {
	if (document.createStyleSheet) {
	document.createStyleSheet().cssText = css;
	} else {
	tag = document.createElement('style');
	tag.type = 'text/css';
	document.getElementsByTagName('head')[0].appendChild(tag);
	tag[(typeof document.body.style.WebkitAppearance == 'string') /* webkit only */ ? 'innerText' : 'innerHTML'] = css;
	}
	};

	// Provide a cross-browser interface to a few simple drawing primitives
	$.fn.simpledraw = function (width, height, useExisting, interact) {
	var target, mhandler;
	if (useExisting && (target = this.data('_jqs_vcanvas'))) {
	return target;
	}
	if (width === undefined) {
	width = $(this).innerWidth();
	}
	if (height === undefined) {
	height = $(this).innerHeight();
	}
	if ($.browser.hasCanvas) {
	target = new VCanvas_canvas(width, height, this, interact);
	} else if ($.browser.msie) {
	target = new VCanvas_vml(width, height, this);
	} else {
	return false;
	}
	mhandler = $(this).data('_jqs_mhandler');
	if (mhandler) {
	mhandler.registerCanvas(target);
	}
	return target;
	};

	$.fn.cleardraw = function () {
	var target = this.data('_jqs_vcanvas');
	if (target) {
	target.reset();
	}
	};

	$.RangeMapClass = RangeMap = createClass({
	init: function (map) {
	var key, range, rangelist = [];
	for (key in map) {
	if (map.hasOwnProperty(key) && typeof key === 'string' && key.indexOf(':') > -1) {
	range = key.split(':');
	range[0] = range[0].length === 0 ? -Infinity : parseFloat(range[0]);
	range[1] = range[1].length === 0 ? Infinity : parseFloat(range[1]);
	range[2] = map[key];
	rangelist.push(range);
	}
	}
	this.map = map;
	this.rangelist = rangelist \|\| false;
	},

	get: function (value) {
	var rangelist = this.rangelist,
	i, range, result;
	if ((result = this.map[value]) !== undefined) {
	return result;
	}
	if (rangelist) {
	for (i = rangelist.length; i--;) {
	range = rangelist[i];
	if (range[0] <= value && range[1] >= value) {
	return range[2];
	}
	}
	}
	return undefined;
	}
	});

	// Convenience function
	$.range_map = function(map) {
	return new RangeMap(map);
	};

	MouseHandler = createClass({
	init: function (el, options) {
	var $el = $(el);
	this.$el = $el;
	this.options = options;
	this.currentPageX = 0;
	this.currentPageY = 0;
	this.el = el;
	this.splist = [];
	this.tooltip = null;
	this.over = false;
	this.displayTooltips = !options.get('disableTooltips');
	this.highlightEnabled = !options.get('disableHighlight');
	},

	registerSparkline: function (sp) {
	this.splist.push(sp);
	if (this.over) {
	this.updateDisplay();
	}
	},

	registerCanvas: function (canvas) {
	var $canvas = $(canvas.canvas);
	this.canvas = canvas;
	this.$canvas = $canvas;
	$canvas.mouseenter($.proxy(this.mouseenter, this));
	$canvas.mouseleave($.proxy(this.mouseleave, this));
	$canvas.click($.proxy(this.mouseclick, this));
	},

	reset: function (removeTooltip) {
	this.splist = [];
	if (this.tooltip && removeTooltip) {
	this.tooltip.remove();
	this.tooltip = undefined;
	}
	},

	mouseclick: function (e) {
	var clickEvent = $.Event('sparklineClick');
	clickEvent.originalEvent = e;
	clickEvent.sparklines = this.splist;
	this.$el.trigger(clickEvent);
	},

	mouseenter: function (e) {
	$(document.body).unbind('mousemove.jqs');
	$(document.body).bind('mousemove.jqs', $.proxy(this.mousemove, this));
	this.over = true;
	this.currentPageX = e.pageX;
	this.currentPageY = e.pageY;
	this.currentEl = e.target;
	if (!this.tooltip && this.displayTooltips) {
	this.tooltip = new Tooltip(this.options);
	this.tooltip.updatePosition(e.pageX, e.pageY);
	}
	this.updateDisplay();
	},

	mouseleave: function () {
	$(document.body).unbind('mousemove.jqs');
	var splist = this.splist,
	spcount = splist.length,
	needsRefresh = false,
	sp, i;
	this.over = false;
	this.currentEl = null;

	if (this.tooltip) {
	this.tooltip.remove();
	this.tooltip = null;
	}

	for (i = 0; i < spcount; i++) {
	sp = splist[i];
	if (sp.clearRegionHighlight()) {
	needsRefresh = true;
	}
	}

	if (needsRefresh) {
	this.canvas.render();
	}
	},

	mousemove: function (e) {
	this.currentPageX = e.pageX;
	this.currentPageY = e.pageY;
	this.currentEl = e.target;
	if (this.tooltip) {
	this.tooltip.updatePosition(e.pageX, e.pageY);
	}
	this.updateDisplay();
	},

	updateDisplay: function () {
	var splist = this.splist,
	spcount = splist.length,
	needsRefresh = false,
	offset = this.$canvas.offset(),
	localX = this.currentPageX - offset.left,
	localY = this.currentPageY - offset.top,
	tooltiphtml, sp, i, result, changeEvent;
	if (!this.over) {
	return;
	}
	for (i = 0; i < spcount; i++) {
	sp = splist[i];
	result = sp.setRegionHighlight(this.currentEl, localX, localY);
	if (result) {
	needsRefresh = true;
	}
	}
	if (needsRefresh) {
	changeEvent = $.Event('sparklineRegionChange');
	changeEvent.sparklines = this.splist;
	this.$el.trigger(changeEvent);
	if (this.tooltip) {
	tooltiphtml = '';
	for (i = 0; i < spcount; i++) {
	sp = splist[i];
	tooltiphtml += sp.getCurrentRegionTooltip();
	}
	this.tooltip.setContent(tooltiphtml);
	}
	if (!this.disableHighlight) {
	this.canvas.render();
	}
	}
	if (result === null) {
	this.mouseleave();
	}
	}
	});


	Tooltip = createClass({
	sizeStyle: 'position: static !important;' +
	'display: block !important;' +
	'visibility: hidden !important;' +
	'float: left !important;',

	init: function (options) {
	var tooltipClassname = options.get('tooltipClassname', 'jqstooltip'),
	sizetipStyle = this.sizeStyle,
	offset;
	this.container = options.get('tooltipContainer') \|\| document.body;
	this.tooltipOffsetX = options.get('tooltipOffsetX', 10);
	this.tooltipOffsetY = options.get('tooltipOffsetY', 12);
	// remove any previous lingering tooltip
	$('#jqssizetip').remove();
	$('#jqstooltip').remove();
	this.sizetip = $('<div/>', {
	id: 'jqssizetip',
	style: sizetipStyle,
	'class': tooltipClassname
	});
	this.tooltip = $('<div/>', {
	id: 'jqstooltip',
	'class': tooltipClassname
	}).appendTo(this.container);
	// account for the container's location
	offset = this.tooltip.offset();
	this.offsetLeft = offset.left;
	this.offsetTop = offset.top;
	this.hidden = true;
	$(window).unbind('resize.jqs scroll.jqs');
	$(window).bind('resize.jqs scroll.jqs', $.proxy(this.updateWindowDims, this));
	this.updateWindowDims();
	},

	updateWindowDims: function () {
	this.scrollTop = $(window).scrollTop();
	this.scrollLeft = $(window).scrollLeft();
	this.scrollRight = this.scrollLeft + $(window).width();
	this.updatePosition();
	},

	getSize: function (content) {
	this.sizetip.html(content).appendTo(this.container);
	this.width = this.sizetip.width() + 1;
	this.height = this.sizetip.height();
	this.sizetip.remove();
	},

	setContent: function (content) {
	if (!content) {
	this.tooltip.css('visibility', 'hidden');
	this.hidden = true;
	return;
	}
	this.getSize(content);
	this.tooltip.html(content)
	.css({
	'width': this.width,
	'height': this.height,
	'visibility': 'visible'
	});
	if (this.hidden) {
	this.hidden = false;
	this.updatePosition();
	}
	},

	updatePosition: function (x, y) {
	if (x === undefined) {
	if (this.mousex === undefined) {
	return;
	}
	x = this.mousex - this.offsetLeft;
	y = this.mousey - this.offsetTop;

	} else {
	this.mousex = x = x - this.offsetLeft;
	this.mousey = y = y - this.offsetTop;
	}
	if (!this.height \|\| !this.width \|\| this.hidden) {
	return;
	}

	y -= this.height + this.tooltipOffsetY;
	x += this.tooltipOffsetX;

	if (y < this.scrollTop) {
	y = this.scrollTop;
	}
	if (x < this.scrollLeft) {
	x = this.scrollLeft;
	} else if (x + this.width > this.scrollRight) {
	x = this.scrollRight - this.width;
	}

	this.tooltip.css({
	'left': x,
	'top': y
	});
	},

	remove: function () {
	this.tooltip.remove();
	this.sizetip.remove();
	this.sizetip = this.tooltip = undefined;
	$(window).unbind('resize.jqs scroll.jqs');
	}
	});

	initStyles = function() {
	addCSS(defaultStyles);
	};

	$(initStyles);

	pending = [];
	$.fn.sparkline = function (userValues, userOptions) {
	return this.each(function () {
	var options = new $.fn.sparkline.options(this, userOptions),
	$this = $(this),
	render, i;
	render = function () {
	var values, width, height, tmp, mhandler, sp, vals;
	if (userValues === 'html' \|\| userValues === undefined) {
	vals = this.getAttribute(options.get('tagValuesAttribute'));
	if (vals === undefined \|\| vals === null) {
	vals = $this.html();
	}
	values = vals.replace(/(^\s<!--)\|(-->\s$)\|\s+/g, '').split(',');
	} else {
	values = userValues;
	}

	width = options.get('width') === 'auto' ? values.length * options.get('defaultPixelsPerValue') : options.get('width');
	if (options.get('height') === 'auto') {
	if (!options.get('composite') \|\| !$.data(this, '_jqs_vcanvas')) {
	// must be a better way to get the line height
	tmp = document.createElement('span');
	tmp.innerHTML = 'a';
	$this.html(tmp);
	height = $(tmp).innerHeight() \|\| $(tmp).height();
	$(tmp).remove();
	tmp = null;
	}
	} else {
	height = options.get('height');
	}

	if (!options.get('disableInteraction')) {
	mhandler = $.data(this, '_jqs_mhandler');
	if (!mhandler) {
	mhandler = new MouseHandler(this, options);
	$.data(this, '_jqs_mhandler', mhandler);
	} else if (!options.get('composite')) {
	mhandler.reset();
	}
	} else {
	mhandler = false;
	}

	if (options.get('composite') && !$.data(this, '_jqs_vcanvas')) {
	if (!$.data(this, '_jqs_errnotify')) {
	alert('Attempted to attach a composite sparkline to an element with no existing sparkline');
	$.data(this, '_jqs_errnotify', true);
	}
	return;
	}

	sp = new $.fn.sparkline[options.get('type')](this, values, options, width, height);

	sp.render();

	if (mhandler) {
	mhandler.registerSparkline(sp);
	}
	};
	// jQuery 1.3.0 completely changed the meaning of :hidden :-/
	if (($(this).html() && !options.get('disableHiddenCheck') && $(this).is(':hidden')) \|\| ($.fn.jquery < '1.3.0' && $(this).parents().is(':hidden')) \|\| !$(this).parents('body').length) {
	if (!options.get('composite') && $.data(this, '_jqs_pending')) {
	// remove any existing references to the element
	for (i = pending.length; i; i--) {
	if (pending[i - 1][0] == this) {
	pending.splice(i - 1, 1);
	}
	}
	}
	pending.push([this, render]);
	$.data(this, '_jqs_pending', true);
	} else {
	render.call(this);
	}
	});
	};

	$.fn.sparkline.defaults = getDefaults();


	$.sparkline_display_visible = function () {
	var el, i, pl;
	var done = [];
	for (i = 0, pl = pending.length; i < pl; i++) {
	el = pending[i][0];
	if ($(el).is(':visible') && !$(el).parents().is(':hidden')) {
	pending[i][1].call(el);
	$.data(pending[i][0], '_jqs_pending', false);
	done.push(i);
	} else if (!$(el).closest('html').length && !$.data(el, '_jqs_pending')) {
	// element has been inserted and removed from the DOM
	// If it was not yet inserted into the dom then the .data request
	// will return true.
	// removing from the dom causes the data to be removed.
	$.data(pending[i][0], '_jqs_pending', false);
	done.push(i);
	}
	}
	for (i = done.length; i; i--) {
	pending.splice(done[i - 1], 1);
	}
	};


	/**
	* User option handler
	*/
	$.fn.sparkline.options = createClass({
	init: function (tag, userOptions) {
	var extendedOptions, defaults, base, tagOptionType;
	this.userOptions = userOptions = userOptions \|\| {};
	this.tag = tag;
	this.tagValCache = {};
	defaults = $.fn.sparkline.defaults;
	base = defaults.common;
	this.tagOptionsPrefix = userOptions.enableTagOptions && (userOptions.tagOptionsPrefix \|\| base.tagOptionsPrefix);

	tagOptionType = this.getTagSetting('type');
	if (tagOptionType === UNSET_OPTION) {
	extendedOptions = defaults[userOptions.type \|\| base.type];
	} else {
	extendedOptions = defaults[tagOptionType];
	}
	this.mergedOptions = $.extend({}, base, extendedOptions, userOptions);
	},


	getTagSetting: function (key) {
	var prefix = this.tagOptionsPrefix,
	val, i, pairs, keyval;
	if (prefix === false \|\| prefix === undefined) {
	return UNSET_OPTION;
	}
	if (this.tagValCache.hasOwnProperty(key)) {
	val = this.tagValCache.key;
	} else {
	val = this.tag.getAttribute(prefix + key);
	if (val === undefined \|\| val === null) {
	val = UNSET_OPTION;
	} else if (val.substr(0, 1) === '[') {
	val = val.substr(1, val.length - 2).split(',');
	for (i = val.length; i--;) {
	val[i] = normalizeValue(val[i].replace(/(^\s)\|(\s$)/g, ''));
	}
	} else if (val.substr(0, 1) === '{') {
	pairs = val.substr(1, val.length - 2).split(',');
	val = {};
	for (i = pairs.length; i--;) {
	keyval = pairs[i].split(':', 2);
	val[keyval[0].replace(/(^\s)\|(\s$)/g, '')] = normalizeValue(keyval[1].replace(/(^\s)\|(\s$)/g, ''));
	}
	} else {
	val = normalizeValue(val);
	}
	this.tagValCache.key = val;
	}
	return val;
	},

	get: function (key, defaultval) {
	var tagOption = this.getTagSetting(key),
	result;
	if (tagOption !== UNSET_OPTION) {
	return tagOption;
	}
	return (result = this.mergedOptions[key]) === undefined ? defaultval : result;
	}
	});


	$.fn.sparkline._base = createClass({
	disabled: false,

	init: function (el, values, options, width, height) {
	this.el = el;
	this.$el = $(el);
	this.values = values;
	this.options = options;
	this.width = width;
	this.height = height;
	this.currentRegion = undefined;
	},

	/**
	* Setup the canvas
	*/
	initTarget: function () {
	var interactive = !this.options.get('disableInteraction');
	if (!(this.target = this.$el.simpledraw(this.width, this.height, this.options.get('composite'), interactive))) {
	this.disabled = true;
	} else {
	this.canvasWidth = this.target.pixelWidth;
	this.canvasHeight = this.target.pixelHeight;
	}
	},

	/**
	* Actually render the chart to the canvas
	*/
	render: function () {
	if (this.disabled) {
	this.el.innerHTML = '';
	return false;
	}
	return true;
	},

	/**
	* Return a region id for a given x/y co-ordinate
	*/
	getRegion: function (x, y) {
	},

	/**
	* Highlight an item based on the moused-over x,y co-ordinate
	*/
	setRegionHighlight: function (el, x, y) {
	var currentRegion = this.currentRegion,
	highlightEnabled = !this.options.get('disableHighlight'),
	newRegion;
	if (x > this.canvasWidth \|\| y > this.canvasHeight \|\| x < 0 \|\| y < 0) {
	return null;
	}
	newRegion = this.getRegion(el, x, y);
	if (currentRegion !== newRegion) {
	if (currentRegion !== undefined && highlightEnabled) {
	this.removeHighlight();
	}
	this.currentRegion = newRegion;
	if (newRegion !== undefined && highlightEnabled) {
	this.renderHighlight();
	}
	return true;
	}
	return false;
	},

	/**
	* Reset any currently highlighted item
	*/
	clearRegionHighlight: function () {
	if (this.currentRegion !== undefined) {
	this.removeHighlight();
	this.currentRegion = undefined;
	return true;
	}
	return false;
	},

	renderHighlight: function () {
	this.changeHighlight(true);
	},

	removeHighlight: function () {
	this.changeHighlight(false);
	},

	changeHighlight: function (highlight) {},

	/**
	* Fetch the HTML to display as a tooltip
	*/
	getCurrentRegionTooltip: function () {
	var options = this.options,
	header = '',
	entries = [],
	fields, formats, formatlen, fclass, text, i,
	showFields, showFieldsKey, newFields, fv,
	formatter, format, fieldlen, j;
	if (this.currentRegion === undefined) {
	return '';
	}
	fields = this.getCurrentRegionFields();
	formatter = options.get('tooltipFormatter');
	if (formatter) {
	return formatter(this, options, fields);
	}
	if (options.get('tooltipChartTitle')) {
	header += '<div class="jqs jqstitle">' + options.get('tooltipChartTitle') + '</div>\n';
	}
	formats = this.options.get('tooltipFormat');
	if (!formats) {
	return '';
	}
	if (!$.isArray(formats)) {
	formats = [formats];
	}
	if (!$.isArray(fields)) {
	fields = [fields];
	}
	showFields = this.options.get('tooltipFormatFieldlist');
	showFieldsKey = this.options.get('tooltipFormatFieldlistKey');
	if (showFields && showFieldsKey) {
	// user-selected ordering of fields
	newFields = [];
	for (i = fields.length; i--;) {
	fv = fields[i][showFieldsKey];
	if ((j = $.inArray(fv, showFields)) != -1) {
	newFields[j] = fields[i];
	}
	}
	fields = newFields;
	}
	formatlen = formats.length;
	fieldlen = fields.length;
	for (i = 0; i < formatlen; i++) {
	format = formats[i];
	if (typeof format === 'string') {
	format = new SPFormat(format);
	}
	fclass = format.fclass \|\| 'jqsfield';
	for (j = 0; j < fieldlen; j++) {
	if (!fields[j].isNull \|\| !options.get('tooltipSkipNull')) {
	$.extend(fields[j], {
	prefix: options.get('tooltipPrefix'),
	suffix: options.get('tooltipSuffix')
	});
	text = format.render(fields[j], options.get('tooltipValueLookups'), options);
	entries.push('<div class="' + fclass + '">' + text + '</div>');
	}
	}
	}
	if (entries.length) {
	return header + entries.join('\n');
	}
	return '';
	},

	getCurrentRegionFields: function () {},

	calcHighlightColor: function (color, options) {
	var highlightColor = options.get('highlightColor'),
	lighten = options.get('highlightLighten'),
	parse, mult, rgbnew, i;
	if (highlightColor) {
	return highlightColor;
	}
	if (lighten) {
	// extract RGB values
	parse = /^#([0-9a-f])([0-9a-f])([0-9a-f])$/i.exec(color) \|\| /^#([0-9a-f]{2})([0-9a-f]{2})([0-9a-f]{2})$/i.exec(color);
	if (parse) {
	rgbnew = [];
	mult = color.length === 4 ? 16 : 1;
	for (i = 0; i < 3; i++) {
	rgbnew[i] = clipval(Math.round(parseInt(parse[i + 1], 16) * mult * lighten), 0, 255);
	}
	return 'rgb(' + rgbnew.join(',') + ')';
	}

	}
	return color;
	}

	});

	barHighlightMixin = {
	changeHighlight: function (highlight) {
	var currentRegion = this.currentRegion,
	target = this.target,
	shapeids = this.regionShapes[currentRegion],
	newShapes;
	// will be null if the region value was null
	if (shapeids) {
	newShapes = this.renderRegion(currentRegion, highlight);
	if ($.isArray(newShapes) \|\| $.isArray(shapeids)) {
	target.replaceWithShapes(shapeids, newShapes);
	this.regionShapes[currentRegion] = $.map(newShapes, function (newShape) {
	return newShape.id;
	});
	} else {
	target.replaceWithShape(shapeids, newShapes);
	this.regionShapes[currentRegion] = newShapes.id;
	}
	}
	},

	render: function () {
	var values = this.values,
	target = this.target,
	regionShapes = this.regionShapes,
	shapes, ids, i, j;

	if (!this.cls._super.render.call(this)) {
	return;
	}
	for (i = values.length; i--;) {
	shapes = this.renderRegion(i);
	if (shapes) {
	if ($.isArray(shapes)) {
	ids = [];
	for (j = shapes.length; j--;) {
	shapes[j].append();
	ids.push(shapes[j].id);
	}
	regionShapes[i] = ids;
	} else {
	shapes.append();
	regionShapes[i] = shapes.id; // store just the shapeid
	}
	} else {
	// null value
	regionShapes[i] = null;
	}
	}
	target.render();
	}
	};

	/**
	* Line charts
	*/
	$.fn.sparkline.line = line = createClass($.fn.sparkline._base, {
	type: 'line',

	init: function (el, values, options, width, height) {
	line._super.init.call(this, el, values, options, width, height);
	this.vertices = [];
	this.regionMap = [];
	this.xvalues = [];
	this.yvalues = [];
	this.yminmax = [];
	this.hightlightSpotId = null;
	this.lastShapeId = null;
	this.initTarget();
	},

	getRegion: function (el, x, y) {
	var i,
	regionMap = this.regionMap; // maps regions to value positions
	for (i = regionMap.length; i--;) {
	if (regionMap[i] !== null && x >= regionMap[i][0] && x <= regionMap[i][1]) {
	return regionMap[i][2];
	}
	}
	return undefined;
	},

	getCurrentRegionFields: function () {
	var currentRegion = this.currentRegion;
	return {
	isNull: this.yvalues[currentRegion] === null,
	x: this.xvalues[currentRegion],
	y: this.yvalues[currentRegion],
	color: this.options.get('lineColor'),
	fillColor: this.options.get('fillColor'),
	offset: currentRegion
	};
	},

	renderHighlight: function () {
	var currentRegion = this.currentRegion,
	target = this.target,
	vertex = this.vertices[currentRegion],
	options = this.options,
	spotRadius = options.get('spotRadius'),
	highlightSpotColor = options.get('highlightSpotColor'),
	highlightLineColor = options.get('highlightLineColor'),
	highlightSpot, highlightLine;

	if (!vertex) {
	return;
	}
	if (spotRadius && highlightSpotColor) {
	highlightSpot = target.drawCircle(vertex[0], vertex[1],
	spotRadius, undefined, highlightSpotColor);
	this.highlightSpotId = highlightSpot.id;
	target.insertAfterShape(this.lastShapeId, highlightSpot);
	}
	if (highlightLineColor) {
	highlightLine = target.drawLine(vertex[0], this.canvasTop, vertex[0],
	this.canvasTop + this.canvasHeight, highlightLineColor);
	this.highlightLineId = highlightLine.id;
	target.insertAfterShape(this.lastShapeId, highlightLine);
	}
	},

	removeHighlight: function () {
	var target = this.target;
	if (this.highlightSpotId) {
	target.removeShapeId(this.highlightSpotId);
	this.highlightSpotId = null;
	}
	if (this.highlightLineId) {
	target.removeShapeId(this.highlightLineId);
	this.highlightLineId = null;
	}
	},

	scanValues: function () {
	var values = this.values,
	valcount = values.length,
	xvalues = this.xvalues,
	yvalues = this.yvalues,
	yminmax = this.yminmax,
	i, val, isStr, isArray, sp;
	for (i = 0; i < valcount; i++) {
	val = values[i];
	isStr = typeof(values[i]) === 'string';
	isArray = typeof(values[i]) === 'object' && values[i] instanceof Array;
	sp = isStr && values[i].split(':');
	if (isStr && sp.length === 2) { // x:y
	xvalues.push(Number(sp[0]));
	yvalues.push(Number(sp[1]));
	yminmax.push(Number(sp[1]));
	} else if (isArray) {
	xvalues.push(val[0]);
	yvalues.push(val[1]);
	yminmax.push(val[1]);
	} else {
	xvalues.push(i);
	if (values[i] === null \|\| values[i] === 'null') {
	yvalues.push(null);
	} else {
	yvalues.push(Number(val));
	yminmax.push(Number(val));
	}
	}
	}
	if (this.options.get('xvalues')) {
	xvalues = this.options.get('xvalues');
	}

	this.maxy = this.maxyorg = Math.max.apply(Math, yminmax);
	this.miny = this.minyorg = Math.min.apply(Math, yminmax);

	this.maxx = Math.max.apply(Math, xvalues);
	this.minx = Math.min.apply(Math, xvalues);

	this.xvalues = xvalues;
	this.yvalues = yvalues;
	this.yminmax = yminmax;

	},

	processRangeOptions: function () {
	var options = this.options,
	normalRangeMin = options.get('normalRangeMin'),
	normalRangeMax = options.get('normalRangeMax');

	if (normalRangeMin !== undefined) {
	if (normalRangeMin < this.miny) {
	this.miny = normalRangeMin;
	}
	if (normalRangeMax > this.maxy) {
	this.maxy = normalRangeMax;
	}
	}
	if (options.get('chartRangeMin') !== undefined && (options.get('chartRangeClip') \|\| options.get('chartRangeMin') < this.miny)) {
	this.miny = options.get('chartRangeMin');
	}
	if (options.get('chartRangeMax') !== undefined && (options.get('chartRangeClip') \|\| options.get('chartRangeMax') > this.maxy)) {
	this.maxy = options.get('chartRangeMax');
	}
	if (options.get('chartRangeMinX') !== undefined && (options.get('chartRangeClipX') \|\| options.get('chartRangeMinX') < this.minx)) {
	this.minx = options.get('chartRangeMinX');
	}
	if (options.get('chartRangeMaxX') !== undefined && (options.get('chartRangeClipX') \|\| options.get('chartRangeMaxX') > this.maxx)) {
	this.maxx = options.get('chartRangeMaxX');
	}

	},

	drawNormalRange: function (canvasLeft, canvasTop, canvasHeight, canvasWidth, rangey) {
	var normalRangeMin = this.options.get('normalRangeMin'),
	normalRangeMax = this.options.get('normalRangeMax'),
	ytop = canvasTop + Math.round(canvasHeight - (canvasHeight * ((normalRangeMax - this.miny) / rangey))),
	height = Math.round((canvasHeight * (normalRangeMax - normalRangeMin)) / rangey);
	this.target.drawRect(canvasLeft, ytop, canvasWidth, height, undefined, this.options.get('normalRangeColor')).append();
	},

	render: function () {
	var options = this.options,
	target = this.target,
	canvasWidth = this.canvasWidth,
	canvasHeight = this.canvasHeight,
	vertices = this.vertices,
	spotRadius = options.get('spotRadius'),
	regionMap = this.regionMap,
	rangex, rangey, yvallast,
	canvasTop, canvasLeft,
	vertex, path, paths, x, y, xnext, xpos, xposnext,
	last, next, yvalcount, lineShapes, fillShapes, plen,
	valueSpots, hlSpotsEnabled, color, xvalues, yvalues, i;

	if (!line._super.render.call(this)) {
	return;
	}

	this.scanValues();
	this.processRangeOptions();

	xvalues = this.xvalues;
	yvalues = this.yvalues;

	if (!this.yminmax.length \|\| this.yvalues.length < 2) {
	// empty or all null valuess
	return;
	}

	canvasTop = canvasLeft = 0;

	rangex = this.maxx - this.minx === 0 ? 1 : this.maxx - this.minx;
	rangey = this.maxy - this.miny === 0 ? 1 : this.maxy - this.miny;
	yvallast = this.yvalues.length - 1;

	if (spotRadius && (canvasWidth < (spotRadius * 4) \|\| canvasHeight < (spotRadius * 4))) {
	spotRadius = 0;
	}
	if (spotRadius) {
	// adjust the canvas size as required so that spots will fit
	hlSpotsEnabled = options.get('highlightSpotColor') && !options.get('disableInteraction');
	if (hlSpotsEnabled \|\| options.get('minSpotColor') \|\| (options.get('spotColor') && yvalues[yvallast] === this.miny)) {
	canvasHeight -= Math.ceil(spotRadius);
	}
	if (hlSpotsEnabled \|\| options.get('maxSpotColor') \|\| (options.get('spotColor') && yvalues[yvallast] === this.maxy)) {
	canvasHeight -= Math.ceil(spotRadius);
	canvasTop += Math.ceil(spotRadius);
	}
	if (hlSpotsEnabled \|\|
	((options.get('minSpotColor') \|\| options.get('maxSpotColor')) && (yvalues[0] === this.miny \|\| yvalues[0] === this.maxy))) {
	canvasLeft += Math.ceil(spotRadius);
	canvasWidth -= Math.ceil(spotRadius);
	}
	if (hlSpotsEnabled \|\| options.get('spotColor') \|\|
	(options.get('minSpotColor') \|\| options.get('maxSpotColor') &&
	(yvalues[yvallast] === this.miny \|\| yvalues[yvallast] === this.maxy))) {
	canvasWidth -= Math.ceil(spotRadius);
	}
	}


	canvasHeight--;

	if (options.get('normalRangeMin') !== undefined && !options.get('drawNormalOnTop')) {
	this.drawNormalRange(canvasLeft, canvasTop, canvasHeight, canvasWidth, rangey);
	}

	path = [];
	paths = [path];
	last = next = null;
	yvalcount = yvalues.length;
	for (i = 0; i < yvalcount; i++) {
	x = xvalues[i];
	xnext = xvalues[i + 1];
	y = yvalues[i];
	xpos = canvasLeft + Math.round((x - this.minx) * (canvasWidth / rangex));
	xposnext = i < yvalcount - 1 ? canvasLeft + Math.round((xnext - this.minx) * (canvasWidth / rangex)) : canvasWidth;
	next = xpos + ((xposnext - xpos) / 2);
	regionMap[i] = [last \|\| 0, next, i];
	last = next;
	if (y === null) {
	if (i) {
	if (yvalues[i - 1] !== null) {
	path = [];
	paths.push(path);
	}
	vertices.push(null);
	}
	} else {
	if (y < this.miny) {
	y = this.miny;
	}
	if (y > this.maxy) {
	y = this.maxy;
	}
	if (!path.length) {
	// previous value was null
	path.push([xpos, canvasTop + canvasHeight]);
	}
	vertex = [xpos, canvasTop + Math.round(canvasHeight - (canvasHeight * ((y - this.miny) / rangey)))];
	path.push(vertex);
	vertices.push(vertex);
	}
	}

	lineShapes = [];
	fillShapes = [];
	plen = paths.length;
	for (i = 0; i < plen; i++) {
	path = paths[i];
	if (path.length) {
	if (options.get('fillColor')) {
	path.push([path[path.length - 1][0], (canvasTop + canvasHeight)]);
	fillShapes.push(path.slice(0));
	path.pop();
	}
	// if there's only a single point in this path, then we want to display it
	// as a vertical line which means we keep path[0] as is
	if (path.length > 2) {
	// else we want the first value
	path[0] = [path[0][0], path[1][1]];
	}
	lineShapes.push(path);
	}
	}

	// draw the fill first, then optionally the normal range, then the line on top of that
	plen = fillShapes.length;
	for (i = 0; i < plen; i++) {
	target.drawShape(fillShapes[i],
	options.get('fillColor'), options.get('fillColor')).append();
	}

	if (options.get('normalRangeMin') !== undefined && options.get('drawNormalOnTop')) {
	this.drawNormalRange(canvasLeft, canvasTop, canvasHeight, canvasWidth, rangey);
	}

	plen = lineShapes.length;
	for (i = 0; i < plen; i++) {
	target.drawShape(lineShapes[i], options.get('lineColor'), undefined,
	options.get('lineWidth')).append();
	}

	if (spotRadius && options.get('valueSpots')) {
	valueSpots = options.get('valueSpots');
	if (valueSpots.get === undefined) {
	valueSpots = new RangeMap(valueSpots);
	}
	for (i = 0; i < yvalcount; i++) {
	color = valueSpots.get(yvalues[i]);
	if (color) {
	target.drawCircle(canvasLeft + Math.round((xvalues[i] - this.minx) * (canvasWidth / rangex)),
	canvasTop + Math.round(canvasHeight - (canvasHeight * ((yvalues[i] - this.miny) / rangey))),
	spotRadius, undefined,
	color).append();
	}
	}

	}
	if (spotRadius && options.get('spotColor')) {
	target.drawCircle(canvasLeft + Math.round((xvalues[xvalues.length - 1] - this.minx) * (canvasWidth / rangex)),
	canvasTop + Math.round(canvasHeight - (canvasHeight * ((yvalues[yvallast] - this.miny) / rangey))),
	spotRadius, undefined,
	options.get('spotColor')).append();
	}
	if (this.maxy !== this.minyorg) {
	if (spotRadius && options.get('minSpotColor')) {
	x = xvalues[$.inArray(this.minyorg, yvalues)];
	target.drawCircle(canvasLeft + Math.round((x - this.minx) * (canvasWidth / rangex)),
	canvasTop + Math.round(canvasHeight - (canvasHeight * ((this.minyorg - this.miny) / rangey))),
	spotRadius, undefined,
	options.get('minSpotColor')).append();
	}
	if (spotRadius && options.get('maxSpotColor')) {
	x = xvalues[$.inArray(this.maxyorg, yvalues)];
	target.drawCircle(canvasLeft + Math.round((x - this.minx) * (canvasWidth / rangex)),
	canvasTop + Math.round(canvasHeight - (canvasHeight * ((this.maxyorg - this.miny) / rangey))),
	spotRadius, undefined,
	options.get('maxSpotColor')).append();
	}
	}

	this.lastShapeId = target.getLastShapeId();
	this.canvasTop = canvasTop;
	target.render();
	}
	});

	/**
	* Bar charts
	*/
	$.fn.sparkline.bar = bar = createClass($.fn.sparkline._base, barHighlightMixin, {
	type: 'bar',

	init: function (el, values, options, width, height) {
	var barWidth = parseInt(options.get('barWidth'), 10),
	barSpacing = parseInt(options.get('barSpacing'), 10),
	chartRangeMin = options.get('chartRangeMin'),
	chartRangeMax = options.get('chartRangeMax'),
	chartRangeClip = options.get('chartRangeClip'),
	stackMin = Infinity,
	stackMax = -Infinity,
	isStackString, groupMin, groupMax, stackRanges,
	numValues, i, vlen, range, zeroAxis, xaxisOffset, min, max, clipMin, clipMax,
	stacked, vlist, j, slen, svals, val, yoffset, yMaxCalc, canvasHeightEf;
	bar._super.init.call(this, el, values, options, width, height);

	// scan values to determine whether to stack bars
	for (i = 0, vlen = values.length; i < vlen; i++) {
	val = values[i];
	isStackString = typeof(val) === 'string' && val.indexOf(':') > -1;
	if (isStackString \|\| $.isArray(val)) {
	stacked = true;
	if (isStackString) {
	val = values[i] = normalizeValues(val.split(':'));
	}
	val = remove(val, null); // min/max will treat null as zero
	groupMin = Math.min.apply(Math, val);
	groupMax = Math.max.apply(Math, val);
	if (groupMin < stackMin) {
	stackMin = groupMin;
	}
	if (groupMax > stackMax) {
	stackMax = groupMax;
	}
	}
	}

	this.stacked = stacked;
	this.regionShapes = {};
	this.barWidth = barWidth;
	this.barSpacing = barSpacing;
	this.totalBarWidth = barWidth + barSpacing;
	this.width = width = (values.length * barWidth) + ((values.length - 1) * barSpacing);

	this.initTarget();

	if (chartRangeClip) {
	clipMin = chartRangeMin === undefined ? -Infinity : chartRangeMin;
	clipMax = chartRangeMax === undefined ? Infinity : chartRangeMax;
	}

	numValues = [];
	stackRanges = stacked ? [] : numValues;
	var stackTotals = [];
	var stackRangesNeg = [];
	for (i = 0, vlen = values.length; i < vlen; i++) {
	if (stacked) {
	vlist = values[i];
	values[i] = svals = [];
	stackTotals[i] = 0;
	stackRanges[i] = stackRangesNeg[i] = 0;
	for (j = 0, slen = vlist.length; j < slen; j++) {
	val = svals[j] = chartRangeClip ? clipval(vlist[j], clipMin, clipMax) : vlist[j];
	if (val !== null) {
	if (val > 0) {
	stackTotals[i] += val;
	}
	if (stackMin < 0 && stackMax > 0) {
	if (val < 0) {
	stackRangesNeg[i] += Math.abs(val);
	} else {
	stackRanges[i] += val;
	}
	} else {
	stackRanges[i] += Math.abs(val - (val < 0 ? stackMax : stackMin));
	}
	numValues.push(val);
	}
	}
	} else {
	val = chartRangeClip ? clipval(values[i], clipMin, clipMax) : values[i];
	val = values[i] = normalizeValue(val);
	if (val !== null) {
	numValues.push(val);
	}
	}
	}
	this.max = max = Math.max.apply(Math, numValues);
	this.min = min = Math.min.apply(Math, numValues);
	this.stackMax = stackMax = stacked ? Math.max.apply(Math, stackTotals) : max;
	this.stackMin = stackMin = stacked ? Math.min.apply(Math, numValues) : min;

	if (options.get('chartRangeMin') !== undefined && (options.get('chartRangeClip') \|\| options.get('chartRangeMin') < min)) {
	min = options.get('chartRangeMin');
	}
	if (options.get('chartRangeMax') !== undefined && (options.get('chartRangeClip') \|\| options.get('chartRangeMax') > max)) {
	max = options.get('chartRangeMax');
	}

	this.zeroAxis = zeroAxis = options.get('zeroAxis', true);
	if (min <= 0 && max >= 0 && zeroAxis) {
	xaxisOffset = 0;
	} else if (zeroAxis == false) {
	xaxisOffset = min;
	} else if (min > 0) {
	xaxisOffset = min;
	} else {
	xaxisOffset = max;
	}
	this.xaxisOffset = xaxisOffset;

	range = stacked ? (Math.max.apply(Math, stackRanges) + Math.max.apply(Math, stackRangesNeg)) : max - min;

	// as we plot zero/min values a single pixel line, we add a pixel to all other
	// values - Reduce the effective canvas size to suit
	this.canvasHeightEf = (zeroAxis && min < 0) ? this.canvasHeight - 2 : this.canvasHeight - 1;

	if (min < xaxisOffset) {
	yMaxCalc = (stacked && max >= 0) ? stackMax : max;
	yoffset = (yMaxCalc - xaxisOffset) / range * this.canvasHeight;
	if (yoffset !== Math.ceil(yoffset)) {
	this.canvasHeightEf -= 2;
	yoffset = Math.ceil(yoffset);
	}
	} else {
	yoffset = this.canvasHeight;
	}
	this.yoffset = yoffset;

	if ($.isArray(options.get('colorMap'))) {
	this.colorMapByIndex = options.get('colorMap');
	this.colorMapByValue = null;
	} else {
	this.colorMapByIndex = null;
	this.colorMapByValue = options.get('colorMap');
	if (this.colorMapByValue && this.colorMapByValue.get === undefined) {
	this.colorMapByValue = new RangeMap(this.colorMapByValue);
	}
	}

	this.range = range;
	},

	getRegion: function (el, x, y) {
	var result = Math.floor(x / this.totalBarWidth);
	return (result < 0 \|\| result >= this.values.length) ? undefined : result;
	},

	getCurrentRegionFields: function () {
	var currentRegion = this.currentRegion,
	values = ensureArray(this.values[currentRegion]),
	result = [],
	value, i;
	for (i = values.length; i--;) {
	value = values[i];
	result.push({
	isNull: value === null,
	value: value,
	color: this.calcColor(i, value, currentRegion),
	offset: currentRegion
	});
	}
	return result;
	},

	calcColor: function (stacknum, value, valuenum) {
	var colorMapByIndex = this.colorMapByIndex,
	colorMapByValue = this.colorMapByValue,
	options = this.options,
	color, newColor;
	if (this.stacked) {
	color = options.get('stackedBarColor');
	} else {
	color = (value < 0) ? options.get('negBarColor') : options.get('barColor');
	}
	if (value === 0 && options.get('zeroColor') !== undefined) {
	color = options.get('zeroColor');
	}
	if (colorMapByValue && (newColor = colorMapByValue.get(value))) {
	color = newColor;
	} else if (colorMapByIndex && colorMapByIndex.length > valuenum) {
	color = colorMapByIndex[valuenum];
	}
	return $.isArray(color) ? color[stacknum % color.length] : color;
	},

	/**
	* Render bar(s) for a region
	*/
	renderRegion: function (valuenum, highlight) {
	var vals = this.values[valuenum],
	options = this.options,
	xaxisOffset = this.xaxisOffset,
	result = [],
	range = this.range,
	stacked = this.stacked,
	target = this.target,
	x = valuenum * this.totalBarWidth,
	canvasHeightEf = this.canvasHeightEf,
	yoffset = this.yoffset,
	y, height, color, isNull, yoffsetNeg, i, valcount, val, minPlotted, allMin;

	vals = $.isArray(vals) ? vals : [vals];
	valcount = vals.length;
	val = vals[0];
	isNull = all(null, vals);
	allMin = all(xaxisOffset, vals, true);

	if (isNull) {
	if (options.get('nullColor')) {
	color = highlight ? options.get('nullColor') : this.calcHighlightColor(options.get('nullColor'), options);
	y = (yoffset > 0) ? yoffset - 1 : yoffset;
	return target.drawRect(x, y, this.barWidth - 1, 0, color, color);
	} else {
	return undefined;
	}
	}
	yoffsetNeg = yoffset;
	for (i = 0; i < valcount; i++) {
	val = vals[i];

	if (stacked && val === xaxisOffset) {
	if (!allMin \|\| minPlotted) {
	continue;
	}
	minPlotted = true;
	}

	if (range > 0) {
	height = Math.floor(canvasHeightEf * ((Math.abs(val - xaxisOffset) / range))) + 1;
	} else {
	height = 1;
	}
	if (val < xaxisOffset \|\| (val === xaxisOffset && yoffset === 0)) {
	y = yoffsetNeg;
	yoffsetNeg += height;
	} else {
	y = yoffset - height;
	yoffset -= height;
	}
	color = this.calcColor(i, val, valuenum);
	if (highlight) {
	color = this.calcHighlightColor(color, options);
	}
	result.push(target.drawRect(x, y, this.barWidth - 1, height - 1, color, color));
	}
	if (result.length === 1) {
	return result[0];
	}
	return result;
	}
	});

	/**
	* Tristate charts
	*/
	$.fn.sparkline.tristate = tristate = createClass($.fn.sparkline._base, barHighlightMixin, {
	type: 'tristate',

	init: function (el, values, options, width, height) {
	var barWidth = parseInt(options.get('barWidth'), 10),
	barSpacing = parseInt(options.get('barSpacing'), 10);
	tristate._super.init.call(this, el, values, options, width, height);

	this.regionShapes = {};
	this.barWidth = barWidth;
	this.barSpacing = barSpacing;
	this.totalBarWidth = barWidth + barSpacing;
	this.values = $.map(values, Number);
	this.width = width = (values.length * barWidth) + ((values.length - 1) * barSpacing);

	if ($.isArray(options.get('colorMap'))) {
	this.colorMapByIndex = options.get('colorMap');
	this.colorMapByValue = null;
	} else {
	this.colorMapByIndex = null;
	this.colorMapByValue = options.get('colorMap');
	if (this.colorMapByValue && this.colorMapByValue.get === undefined) {
	this.colorMapByValue = new RangeMap(this.colorMapByValue);
	}
	}
	this.initTarget();
	},

	getRegion: function (el, x, y) {
	return Math.floor(x / this.totalBarWidth);
	},

	getCurrentRegionFields: function () {
	var currentRegion = this.currentRegion;
	return {
	isNull: this.values[currentRegion] === undefined,
	value: this.values[currentRegion],
	color: this.calcColor(this.values[currentRegion], currentRegion),
	offset: currentRegion
	};
	},

	calcColor: function (value, valuenum) {
	var values = this.values,
	options = this.options,
	colorMapByIndex = this.colorMapByIndex,
	colorMapByValue = this.colorMapByValue,
	color, newColor;

	if (colorMapByValue && (newColor = colorMapByValue.get(value))) {
	color = newColor;
	} else if (colorMapByIndex && colorMapByIndex.length > valuenum) {
	color = colorMapByIndex[valuenum];
	} else if (values[valuenum] < 0) {
	color = options.get('negBarColor');
	} else if (values[valuenum] > 0) {
	color = options.get('posBarColor');
	} else {
	color = options.get('zeroBarColor');
	}
	return color;
	},

	renderRegion: function (valuenum, highlight) {
	var values = this.values,
	options = this.options,
	target = this.target,
	canvasHeight, height, halfHeight,
	x, y, color;

	canvasHeight = target.pixelHeight;
	halfHeight = Math.round(canvasHeight / 2);

	x = valuenum * this.totalBarWidth;
	if (values[valuenum] < 0) {
	y = halfHeight;
	height = halfHeight - 1;
	} else if (values[valuenum] > 0) {
	y = 0;
	height = halfHeight - 1;
	} else {
	y = halfHeight - 1;
	height = 2;
	}
	color = this.calcColor(values[valuenum], valuenum);
	if (color === null) {
	return;
	}
	if (highlight) {
	color = this.calcHighlightColor(color, options);
	}
	return target.drawRect(x, y, this.barWidth - 1, height - 1, color, color);
	}
	});

	/**
	* Discrete charts
	*/
	$.fn.sparkline.discrete = discrete = createClass($.fn.sparkline._base, barHighlightMixin, {
	type: 'discrete',

	init: function (el, values, options, width, height) {
	discrete._super.init.call(this, el, values, options, width, height);

	this.regionShapes = {};
	this.values = values = $.map(values, Number);
	this.min = Math.min.apply(Math, values);
	this.max = Math.max.apply(Math, values);
	this.range = this.max - this.min;
	this.width = width = options.get('width') === 'auto' ? values.length * 2 : this.width;
	this.interval = Math.floor(width / values.length);
	this.itemWidth = width / values.length;
	if (options.get('chartRangeMin') !== undefined && (options.get('chartRangeClip') \|\| options.get('chartRangeMin') < this.min)) {
	this.min = options.get('chartRangeMin');
	}
	if (options.get('chartRangeMax') !== undefined && (options.get('chartRangeClip') \|\| options.get('chartRangeMax') > this.max)) {
	this.max = options.get('chartRangeMax');
	}
	this.initTarget();
	if (this.target) {
	this.lineHeight = options.get('lineHeight') === 'auto' ? Math.round(this.canvasHeight * 0.3) : options.get('lineHeight');
	}
	},

	getRegion: function (el, x, y) {
	return Math.floor(x / this.itemWidth);
	},

	getCurrentRegionFields: function () {
	var currentRegion = this.currentRegion;
	return {
	isNull: this.values[currentRegion] === undefined,
	value: this.values[currentRegion],
	offset: currentRegion
	};
	},

	renderRegion: function (valuenum, highlight) {
	var values = this.values,
	options = this.options,
	min = this.min,
	max = this.max,
	range = this.range,
	interval = this.interval,
	target = this.target,
	canvasHeight = this.canvasHeight,
	lineHeight = this.lineHeight,
	pheight = canvasHeight - lineHeight,
	ytop, val, color, x;

	val = clipval(values[valuenum], min, max);
	x = valuenum * interval;
	ytop = Math.round(pheight - pheight * ((val - min) / range));
	color = (options.get('thresholdColor') && val < options.get('thresholdValue')) ? options.get('thresholdColor') : options.get('lineColor');
	if (highlight) {
	color = this.calcHighlightColor(color, options);
	}
	return target.drawLine(x, ytop, x, ytop + lineHeight, color);
	}
	});

	/**
	* Bullet charts
	*/
	$.fn.sparkline.bullet = bullet = createClass($.fn.sparkline._base, {
	type: 'bullet',

	init: function (el, values, options, width, height) {
	var min, max, vals;
	bullet._super.init.call(this, el, values, options, width, height);

	// values: target, performance, range1, range2, range3
	this.values = values = normalizeValues(values);
	// target or performance could be null
	vals = values.slice();
	vals[0] = vals[0] === null ? vals[2] : vals[0];
	vals[1] = values[1] === null ? vals[2] : vals[1];
	min = Math.min.apply(Math, values);
	max = Math.max.apply(Math, values);
	if (options.get('base') === undefined) {
	min = min < 0 ? min : 0;
	} else {
	min = options.get('base');
	}
	this.min = min;
	this.max = max;
	this.range = max - min;
	this.shapes = {};
	this.valueShapes = {};
	this.regiondata = {};
	this.width = width = options.get('width') === 'auto' ? '4.0em' : width;
	this.target = this.$el.simpledraw(width, height, options.get('composite'));
	if (!values.length) {
	this.disabled = true;
	}
	this.initTarget();
	},

	getRegion: function (el, x, y) {
	var shapeid = this.target.getShapeAt(el, x, y);
	return (shapeid !== undefined && this.shapes[shapeid] !== undefined) ? this.shapes[shapeid] : undefined;
	},

	getCurrentRegionFields: function () {
	var currentRegion = this.currentRegion;
	return {
	fieldkey: currentRegion.substr(0, 1),
	value: this.values[currentRegion.substr(1)],
	region: currentRegion
	};
	},

	changeHighlight: function (highlight) {
	var currentRegion = this.currentRegion,
	shapeid = this.valueShapes[currentRegion],
	shape;
	delete this.shapes[shapeid];
	switch (currentRegion.substr(0, 1)) {
	case 'r':
	shape = this.renderRange(currentRegion.substr(1), highlight);
	break;
	case 'p':
	shape = this.renderPerformance(highlight);
	break;
	case 't':
	shape = this.renderTarget(highlight);
	break;
	}
	this.valueShapes[currentRegion] = shape.id;
	this.shapes[shape.id] = currentRegion;
	this.target.replaceWithShape(shapeid, shape);
	},

	renderRange: function (rn, highlight) {
	var rangeval = this.values[rn],
	rangewidth = Math.round(this.canvasWidth * ((rangeval - this.min) / this.range)),
	color = this.options.get('rangeColors')[rn - 2];
	if (highlight) {
	color = this.calcHighlightColor(color, this.options);
	}
	return this.target.drawRect(0, 0, rangewidth - 1, this.canvasHeight - 1, color, color);
	},

	renderPerformance: function (highlight) {
	var perfval = this.values[1],
	perfwidth = Math.round(this.canvasWidth * ((perfval - this.min) / this.range)),
	color = this.options.get('performanceColor');
	if (highlight) {
	color = this.calcHighlightColor(color, this.options);
	}
	return this.target.drawRect(0, Math.round(this.canvasHeight * 0.3), perfwidth - 1,
	Math.round(this.canvasHeight * 0.4) - 1, color, color);
	},

	renderTarget: function (highlight) {
	var targetval = this.values[0],
	x = Math.round(this.canvasWidth * ((targetval - this.min) / this.range) - (this.options.get('targetWidth') / 2)),
	targettop = Math.round(this.canvasHeight * 0.10),
	targetheight = this.canvasHeight - (targettop * 2),
	color = this.options.get('targetColor');
	if (highlight) {
	color = this.calcHighlightColor(color, this.options);
	}
	return this.target.drawRect(x, targettop, this.options.get('targetWidth') - 1, targetheight - 1, color, color);
	},

	render: function () {
	var vlen = this.values.length,
	target = this.target,
	i, shape;
	if (!bullet._super.render.call(this)) {
	return;
	}
	for (i = 2; i < vlen; i++) {
	shape = this.renderRange(i).append();
	this.shapes[shape.id] = 'r' + i;
	this.valueShapes['r' + i] = shape.id;
	}
	if (this.values[1] !== null) {
	shape = this.renderPerformance().append();
	this.shapes[shape.id] = 'p1';
	this.valueShapes.p1 = shape.id;
	}
	if (this.values[0] !== null) {
	shape = this.renderTarget().append();
	this.shapes[shape.id] = 't0';
	this.valueShapes.t0 = shape.id;
	}
	target.render();
	}
	});

	/**
	* Pie charts
	*/
	$.fn.sparkline.pie = pie = createClass($.fn.sparkline._base, {
	type: 'pie',

	init: function (el, values, options, width, height) {
	var total = 0, i;

	pie._super.init.call(this, el, values, options, width, height);

	this.shapes = {}; // map shape ids to value offsets
	this.valueShapes = {}; // maps value offsets to shape ids
	this.values = values = $.map(values, Number);

	if (options.get('width') === 'auto') {
	this.width = this.height;
	}

	if (values.length > 0) {
	for (i = values.length; i--;) {
	total += values[i];
	}
	}
	this.total = total;
	this.initTarget();
	this.radius = Math.floor(Math.min(this.canvasWidth, this.canvasHeight) / 2);
	},

	getRegion: function (el, x, y) {
	var shapeid = this.target.getShapeAt(el, x, y);
	return (shapeid !== undefined && this.shapes[shapeid] !== undefined) ? this.shapes[shapeid] : undefined;
	},

	getCurrentRegionFields: function () {
	var currentRegion = this.currentRegion;
	return {
	isNull: this.values[currentRegion] === undefined,
	value: this.values[currentRegion],
	percent: this.values[currentRegion] / this.total * 100,
	color: this.options.get('sliceColors')[currentRegion % this.options.get('sliceColors').length],
	offset: currentRegion
	};
	},

	changeHighlight: function (highlight) {
	var currentRegion = this.currentRegion,
	newslice = this.renderSlice(currentRegion, highlight),
	shapeid = this.valueShapes[currentRegion];
	delete this.shapes[shapeid];
	this.target.replaceWithShape(shapeid, newslice);
	this.valueShapes[currentRegion] = newslice.id;
	this.shapes[newslice.id] = currentRegion;
	},

	renderSlice: function (valuenum, highlight) {
	var target = this.target,
	options = this.options,
	radius = this.radius,
	borderWidth = options.get('borderWidth'),
	offset = options.get('offset'),
	circle = 2 * Math.PI,
	values = this.values,
	total = this.total,
	next = offset ? (2Math.PI)(offset/360) : 0,
	start, end, i, vlen, color;

	vlen = values.length;
	for (i = 0; i < vlen; i++) {
	start = next;
	end = next;
	if (total > 0) { // avoid divide by zero
	end = next + (circle * (values[i] / total));
	}
	if (valuenum === i) {
	color = options.get('sliceColors')[i % options.get('sliceColors').length];
	if (highlight) {
	color = this.calcHighlightColor(color, options);
	}

	return target.drawPieSlice(radius, radius, radius - borderWidth, start, end, undefined, color);
	}
	next = end;
	}
	},

	render: function () {
	var target = this.target,
	values = this.values,
	options = this.options,
	radius = this.radius,
	borderWidth = options.get('borderWidth'),
	shape, i;

	if (!pie._super.render.call(this)) {
	return;
	}
	if (borderWidth) {
	target.drawCircle(radius, radius, Math.floor(radius - (borderWidth / 2)),
	options.get('borderColor'), undefined, borderWidth).append();
	}
	for (i = values.length; i--;) {
	if (values[i]) { // don't render zero values
	shape = this.renderSlice(i).append();
	this.valueShapes[i] = shape.id; // store just the shapeid
	this.shapes[shape.id] = i;
	}
	}
	target.render();
	}
	});

	/**
	* Box plots
	*/
	$.fn.sparkline.box = box = createClass($.fn.sparkline._base, {
	type: 'box',

	init: function (el, values, options, width, height) {
	box._super.init.call(this, el, values, options, width, height);
	this.values = $.map(values, Number);
	this.width = options.get('width') === 'auto' ? '4.0em' : width;
	this.initTarget();
	if (!this.values.length) {
	this.disabled = 1;
	}
	},

	/**
	* Simulate a single region
	*/
	getRegion: function () {
	return 1;
	},

	getCurrentRegionFields: function () {
	var result = [
	{ field: 'lq', value: this.quartiles[0] },
	{ field: 'med', value: this.quartiles[1] },
	{ field: 'uq', value: this.quartiles[2] }
	];
	if (this.loutlier !== undefined) {
	result.push({ field: 'lo', value: this.loutlier});
	}
	if (this.routlier !== undefined) {
	result.push({ field: 'ro', value: this.routlier});
	}
	if (this.lwhisker !== undefined) {
	result.push({ field: 'lw', value: this.lwhisker});
	}
	if (this.rwhisker !== undefined) {
	result.push({ field: 'rw', value: this.rwhisker});
	}
	return result;
	},

	render: function () {
	var target = this.target,
	values = this.values,
	vlen = values.length,
	options = this.options,
	canvasWidth = this.canvasWidth,
	canvasHeight = this.canvasHeight,
	minValue = options.get('chartRangeMin') === undefined ? Math.min.apply(Math, values) : options.get('chartRangeMin'),
	maxValue = options.get('chartRangeMax') === undefined ? Math.max.apply(Math, values) : options.get('chartRangeMax'),
	canvasLeft = 0,
	lwhisker, loutlier, iqr, q1, q2, q3, rwhisker, routlier, i,
	size, unitSize;

	if (!box._super.render.call(this)) {
	return;
	}

	if (options.get('raw')) {
	if (options.get('showOutliers') && values.length > 5) {
	loutlier = values[0];
	lwhisker = values[1];
	q1 = values[2];
	q2 = values[3];
	q3 = values[4];
	rwhisker = values[5];
	routlier = values[6];
	} else {
	lwhisker = values[0];
	q1 = values[1];
	q2 = values[2];
	q3 = values[3];
	rwhisker = values[4];
	}
	} else {
	values.sort(function (a, b) { return a - b; });
	q1 = quartile(values, 1);
	q2 = quartile(values, 2);
	q3 = quartile(values, 3);
	iqr = q3 - q1;
	if (options.get('showOutliers')) {
	lwhisker = rwhisker = undefined;
	for (i = 0; i < vlen; i++) {
	if (lwhisker === undefined && values[i] > q1 - (iqr * options.get('outlierIQR'))) {
	lwhisker = values[i];
	}
	if (values[i] < q3 + (iqr * options.get('outlierIQR'))) {
	rwhisker = values[i];
	}
	}
	loutlier = values[0];
	routlier = values[vlen - 1];
	} else {
	lwhisker = values[0];
	rwhisker = values[vlen - 1];
	}
	}
	this.quartiles = [q1, q2, q3];
	this.lwhisker = lwhisker;
	this.rwhisker = rwhisker;
	this.loutlier = loutlier;
	this.routlier = routlier;

	unitSize = canvasWidth / (maxValue - minValue + 1);
	if (options.get('showOutliers')) {
	canvasLeft = Math.ceil(options.get('spotRadius'));
	canvasWidth -= 2 * Math.ceil(options.get('spotRadius'));
	unitSize = canvasWidth / (maValue - minValue + 1);
	if (loutlier < lwhisker) {
	target.drawCircle((loutlier - minValue) * unitSize + canvasLeft,
	canvasHeight / 2,
	options.get('spotRadius'),
	options.get('outlierLineColor'),
	options.get('outlierFillColor')).append();
	}
	if (routlier > rwhisker) {
	target.drawCircle((routlier - minValue) * unitSize + canvasLeft,
	canvasHeight / 2,
	options.get('spotRadius'),
	options.get('outlierLineColor'),
	options.get('outlierFillColor')).append();
	}
	}

	// box
	target.drawRect(
	Math.round((q1 - minValue) * unitSize + canvasLeft),
	Math.round(canvasHeight * 0.1),
	Math.round((q3 - q1) * unitSize),
	Math.round(canvasHeight * 0.8),
	options.get('boxLineColor'),
	options.get('boxFillColor')).append();
	// left whisker
	target.drawLine(
	Math.round((lwhisker - minValue) * unitSize + canvasLeft),
	Math.round(canvasHeight / 2),
	Math.round((q1 - minValue) * unitSize + canvasLeft),
	Math.round(canvasHeight / 2),
	options.get('lineColor')).append();
	target.drawLine(
	Math.round((lwhisker - minValue) * unitSize + canvasLeft),
	Math.round(canvasHeight / 4),
	Math.round((lwhisker - minValue) * unitSize + canvasLeft),
	Math.round(canvasHeight - canvasHeight / 4),
	options.get('whiskerColor')).append();
	// right whisker
	target.drawLine(Math.round((rwhisker - minValue) * unitSize + canvasLeft),
	Math.round(canvasHeight / 2),
	Math.round((q3 - minValue) * unitSize + canvasLeft),
	Math.round(canvasHeight / 2),
	options.get('lineColor')).append();
	target.drawLine(
	Math.round((rwhisker - minValue) * unitSize + canvasLeft),
	Math.round(canvasHeight / 4),
	Math.round((rwhisker - minValue) * unitSize + canvasLeft),
	Math.round(canvasHeight - canvasHeight / 4),
	options.get('whiskerColor')).append();
	// median line
	target.drawLine(
	Math.round((q2 - minValue) * unitSize + canvasLeft),
	Math.round(canvasHeight * 0.1),
	Math.round((q2 - minValue) * unitSize + canvasLeft),
	Math.round(canvasHeight * 0.9),
	options.get('medianColor')).append();
	if (options.get('target')) {
	size = Math.ceil(options.get('spotRadius'));
	target.drawLine(
	Math.round((options.get('target') - minValue) * unitSize + canvasLeft),
	Math.round((canvasHeight / 2) - size),
	Math.round((options.get('target') - minValue) * unitSize + canvasLeft),
	Math.round((canvasHeight / 2) + size),
	options.get('targetColor')).append();
	target.drawLine(
	Math.round((options.get('target') - minValue) * unitSize + canvasLeft - size),
	Math.round(canvasHeight / 2),
	Math.round((options.get('target') - minValue) * unitSize + canvasLeft + size),
	Math.round(canvasHeight / 2),
	options.get('targetColor')).append();
	}
	target.render();
	}
	});

	// Setup a very simple "virtual canvas" to make drawing the few shapes we need easier
	// This is accessible as $(foo).simpledraw()

	if ($.browser.msie && document.namespaces && !document.namespaces.v) {
	document.namespaces.add('v', 'urn:schemas-microsoft-com:vml', '#default#VML');
	}

	if ($.browser.hasCanvas === undefined) {
	$.browser.hasCanvas = document.createElement('canvas').getContext !== undefined;
	}

	VShape = createClass({
	init: function (target, id, type, args) {
	this.target = target;
	this.id = id;
	this.type = type;
	this.args = args;
	},
	append: function () {
	this.target.appendShape(this);
	return this;
	}
	});

	VCanvas_base = createClass({
	_pxregex: /(\d+)(px)?\s*$/i,

	init: function (width, height, target) {
	if (!width) {
	return;
	}
	this.width = width;
	this.height = height;
	this.target = target;
	this.lastShapeId = null;
	if (target[0]) {
	target = target[0];
	}
	$.data(target, '_jqs_vcanvas', this);
	},

	drawLine: function (x1, y1, x2, y2, lineColor, lineWidth) {
	return this.drawShape([[x1, y1], [x2, y2]], lineColor, lineWidth);
	},

	drawShape: function (path, lineColor, fillColor, lineWidth) {
	return this._genShape('Shape', [path, lineColor, fillColor, lineWidth]);
	},

	drawCircle: function (x, y, radius, lineColor, fillColor, lineWidth) {
	return this._genShape('Circle', [x, y, radius, lineColor, fillColor, lineWidth]);
	},

	drawPieSlice: function (x, y, radius, startAngle, endAngle, lineColor, fillColor) {
	return this._genShape('PieSlice', [x, y, radius, startAngle, endAngle, lineColor, fillColor]);
	},

	drawRect: function (x, y, width, height, lineColor, fillColor) {
	return this._genShape('Rect', [x, y, width, height, lineColor, fillColor]);
	},

	getElement: function () {
	return this.canvas;
	},

	/**
	* Return the most recently inserted shape id
	*/
	getLastShapeId: function () {
	return this.lastShapeId;
	},

	/**
	* Clear and reset the canvas
	*/
	reset: function () {
	alert('reset not implemented');
	},

	_insert: function (el, target) {
	$(target).html(el);
	},

	/**
	* Calculate the pixel dimensions of the canvas
	*/
	_calculatePixelDims: function (width, height, canvas) {
	// XXX This should probably be a configurable option
	var match;
	match = this._pxregex.exec(height);
	if (match) {
	this.pixelHeight = match[1];
	} else {
	this.pixelHeight = $(canvas).height();
	}
	match = this._pxregex.exec(width);
	if (match) {
	this.pixelWidth = match[1];
	} else {
	this.pixelWidth = $(canvas).width();
	}
	},

	/**
	* Generate a shape object and id for later rendering
	*/
	_genShape: function (shapetype, shapeargs) {
	var id = shapeCount++;
	shapeargs.unshift(id);
	return new VShape(this, id, shapetype, shapeargs);
	},

	/**
	* Add a shape to the end of the render queue
	*/
	appendShape: function (shape) {
	alert('appendShape not implemented');
	},

	/**
	* Replace one shape with another
	*/
	replaceWithShape: function (shapeid, shape) {
	alert('replaceWithShape not implemented');
	},

	/**
	* Insert one shape after another in the render queue
	*/
	insertAfterShape: function (shapeid, shape) {
	alert('insertAfterShape not implemented');
	},

	/**
	* Remove a shape from the queue
	*/
	removeShapeId: function (shapeid) {
	alert('removeShapeId not implemented');
	},

	/**
	* Find a shape at the specified x/y co-ordinates
	*/
	getShapeAt: function (el, x, y) {
	alert('getShapeAt not implemented');
	},

	/**
	* Render all queued shapes onto the canvas
	*/
	render: function () {
	alert('render not implemented');
	}
	});

	VCanvas_canvas = createClass(VCanvas_base, {
	init: function (width, height, target, interact) {
	VCanvas_canvas._super.init.call(this, width, height, target);
	this.canvas = document.createElement('canvas');
	if (target[0]) {
	target = target[0];
	}
	$.data(target, '_jqs_vcanvas', this);
	$(this.canvas).css({ display: 'inline-block', width: width, height: height, verticalAlign: 'top' });
	this._insert(this.canvas, target);
	this._calculatePixelDims(width, height, this.canvas);
	this.canvas.width = this.pixelWidth;
	this.canvas.height = this.pixelHeight;
	this.interact = interact;
	this.shapes = {};
	this.shapeseq = [];
	this.currentTargetShapeId = undefined;
	$(this.canvas).css({width: this.pixelWidth, height: this.pixelHeight});
	},

	_getContext: function (lineColor, fillColor, lineWidth) {
	var context = this.canvas.getContext('2d');
	if (lineColor !== undefined) {
	context.strokeStyle = lineColor;
	}
	context.lineWidth = lineWidth === undefined ? 1 : lineWidth;
	if (fillColor !== undefined) {
	context.fillStyle = fillColor;
	}
	return context;
	},

	reset: function () {
	var context = this._getContext();
	context.clearRect(0, 0, this.pixelWidth, this.pixelHeight);
	this.shapes = {};
	this.shapeseq = [];
	this.currentTargetShapeId = undefined;
	},

	_drawShape: function (shapeid, path, lineColor, fillColor, lineWidth) {
	var context = this._getContext(lineColor, fillColor, lineWidth),
	i, plen;
	context.beginPath();
	context.moveTo(path[0][0] + 0.5, path[0][1] + 0.5);
	for (i = 1, plen = path.length; i < plen; i++) {
	context.lineTo(path[i][0] + 0.5, path[i][1] + 0.5); // the 0.5 offset gives us crisp pixel-width lines
	}
	if (lineColor !== undefined) {
	context.stroke();
	}
	if (fillColor !== undefined) {
	context.fill();
	}
	if (this.targetX !== undefined && this.targetY !== undefined &&
	context.isPointInPath(this.targetX, this.targetY)) {
	this.currentTargetShapeId = shapeid;
	}
	},

	_drawCircle: function (shapeid, x, y, radius, lineColor, fillColor, lineWidth) {
	var context = this._getContext(lineColor, fillColor, lineWidth);
	context.beginPath();
	context.arc(x, y, radius, 0, 2 * Math.PI, false);
	if (this.targetX !== undefined && this.targetY !== undefined &&
	context.isPointInPath(this.targetX, this.targetY)) {
	this.currentTargetShapeId = shapeid;
	}
	if (lineColor !== undefined) {
	context.stroke();
	}
	if (fillColor !== undefined) {
	context.fill();
	}
	},

	_drawPieSlice: function (shapeid, x, y, radius, startAngle, endAngle, lineColor, fillColor) {
	var context = this._getContext(lineColor, fillColor);
	context.beginPath();
	context.moveTo(x, y);
	context.arc(x, y, radius, startAngle, endAngle, false);
	context.lineTo(x, y);
	context.closePath();
	if (lineColor !== undefined) {
	context.stroke();
	}
	if (fillColor) {
	context.fill();
	}
	if (this.targetX !== undefined && this.targetY !== undefined &&
	context.isPointInPath(this.targetX, this.targetY)) {
	this.currentTargetShapeId = shapeid;
	}
	},

	_drawRect: function (shapeid, x, y, width, height, lineColor, fillColor) {
	return this._drawShape(shapeid, [[x, y], [x + width, y], [x + width, y + height], [x, y + height], [x, y]], lineColor, fillColor);
	},

	appendShape: function (shape) {
	this.shapes[shape.id] = shape;
	this.shapeseq.push(shape.id);
	this.lastShapeId = shape.id;
	return shape.id;
	},

	replaceWithShape: function (shapeid, shape) {
	var shapeseq = this.shapeseq,
	i;
	this.shapes[shape.id] = shape;
	for (i = shapeseq.length; i--;) {
	if (shapeseq[i] == shapeid) {
	shapeseq[i] = shape.id;
	}
	}
	delete this.shapes[shapeid];
	},

	replaceWithShapes: function (shapeids, shapes) {
	var shapeseq = this.shapeseq,
	shapemap = {},
	sid, i, first;

	for (i = shapeids.length; i--;) {
	shapemap[shapeids[i]] = true;
	}
	for (i = shapeseq.length; i--;) {
	sid = shapeseq[i];
	if (shapemap[sid]) {
	shapeseq.splice(i, 1);
	delete this.shapes[sid];
	first = i;
	}
	}
	for (i = shapes.length; i--;) {
	shapeseq.splice(first, 0, shapes[i].id);
	this.shapes[shapes[i].id] = shapes[i];
	}

	},

	insertAfterShape: function (shapeid, shape) {
	var shapeseq = this.shapeseq,
	i;
	for (i = shapeseq.length; i--;) {
	if (shapeseq[i] === shapeid) {
	shapeseq.splice(i + 1, 0, shape.id);
	this.shapes[shape.id] = shape;
	return;
	}
	}
	},

	removeShapeId: function (shapeid) {
	var shapeseq = this.shapeseq,
	i;
	for (i = shapeseq.length; i--;) {
	if (shapeseq[i] === shapeid) {
	shapeseq.splice(i, 1);
	break;
	}
	}
	delete this.shapes[shapeid];
	},

	getShapeAt: function (el, x, y) {
	this.targetX = x;
	this.targetY = y;
	this.render();
	return this.currentTargetShapeId;
	},

	render: function () {
	var shapeseq = this.shapeseq,
	shapes = this.shapes,
	shapeCount = shapeseq.length,
	context = this._getContext(),
	shapeid, shape, i;
	context.clearRect(0, 0, this.pixelWidth, this.pixelHeight);
	for (i = 0; i < shapeCount; i++) {
	shapeid = shapeseq[i];
	shape = shapes[shapeid];
	this['_draw' + shape.type].apply(this, shape.args);
	}
	if (!this.interact) {
	// not interactive so no need to keep the shapes array
	this.shapes = {};
	this.shapeseq = [];
	}
	}

	});

	VCanvas_vml = createClass(VCanvas_base, {
	init: function (width, height, target) {
	var groupel;
	VCanvas_vml._super.init.call(this, width, height, target);
	if (target[0]) {
	target = target[0];
	}
	$.data(target, '_jqs_vcanvas', this);
	this.canvas = document.createElement('span');
	$(this.canvas).css({ display: 'inline-block', position: 'relative', overflow: 'hidden', width: width, height: height, margin: '0px', padding: '0px', verticalAlign: 'top'});
	this._insert(this.canvas, target);
	this._calculatePixelDims(width, height, this.canvas);
	this.canvas.width = this.pixelWidth;
	this.canvas.height = this.pixelHeight;
	groupel = '<v:group coordorigin="0 0" coordsize="' + this.pixelWidth + ' ' + this.pixelHeight + '"' +
	' style="position:absolute;top:0;left:0;width:' + this.pixelWidth + 'px;height=' + this.pixelHeight + 'px;"></v:group>';
	this.canvas.insertAdjacentHTML('beforeEnd', groupel);
	this.group = $(this.canvas).children()[0];
	this.rendered = false;
	this.prerender = '';
	},

	_drawShape: function (shapeid, path, lineColor, fillColor, lineWidth) {
	var vpath = [],
	initial, stroke, fill, closed, vel, plen, i;
	for (i = 0, plen = path.length; i < plen; i++) {
	vpath[i] = '' + (path[i][0]) + ',' + (path[i][1]);
	}
	initial = vpath.splice(0, 1);
	lineWidth = lineWidth === undefined ? 1 : lineWidth;
	stroke = lineColor === undefined ? ' stroked="false" ' : ' strokeWeight="' + lineWidth + 'px" strokeColor="' + lineColor + '" ';
	fill = fillColor === undefined ? ' filled="false"' : ' fillColor="' + fillColor + '" filled="true" ';
	closed = vpath[0] === vpath[vpath.length - 1] ? 'x ' : '';
	vel = '<v:shape coordorigin="0 0" coordsize="' + this.pixelWidth + ' ' + this.pixelHeight + '" ' +
	' id="jqsshape' + shapeid + '" ' +
	stroke +
	fill +
	' style="position:absolute;left:0px;top:0px;height:' + this.pixelHeight + 'px;width:' + this.pixelWidth + 'px;padding:0px;margin:0px;" ' +
	' path="m ' + initial + ' l ' + vpath.join(', ') + ' ' + closed + 'e">' +
	' </v:shape>';
	return vel;
	},

	_drawCircle: function (shapeid, x, y, radius, lineColor, fillColor, lineWidth) {
	var stroke, fill, vel;
	x -= radius;
	y -= radius;
	stroke = lineColor === undefined ? ' stroked="false" ' : ' strokeWeight="' + lineWidth + 'px" strokeColor="' + lineColor + '" ';
	fill = fillColor === undefined ? ' filled="false"' : ' fillColor="' + fillColor + '" filled="true" ';
	vel = '<v:oval ' +
	' id="jqsshape' + shapeid + '" ' +
	stroke +
	fill +
	' style="position:absolute;top:' + y + 'px; left:' + x + 'px; width:' + (radius * 2) + 'px; height:' + (radius * 2) + 'px"></v:oval>';
	return vel;

	},

	_drawPieSlice: function (shapeid, x, y, radius, startAngle, endAngle, lineColor, fillColor) {
	var vpath, startx, starty, endx, endy, stroke, fill, vel;
	if (startAngle === endAngle) {
	return; // VML seems to have problem when start angle equals end angle.
	}
	if ((endAngle - startAngle) === (2 * Math.PI)) {
	startAngle = 0.0; // VML seems to have a problem when drawing a full circle that doesn't start 0
	endAngle = (2 * Math.PI);
	}

	startx = x + Math.round(Math.cos(startAngle) * radius);
	starty = y + Math.round(Math.sin(startAngle) * radius);
	endx = x + Math.round(Math.cos(endAngle) * radius);
	endy = y + Math.round(Math.sin(endAngle) * radius);

	// Prevent very small slices from being mistaken as a whole pie
	if (startx === endx && starty === endy && (endAngle - startAngle) < Math.PI) {
	return;
	}

	vpath = [x - radius, y - radius, x + radius, y + radius, startx, starty, endx, endy];
	stroke = lineColor === undefined ? ' stroked="false" ' : ' strokeWeight="1px" strokeColor="' + lineColor + '" ';
	fill = fillColor === undefined ? ' filled="false"' : ' fillColor="' + fillColor + '" filled="true" ';
	vel = '<v:shape coordorigin="0 0" coordsize="' + this.pixelWidth + ' ' + this.pixelHeight + '" ' +
	' id="jqsshape' + shapeid + '" ' +
	stroke +
	fill +
	' style="position:absolute;left:0px;top:0px;height:' + this.pixelHeight + 'px;width:' + this.pixelWidth + 'px;padding:0px;margin:0px;" ' +
	' path="m ' + x + ',' + y + ' wa ' + vpath.join(', ') + ' x e">' +
	' </v:shape>';
	return vel;
	},

	_drawRect: function (shapeid, x, y, width, height, lineColor, fillColor) {
	return this._drawShape(shapeid, [[x, y], [x, y + height], [x + width, y + height], [x + width, y], [x, y]], lineColor, fillColor);
	},

	reset: function () {
	this.group.innerHTML = '';
	},

	appendShape: function (shape) {
	var vel = this['_draw' + shape.type].apply(this, shape.args);
	if (this.rendered) {
	this.group.insertAdjacentHTML('beforeEnd', vel);
	} else {
	this.prerender += vel;
	}
	this.lastShapeId = shape.id;
	return shape.id;
	},

	replaceWithShape: function (shapeid, shape) {
	var existing = $('#jqsshape' + shapeid),
	vel = this['_draw' + shape.type].apply(this, shape.args);
	existing[0].outerHTML = vel;
	},

	replaceWithShapes: function (shapeids, shapes) {
	// replace the first shapeid with all the new shapes then toast the remaining old shapes
	var existing = $('#jqsshape' + shapeids[0]),
	replace = '',
	slen = shapes.length,
	i;
	for (i = 0; i < slen; i++) {
	replace += this['_draw' + shapes[i].type].apply(this, shapes[i].args);
	}
	existing[0].outerHTML = replace;
	for (i = 1; i < shapeids.length; i++) {
	$('#jqsshape' + shapeids[i]).remove();
	}
	},

	insertAfterShape: function (shapeid, shape) {
	var existing = $('#jqsshape' + shapeid),
	vel = this['_draw' + shape.type].apply(this, shape.args);
	existing[0].insertAdjacentHTML('afterEnd', vel);
	},

	removeShapeId: function (shapeid) {
	var existing = $('#jqsshape' + shapeid);
	this.group.removeChild(existing[0]);
	},

	getShapeAt: function (el, x, y) {
	var shapeid = el.id.substr(8);
	return shapeid;
	},

	render: function () {
	if (!this.rendered) {
	// batch the intial render into a single repaint
	this.group.innerHTML = this.prerender;
	this.rendered = true;
	}
	}
	});

	}));

file:a/ckanext/ga_report/report_model.py (deleted)

file:b/ckanext/ga_report/templates/ga_report/ga_popular_datasets.html (new)

	<html xmlns:py="http://genshi.edgewall.org/"
	xmlns:i18n="http://genshi.edgewall.org/i18n"
	xmlns:xi="http://www.w3.org/2001/XInclude"
	py:strip="">

	<div class="popular_datasets">
	<div class="pull-right">
	<a href="${h.url_for(controller='ckanext.ga_report.controller:GaDatasetReport',action='read')}" class="btn btn-primary">More popular datasets »</a>
	</div>
	<h2>Popular datasets</h2>
	<h4>${publisher.title}</h4>
	<ul>
	<py:for each="dataset, _, _ in datasets">
	<li>
	<span>${h.link_to(dataset.title, h.url_for(controller='package', action='read', id=dataset.name))}</span>
	<div>${h.truncate(dataset.notes, length=80, whole_word=True)}</div>
	</li>
	</py:for>
	</ul>

	</div>

	</html>

file:b/ckanext/ga_report/templates/ga_report/ga_popular_single.html (new)

	<html xmlns:py="http://genshi.edgewall.org/"
	xmlns:i18n="http://genshi.edgewall.org/i18n"
	xmlns:xi="http://www.w3.org/2001/XInclude"
	py:strip="">


	<div class="popular_datasets">
	<h2>Featured dataset</h2>

	<div class="dataset-summary boxed">
	<a class="dataset-header" href="${h.url_for(controller='package', action='read', id=dataset['name'])}">
	<h3>${dataset['title']}</h3>
	</a>
	<h4>
	<strong>Publisher</strong> :
	<a href="/publisher/${publisher['name']}">${publisher['title']}</a>
	</h4>
	<div>${h.truncate(dataset['notes_rendered'], length=200, whole_word=True)}</div>
	</div>
	<div>
	<a href="${h.url_for(controller='ckanext.ga_report.controller:GaDatasetReport',action='read')}" class="btn">Other popular datasets</a>
	<a href="${h.url_for(controller='ckanext.ga_report.controller:GaReport',action='index')}" class="btn">All usage data</a>
	</div>
	</div>


	</html>

file:b/ckanext/ga_report/templates/ga_report/ga_util.html (new)

	<html
	xmlns="http://www.w3.org/1999/xhtml"
	xmlns:i18n="http://genshi.edgewall.org/i18n"
	xmlns:py="http://genshi.edgewall.org/"
	xmlns:xi="http://www.w3.org/2001/XInclude"
	py:strip=""
	>

	<select name="month" py:def="month_selector(current_month, months, day)">
	<option value='' py:attrs="{'selected': 'selected' if not current_month else None}">All months</option>
	<py:for each="i, (val,desc) in enumerate(months)">
	<option value='${val}' py:attrs="{'selected': 'selected' if current_month == val else None}">${desc}<py:if test="i == 0 and day"> (up to ${day})</py:if></option>
	</py:for>
	</select>


	<table py:def="social_table(items, with_source=False)" class="table table-condensed table-bordered table-striped">
	<tr>
	<th>Name</th>
	<th py:if="with_source">Source</th>
	<th>Visits</th>
	</tr>
	<py:for each="name, url, source, count in items">
	<tr>
	<td><a href="${url}">${name}</a></td>
	<td py:if="with_source">${source}</td>
	<td>${count}</td>
	</tr>
	</py:for>
	</table>


	<table py:def="stat_table(items, title='Views')" class="table table-condensed table-bordered table-striped">
	<tr>
	<th>Name</th>
	<th>% ${title}</th>
	</tr>
	<py:for each="name, value in items">
	<tr>
	<td>${name}</td>
	<td>${value}</td>
	</tr>
	</py:for>
	</table>


	<div py:def="usage_nav(active_name)" id="minornavigation">
	<div id="minornavigation-bg-left">
	<div id="minornavigation-bg-right">
	<ul class="nav nav-pills">
	<li py:attrs="{'class': 'active' if active_name=='Site-wide' else None}"><a py:attrs="{'class': 'active' if active_name=='Site-wide' else None}" href="${h.url_for(controller='ckanext.ga_report.controller:GaReport',action='index')}"><img src="/images/icons/page_white.png" height="16px" width="16px" alt="None" class="inline-icon "/> Site-wide</a></li>
	<li py:attrs="{'class': 'active' if active_name=='Publishers' else None}">
	<a py:attrs="{'class': 'active' if active_name=='Publishers' else None}" href="${h.url_for(controller='ckanext.ga_report.controller:GaDatasetReport',action='publishers')}"><img src="/images/icons/page_white.png" height="16px" width="16px" alt="None" class="inline-icon "/> Publishers</a>
	</li>
	<li py:attrs="{'class': 'active' if active_name=='Datasets' else None}">
	<a py:attrs="{'class': 'active' if active_name=='Datasets' else None}" href="${h.url_for(controller='ckanext.ga_report.controller:GaDatasetReport',action='read')}"><img src="/images/icons/page_white.png" height="16px" width="16px" alt="None" class="inline-icon "/> Datasets</a>
	</li>
	</ul>
	</div>
	</div>
	</div>


	</html>

file:b/ckanext/ga_report/templates/ga_report/notes.html (new)

	<html xmlns:py="http://genshi.edgewall.org/"
	xmlns:i18n="http://genshi.edgewall.org/i18n"
	xmlns:xi="http://www.w3.org/2001/XInclude"
	py:strip="">

	<li class="widget-container boxed widget_text">
	<h4>Notes</h4>
	<ul>
	<li>"Views" is the number of times a page was loaded in users' browsers.</li>
	<li>"Downloads" is the number of times a user has clicked to download either an original or cached resource for a particular dataset since December 2012</li>
	<li>These usage statistics are confined to users with javascript enabled, which excludes web crawlers and API calls.</li>
	<li>The results are not shown when the number of views/visits is tiny. Where these relate to site pages, results are available in full in the CSV download. Where these relate to users' web browser information, results are not disclosed, for privacy reasons.</li>
	</ul>
	</li>
	</html>

file:a/ckanext/ga_report/templates/ga_report/publisher/index.html -> file:b/ckanext/ga_report/templates/ga_report/publisher/index.html

<html xmlns:py="http://genshi.edgewall.org/"	<html xmlns:py="http://genshi.edgewall.org/"
xmlns:i18n="http://genshi.edgewall.org/i18n"	xmlns:i18n="http://genshi.edgewall.org/i18n"
xmlns:xi="http://www.w3.org/2001/XInclude"	xmlns:xi="http://www.w3.org/2001/XInclude"
py:strip="">	py:strip="">

<py:def function="page_title">Publisher Analytics for ${g.site_title}</py:def>	<xi:include href="../ga_util.html" />

	<py:def function="page_title">Usage by Publisher</py:def>

<py:match path="primarysidebar">	<py:match path="primarysidebar">
<li class="widget-container boxed widget_text">	<li class="widget-container boxed widget_text">
<h4>Publishers</h4>	<h4>Download</h4>
<p>The table shows the top 20 publishers as recorded by page views of datasets owned by that publisher, and the number of visits to each publisher's home page.</p>	<p><center>
	<a class="btn button btn-primary" href="${h.url_for(controller='ckanext.ga_report.controller:GaDatasetReport',action='publisher_csv',month=c.month or 'all')}">Download as CSV</a></center>
	</p>
</li>	</li>
	<xi:include href="../notes.html" />
</py:match>	</py:match>


<div py:match="content">	<div py:match="content">
<h1>Publisher Analytics</h1>
<h2>The top 20 publishers</h2>

<form class="form-inline" action="${h.url_for(controller='ckanext.ga_report.controller:GaPublisherReport',action='index')}" method="get">	<h1>Site Usage</h1>

	${usage_nav('Publishers')}


	<form class="form-inline" action="${h.url_for(controller='ckanext.ga_report.controller:GaDatasetReport',action='publishers')}" method="get">
<div class="controls">	<div class="controls">
<select name="month">
<py:for each="val,desc in c.months">	${month_selector(c.month, c.months, c.day)}
<option value='${val}' py:attrs="{'selected': 'selected' if c.month == val else None}">${desc}</option>
</py:for>	<input class="btn button btn-primary" type='submit' value="Update"/>
</select>
<input class="btn button" type='submit' value="Update"/>
</div>	</div>
</form>	</form>

<table class="table table-condensed table-bordered table-striped">	<table class="table table-condensed table-bordered table-striped">
<tr>	<tr>
<th>Publisher</th>	<th>Publisher</th>
<th>Dataset Views</th>	<th>Dataset Views</th>
<th>Visits</th>
</tr>	</tr>
<py:for each="publisher, views, visits in c.top_publishers">	<py:for each="publisher, views, visits in c.top_publishers">
<tr>	<tr>
<td>${h.link_to(publisher.title, h.url_for(controller='ckanext.ga_report.controller:GaPublisherReport', action='read', id=publisher.name))}	<td>
	${h.link_to(publisher.title, h.url_for(controller='ckanext.ga_report.controller:GaDatasetReport', action='read_publisher', id=publisher.name) + (("?month=" + c.month) if c.month else ''))}
</td>	</td>
<td>${views}</td>	<td>${views}</td>
<td>${visits}</td>
</tr>	</tr>
</py:for>	</py:for>
</table>	</table>


</div>	</div>

<xi:include href="../../layout.html" />	<xi:include href="../../layout.html" />

<py:def function="optional_footer">	<py:def function="optional_footer">
<script type='text/javascript'>	<script type='text/javascript'>
$('.nav-tabs li a').click(function (e) {	$('.nav-tabs li a').click(function (e) {
e.preventDefault();	e.preventDefault();
$(this).tab('show');	$(this).tab('show');
})	})
</script>	</script>
</py:def>	</py:def>

</html>	</html>

file:b/ckanext/ga_report/templates/ga_report/publisher/popular.html (new)

	<html xmlns:py="http://genshi.edgewall.org/"
	xmlns:i18n="http://genshi.edgewall.org/i18n"
	xmlns:xi="http://www.w3.org/2001/XInclude"
	py:strip="">
	<py:if test="dataset_count == 0">
	<p>We do not currently have usage data for ${publisher.title}</p>
	</py:if>

	<py:if test="dataset_count > 0">
	<div class="popular_datasets">
	<ul>
	<py:for each="dataset,views,visits in datasets">
	<li>
	${h.link_to(dataset.title, h.url_for(controller='package', action='read', id=dataset.name))}
	</li>
	</py:for>
	</ul>
	<p class="">${h.link_to("More usage data for " + publisher.title, h.url_for(controller='ckanext.ga_report.controller:GaDatasetReport',action='read_publisher',id=publisher.name))}</p>
	</div>
	</py:if>
	</html>

file:a/ckanext/ga_report/templates/ga_report/publisher/read.html -> file:b/ckanext/ga_report/templates/ga_report/publisher/read.html

<html xmlns:py="http://genshi.edgewall.org/"	<html xmlns:py="http://genshi.edgewall.org/"
xmlns:i18n="http://genshi.edgewall.org/i18n"	xmlns:i18n="http://genshi.edgewall.org/i18n"
xmlns:xi="http://www.w3.org/2001/XInclude"	xmlns:xi="http://www.w3.org/2001/XInclude"
py:strip="">	py:strip="">

<py:def function="page_title">Analytics for ${g.site_title}</py:def>	<xi:include href="../ga_util.html" />

<py:match path="primarysidebar">	<py:def function="page_title">Usage by Dataset</py:def>

	<py:match path="primarysidebar">
<li class="widget-container boxed widget_text">	<li class="widget-container boxed widget_text">
<h4>${c.publisher.title}</h4>	<h4>Download</h4>
<p>	<p><center>
The table shows the top 20 most viewed datasets belonging to ${c.publisher.title}.	<a class="btn button btn-primary" href="${h.url_for(controller='ckanext.ga_report.controller:GaDatasetReport',action='dataset_csv',id=c.publisher_name or 'all',month=c.month or 'all')}">Download as CSV</a></center>
</p>	</p>
<p>
As well as showing the number of views within ${c.month_desc}, it will also show the
number of visitors that viewed each dataset.
</p>
<p>The dataset list page for <a href="${h.url_for(controller='ckanext.dgu.controllers.publisher:PublisherController', action='read', id=c.publisher.name)}">${c.publisher.title}</a> was viewed ${c.publisher_page_views} times during ${c.month_desc}</p>
<p>View the <a href="${h.url_for(controller='ckanext.ga_report.controller:GaPublisherReport', action='index')}">publishers</a> leaderboard</p>
</li>	</li>
	<xi:include href="../notes.html" />
</py:match>	</py:match>


<div py:match="content">	<div py:match="content">
<h1>Analytics for ${c.publisher.title}</h1>	<h1>Site Usage</h1>

<h2>Top 20 most viewed datasets</h2>	${usage_nav('Datasets')}
<p><em>Note: this data does not include API calls</em></p>

<form class="form-inline" action="${h.url_for(controller='ckanext.ga_report.controller:GaPublisherReport',action='read',id=c.publisher.name)}" method="get">	<form class="form-inline" action="${h.url_for(controller='ckanext.ga_report.controller:GaDatasetReport',action='read')}" method="get">
<div class="controls">	<div class="controls">
<select name="month">
<py:for each="val,desc in c.months">	${month_selector(c.month, c.months, c.day)}
<option value='${val}' py:attrs="{'selected': 'selected' if c.month == val else None}">${desc}</option>
	<select name="publisher">
	<option value='' py:attrs="{'selected': 'selected' if not c.publisher else None}">All publishers</option>
	<py:for each="val,desc in c.publishers">
	<option value='${val}' py:attrs="{'selected': 'selected' if c.publisher_name == val else None}">${desc}</option>
</py:for>	</py:for>
</select>	</select>
<input class="btn button" type='submit' value="Update"/>	<input class="btn button btn-primary" type='submit' value="Update"/>
</div>	</div>
</form>	</form>

<table class="table table-condensed table-bordered table-striped">	<h3 py:if="c.publisher"><a href="${h.url_for(controller='ckanext.dgu.controllers.publisher:PublisherController',action='read',id=c.publisher.name)}">${c.publisher.title}</a></h3>

	<p py:if="not c.top_packages">No page views in this period</p>
	<table py:if="c.top_packages" class="table table-condensed table-bordered table-striped">
<tr>	<tr>
<th>Dataset</th>	<th>Dataset</th>
<th>Views</th>	<th>Views</th>
<th>Visits</th>	<th>Downloads</th>
</tr>	</tr>
<py:for each="package, views, visits in c.top_packages">	<py:for each="package, views, visits,downloads in c.top_packages">
<tr>	<tr>
<td>${h.link_to(package.title or package.name, h.url_for(controller='package', action='read', id=package.name))}	<td>${h.link_to(package.title or package.name, h.url_for(controller='package', action='read', id=package.name))}
</td>	</td>
<td>${views}</td>	<td>${views}</td>
<td>${visits}</td>	<td>${downloads}</td>
</tr>	</tr>
</py:for>	</py:for>
</table>	</table>


</div>	</div>

<xi:include href="../../layout.html" />	<xi:include href="../../layout.html" />
</html>	</html>

file:b/ckanext/ga_report/templates/ga_report/site/downloads.html (new)

	<html xmlns:py="http://genshi.edgewall.org/"
	xmlns:i18n="http://genshi.edgewall.org/i18n"
	xmlns:xi="http://www.w3.org/2001/XInclude"
	py:strip="">

	<xi:include href="../ga_util.html" />

	<py:def function="page_title">Downloads</py:def>

	<py:match path="primarysidebar">
	<li py:if="c.downloads" class="widget-container boxed widget_text">
	<h4>Download</h4>
	<p><center>
	<a class="btn button btn-primary" href="${h.url_for(controller='ckanext.ga_report.controller:GaReport',action='csv_downloads',month=c.month or 'all')}">Download as CSV</a></center>
	</p>
	</li>
	<xi:include href="../notes.html" />

	</py:match>

	<div py:match="content">
	<h1>Downloads</h1>
	${usage_nav('Downloads')}

	<form class="form-inline" action="${h.url_for(controller='ckanext.ga_report.controller:GaReport',action='downloads')}" method="get">
	<div class="controls">

	${month_selector(c.month, c.months, c.day)}

	<input class="btn button btn-primary" type='submit' value="Update"/>
	</div>
	</form>

	<py:if test="c.downloads">
	${downloads_table(c.downloads)}
	</py:if>
	<py:if test="not c.downloads">
	<h4>No data</h4>
	<p>There is no download data available for this month</p>
	</py:if>
	</div>

	<xi:include href="../../layout.html" />

	<py:def function="optional_footer">
	<script type='text/javascript'>
	$('.dropdown-toggle').dropdown();
	$('.nav-tabs li a').click(function (e) {
	e.preventDefault();
	$(this).tab('show');
	})
	alert(window.location.hash);
	</script>
	</py:def>
	</html>

file:a/ckanext/ga_report/templates/ga_report/site/index.html -> file:b/ckanext/ga_report/templates/ga_report/site/index.html

<html xmlns:py="http://genshi.edgewall.org/"	<html xmlns:py="http://genshi.edgewall.org/"
xmlns:i18n="http://genshi.edgewall.org/i18n"	xmlns:i18n="http://genshi.edgewall.org/i18n"
xmlns:xi="http://www.w3.org/2001/XInclude"	xmlns:xi="http://www.w3.org/2001/XInclude"
py:strip="">	py:strip="">

<py:def function="page_title">Site analytics</py:def>	<xi:include href="../ga_util.html" />

	<py:def function="page_title">Site usage</py:def>

	<py:def function="optional_head">
	<script type="text/javascript" src="/scripts/vendor/jquery.sparkline.modified.js"></script>
	<style type="text/css">
	.table-condensed td.sparkline-cell {
	padding: 1px 0 0 0;
	width: 108px;
	text-align: center;
	}
	</style>
	</py:def>

<py:match path="primarysidebar">	<py:match path="primarysidebar">
<li class="widget-container boxed widget_text">	<li class="widget-container boxed widget_text">
<h4>Statistics</h4>	<h4>Download</h4>
<p>It is possible to <a href="${h.url_for(controller='ckanext.ga_report.controller:GaReport',action='csv',month=c.month)}">export the analytics data</a> as a CSV file, which contains all of the information for ${c.month_desc}</p>	<p><center>
	<a class="btn button btn-primary" href="${h.url_for(controller='ckanext.ga_report.controller:GaReport',action='csv',month=c.month or 'all')}">Download as CSV</a></center>
	</p>
	</li>
	<xi:include href="../notes.html" />

</li>
</py:match>	</py:match>

<div py:match="content">	<div py:match="content">
<h1>Site statistics</h1>	<h1>Site Usage</h1>
	${usage_nav('Site-wide')}

<form class="form-inline" action="${h.url_for(controller='ckanext.ga_report.controller:GaReport',action='index')}" method="get">	<form class="form-inline" action="${h.url_for(controller='ckanext.ga_report.controller:GaReport',action='index')}" method="get">
<div class="controls">	<div class="controls">
<select name="month">
<py:for each="val,desc in c.months">	${month_selector(c.month, c.months, c.day)}
<option value='${val}' py:attrs="{'selected': 'selected' if c.month == val else None}">${desc}</option>
</py:for>	<input class="btn button btn-primary" type='submit' value="Update"/>
</select>
<input class="btn button" type='submit' value="Update"/>
</div>	</div>
</form>	</form>

<div class="tabbable">	<div class="tabbable">
<ul class="nav nav-tabs">	<ul class="nav nav-tabs">
<li class="active"><a href="#totals" data-toggle="tab">Totals</a></li>	<li class="active"><a href="#totals" data-toggle="tab">Totals</a></li>
<li><a href="#browsers" data-toggle="tab">Browsers</a></li>	<li class="dropdown">
<li><a href="#os" data-toggle="tab">Operating Systems</a></li>	<a href="#browsers" class="dropdown-toggle" data-toggle="dropdown">Browsers
<li><a href="#social_networks" data-toggle="tab">Social Networks</a></li>	<b class="caret"></b></a>
	<ul class="dropdown-menu">
	<li><a href="#browsers_names" data-toggle="tab">Browsers</a></li>
	<li><a href="#browsers_versions" data-toggle="tab">Versions</a></li>
	</ul>
	</li>
	<li class="dropdown">
	<a href="#browsers" class="dropdown-toggle" data-toggle="dropdown">Operating<br/>Systems
	<b class="caret"></b></a>
	<ul class="dropdown-menu">
	<li><a href="#os" data-toggle="tab">Operating Systems</a></li>
	<li><a href="#os_versions" data-toggle="tab">Versions</a></li>
	</ul>
	</li>
	<li class="dropdown">
	<a href="#browsers" class="dropdown-toggle" data-toggle="dropdown">Social Networks
	<b class="caret"></b></a>
	<ul class="dropdown-menu">
	<li><a href="#social_networks" data-toggle="tab">All networks</a></li>
	<li><a href="#social_referrals_totals" data-toggle="tab">Referral links</a></li>
	</ul>
	</li>

	<li><a href="#social_networks" data-toggle="tab"></a></li>
<li><a href="#languages" data-toggle="tab">Languages</a></li>	<li><a href="#languages" data-toggle="tab">Languages</a></li>
<li><a href="#country" data-toggle="tab">Country</a></li>	<li><a href="#country" data-toggle="tab">Country</a></li>
</ul>	</ul>
<div class="tab-content">	<div class="tab-content">
<div class="tab-pane active" id="totals">	<div class="tab-pane active" id="totals">
<table class="table table-condensed table-bordered table-striped">	<table class="table table-condensed table-bordered table-striped">
<tr>	<tr>
<th>Name</th>	<th>Name</th>
<th>Value</th>	<th>Value</th>
	<th>History</th>
</tr>	</tr>
<py:for each="name, value in c.global_totals">	<py:for each="name, value, graph in c.global_totals">
<tr>	<tr>
<td>${name}</td>	<td>${name}</td>
<td>${value}</td>	<td>${value}</td>
	<td class="sparkline-cell">
	<span class="sparkline" sparkTooltips="${','.join([x for x,y in graph])}">
	${','.join([y for x,y in graph])}
	</span>
	</td>
</tr>	</tr>
</py:for>	</py:for>
</table>	</table>
</div>	</div>
<div class="tab-pane" id="browsers">	<div class="tab-pane" id="browsers_versions">
<table class="table table-condensed table-bordered table-striped">	<p>Note: Where a browser has a large number of versions, these have been grouped together.</p>
<tr>	${stat_table(c.browser_versions)}
<th>Name</th>	</div>
<th>Value</th>	<div class="tab-pane" id="browsers_names">
</tr>	${stat_table(c.browsers)}
<py:for each="name, value in c.browsers">	</div>
<tr>	<div class="tab-pane" id="os">
<td>${name}</td>	${stat_table(c.os)}
<td>${value}</td>	</div>
</tr>	<div class="tab-pane" id="os_versions">
</py:for>	${stat_table(c.os_versions)}
</table>	</div>
</div>	<div class="tab-pane" id="social_referrals_totals">
<div class="tab-pane" id="os">	<p>Number of visits that were referred from social networks</p>
<table class="table table-condensed table-bordered table-striped">	${social_table(c.social_referrer_totals)}
<tr>
<th>Name</th>
<th>Value</th>
</tr>
<py:for each="name, value in c.os">
<tr>
<td>${name}</td>
<td>${value}</td>
</tr>
</py:for>
</table>
</div>	</div>
<div class="tab-pane" id="social_networks">	<div class="tab-pane" id="social_networks">
<table class="table table-condensed table-bordered table-striped">	<p>Percentage of visits that were referred from these social networks</p>
<tr>
<th>Name</th>	${stat_table(c.social_networks, 'Visits')}
<th>Value</th>
</tr>
<py:for each="name, value in c.social_networks">
<tr>
<td>${name}</td>
<td>${value}</td>
</tr>
</py:for>
</table>
</div>	</div>
<div class="tab-pane" id="languages">	<div class="tab-pane" id="languages">
<table class="table table-condensed table-bordered table-striped">	${stat_table(c.languages)}
<tr>
<th>Name</th>
<th>Value</th>
</tr>
<py:for each="name, value in c.languages">
<tr>
<td>${name}</td>
<td>${value}</td>
</tr>
</py:for>
</table>
</div>	</div>
<div class="tab-pane" id="country">	<div class="tab-pane" id="country">
<table class="table table-condensed table-bordered table-striped">	${stat_table(c.country)}
<tr>
<th>Name</th>
<th>Value</th>
</tr>
<py:for each="name, value in c.country">
<tr>
<td>${name}</td>
<td>${value}</td>
</tr>
</py:for>
</table>
</div>	</div>


</div>	</div>
</div>	</div>



</div>	</div>

<xi:include href="../../layout.html" />

<py:def function="optional_footer">	<py:def function="optional_footer">
<script type='text/javascript'>	<script type="text/javascript">
$('.nav-tabs li a').click(function (e) {	$(function() {
e.preventDefault();	var sparkOptions = {
$(this).tab('show');	enableTagOptions: true,
})	type: 'line',
	width: 100,
	height: 26,
	chartRangeMin: 0,
	spotColor: '',
	maxSpotColor: '',
	minSpotColor: '',
	highlightSpotColor: '000000',
	lineColor: '3F8E6D',
	fillColor: 'B7E66B'
	};
	$('.sparkline').sparkline('html',sparkOptions);
	});
</script>	</script>
</py:def>	</py:def>

	<xi:include href="../../layout.html" />
</html>	</html>

file:b/ckanext/ga_report/tests/test_download.py (new)

	from nose.tools import assert_equal

	from ckanext.ga_report.download_analytics import DownloadAnalytics

	_filter_browser_version = DownloadAnalytics._filter_browser_version

	class TestBrowserVersionFilter:
	def test_chrome(self):
	assert_equal(_filter_browser_version('Chrome', u'6.0.472.0'), '6')
	def test_firefox(self):
	assert_equal(_filter_browser_version('Firefox', u'16.1'), '16')
	def test_safari(self):
	assert_equal(_filter_browser_version('Safari', u'534.55.3'), '53X')
	assert_equal(_filter_browser_version('Safari', u'1534.55.3'), '15XX')
	def test_ie(self):
	assert_equal(_filter_browser_version('Internet Explorer', u'8.0'), '8')
	def test_opera_mini(self):
	assert_equal(_filter_browser_version('Opera Mini', u'6.5.27431'), '6')
	def test_opera(self):
	assert_equal(_filter_browser_version('Opera', u'11.60'), '11')

	class TestDownloadAnalytics:
	def test_filter_out_long_tail(self):
	data = {'Firefox': 100,
	'Obscure Browser': 5,
	'Chrome': 150}
	DownloadAnalytics._filter_out_long_tail(data, 10)
	assert_equal(data, {'Firefox': 100,
	'Chrome': 150})

file:b/ckanext/ga_report/tests/test_model.py (new)

	from nose.tools import assert_equal

	from ckanext.ga_report.ga_model import _normalize_url

	class TestNormalizeUrl:
	def test_normal(self):
	assert_equal(_normalize_url('http://data.gov.uk/dataset/weekly_fuel_prices'),
	'/dataset/weekly_fuel_prices')

	def test_www_dot(self):
	assert_equal(_normalize_url('http://www.data.gov.uk/dataset/weekly_fuel_prices'),
	'/dataset/weekly_fuel_prices')

	def test_https(self):
	assert_equal(_normalize_url('https://data.gov.uk/dataset/weekly_fuel_prices'),
	'/dataset/weekly_fuel_prices')

file:a/setup.py -> file:b/setup.py

from setuptools import setup, find_packages	from setuptools import setup, find_packages
import sys, os	import sys, os

version = '0.1'	version = '0.1'

setup(	setup(
name='ckanext-ga-report',	name='ckanext-ga-report',
version=version,	version=version,
description="GA reporting for CKAN",	description="GA reporting for CKAN",
long_description="""\	long_description="""\
""",	""",
classifiers=[], # Get strings from http://pypi.python.org/pypi?%3Aaction=list_classifiers	classifiers=[], # Get strings from http://pypi.python.org/pypi?%3Aaction=list_classifiers
keywords='',	keywords='',
author='David Read',	author='David Read',
author_email='david.read@hackneyworkshop.com',	author_email='david.read@hackneyworkshop.com',
url='',	url='',
license='',	license='',
packages=find_packages(exclude=['ez_setup', 'examples', 'tests']),	packages=find_packages(exclude=['ez_setup', 'examples', 'tests']),
namespace_packages=['ckanext', 'ckanext.ga_report'],	namespace_packages=['ckanext', 'ckanext.ga_report'],
include_package_data=True,	include_package_data=True,
zip_safe=False,	zip_safe=False,
install_requires=[	install_requires=[
'gdata',	'gdata',
'google-api-python-client'	'google-api-python-client'
],	],
entry_points=\	entry_points=\
"""	"""
[ckan.plugins]	[ckan.plugins]
# Add plugins here, eg	# Add plugins here
ga-report=ckanext.ga_report.plugin:GAReportPlugin	ga-report=ckanext.ga_report.plugin:GAReportPlugin

[paste.paster_command]	[paste.paster_command]
loadanalytics = ckanext.ga_report.command:LoadAnalytics	loadanalytics = ckanext.ga_report.command:LoadAnalytics
initdb = ckanext.ga_report.command:InitDB	initdb = ckanext.ga_report.command:InitDB
getauthtoken = ckanext.ga_report.command:GetAuthToken	getauthtoken = ckanext.ga_report.command:GetAuthToken
	fixtimeperiods = ckanext.ga_report.command:FixTimePeriods
""",	""",
)	)

import os	import os
import httplib2	import httplib2
from apiclient.discovery import build	from apiclient.discovery import build
from oauth2client.client import flow_from_clientsecrets	from oauth2client.client import flow_from_clientsecrets
from oauth2client.file import Storage	from oauth2client.file import Storage
from oauth2client.tools import run	from oauth2client.tools import run

from pylons import config	from pylons import config


def _prepare_credentials(token_filename, credentials_filename):	def _prepare_credentials(token_filename, credentials_filename):
"""	"""
Either returns the user's oauth credentials or uses the credentials	Either returns the user's oauth credentials or uses the credentials
file to generate a token (by forcing the user to login in the browser)	file to generate a token (by forcing the user to login in the browser)
"""	"""
storage = Storage(token_filename)	storage = Storage(token_filename)
credentials = storage.get()	credentials = storage.get()

if credentials is None or credentials.invalid:	if credentials is None or credentials.invalid:
flow = flow_from_clientsecrets(credentials_filename,	flow = flow_from_clientsecrets(credentials_filename,
scope='https://www.googleapis.com/auth/analytics.readonly',	scope='https://www.googleapis.com/auth/analytics.readonly',
message="Can't find the credentials file")	message="Can't find the credentials file")
credentials = run(flow, storage)	credentials = run(flow, storage)

return credentials	return credentials


def init_service(token_file, credentials_file):	def init_service(token_file, credentials_file):
"""	"""
Given a file containing the user's oauth token (and another with	Given a file containing the user's oauth token (and another with
credentials in case we need to generate the token) will return a	credentials in case we need to generate the token) will return a
service object representing the analytics API.	service object representing the analytics API.
"""	"""
http = httplib2.Http()	http = httplib2.Http()

credentials = _prepare_credentials(token_file, credentials_file)	credentials = _prepare_credentials(token_file, credentials_file)
http = credentials.authorize(http) # authorize the http object	http = credentials.authorize(http) # authorize the http object

return build('analytics', 'v3', http=http)	return build('analytics', 'v3', http=http)


def get_profile_id(service):	def get_profile_id(service):
"""	"""
Get the profile ID for this user and the service specified by the	Get the profile ID for this user and the service specified by the
'googleanalytics.id' configuration option. This function iterates	'googleanalytics.id' configuration option. This function iterates
over all of the accounts available to the user who invoked the	over all of the accounts available to the user who invoked the
service to find one where the account name matches (in case the	service to find one where the account name matches (in case the
user has several).	user has several).
"""	"""
accounts = service.management().accounts().list().execute()	accounts = service.management().accounts().list().execute()

if not accounts.get('items'):	if not accounts.get('items'):
return None	return None

accountName = config.get('googleanalytics.account')	accountName = config.get('googleanalytics.account')
	if not accountName:
	raise Exception('googleanalytics.account needs to be configured')
webPropertyId = config.get('googleanalytics.id')	webPropertyId = config.get('googleanalytics.id')
	if not webPropertyId:
	raise Exception('googleanalytics.id needs to be configured')
for acc in accounts.get('items'):	for acc in accounts.get('items'):
if acc.get('name') == accountName:	if acc.get('name') == accountName:
accountId = acc.get('id')	accountId = acc.get('id')

webproperties = service.management().webproperties().list(accountId=accountId).execute()	webproperties = service.management().webproperties().list(accountId=accountId).execute()

profiles = service.management().profiles().list(	profiles = service.management().profiles().list(
accountId=accountId, webPropertyId=webPropertyId).execute()	accountId=accountId, webPropertyId=webPropertyId).execute()

if profiles.get('items'):	if profiles.get('items'):
return profiles.get('items')[0].get('id')	return profiles.get('items')[0].get('id')

return None	return None