Merge branch 'master' of ssh://apples.lambdacomplex.org/git/disclosr
Former-commit-id: 3df5c7f8ae2e72a7e4ef02c533d19f602f90e201
--- a/about.php
+++ b/about.php
@@ -4,19 +4,16 @@
?>
<div class="foundation-header">
<h1><a href="about.php">About/FAQ</a></h1>
- <h4 class="subheader">Lorem ipsum.</h4>
</div>
-<h2> What is this? </h2>
-Disclo.gs is a project to monitor Australian Federal Government agencies
-compliance with their <a href="http://www.oaic.gov.au/publications/other_operational/foi_policy_frequently_asked_questions.html#_Toc291837571">"proactive disclosure requirements" to make a transparency league table as suggested by gov2 taskforce http://gov2.net.au/blog/2009/09/19/a-league-ladder-of-psi-openness/</a>.
+<a href="http://thenounproject.com/noun/document-dump/">Document Dump icon designed by Iconathon, 2013</a>
+Contact us: maxious@lambdacomplex.org
+
+Exports: <a href="admin/exportAll.csv.php">All Agencies</a> <a href="admin/exportEmployees.csv.php">Agency Employee Headcounts</a>
<h2> Attributions </h2>
National Archives of Australia, Australian Governments’ Interactive Functions Thesaurus, 2nd edition, September 2005, published at http://www.naa.gov.au/recordkeeping/thesaurus/index.htm <br/>
data.gov.au http://data.gov.au/dataset/directory-gov-au-full-data-export/ <br/>
directory.gov.au <br/>
australia.gov.au http://australia.gov.au/about/copyright <br/>
-<h2> Open everything </h2>
-All documents released CC-BY 3 AU
-Open source git @
<h2>Organisational Data Sources</h2>
@@ -25,17 +22,17 @@
http://www.finance.gov.au/publications/flipchart/docs/FMACACFlipchart.pdf summarises these. view-source:https://www.tenders.gov.au/?event=public.advancedsearch.home is great for the suspended/active status<br>
-Fraud in gov depts by Fairfax Media http://www.smh.com.au/national/public-service-keeps-fraud-cases-private-20110923-1kpdr.html
+Fraud in gov depts by Fairfax Media http://www.smh.com.au/national/public-service-keeps-fraud-cases-private-20110923-1kpdr.html <br>
When defining the hierachy, this system is designed towards monitoring accountablity. Thus large agencies that have registered their own ABN
-and have their own accountablity mechanisms/website receive a seperate record as a child of their department.
+and have their own accountablity mechanisms/website receive a seperate record as a child of their department.<br>
Some small agencies will choose to simply rely on their parent department's accountablity measures.<br>
-This flows through to organisation name and other/past names. A department that completely accounts for an agency will list that agency as an other child name.
-As agencies themselves shift between departments, there may be scope for providing time ranges but typically the newest hierarchy will be the one recorded.
+This flows through to organisation name and other/past names. A department that completely accounts for an agency will list that agency as an other child name.<br>
+As agencies themselves shift between departments, there may be scope for providing time ranges but typically the newest hierarchy will be the one recorded.<br>
A department/agency name will be the newest active name assigned to that ABN.<br>
-ABN information is derived from the ABR. This is the definitive umpire about which former name should be linked to which current name.
+ABN information is derived from the ABR. This is the definitive umpire about which former name should be linked to which current name. <br>
For example "Department of Transport and Regional Services" became "Department of Infrastructure, Transport, Regional Development and Local Government" (same ABN)
however it later split into "Department of Infrastructure and Transport" (same ABN)
and "Department of Regional Australia, Regional Development and Local Government" (new ABN).<br>
@@ -43,24 +40,7 @@
Statistical information from http://www.apsc.gov.au/stateoftheservice/1011/statsbulletin/section1.html#t2total https://www.apsedii.gov.au/apsedii/CustomQueryx33.shtml
and individual annual reports.<br>
-<h2>Webpage Assessment</h2>
-Much due care has been put into correctly recording disclosure URLs. Typically the "About", "Corporate", "Publications" and "Sitemap" sections are checked at the very least.
-Occasionally it is nessicary to use a site or Google search. In several rare cases, there is a secret "Disclosure" navigation menu you can find if you find one of the mandatory publishing obligations in that category (seriously).<br>
-Some rules about leniency:<br>
-<ul>
- <li>An empty FOI disclosure log counts, a page outlining what the FOI Act is does not.</li>
- <li>A disclosure log in PDF or Word format counts :(</li>
- <li>An empty File/Record list counts (although that's very minimalistic that you have no files, electronic or paper)</li>
- <li>Only a current information publication scheme page counts, not a s.9 FOI Act page or an organisation chart.</li>
- <li>If there isn't a page easily listing all current and past Annual Reports, the most current one (html, pdf) counts.</li>
- <li>Consultancy contracts might not need it's own webpage (if in Annual Report), grants/appointments might not apply to all organisations but Legal Services Expenditure (and all other obligations) does need a webpage. </li>
-
-<h2>Open Government Scoring</h2>
-+1 point for every true Has... attribute<br>
--1 point for every false Has... (ie. Has Not) attribute</br>
-
-Don't like this? Make your own score, suggest a better scoring mechanism.</br>
-
<?php
include_footer();
?>
+
--- a/admin/refreshDesignDoc.php
+++ b/admin/refreshDesignDoc.php
@@ -40,6 +40,9 @@
$obj->views->byURL->map = "function(doc) {\n emit(doc.url, doc);\n}";
$obj->views->agency->map = "function(doc) {\n emit(doc.agencyID, doc);\n}";
$obj->views->byWebServer->map = "function(doc) {\n emit(doc.web_server, doc);\n}";
+
+$obj->views->datasets->map = "function(doc) {\nif (doc.fieldName == \"data\") {\n emit(doc._id, doc);\n}\n}";
+$obj->views->datasetGroups->map = "function(doc) {\nif (doc.fieldName == \"data\") {\n doc.metadata[\"data.gov.au Category\"] && doc.metadata[\"data.gov.au Category\"].forEach(function(tag) {\n emit(tag, doc.url); \n });\n}\n}";
$obj->views->getValidationRequired->map = "function(doc) {\nif (doc.mime_type == \"text/html\" \n&& typeof(doc.validation) == \"undefined\") {\n emit(doc._id, doc._attachments);\n}\n}";
$docdb->save($obj, true);
--- a/bubbletree.php
+++ /dev/null
@@ -1,126 +1,1 @@
-<!DOCTYPE html>
-<html xmlns="http://www.w3.org/1999/xhtml">
- <head>
- <meta charset="UTF-8"/>
- <title>Minimal BubbleTree Demo</title>
- <script type="text/javascript" src="http://code.jquery.com/jquery-1.7.2.js"></script>
- <script type="text/javascript" src="js/bubbletree/lib/jquery.history.js"></script>
- <script type="text/javascript" src="js/bubbletree/lib/raphael.js"></script>
- <script type="text/javascript" src="js/bubbletree/lib/vis4.js"></script>
- <script type="text/javascript" src="js/bubbletree/lib/Tween.js"></script>
- <script type="text/javascript" src="js/bubbletree/build/bubbletree.js"></script>
- <link rel="stylesheet" type="text/css" href="js/bubbletree/build/bubbletree.css" />
- <script type="text/javascript" src="js/bubbletree/styles/cofog.js"></script>
-
-
- <script type="text/javascript">
-
- $(function() {
-<?php
-include_once('include/common.inc.php');
-
-include("lib/Color.php");
-$color = new Lux_Color();
-
-$portfolios = Array();
-$total = 0;
-$db = $server->get_db('disclosr-agencies');
-try {
- $rows = $db->get_view("app", "byDeptStateName", null, true)->rows;
- foreach ($rows as $row) {
- $portfolios[trim(str_replace(Array("Department of", "Department", "the", "'", "`"), "", $row->key))] = $row->value;
- }
-} catch (SetteeRestClientException $e) {
- setteErrorHandler($e);
-}
-
-$agencies = Array();
-try {
- $rows = $db->get_view("app", "byCanonicalName", null, true)->rows;
-//print_r($rows);
- foreach ($rows as $row) {
- $employees = 0;
- $portfolioid = 0;
- if (isset($row->value->employees)) {
- $employees = $row->value->employees;
- }
- if (isset($row->value->statistics->employees)) {
- $agencyEmployeesArray = object_to_array($row->value->statistics->employees);
- if (isset($agencyEmployeesArray["2010-2011"]["value"])) {
- $employees = $agencyEmployeesArray["2010-2011"]["value"];
- } else {
- // bailout for agencies that are closed for business
- continue;
- }
- }
- if (!($employees > 0)) {
- $employees = 0;
- }
- if (isset($row->value->parentOrg)) {
- $portfolioid = $row->value->parentOrg;
- }
- if (isset($row->value->orgType) && $row->value->orgType == "FMA-DepartmentOfState") {
- $portfolioid = $row->id;
- }
- $agencies[$portfolioid][$row->value->name] = $employees;
- }
-} catch (SetteeRestClientException $e) {
- setteErrorHandler($e);
-}
-//print_r($portfolios);
-//print_r($agencies);
-
-// http://martin.ankerl.com/2009/12/09/how-to-create-random-colors-programmatically/
-$golden_ratio_conjugate = 0.618033988749895;
-$h = 0.00+rand(0,10)/10; # use random start value
-foreach ($portfolios as $portfolioName => $portfolioID) {
- $h += $golden_ratio_conjugate;
-
- $h = fmod($h,1);
- $portfolioColor = $color->hsv2hex(Array($h, .3, .99));
- $subnodes = Array();
- $portfolioEmployees = 0;
- foreach ($agencies[$portfolioID] as $agencyName => $agencyEmployees) {
- $agencyColor = $color->hsv2hex(Array($h / 10, rand(1, 10) / 10, abs(($h * (1 / 10)) - .5) + .5));
- $subnodes[] = Array(
- "label" => str_replace(Array("'", "`"), "", $agencyName),
- "amount" => $agencyEmployees,
- //"color" => "#" . $agencyColor
- );
- $portfolioEmployees += $agencyEmployees;
- }
- $nodes[] = Array(
- "label" => $portfolioName,
- "amount" => $portfolioEmployees,
- //"color" => "#" . $portfolioColor,
- "children" => $subnodes
- );
- $total += $portfolioEmployees;
-}
-$data = Array(
- "label" => "Australian Federal Government",
- "amount" => $total,
- //"color" => "#000000",
- "children" => $nodes
-);
-echo "var data =eval('('+'" . json_encode($data) . "'+')');";
-?>
-
- new BubbleTree({
- data: data,
- container: '.bubbletree'
- });
-
-
- });
-
- </script>
- </head>
- <body>
- <div class="bubbletree-wrapper">
- <div class="bubbletree"></div>
- </div>
- </body>
-</html>
-
--- /dev/null
+++ b/budget.php
@@ -1,1 +1,124 @@
+<!DOCTYPE html>
+<html xmlns="http://www.w3.org/1999/xhtml">
+ <head>
+ <meta charset="UTF-8"/>
+ <title>Minimal BubbleTree Demo</title>
+ <script type="text/javascript" src="http://code.jquery.com/jquery-1.7.2.js"></script>
+ <script type="text/javascript" src="js/bubbletree/lib/jquery.history.js"></script>
+ <script type="text/javascript" src="js/bubbletree/lib/raphael.js"></script>
+ <script type="text/javascript" src="js/bubbletree/lib/vis4.js"></script>
+ <script type="text/javascript" src="js/bubbletree/lib/Tween.js"></script>
+ <script type="text/javascript" src="js/bubbletree/build/bubbletree.js"></script>
+ <link rel="stylesheet" type="text/css" href="js/bubbletree/build/bubbletree.css" />
+ <script type="text/javascript" src="js/bubbletree/styles/cofog.js"></script>
+
+
+ <script type="text/javascript">
+
+ $(function() {
+<?php
+include_once('include/common.inc.php');
+
+include("lib/Color.php");
+$color = new Lux_Color();
+
+$portfolios = Array();
+$total = 0;
+$db = $server->get_db('disclosr-agencies');
+try {
+ $rows = $db->get_view("app", "byDeptStateName", null, true)->rows;
+ foreach ($rows as $row) {
+ $portfolios[trim(str_replace(Array("Department of", "Department", "the", "'", "`"), "", $row->key))] = $row->value;
+ }
+} catch (SetteeRestClientException $e) {
+ setteErrorHandler($e);
+}
+
+$agencies = Array();
+try {
+ $rows = $db->get_view("app", "byCanonicalName", null, true)->rows;
+//print_r($rows);
+ foreach ($rows as $row) {
+ $employees = 0;
+ $portfolioid = 0;
+ if (isset($row->value->statistics->budget)) {
+ $agencyEmployeesArray = object_to_array($row->value->statistics->budget);
+ //print_r($agencyEmployeesArray);
+ if (isset($agencyEmployeesArray["2011-2012"]["value"])) {
+ $employees = $agencyEmployeesArray["2011-2012"]["value"];
+ } else {
+ // bailout for agencies that are closed for business
+ continue;
+ }
+ }
+ if (!($employees > 0)) {
+ $employees = 0;
+ }
+ if (isset($row->value->parentOrg)) {
+ $portfolioid = $row->value->parentOrg;
+ }
+ if (isset($row->value->orgType) && $row->value->orgType == "FMA-DepartmentOfState") {
+ $portfolioid = $row->id;
+ }
+ $agencies[$portfolioid][$row->value->name] = $employees;
+ }
+} catch (SetteeRestClientException $e) {
+ setteErrorHandler($e);
+}
+//print_r($portfolios);
+//print_r($agencies);
+
+// http://martin.ankerl.com/2009/12/09/how-to-create-random-colors-programmatically/
+$golden_ratio_conjugate = 0.618033988749895;
+$h = 0.00+rand(0,10)/10; # use random start value
+foreach ($portfolios as $portfolioName => $portfolioID) {
+ $h += $golden_ratio_conjugate;
+
+ $h = fmod($h,1);
+ $portfolioColor = $color->hsv2hex(Array($h, .3, .99));
+ $subnodes = Array();
+ $portfolioEmployees = 0;
+ foreach ($agencies[$portfolioID] as $agencyName => $agencyEmployees) {
+ $agencyColor = $color->hsv2hex(Array($h / 10, rand(1, 10) / 10, abs(($h * (1 / 10)) - .5) + .5));
+ $subnodes[] = Array(
+ "label" => str_replace(Array("'", "`"), "", $agencyName),
+ "amount" => $agencyEmployees,
+ //"color" => "#" . $agencyColor
+ );
+ $portfolioEmployees += $agencyEmployees;
+ }
+ $nodes[] = Array(
+ "label" => $portfolioName,
+ "amount" => $portfolioEmployees,
+ //"color" => "#" . $portfolioColor,
+ "children" => $subnodes
+ );
+ $total += $portfolioEmployees;
+}
+$data = Array(
+ "label" => "Australian Federal Government",
+ "amount" => $total,
+ //"color" => "#000000",
+ "children" => $nodes
+);
+echo "var data =eval('('+'" . json_encode($data) . "'+')');";
+?>
+
+ new BubbleTree({
+ data: data,
+ container: '.bubbletree'
+ });
+
+
+ });
+
+ </script>
+ </head>
+ <body>
+ <div class="bubbletree-wrapper">
+ <div class="bubbletree"></div>
+ </div>
+ </body>
+</html>
+
--- a/charts.php
+++ /dev/null
@@ -1,133 +1,1 @@
-<?php
-include_once('include/common.inc.php');
-include_header('Charts');
-$db = $server->get_db('disclosr-agencies');
-?>
-<div class="foundation-header">
- <h1><a href="about.php">Charts</a></h1>
- <h4 class="subheader">Lorem ipsum.</h4>
-</div>
-<div id="scores" style="width:900px;height:500px;"></div>
-<div id="employees" style="width:1000px;height:900px;"></div>
-<script id="source">
- window.onload = function() {
- $(document).ready(function() {
- var d1 = [];
- var scorelabels = [];
- <?php
- try {
- $rows = $db->get_view("app", "scoreHas?group=true", null, true)->rows;
-
- $dataValues = Array();
- foreach ($rows as $row) {
- $dataValues[$row->value] = $row->key;
- }
- $i = 0;
- ksort($dataValues);
- foreach ($dataValues as $value => $key) {
-
- echo " d1.push([$i, $value]);" . PHP_EOL;
- echo " scorelabels.push('$key');" . PHP_EOL;
- $i++;
- }
- } catch (SetteeRestClientException $e) {
- setteErrorHandler($e);
- }
- ?>
- function scoretrackformatter(obj) {
- if (scorelabels[Math.floor(obj.x)]) {
- return (scorelabels[Math.floor(obj.x)])+"="+obj.y;
-
- } else {
- return "";
- }
- }
- function scoretickformatter(val, axis) {
- if (scorelabels[Math.floor(val)]) {
- return '<p style="margin-top:8em;-webkit-transform:rotate(-90deg);">'+(scorelabels[Math.floor(val)])+"</b>";
-
- } else {
- return "";
- }
- }
- Flotr.draw(document.getElementById("scores"), [ {data: d1}], {
- HtmlText: true,
- bars : {
- show : true
- },
- mouse : {
- track : true,
- relative : true,
- trackFormatter: scoretrackformatter
- },yaxis: {
- autoscaling: true
- },
- xaxis: {
- autoscaling: true,
- minorTickFreq: 0.6,
- noTicks : scorelabels.length,
- tickFormatter: scoretickformatter
- }
- });
-
-
-
-
-
-
-
-
-var emplabels = [];
-function emptrackformatter(obj) {
-
- return (obj.series.label)+" = "+obj.y+" in "+emplabels[Math.floor(obj.x)];
-
- }
- function emptickformatter(val, axis) {
- if (emplabels[Math.floor(val)]) {
- return '<p style="margin-top:8em;-webkit-transform:rotate(-90deg);">'+(emplabels[Math.floor(val)])+"</b>";
-
- } else {
- return "";
- }
- }
-function onDataReceived(series) {
- emplabels = series.labels;
- Flotr.draw(document.getElementById("employees"), series.data, {
- mouse : {
- track : true,
- relative : true,
- trackFormatter: emptrackformatter
- },yaxis: {
- max: 10000,
- scaling: 'logarithmic'
- },
- xaxis: {
- minorTickFreq: 1,
- noTicks: emplabels.length,
- showMinorLabels: true,
- tickFormatter: emptickformatter
- },
- legend: {
- show: false
- }
- });
- }
-
- $.ajax({
- url: "admin/exportEmployees.csv.php?format=json",
- method: 'GET',
- dataType: 'json',
- success: onDataReceived
- });
-
-
- });
- };
-
-</script>
-
-<?php
-include_footer();
-?>
--- /dev/null
+++ b/disclosr.iml
@@ -1,1 +1,16 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<module type="WEB_MODULE" version="4">
+ <component name="FacetManager">
+ <facet type="Python" name="Python">
+ <configuration sdkName="" />
+ </facet>
+ </component>
+ <component name="NewModuleRootManager" inherit-compiler-output="true">
+ <exclude-output />
+ <content url="file://$MODULE_DIR$" />
+ <orderEntry type="inheritedJdk" />
+ <orderEntry type="sourceFolder" forTests="false" />
+ </component>
+</module>
+
--- a/documents/charts.php
+++ b/documents/charts.php
@@ -13,10 +13,10 @@
?>
<div class="foundation-header">
<h1><a href="about.php">Charts</a></h1>
- <h4 class="subheader">Lorem ipsum.</h4>
+ <h4 class="subheader"></h4>
</div>
<div id="bydate" style="width:1000px;height:300px;"></div>
-<div id="byagency" style="width:1200px;height:300px;"></div>
+<div id="byagency" style="width:1200px;height:800px;"></div>
<script id="source">
window.onload = function () {
$(document).ready(function () {
@@ -59,7 +59,7 @@
mode: 'x'
},
HtmlText: false,
- title: 'Time'
+ title: 'Disclosure Log entries added by Date'
};
// Draw graph with default options, overwriting with passed options
@@ -98,12 +98,12 @@
var agencylabels = [];
function agencytrackformatter(obj) {
- return agencylabels[Math.floor(obj.x)] + " = " + obj.y;
+ return agencylabels[Math.floor(obj.y)] + " = " + obj.x;
}
function agencytickformatter(val, axis) {
if (agencylabels[Math.floor(val)]) {
- return '<p style="margin-top:8em;-webkit-transform:rotate(-90deg);">' + (agencylabels[Math.floor(val)]) + "</b>";
+ return (agencylabels[Math.floor(val)]) ;
} else {
return "";
@@ -117,7 +117,7 @@
$dataValues = Array();
$i = 0;
foreach ($rows as $row) {
- echo " d2.push([".$i.", $row->value]);" . PHP_EOL;
+ echo " d2.push([ $row->value,$i]);" . PHP_EOL;
echo " agencylabels.push(['".str_replace("'","",$idtoname[$row->key])."']);" . PHP_EOL;
$i++;
@@ -131,9 +131,10 @@
document.getElementById("byagency"),
[d2],
{
+ title: "Disclosure Log entries by Agency",
bars: {
show: true,
- horizontal: false,
+ horizontal: true,
shadowSize: 0,
barWidth: 0.5
},
@@ -143,14 +144,14 @@
trackFormatter: agencytrackformatter
},
yaxis: {
- min: 0,
- autoscaleMargin: 1
- },
- xaxis: {
minorTickFreq: 1,
noTicks: agencylabels.length,
showMinorLabels: true,
tickFormatter: agencytickformatter
+ },
+ xaxis: {
+ min: 0,
+ autoscaleMargin: 1
},
legend: {
show: false
--- /dev/null
+++ b/documents/datagov-export-groups.py
@@ -1,1 +1,81 @@
+import ckanclient
+import couchdb
+from ckanclient import CkanApiError
+import re
+
+class LoaderError(Exception):
+ pass
+
+# Instantiate the CKAN client.
+#ckan = ckanclient.CkanClient(base_location='http://localhost:5000/api', api_key='b47b24cd-591d-40c1-8677-d73101d56d1b')
+api_key = 'ff34526e-f794-4068-8235-fcbba38cd8bc'
+ckan = ckanclient.CkanClient(base_location='http://data.disclosurelo.gs/api',
+ api_key=api_key)
+couch = couchdb.Server('http://127.0.0.1:5984/')
+#couch = couchdb.Server('http://192.168.1.113:5984/')
+
+# https://github.com/okfn/ckanext-importlib
+def munge(name):
+ # convert spaces to underscores
+ name = re.sub(' ', '_', name).lower()
+ # convert symbols to dashes
+ name = re.sub('[:]', '_-', name).lower()
+ name = re.sub('[/]', '-', name).lower()
+ # take out not-allowed characters
+ name = re.sub('[^a-zA-Z0-9-_]', '', name).lower()
+ # remove double underscores
+ name = re.sub('__', '_', name).lower()
+ return name
+
+
+def name_munge(input_name):
+ return munge(input_name.replace(' ', '').replace('.', '_').replace('&', 'and'))
+
+
+docsdb = couch['disclosr-documents']
+
+if __name__ == "__main__":
+ groups = {}
+ for doc in docsdb.view('app/datasetGroups'):
+ group_name = doc.key
+ if group_name != "Not specified":
+ pkg_name = filter(lambda x: x in '0123456789abcdefghijklmnopqrstuvwxyz-_',
+ doc.value.replace("http://data.gov.au/dataset/", '').replace('/', '')[:100]);
+ if group_name in groups.keys():
+ groups[group_name] = list(set(groups[group_name] + [pkg_name]))
+ else:
+ groups[group_name] = [pkg_name]
+
+ # add dataset to group(s)
+ for group_name in groups.keys():
+ if group_name != "Not specified":
+ group_url = name_munge(group_name[:100])
+ print group_name
+ print groups[group_name]
+ try:
+ # Update the group details
+ group_entity = ckan.group_entity_get(group_url)
+ print "group "+group_name+" exists"
+ if 'packages' in group_entity.keys():
+ group_entity['packages'] = list(set(group_entity['packages'] + groups[group_name]))
+ else:
+ group_entity['packages'] = groups[group_name]
+ ckan.group_entity_put(group_entity)
+ except CkanApiError, e:
+ if ckan.last_status == 404:
+ print "group "+group_name+" does not exist, creating"
+ group_entity = {
+ 'name': group_url,
+ 'title': group_name,
+ 'description': group_name,
+ 'packages': groups[group_name]
+ }
+ #print group_entity
+ ckan.group_register_post(group_entity)
+ elif ckan.last_status == 409:
+ print "group already exists"
+ else:
+ raise LoaderError('Unexpected status %s adding to group under \'%s\': %r' % (
+ ckan.last_status, pkg_name, e.args))
+
--- /dev/null
+++ b/documents/datagov-export.py
@@ -1,1 +1,318 @@
-
+# coding=utf-8
+import ckanclient
+import couchdb
+from ckanclient import CkanApiError
+import re
+import html2text # aaronsw :(
+import ckanapi # https://github.com/open-data/ckanapi
+import scrape
+import datetime, os, hashlib
+import urllib2
+
+class LoaderError(Exception):
+ pass
+
+import tempfile
+def add_package_resource_cachedurl(ckan, package_name, url, name, format, license_id, size,**kwargs):
+ if "xls" in url:
+ format = "xls"
+ if "pdf" in url:
+ format = "pdf"
+ if "xlsx" in url:
+ format = "xlsx"
+ (returned_url, mime_type, content) = scrape.fetchURL(scrape.docsdb,
+ url, "dataset_resource", "AGIMO", False)
+ if mime_type in ["application/vnd.ms-excel","application/msexcel","application/x-msexcel","application/x-ms-excel","application/x-excel","application/x-dos_ms_excel","application/xls","application/x-xls"]:
+ format = "xls"
+ if mime_type in ["application/xlsx","application/x-xlsx","application/vnd.openxmlformats-officedocument.spreadsheetml.sheet"]:
+ format = "xlsx"
+
+ if content != None:
+ tf = tempfile.NamedTemporaryFile(delete=False)
+ tfName = os.path.abspath(tf.name)
+ print tfName
+ tf.seek(0)
+ tf.write(content)
+ tf.flush()
+ ckan.add_package_resource (package_name, tfName, name=name, format=format, license_id=license_id)
+ else:
+ print "fetch error"
+ return ckan.add_package_resource(package_name, url, name=name, resource_type='data',
+ format=format,
+ size=size, mimetype=mime_type, license_id=license_id)
+
+# Instantiate the CKAN client.
+api_key = 'ff34526e-f794-4068-8235-fcbba38cd8bc'
+server = 'data.disclosurelo.gs'
+
+ckan = ckanclient.CkanClient(base_location='http://' + server + '/api',
+ api_key=api_key)
+ckandirect = ckanapi.RemoteCKAN('http://' + server, api_key=api_key)
+couch = couchdb.Server('http://127.0.0.1:5984/')
+#couch = couchdb.Server('http://192.168.1.113:5984/')
+
+import urllib
+import urlparse
+
+
+def url_fix(s, charset='utf-8'):
+ """Sometimes you get an URL by a user that just isn't a real
+ URL because it contains unsafe characters like ' ' and so on. This
+ function can fix some of the problems in a similar way browsers
+ handle data entered by the user:
+
+ >>> url_fix(u'http://de.wikipedia.org/wiki/Elf (Begriffsklärung)')
+ 'http://de.wikipedia.org/wiki/Elf%20%28Begriffskl%C3%A4rung%29'
+
+ :param charset: The target charset for the URL if the url was
+ given as unicode string.
+ """
+ if isinstance(s, unicode):
+ s = s.encode(charset, 'ignore')
+ if not urlparse.urlparse(s).scheme:
+ s = "http://" + s
+ scheme, netloc, path, qs, anchor = urlparse.urlsplit(s)
+ path = urllib.quote(path, '/%')
+ qs = urllib.quote_plus(qs, ':&=')
+ return urlparse.urlunsplit((scheme, netloc, path, qs, anchor))
+