Handle more http edge cases in scraper
Former-commit-id: 994d782d8883843a55bf2558f8e6a6c9ffbcebde
--- a/.gitmodules
+++ b/.gitmodules
@@ -10,4 +10,7 @@
[submodule "lib/php-diff"]
path = lib/php-diff
url = https://github.com/chrisboulton/php-diff.git
+[submodule "javascripts/flot"]
+ path = javascripts/flot
+ url = https://github.com/paradoxxxzero/flot.git
--- a/about.php
+++ b/about.php
@@ -13,48 +13,49 @@
Prometheus is the agent which polls agency websites to assess compliance.
<h2> Open everything </h2>
-all documents released CC-BY 3 AU
+All documents released CC-BY 3 AU
Open source git @
<h2>Organisational Data Sources</h2>
http://www.comlaw.gov.au/Browse/Results/ByTitle/AdministrativeArrangementsOrders/Current/Ad/0 defines departments
-Agencies can be found in the Schedule to an Appropriation Bill (budget), Schedule to FMA Regulations and/or Public Service Act.
+Agencies can be found in the Schedule to an Appropriation Bill (budget), Schedule to FMA Regulations and/or Public Service Act.<br>
-http://www.finance.gov.au/publications/flipchart/docs/FMACACFlipchart.pdf summarises these. view-source:https://www.tenders.gov.au/?event=public.advancedsearch.home is great for the suspended/active status
+http://www.finance.gov.au/publications/flipchart/docs/FMACACFlipchart.pdf summarises these. view-source:https://www.tenders.gov.au/?event=public.advancedsearch.home is great for the suspended/active status<br>
When defining the hierachy, this system is designed towards monitoring accountablity. Thus large agencies that have registered their own ABN
and have their own accountablity mechanisms/website receive a seperate record as a child of their department.
-Some small agencies will choose to simply rely on their parent department's accountablity measures.
+Some small agencies will choose to simply rely on their parent department's accountablity measures.<br>
This flows through to organisation name and other/past names. A department that completely accounts for an agency will list that agency as an other child name.
As agencies themselves shift between departments, there may be scope for providing time ranges but typically the newest hierarchy will be the one recorded.
-A department/agency name will be the newest active name assigned to that ABN.
+A department/agency name will be the newest active name assigned to that ABN.<br>
ABN information is derived from the ABR. This is the definitive umpire about which former name should be linked to which current name.
For example "Department of Transport and Regional Services" became "Department of Infrastructure, Transport, Regional Development and Local Government" (same ABN)
however it later split into "Department of Infrastructure and Transport" (same ABN)
-and "Department of Regional Australia, Regional Development and Local Government" (new ABN).
+and "Department of Regional Australia, Regional Development and Local Government" (new ABN).<br>
Statistical information from http://www.apsc.gov.au/stateoftheservice/1011/statsbulletin/section1.html#t2total https://www.apsedii.gov.au/apsedii/CustomQueryx33.shtml
-and individual annual reports.
+and individual annual reports.<br>
-Webpage Assessment
+<h2>Webpage Assessment</h2>
Much due care has been put into correctly recording disclosure URLs. Typically the "About", "Corporate", "Publications" and "Sitemap" sections are checked at the very least.
-Occasionally it is nessicary to use a site or Google search. In several rare cases, there is a secret "Disclosure" navigation menu you can find if you find one of the mandatory publishing obligations in that category (seriously).
-Some rules about leniency:
- An empty FOI disclosure log counts, a page outlining what the FOI Act is does not.
- A disclosure log in PDF or Word format counts :(
- An empty File/Record list counts (although that's very minimalistic that you have no files, electronic or paper)
- Only a current information publication scheme page counts, not a s.9 FOI Act page or an organisation chart.
- If there isn't a page easily listing all current and past Annual Reports, the most current one (html, pdf) counts.
- Consultancy contracts might not need it's own webpage (if in Annual Report), grants/appointments might not apply to all organisations but Legal Services Expenditure (and all other obligations) does need a webpage.
+Occasionally it is nessicary to use a site or Google search. In several rare cases, there is a secret "Disclosure" navigation menu you can find if you find one of the mandatory publishing obligations in that category (seriously).<br>
+Some rules about leniency:<br>
+<ul>
+ <li>An empty FOI disclosure log counts, a page outlining what the FOI Act is does not.</li>
+ <li>A disclosure log in PDF or Word format counts :(</li>
+ <li>An empty File/Record list counts (although that's very minimalistic that you have no files, electronic or paper)</li>
+ <li>Only a current information publication scheme page counts, not a s.9 FOI Act page or an organisation chart.</li>
+ <li>If there isn't a page easily listing all current and past Annual Reports, the most current one (html, pdf) counts.</li>
+ <li>Consultancy contracts might not need it's own webpage (if in Annual Report), grants/appointments might not apply to all organisations but Legal Services Expenditure (and all other obligations) does need a webpage. </li>
<h2>Open Government Scoring</h2>
-+1 point for every true Has... attribute
--1 point for every false Has... (ie. Has Not) attribute
++1 point for every true Has... attribute<br>
+-1 point for every false Has... (ie. Has Not) attribute</br>
-Don't like this? Make your own score, suggest a better scoring mechanism.
+Don't like this? Make your own score, suggest a better scoring mechanism.</br>
<?php
include_footer();
--- /dev/null
+++ b/admin/exportAll.csv.php
@@ -1,1 +1,63 @@
+<?php
+include_once("../include/common.inc.php");
+
+
+setlocale(LC_CTYPE, 'C');
+
+$headers = Array("#id", "name", "request_email", "short_name", "notes", "publication_scheme", "home_page", "tag_string");
+
+$db = $server->get_db('disclosr-agencies');
+$headers = Array();
+ try {
+ $rows = $db->get_view("app", "fieldNames?group=true", null, true)->rows;
+
+ $dataValues = Array();
+ foreach ($rows as $row) {
+ $headers[] = $row->key;
+ }
+} catch (SetteeRestClientException $e) {
+ setteErrorHandler($e);
+}
+
+$fp = fopen('php://output', 'w');
+if ($fp && $db) {
+ header('Content-Type: text/csv; charset=utf-8');
+ header('Content-Disposition: attachment; filename="export.' . date("c") . '.csv"');
+ header('Pragma: no-cache');
+ header('Expires: 0');
+ fputcsv($fp, $headers);
+ try {
+ $agencies = $db->get_view("app", "byCanonicalName", null, true)->rows;
+ //print_r($rows);
+ foreach ($agencies as $agency) {
+ // print_r($agency);
+
+ if ( !isset($agency->value->status)) {
+ $row = Array();
+ $agencyArray = object_to_array($agency->value);
+ foreach ($headers as $fieldName) {
+ if (isset($agencyArray[$fieldName])) {
+ if (is_array($agencyArray[$fieldName])) {
+ $row[] = implode(";",$agencyArray[$fieldName]);
+ } else {
+ $row[] = $agencyArray[$fieldName];
+ }
+ } else {
+ $row[] = "";
+ }
+ }
+
+ fputcsv($fp, array_values($row));
+
+
+ }
+ }
+ } catch (SetteeRestClientException $e) {
+ setteErrorHandler($e);
+ }
+
+ die;
+}
+?>
+
--- /dev/null
+++ b/charts.php
@@ -1,1 +1,102 @@
+<?php
+include_once('include/common.inc.php');
+include_header();
+$db = $server->get_db('disclosr-agencies');
+
+?>
+<div class="foundation-header">
+ <h1><a href="about.php">Charts</a></h1>
+ <h4 class="subheader">Lorem ipsum.</h4>
+</div>
+<div id="placeholder" style="width:900px;height:600px;"></div>
+<script id="source">
+window.onload = function() {
+ $(document).ready(function() {
+ var d1 = [];
+ var labels = [];
+ <?php
+ try {
+ $rows = $db->get_view("app", "scoreHas?group=true", null, true)->rows;
+
+ /*foreach ($rows as $key => $row) {
+ echo " d1.push([$key, {$row->value}]);".PHP_EOL;
+ echo " labels.push('{$row->key}');".PHP_EOL;
+ }*/
+ $dataValues = Array();
+ foreach ($rows as $row) {
+ $dataValues[$row->value] = $row->key;
+ }
+ $i = 0;
+ ksort($dataValues);
+ foreach($dataValues as $value => $key) {
+
+ echo " d1.push([$i, $value]);".PHP_EOL;
+ echo " labels.push('$key');".PHP_EOL;
+ $i++;
+ }
+} catch (SetteeRestClientException $e) {
+ setteErrorHandler($e);
+}
+?>
+
+ $.plot($("#placeholder"), [ d1], {
+ grid: { hoverable: true },
+
+ series: {
+ bars: { show: true, barWidth: 0.6 }
+ },
+ xaxis: {
+ tickFormatter: function formatter(val, axis) {
+ if (labels[val]) {
+ return(labels[val]);
+
+ } else {
+ return "";
+ }
+
+ },
+ labelAngle: 90
+ }
+ });
+
+var previousPoint = null;
+$("#placeholder").bind("plothover", function (event, pos, item) {
+ if (item) {
+ if (previousPoint != item.datapoint) {
+ previousPoint = item.datapoint;
+
+ $("#tooltip").remove();
+ var x = item.datapoint[0],
+ y = item.datapoint[1] - item.datapoint[2];
+
+ showTooltip(item.pageX, item.pageY, y );
+ }
+ }
+ else {
+ $("#tooltip").remove();
+ previousPoint = null;
+ }
+});
+
+
+
+});
+};
+ function showTooltip(x, y, contents) {
+ $('<div id="tooltip">' + contents + '</div>').css( {
+ position: 'absolute',
+ display: 'none',
+ top: y + 5,
+ left: x + 5,
+ border: '1px solid #fdd',
+ padding: '2px',
+ 'background-color': '#fee',
+ opacity: 0.80
+ }).appendTo("body").fadeIn(200);
+ }
+</script>
+
+<?php
+include_footer();
+?>
--- /dev/null
+++ b/couchdb/SetteeDatabase.class.php
@@ -1,1 +1,306 @@
-
+<?php
+
+/**
+* Databaase class.
+*/
+class SetteeDatabase {
+
+ /**
+ * Base URL of the CouchDB REST API
+ */
+ private $conn_url;
+
+ /**
+ * HTTP REST Client instance
+ */
+ protected $rest_client;
+
+ /**
+ * Name of the database
+ */
+ private $dbname;
+
+ /**
+ * Default constructor
+ */
+ function __construct($conn_url, $dbname) {
+ $this->conn_url = $conn_url;
+ $this->dbname = $dbname;
+ $this->rest_client = SetteeRestClient::get_instance($this->conn_url);
+ }
+
+
+ /**
+ * Get UUID from CouchDB
+ *
+ * @return
+ * CouchDB-generated UUID string
+ *
+ */
+ function gen_uuid() {
+ $ret = $this->rest_client->http_get('_uuids');
+ return $ret['decoded']->uuids[0]; // should never be empty at this point, so no checking
+ }
+
+ /**
+ * Create or update a document database
+ *
+ * @param $document
+ * PHP object, a PHP associative array, or a JSON String representing the document to be saved. PHP Objects and arrays are JSON-encoded automatically.
+ *
+ * <p>If $document has a an "_id" property set, it will be used as document's unique id (even for "create" operation).
+ * If "_id" is missing, CouchDB will be used to generate a UUID.
+ *
+ * <p>If $document has a "_rev" property (revision), document will be updated, rather than creating a new document.
+ * You have to provide "_rev" if you want to update an existing document, otherwise operation will be assumed to be
+ * one of creation and you will get a duplicate document exception from CouchDB. Also, you may not provide "_rev" but
+ * not provide "_id" since that is an invalid input.
+ *
+ * @param $allowRevAutoDetection
+ * Default: false. When true and _rev is missing from the document, save() function will auto-detect latest revision
+ * for a document and use it. This option is "false" by default because it involves an extra http HEAD request and
+ * therefore can make save() operation slightly slower if such auto-detection is not required.
+ *
+ * @return
+ * document object with the database id (uuid) and revision attached;
+ *
+ * @throws SetteeCreateDatabaseException
+ */
+ function save($document, $allowRevAutoDetection = false) {
+ if (is_string($document)) {
+ $document = json_decode($document);
+ }
+
+ // Allow passing of $document as an array (for syntactic simplicity and also because in JSON world it does not matter)
+ if(is_array($document)) {
+ $document = (object) $document;
+ }
+
+ if (empty($document->_id) && empty($document->_rev)) {
+ $id = $this->gen_uuid();
+ }
+ elseif (empty($document->_id) && !empty($document->_rev)) {
+ throw new SetteeWrongInputException("Error: You can not save a document with a revision provided, but missing id");
+ }
+ else {
+ $id = $document->_id;
+
+ if ($allowRevAutoDetection) {
+ try {
+ $rev = $this->get_rev($id);
+ } catch (SetteeRestClientException $e) {
+ // auto-detection may fail legitimately, if a document has never been saved before (new doc), so skipping error
+ }
+ if (!empty($rev)) {
+ $document->_rev = $rev;
+ }
+ }
+ }
+
+ $full_uri = $this->dbname . "/" . $this->safe_urlencode($id);
+ $document_json = json_encode($document, JSON_NUMERIC_CHECK);
+
+ $ret = $this->rest_client->http_put($full_uri, $document_json);
+
+ $document->_id = $ret['decoded']->id;
+ $document->_rev = $ret['decoded']->rev;
+
+ return $document;
+ }
+
+ /**
+ * @param $doc
+ * @param $name
+ * @param $content
+ * Content of the attachment in a string-buffer format. This function will automatically base64-encode content for
+ * you, so you don't have to do it.
+ * @param $mime_type
+ * Optional. Will be auto-detected if not provided
+ * @return void
+ */
+ public function add_attachment($doc, $name, $content, $mime_type = null) {
+ if (empty($doc->_attachments) || !is_object($doc->_attachments)) {
+ $doc->_attachments = new stdClass();
+ }
+
+ if (empty($mime_type)) {
+ $mime_type = $this->rest_client->content_mime_type($content);
+ }
+
+ $doc->_attachments->$name = new stdClass();
+ $doc->_attachments->$name->content_type = $mime_type;
+ $doc->_attachments->$name->data = base64_encode($content);
+ }
+
+ /**
+ * @param $doc
+ * @param $name
+ * @param $file
+ * Full path to a file (e.g. as returned by PHP's realpath function).
+ * @param $mime_type
+ * Optional. Will be auto-detected if not provided
+ * @return void
+ */
+ public function add_attachment_file($doc, $name, $file, $mime_type = null) {
+ $content = file_get_contents($file);
+ $this->add_attachment($doc, $name, $content, $mime_type);
+ }
+
+ /**
+ *
+ * Retrieve a document from CouchDB
+ *
+ * @throws SetteeWrongInputException
+ *
+ * @param $id
+ * Unique ID (usually: UUID) of the document to be retrieved.
+ * @return
+ * database document in PHP object format.
+ */
+ function get($id) {
+ if (empty($id)) {
+ throw new SetteeWrongInputException("Error: Can't retrieve a document without a uuid.");
+ }
+
+ $full_uri = $this->dbname . "/" . $this->safe_urlencode($id);
+$full_uri = str_replace("%3Frev%3D","?rev=",$full_uri);
+ $ret = $this->rest_client->http_get($full_uri);
+ return $ret['decoded'];
+ }
+
+ /**
+ *
+ * Get the latest revision of a document with document id: $id in CouchDB.
+ *
+ * @throws SetteeWrongInputException
+ *
+ * @param $id
+ * Unique ID (usually: UUID) of the document to be retrieved.
+ * @return
+ * database document in PHP object format.
+ */
+ function get_rev($id) {
+ if (empty($id)) {
+ throw new SetteeWrongInputException("Error: Can't query a document without a uuid.");
+ }
+
+ $full_uri = $this->dbname . "/" . $this->safe_urlencode($id);
+ $headers = $this->rest_client->http_head($full_uri);
+ if (empty($headers['Etag'])) {
+ throw new SetteeRestClientException("Error: could not retrieve revision. Server unexpectedly returned empty Etag");
+ }
+ $etag = str_replace('"', '', $headers['Etag']);
+ return $etag;
+ }
+
+ /**
+ * Delete a document
+ *
+ * @param $document
+ * a PHP object or JSON representation of the document that has _id and _rev fields.
+ *
+ * @return void
+ */
+ function delete($document) {
+ if (!is_object($document)) {
+ $document = json_decode($document);
+ }
+
+ $full_uri = $this->dbname . "/" . $this->safe_urlencode($document->_id) . "?rev=" . $document->_rev;
+ $this->rest_client->http_delete($full_uri);
+ }
+
+
+ /*----------------- View-related functions --------------*/
+
+ /**
+ * Create a new view or update an existing one.
+ *
+ * @param $design_doc
+ * @param $view_name
+ * @param $map_src
+ * Source code of the map function in Javascript
+ * @param $reduce_src
+ * Source code of the reduce function in Javascript (optional)
+ * @return void
+ */
+ function save_view($design_doc, $view_name, $map_src, $reduce_src = null) {
+ $obj = new stdClass();
+ $obj->_id = "_design/" . urlencode($design_doc);
+ $view_name = urlencode($view_name);
+ $obj->views->$view_name->map = $map_src;
+ if (!empty($reduce_src)) {
+ $obj->views->$view_name->reduce = $reduce_src;
+ }
+
+ // allow safe updates (even if slightly slower due to extra: rev-detection check).
+ return $this->save($obj, true);
+ }
+
+ /**
+ * Create a new view or update an existing one.
+ *
+ * @param $design_doc
+ * @param $view_name
+ * @param $key
+ * key parameter to a view. Can be a single value or an array (for a range). If passed an array, function assumes
+ * that first element is startkey, second: endkey.
+ * @param $descending
+ * return results in descending order. Please don't forget that if you are using a startkey/endkey, when you change
+ * order you also need to swap startkey and endkey values!
+ *
+ * @return void
+ */
+ function get_view($design_doc, $view_name, $key = null, $descending = false) {
+ $id = "_design/" . urlencode($design_doc);
+ $view_name = urlencode($view_name);
+ $id .= "/_view/$view_name";
+
+ $data = array();
+ if (!empty($key)) {
+ if (is_string($key)) {
+ $data = "key=" . '"' . $key . '"';
+ }
+ elseif (is_array($key)) {
+ list($startkey, $endkey) = $key;
+ $data = "startkey=" . '"' . $startkey . '"&' . "endkey=" . '"' . $endkey . '"';
+ }
+
+ if ($descending) {
+ $data .= "&descending=true";
+ }
+ }
+
+
+
+ if (empty($id)) {
+ throw new SetteeWrongInputException("Error: Can't retrieve a document without a uuid.");
+ }
+
+ $full_uri = $this->dbname . "/" . $this->safe_urlencode($id);
+$full_uri = str_replace("%253Fgroup%253Dtrue","?group=true",$full_uri);
+ $ret = $this->rest_client->http_get($full_uri, $data);
+ return $ret['decoded'];
+
+ }
+
+ /**
+ * @param $id
+ * @return
+ * return a properly url-encoded id.
+ */
+ private function safe_urlencode($id) {
+ //-- System views like _design can have "/" in their URLs.
+ $id = rawurlencode($id);
+ if (substr($id, 0, 1) == '_') {
+ $id = str_replace('%2F', '/', $id);
+ }
+ return $id;
+ }
+
+ /** Getter for a database name */
+ function get_name() {
+ return $this->dbname;
+ }
+
+}
--- a/getAgency.php
+++ b/getAgency.php
@@ -58,9 +58,7 @@
echo "<option value='{$row->value}'" . (($row->value == $value) ? "SELECTED" : "") . " >" . str_replace("Department of ", "", $row->key) . "</option>";
}
echo" </select>";
- } else if (strpos($key, "has") === 0) {
- echo "<label for='$key'><input type='checkbox' id='$key' name='$key' " . (($value == 'on' || $value == 'true') ? "checked='$value'" : "") . "> $key</label>";
- } else {
+ } else {
echo "<label>$key</label><input class='input-text' type='text' id='$key' name='$key' value='$value'/>";
if ((strpos($key, "URL") > 0 || $key == 'website') && $value != "") {
echo "<a href='$value'>view</a>";
@@ -80,11 +78,9 @@
foreach ($defaultFields as $defaultField) {
if (!isset($row[$defaultField])) {
if ($schemas['agency']['properties'][$defaultField]['type'] == "string") {
- if (strpos($defaultField, "has") === 0) {
- $row[$defaultField] = "false";
- } else {
+
$row[$defaultField] = "";
- }
+
}
if ($schemas['agency']['properties'][$defaultField]['type'] == "array") {
@@ -124,7 +120,7 @@
}
}
- $mode = "view";
+ $mode = "edit";
if ($mode == "edit") {
$row = addDefaultFields(object_to_array($row));
} else {
--- a/include/couchdb.inc.php
+++ b/include/couchdb.inc.php
@@ -26,14 +26,16 @@
emit(doc._id, doc.parentOrg);
}
};";
- $obj->views->byName->map = "function(doc) {
+ $obj->views->byName->map = 'function(doc) {
+ if (typeof(doc["status"]) == "undefined" || doc["status"] != "suspended") {
emit(doc.name, doc._id);
for (name in doc.otherNames) {
-if (doc.otherNames[name] != '' && doc.otherNames[name] != doc.name) {
+if (doc.otherNames[name] != "" && doc.otherNames[name] != doc.name) {
emit(doc.otherNames[name], doc._id);
}
}
-};";
+ }
+};';
$obj->views->foiEmails->map = "function(doc) {
emit(doc._id, doc.foiEmail);
@@ -80,24 +82,39 @@
emit("total", 1);
}
}';
- $obj->views->score->map = 'if(!String.prototype.startsWith){
+ $obj->views->scoreHas->map = 'if(!String.prototype.startsWith){
String.prototype.startsWith = function (str) {
return !this.indexOf(str);
}
}
-
+if(!String.prototype.endsWith){
+ String.prototype.endsWith = function(suffix) {
+ return this.indexOf(suffix, this.length - suffix.length) !== -1;
+ };
+}
function(doc) {
-count = 0;
if (typeof(doc["status"]) == "undefined" || doc["status"] != "suspended") {
for(var propName in doc) {
- if(typeof(doc[propName]) != "undefined" && propName.startsWith("l")) {
- count++
+ if(typeof(doc[propName]) != "undefined" && (propName.startsWith("has") || propName.endsWith("URL"))) {
+ emit(propName, 1);
}
}
- emit(count+doc._id, {id:doc._id, name: doc.name, score:count});
+ emit("total", 1);
}
}';
-
+ $obj->views->scoreHas->reduce = 'function (key, values, rereduce) {
+ return sum(values);
+}';
+ $obj->views->fieldNames->map = '
+function(doc) {
+for(var propName in doc) {
+ emit(propName, doc._id);
+ }
+
+}';
+ $obj->views->fieldNames->reduce = 'function (key, values, rereduce) {
+ return values.length;
+}';
// allow safe updates (even if slightly slower due to extra: rev-detection check).
return $db->save($obj, true);
}
--- a/include/template.inc.php
+++ b/include/template.inc.php
@@ -69,6 +69,9 @@
<script src="<?php echo $basePath; ?>javascripts/foundation.js"></script>
<script src="<?php echo $basePath; ?>javascripts/app.js"></script>
<script src="http://code.jquery.com/jquery-1.7.1.min.js"></script>
+
+ <!--<script language="javascript" type="text/javascript" src="javascripts/jquery.js"></script>-->
+ <script language="javascript" type="text/javascript" src="javascripts/flot/jquery.flot.js"></script>
</body>
</html>
--- a/javascripts/app.js
+++ b/javascripts/app.js
@@ -43,7 +43,7 @@
/* PLACEHOLDER FOR FORMS ------------- */
/* Remove this and jquery.placeholder.min.js if you don't need :) */
- $('input, textarea').placeholder();
+ //$('input, textarea').placeholder();
--- /dev/null
+++ b/javascripts/flot
--- a/schemas/agency.json.php
+++ b/schemas/agency.json.php
@@ -24,17 +24,30 @@
"consultanciesURL" => Array("type" => "string", "required" => true, "x-title" => "Consultants Hired", "description" => ""),
"legalExpenditureURL" => Array("type" => "string", "required" => true, "x-title" => "Legal Services Expenditure", "description" => "Legal Services Expenditure mandated by Legal Services Directions 2005"),
"recordsListURL" => Array("type" => "string", "required" => true, "x-title" => "Files/Records Held", "description" => "Indexed lists of departmental and agency files, <a href='http://www.aph.gov.au/senate/pubs/standing_orders/d05.htm'>mandated by the Senate</a>"),
- "FOIDocumentsURL" => Array("type" => "string", "required" => true, "x-title" => "FOI Documents Released", "description" => ""),
- "infoPublicationSchemeURL" => Array("type" => "string", "required" => true, "x-title" => "Information Publication Scheme", "description" => ""),
+ "FOIDocumentsURL" => Array("type" => "string", "required" => true, "x-title" => "FOI Documents Released", "description" => "FOI Disclosure Log URL"),
+ "FOIDocumentsRSSURL" => Array("type" => "string", "required" => false, "x-title" => "RSS Feed of FOI Documents Released", "description" => "FOI Disclosure Log in RSS format"),
+ "hasFOIPDF" => Array("type" => "array", "required" => false, "x-title" => "Has FOI Documents Released in PDF", "description" => "FOI Disclosure Log contains any PDFs",
+ "items" => Array("type" => "string")),
+ "infoPublicationSchemeURL" => Array("type" => "string", "required" => true, "x-title" => "Information Publication Scheme", "description" => ""),
"appointmentsURL" => Array("type" => "string", "required" => true, "x-title" => "Agency Appointments/Boards", "description" => "Departmental and agency appointments and vacancies , <a href='http://www.aph.gov.au/senate/pubs/standing_orders/d05.htm'>mandated by the Senate</a>"),
"advertisingURL" => Array("type" => "string", "required" => true, "x-title" => "Approved Advertising Campaigns", "description" => " Agency advertising and public information projects, <a href='http://www.aph.gov.au/senate/pubs/standing_orders/d05.htm'>mandated by the Senate</a> "),
- "hasRSS" => Array("type" => "string", "required" => true, "x-title" => "Has RSS", "description" => ""),
- "hasMailingList" => Array("type" => "string", "required" => true, "x-title" => "Has Mailing List", "description" => ""),
- "hasTwitter" => Array("type" => "string", "required" => true, "x-title" => "Has Twitter", "description" => ""),
- "hasFacebook" => Array("type" => "string", "required" => true, "x-title" => "Has Facebook", "description" => ""),
- "hasYouTube" => Array("type" => "string", "required" => true, "x-title" => "Has YouTube", "description" => ""),
- "hasFlickr" => Array("type" => "string", "required" => true, "x-title" => "Has Flickr", "description" => ""),
- "hasCCBY" => Array("type" => "string", "required" => true, "x-title" => "Has CC-BY", "description" => "Has any page licenced Creative Commons - Attribution"),
+ "hasRSS" => Array("type" => "array", "required" => true, "x-title" => "Has RSS", "description" => ""),
+ "hasMailingList" => Array("type" => "array", "required" => true, "x-title" => "Has Mailing List", "description" => "",
+ "items" => Array("type" => "string")),
+ "hasTwitter" => Array("type" => "array", "required" => true, "x-title" => "Has Twitter", "description" => "",
+ "items" => Array("type&