<?php | <?php |
include_once('include/common.inc.php'); | include_once('include/common.inc.php'); |
include_header('About'); | include_header('About'); |
?> | ?> |
<div class="foundation-header"> | <div class="foundation-header"> |
<h1><a href="about.php">About/FAQ</a></h1> | <h1><a href="about.php">About/FAQ</a></h1> |
</div> | </div> |
<a href="http://thenounproject.com/noun/document-dump/">Document Dump icon designed by Iconathon, 2013</a> | |
Contact us: maxious@lambdacomplex.org | |
Exports: <a href="admin/exportAll.csv.php">All Agencies</a> <a href="admin/exportEmployees.csv.php">Agency Employee Headcounts</a> | |
<h2> Attributions </h2> | <h2> Attributions </h2> |
National Archives of Australia, Australian Governments’ Interactive Functions Thesaurus, 2nd edition, September 2005, published at http://www.naa.gov.au/recordkeeping/thesaurus/index.htm <br/> | National Archives of Australia, Australian Governments’ Interactive Functions Thesaurus, 2nd edition, September 2005, published at http://www.naa.gov.au/recordkeeping/thesaurus/index.htm <br/> |
data.gov.au http://data.gov.au/dataset/directory-gov-au-full-data-export/ <br/> | data.gov.au http://data.gov.au/dataset/directory-gov-au-full-data-export/ <br/> |
directory.gov.au <br/> | directory.gov.au <br/> |
australia.gov.au http://australia.gov.au/about/copyright <br/> | australia.gov.au http://australia.gov.au/about/copyright <br/> |
<h2>Organisational Data Sources</h2> | <h2>Organisational Data Sources</h2> |
http://www.comlaw.gov.au/Browse/Results/ByTitle/AdministrativeArrangementsOrders/Current/Ad/0 defines departments | http://www.comlaw.gov.au/Browse/Results/ByTitle/AdministrativeArrangementsOrders/Current/Ad/0 defines departments |
Agencies can be found in the Schedule to an Appropriation Bill (budget), Schedule to FMA Regulations and/or Public Service Act.<br> | Agencies can be found in the Schedule to an Appropriation Bill (budget), Schedule to FMA Regulations and/or Public Service Act.<br> |
http://www.finance.gov.au/publications/flipchart/docs/FMACACFlipchart.pdf summarises these. view-source:https://www.tenders.gov.au/?event=public.advancedsearch.home is great for the suspended/active status<br> | http://www.finance.gov.au/publications/flipchart/docs/FMACACFlipchart.pdf summarises these. view-source:https://www.tenders.gov.au/?event=public.advancedsearch.home is great for the suspended/active status<br> |
Fraud in gov depts by Fairfax Media http://www.smh.com.au/national/public-service-keeps-fraud-cases-private-20110923-1kpdr.html <br> | Fraud in gov depts by Fairfax Media http://www.smh.com.au/national/public-service-keeps-fraud-cases-private-20110923-1kpdr.html <br> |
When defining the hierachy, this system is designed towards monitoring accountablity. Thus large agencies that have registered their own ABN | When defining the hierachy, this system is designed towards monitoring accountablity. Thus large agencies that have registered their own ABN |
and have their own accountablity mechanisms/website receive a seperate record as a child of their department.<br> | and have their own accountablity mechanisms/website receive a seperate record as a child of their department.<br> |
Some small agencies will choose to simply rely on their parent department's accountablity measures.<br> | Some small agencies will choose to simply rely on their parent department's accountablity measures.<br> |
This flows through to organisation name and other/past names. A department that completely accounts for an agency will list that agency as an other child name.<br> | This flows through to organisation name and other/past names. A department that completely accounts for an agency will list that agency as an other child name.<br> |
As agencies themselves shift between departments, there may be scope for providing time ranges but typically the newest hierarchy will be the one recorded.<br> | As agencies themselves shift between departments, there may be scope for providing time ranges but typically the newest hierarchy will be the one recorded.<br> |
A department/agency name will be the newest active name assigned to that ABN.<br> | A department/agency name will be the newest active name assigned to that ABN.<br> |
ABN information is derived from the ABR. This is the definitive umpire about which former name should be linked to which current name. <br> | ABN information is derived from the ABR. This is the definitive umpire about which former name should be linked to which current name. <br> |
For example "Department of Transport and Regional Services" became "Department of Infrastructure, Transport, Regional Development and Local Government" (same ABN) | For example "Department of Transport and Regional Services" became "Department of Infrastructure, Transport, Regional Development and Local Government" (same ABN) |
however it later split into "Department of Infrastructure and Transport" (same ABN) | however it later split into "Department of Infrastructure and Transport" (same ABN) |
and "Department of Regional Australia, Regional Development and Local Government" (new ABN).<br> | and "Department of Regional Australia, Regional Development and Local Government" (new ABN).<br> |
Statistical information from http://www.apsc.gov.au/stateoftheservice/1011/statsbulletin/section1.html#t2total https://www.apsedii.gov.au/apsedii/CustomQueryx33.shtml | Statistical information from http://www.apsc.gov.au/stateoftheservice/1011/statsbulletin/section1.html#t2total https://www.apsedii.gov.au/apsedii/CustomQueryx33.shtml |
and individual annual reports.<br> | and individual annual reports.<br> |
<?php | <?php |
include_footer(); | include_footer(); |
?> | ?> |
1 0 obj <</Metadata 2 0 R/OCProperties<</D<</ON[5 0 R 6 0 R]/Order 7 0 R/RBGroups[]>>/OCGs[5 0 R 6 0 R]>>/Pages 3 0 R/Type/Catalog>> endobj 2 0 obj <</Length 21938/Subtype/XML/Type/Metadata>>stream | |
<?xpacket begin="" id="W5M0MpCehiHzreSzNTczkc9d"?> | |
<x:xmpmeta xmlns:x="adobe:ns:meta/" x:xmptk="Adobe XMP Core 5.0-c060 61.134777, 2010/02/12-17:32:00 "> | |
<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"> | |
<rdf:Description rdf:about="" | |
xmlns:xmp="http://ns.adobe.com/xap/1.0/" | |
xmlns:xmpGImg="http://ns.adobe.com/xap/1.0/g/img/"> | |
<xmp:CreatorTool>Adobe Illustrator CS5</xmp:CreatorTool> | |
<xmp:CreateDate>2013-06-22T15:58:27+10:00</xmp:CreateDate> | |
<xmp:ModifyDate>2013-06-22T15:58:27+10:00</xmp:ModifyDate> | |
<xmp:MetadataDate>2013-06-22T15:58:27+10:00</xmp:MetadataDate> | |
<xmp:Thumbnails> | |
<rdf:Alt> | |
<rdf:li rdf:parseType="Resource"> | |
<xmpGImg:width>256</xmpGImg:width> | |
<xmpGImg:height>196</xmpGImg:height> | |
<xmpGImg:format>JPEG</xmpGImg:format> | |
<xmpGImg:image>/9j/4AAQSkZJRgABAgEASABIAAD/7QAsUGhvdG9zaG9wIDMuMAA4QklNA+0AAAAAABAASAAAAAEA
AQBIAAAAAQAB/+4ADkFkb2JlAGTAAAAAAf/bAIQABgQEBAUEBgUFBgkGBQYJCwgGBggLDAoKCwoK
DBAMDAwMDAwQDA4PEA8ODBMTFBQTExwbGxscHx8fHx8fHx8fHwEHBwcNDA0YEBAYGhURFRofHx8f
Hx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8fHx8f/8AAEQgAxAEAAwER
AAIRAQMRAf/EAaIAAAAHAQEBAQEAAAAAAAAAAAQFAwIGAQAHCAkKCwEAAgIDAQEBAQEAAAAAAAAA
AQACAwQFBgcICQoLEAACAQMDAgQCBgcDBAIGAnMBAgMRBAAFIRIxQVEGE2EicYEUMpGhBxWxQiPB
UtHhMxZi8CRygvElQzRTkqKyY3PCNUQnk6OzNhdUZHTD0uIIJoMJChgZhJRFRqS0VtNVKBry4/PE
1OT0ZXWFlaW1xdXl9WZ2hpamtsbW5vY3R1dnd4eXp7fH1+f3OEhYaHiImKi4yNjo+Ck5SVlpeYmZ
qbnJ2en5KjpKWmp6ipqqusra6voRAAICAQIDBQUEBQYECAMDbQEAAhEDBCESMUEFURNhIgZxgZEy
obHwFMHR4SNCFVJicvEzJDRDghaSUyWiY7LCB3PSNeJEgxdUkwgJChgZJjZFGidkdFU38qOzwygp
0+PzhJSktMTU5PRldYWVpbXF1eX1RlZmdoaWprbG1ub2R1dnd4eXp7fH1+f3OEhYaHiImKi4yNjo
+DlJWWl5iZmpucnZ6fkqOkpaanqKmqq6ytrq+v/aAAwDAQACEQMRAD8A9U4q7FXYq7FXYq7FXYq7
FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYqkOv+cNN0kmEf6TeD/dCGgX/Xben68pyZ
hH3tsMRkwrUPPOv3ZISYWsZ6JCKH/gjVvxzElnkXIjhiElnvby4NZ55JSepd2b9ZyoyJ5tgACeeS
Na/R+rCGVqWt3SN69A/7Dfft9OXYJ8Mve15oWHqObBwnYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq
7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq4kAVOwHU4qwbzV54IL2OkvuPhlux+Ij/5q+7xzEzZ+kXJx
4epYMSWJJNSdyT1JzDclrFXYq7FXrPlLWv0ppEbyGtzB+6n8SQNm/wBkPxzZYZ8UXByw4SnWWtbs
VdirsVdirsVdirsVdirsVdirsVdirsVdirsVdirsVdirsVdirsVdirsVdirH/PEWpvojmycqiGt1
Gv2mipvv4DuPDKM4PDs24SOLd5Zmvc12KuxV2KuxVPPJ+tDS9XQyNS1uP3U/gKn4W/2J/DLsM+GT
XlhxB6vmxcF2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxVCatqUOmafLezAtHFSqrS
p5MFFK/PIzlwi2UY2aSq189+XJ6BpmgY/syoR+K8l/HKxqIlmcMgnFrqFhdCttcRTf8AGN1b9Ryw
SB5NZiRzRGSQ7FXYq4gEEEVB6jFXlnnHy8dKv/VhX/QbgloqdEbqU/p7Zrs2PhPk5uLJxDzY/lLa
7FXYqq21pc3UohtommlPREBY/hhAJ5IJA5so078udTmAe9mS1U/sD94/00IX8cyI6Ynns0yzjo9A
srdra0it2lMxiUJ6rUDNTYVpmbEUKcUmyrYUOxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxVxI
AqdgOpxVZHNDIWEciuV+0FIJHzpgtaYr+Y936ekwWwNGnlqfdYxU/iRmPqT6ab9ON7ec5guW4Egg
g0I6HFUytfMeu2tPRvpQB0Vm5r/wL8hlgyyHViccT0Ti1/MXWoqCeOG4XuSCjfepp+GWjUy6tZwB
PNO/MSxuZo4JrSWKWVgi8Csi8mNB14H8MtjqQejVLAQy3MloQeraZb6nYS2c4+Fx8Ld1YfZYfLIz
gJCiyjKjbyDULC4sLyW0uF4yxNQ+BHYj2I3zWSiQaLnxlYtD5FKdeW/LF3rM9d4rND+9np/wq+Lf
qy3FiMj5NeTIIvTtN0qw023EFnEI1/abqzHxZupzYRgIjZw5SJ5ovJMXYq7FXYq7FXYq7FXYq7FX
Yq7FXYq7FXYqp3DTLBI0Ch5gpMaN0LAbA/M4DySHnc35ja41QkMEX+xYkfe38MwjqZOWMEUFN548
yydLoRg9kjQfiQTkDnn3shhigpfMOuy/bv56HqBIyj7lIyBySPVkMce5BS3FxMayyvIf8ti368iS
SyAT3yNqRs9djjY0iux6LD/KO6H/AILb6cu08ql72rNG4oz8yLv1NWgtgarBFU/60hqfwUZLUn1U
x042tiWYze7FVe2sby5SV7eFpEgQyTMo2VQKkk5IRJ5IMgFDIpTrydafWfMdmpFVjYyt7emCw/4a
mW4Bcg15TUS9ZzZOC7FWM+d/Lv6Rs/rlutb22HQdXj6lfmOozHz4uIWObdhyUa6MK8teXZ9ZvOO6
WkdDcTDsP5V/yjmLixGR8nIyZOEPVrS0t7O2jtrdBHDEOKKPDNjEACg4RNm1XCh2KuxV2KuxV2Ku
xV2KuxV2KuxV2KuxV2KuxV2KvJvOGn/UdfuUApHMfXj+Um5/4aozW5o1IudilcUlypsdirsVXRSP
FIkiGjoQynwINRhBVHa9qI1HV7i8H2ZGHEeyqFH6slklxStjCNCkAiM7BEBZmNFUCpJPYDIMmX6D
5AurjjPqhNvD1EA/vG/1v5f15lY9OTuWiecDkzu30+yt7T6nBCsdtQqY1FAQRQ18a+OZYiAKDimR
Jt45qNm9lf3Fo/WCRkqe4B2P0jNZKNGnYRNi0/8AIl7plje3N1fTrDSMRx8qknk1TQAHpxy7TyAJ
JaswJFBmf+MvLW1L0Ek0AVJCST7BcyvHh3uP4Uu5OQQQD2O++368ta0LY6pp9/6n1SdZvSYpIF6g
g0+7wORjMHkylEjmq21rbWyGO3jWJGZnKqKDkxqThAA5IJtVwodirsVdirsVdirsVdirsVdirsVd
irsVdirsVdirsVS7WU0CQQw6ubdTcP6VsJ2VGeQgnhESQxag6LvkJiJ5tuOMzZiCa51+ljup/lvb
PV9OuDC3aKX4k+hh8Q/HKJ6UdGcdQerHJvJHmWNiPqnNQaBkeMg/RyB/DKDgn3NwzR71A+VPMQJH
1CXb5H+OR8GXcnxY96rF5M8yy9LIqPF2Rf1thGCfcg5Y96aWP5calIwN5PHBH3CVd/4L+OWR0x6s
DqB0Zdpfl7RdGiMkSAOo/eXUpBanf4jQKPlTMqGOMXHlMySXXfzBtYA0GlgXE3Qzt/dr8h1b9Xzy
nJqQOTbDATzd5C167v3vbe8maWeonRm8D8LADsBtsMdPkJsFc0AKpJ/zF0/0NXju1FEu4/iP+XH8
J/4XjlepjUr72eCVimKZjN7NfIXlv1HGr3S/u0P+iIe7DYv9Hb3zL0+L+IuPmydAmHnrzJ9UgOmW
rf6TMv79x1SM9vm36snqMtCgww473LAbK+u7G4W4tZDFMnRl/UR3HtmFGRBsOUQDzejeW/OtpqXG
2u+NvfHYdo5D/kk9D7HM7FnEtjzcTJhI3HJk2ZDS7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7
FXYq+W/+cm/Nf6R832+gwPW20aKswHQ3NwA7fPjHwHsa5rNZO5V3PoXsto+DAch5zP2D9tsP8qfn
B5/8scI7HU3ns02Fjefv4aDsoY8kH+owyqGeceRdnrOxdNqN5RqXeNj+34vZvKn/ADlD5evOEHmS
wk0yY0Bu7es9v7lloJU+QDZlw1gP1bPLaz2Uyx3xSEx3HY/q+567ofmPQddtfrWjX8F/BtyaB1cq
TvRwN1PswrmXGYlyLzWfTZMJ4ckTE+aMuppYYS8cLXDjpEhUE/8ABlRhJppAYfqv5gX1q7QrpjW8
3Y3JP/EQFr/wWYs9QR0ciOAHqxHVNe1XVGreTs6A1WIfCg+Sjb78xp5DLm3xgI8kvyDJOPKWoGx1
+1kJpHI3oyeHGT4d/kaHLcMqkGGWNxZx590/61oLyqKyWrCUePH7Lfga/RmXqI3H3ONhlUmEeV/L
8msagEYEWkNGuXHh2Ue7ZiYsfEfJyMk+EPRNd1i10LSuaqocD07SAbAkCg2H7KjrmdkmIBxYQMi8
muLia5nknncvNKxZ3PUk5rSbNucBSngV2Ksv8t+e57Xha6mTNbDZLjq6D/K/mH45k4tRWxaMmG9w
9At7iC4hSaCRZInFUdTUEZmg3ycUilTCh2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KoLW9XtdH0e9
1W7NLaxgkuJfErGpag9zSgyMpULbcGE5ZxhHnI0+ENY1W71bVrzVLtuV1fTSXEx7cpGLECvYV2zS
SlZt9gw4Y44CEeURSDwNrsVROn6lqOm3SXen3UtndR/Yngdo3HyZSDhBI5NeTFGY4ZASHm9U8qf8
5K+dtK4Q6xHFrlquxaT9zcUHhKgKn/ZIT75kw1chz3ee1nsvp8m+O8Z+Y+X7XsPl387/AMs/NUS2
t5cLp9w+xtNTVUUnp8MtWi+XxA+2ZcdRCex+15bV9garBuBxjvj+rmnuofl/pF2nradKbYuAyUPq
REHcEVNd/Y4y00TydWM0hsWKan5O12wqxg+sRD/dsFXFPdftD7sx54JBujliUl3B8CMpbHsOmXEe
r6FDJJ8S3UPCYDxI4OPvrm0geKLgSHDJbp9jYaBpBTkFhhUyTzHYs3dj/AYIxEIrKRkXmXmHW5tY
1Brh6rEvwwRfyp/U98wMmTiNuZCHCEsytm7FU+sPJWu3dq9wIhCoUtGkvwu5HZV7V96ZdHBIi2uW
aIKROjIxRwVZSQynYgjqDlLYmmheY9R0eblA3OBjWW3b7DfL+U+4yzHlMeTCeMSep6TqUWpWEV5E
jRpKD8LihBBofmPfNjCXELcKUaNIvJMXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXi//OTvmv6h5VtP
L0D0uNXl53AB3FvbkNQ/60nH7jmHrJ1Gu96r2V0fHmOU8oDb3n9lvl/Na+gOxV2KuxV2KuxVkflf
8wvOXldwdF1Sa3hBq1qT6kDeNYn5Jv4gVyyGWUeRcHV9m4NR/eRBPf1+b2Tyn/zlNGeEHmrSyp2D
X1huPm0Eh+/i/wBGZcNb/ODy+s9kzzwy+Ev1j9T1LS9a/LnzzF6mnXdtezkVZUb0rpR4tGeEn/BC
mZH7vI81qNHqNMfXEx+758k/0TRo9ItXtIpWlh5l4+dOShgKrUUB3FemWY4cIpw5z4jbEfzD1S/a
VbARSRWSUZpSpCyv12PQhf15jamZ5dG/BEc+rCsxHITTRfLep6vJ/o8fGAGj3D7IP6n2GWQxGXJh
PIIvQ9E8o6VpIEvH17oCpuJANj/kr0X9eZuPDGPvcWeUyREPmbRZtSGnQ3AkuCDQruhYfshuhPyy
QyxJpicZAtIPPHlb11fVbJP3yit1Eo+0o/bHuO/jlOfFfqDbhyVsWM+VvL8msX4VgRZw0a4f27IP
dsx8WPiPk3ZJ8IerxxxxRrHGoSNAFRRsABsAM2QDgrsVdirsVdirsVdirsVdirsVdirsVdirsVfG
X51+a/8AEn5hajPE/OysT9Rs/DhASGYezSF2Hsc0+onxTL6l2Fo/A0sQfql6j8f2UwTKXcOxV2Ku
xV2KuxV2KuxVfFLLDIssTtHKhDI6EqwI6EEbjFBAIovSPKf/ADkD+YWg8Ibi5XWbJaD0b4F5AP8A
JnFJK/6xYe2ZENVOPm6LWezumzbgcEv6P6uXyp7J5X/5yM8g64i22sK+jXEnwslyPVtmJ7eqo6f6
6qMzIauEuezy+r9mdTi3h+8Hlsfl+q2axeVvKGqejqNl6c1s55q9rKGgkH+xLLT/AFSMn4MDuHRT
lkgeGQIPnzTu4kSxsmaC2aRYVpHbwKK+wVdstJoNI3LzLzD5p1jUZHglraW4NDarUH/Zk0Lfq9sw
MmWUtuTmwxgJGjsjq6Eq6kFWGxBHQjKWx6j5R8zpq1t6E5Av4R+8HTmvTmP45sMOXiFHm4WXHwny
TqzsLOyjaO1iWFHdpGVdgWbqcujEDk1mRPNXwoQt9qmn2EfO8uEhXsGO5+Sjc/RkZTA5soxJ5N6d
qFtqFnHd2xJhk5cSRQ/CxU7fMYxkJCwiUSDRROSQ7FXYqgZNf0KORo5NRtUkQlXRpowQRsQQW2Iy
PEO9uGnyEWIy+RW/4i8v/wDVztP+R8f/ADVjxjvT+Wy/zZfIu/xF5f8A+rnaf8j4/wDmrHjHev5b
L/Nl8i7/ABF5f/6udp/yPj/5qx4x3r+Wy/zZfIu/xF5f/wCrnaf8j4/+aseMd6/lsv8ANl8iiLTU
tOvCws7qG5KULiGRXpXpXiTTCCCwnilH6gR70Rha2Kfml5r/AMLeRdU1VH43Yj9Cy8fXm+BCP9Wv
P6MqzT4Ykux7J0f5jURh0uz7h+KfEZNdzmmfWWsVdirsVdirsVdirsVdirsVdirsVTXQPNXmPy9c
/WdF1GewlqC3ouQjU/nQ1Rx7MDkozMeRcfUaTFmFZIiXvev+U/8AnKPWbbhB5n09L+IUDXlpSGen
dmjP7tz8uGZcNaR9QeZ1nsnjlvhlwnuO4+fP73rehfmF+WXnhEghu4JLtqBbK6HoXIJ7JyoW/wCe
bHMkZMeR5fVdlarTbyieHvG4/HvSzzj5Vi0kxXNnyazkPFwxqUfqBXwIyjNh4dxycfFk4tix6zvL
izuo7q3cpNEeSMP1H2PfKIyINhtIsU9a8va9b6xYieOizJ8NxD3Vv6Htmyx5BIODOHCWO+d9W8y2
EwWGQRWE20csS0atN1ZjWh8KUyjPOY9zbhjE+9gksssshkldpJG3Z2JYn5k5iE25QD0j8u5/U0Fo
z1hmdQPYgN/xtmdpj6XDzj1MozIaXYq8P/O/87xpIn8seWJwdVIMeoahGdrYdDHGR/u3xP7H+t9n
C1Opr0x5vW9g9g+JWbMPR0Hf5ny+/wB3PzQzFiWYksTUk9Sc1z3rWKuxV2KuxV75/wA4n/8AHQ8x
/wDGG1/4lLmdouZeN9r/AKcfvl+h9GZsHh3zd/zlL5r+sarpvleB6x2SfXbwA7etKCsSn3WOp/2e
a7Wz3EXuvZPR1CWY/wAXpHuHP7fueEZhPYOxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KvqD8h/O
kXm7yhceU9XlMmo6bHxikc8nktSf3b79Whai/Ljmx08+OPCXz32j7P8AAzDLAeif2S/bz+ay/sp7
G8mtJxSWFire/gR7Eb5jyjRp1ETYtX0XWLrSb5LqA1A2ljJoHTup/hhhMxNhE4CQp6mDpnmDR609
S1uV6HZlYfqZTmw2nHycLeJeW65otzpF+9rNuv2oZR0dOx/rmvyQMTTmwmJC2VfllP8A7325/wCK
5F/4YH+GZOlPMNGoHJnOZbjPD/zv/O8aSJ/LHlicHVSDHqGoRna2HQxxkf7t8T+x/rfZwtTqa9Me
b1vYPYPiVmzD0dB3+Z8vv93PzQzFiWYksTUk9Sc1z3rWKuxV2KuxV2KvfP8AnE//AI6HmP8A4w2v
/EpcztFzLxvtf9OP3y/Q+h7y7t7O0nvLlxFbW0bTTyN0VEUszH5AZnk08TCBlIRHM7PhPzd5huPM
fmbUtbnqHv53lVT1WOtI0/2CAL9GaSc+KRL6/o9MMGKOMfwj+37UoyLkuxV2KuxV2KuxV2KuxV2K
uxV2KuxV2KuxV2Kp55L81X/lXzLY65Zkl7V/3sVaCWJtpIz/AKy/cd8njmYysOJrtJHUYpY5dfsP
QvrfWNLt/N2nabr2hukiXkaMHYhQYnFQW6/Eh2I69u2bHLj4wJRfLDGWGcsc+cSj9G8h6VZBZLsf
XLjvzH7sH2Tv/sslDTxHPdpnmJ5JjqHmPQtLHpTXCK6CggiHJh7cV+z9OWSyxiwjjlJhXmnzfaav
bfVorOiq3KO4lPxr40VdhUe5zEy5hIVTk48RjvbX5dz+nrzRnpNC6ge4Ib/jXHTH1LnHpY1+d/53
jSRP5Y8sTg6qQY9Q1CM7Ww6GOMj/AHb4n9j/AFvsy1Opr0x5vQ9g9g+JWbMPR0Hf5ny+/wB3PzQz
FiWYksTUk9Sc1z3rWKuxV2KuxV2KuxV75/zif/x0PMf/ABhtf+JS5naLmXjfa/6cfvl+hnH/ADkb
5r/Q3kJtNhfjea5J9WUDY+glHnPyI4of9bLtXOoV3up9mdH4up4z9OPf49P1/B8l5q30h2KuxV2K
uxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV79/zjN+YXpTS+TNQlpHKWuNIZj0f7U0I/1vtr78vHM7
R5f4S8Z7U9m2BqIjltL9B/R8npXnjXdcgv3sA31a0Kho2jqGkU9y3XrUUGTz5JA10eSwwjVsMzFc
h2Kpt5TuPQ8x2D1pyk9P/kYCn/G2W4TUgwyi4l8qsxYlmJLE1JPUnMN9baxV2KuxV2KuxV2KuxV7
5/zif/x0PMf/ABhtf+JS5naLmXjfa/6cfvl+h61+YX5VeWfPKQtqjTw3lsrJbXUEhBQMakem3KM1
PX4a++ZWXBGfN5vs3tfLpL4KMTzB/FvCPNf/ADjR500vnNossWt2o3CJ+4uAPeNyUP8AsXJ9swp6
OQ5bvYaP2o0+TbIDjPzHz/Y8q1HS9S0y6a01G1ms7pPtwTo0bj/YsAcxTEjm9FiywyDigRIeW6Fw
NjsVdirsVdirsVdirsVdirsVdirsVdirsVdiqJ07ULzTr+3v7KUw3drIs0Eq9VdDyU/eMINGwwyY
4ziYyFxOz69sdVtPzD8gWmvWSgahEp9aBdyk6ACaHx3+0viKeObOX72Fjm+V6zSy0mc4zy6eY6Fh
2YSHYqr2LyR3sEkalpI5FdVUVJKmuw+jDHmg8nzHmO+suxV2KuxV2KuxVMrby35iuo/VttKvJ46c
ucUErrTxqqnJCBPRolqsUTRlEfEIO5tLq1k9K5heCTrwkUo33MBkSKbYzEhYNvdv+cT/APjoeY/+
MNr/AMSlzO0XMvH+1/04/fL9D6MzYPDuxVL9a8vaFrlqbXWLCC/t96JPGr8SdqqSKqfcb5GUBLmG
/BqcmI8UJGJ8nkXmv/nF/wAuXvOfy5eyaVMakWs1bi3+QYkSr8yW+WYk9GDy2el0ftXljtliJjvG
x/V9zxrzX+Tn5g+Wecl5pjXVmnW9sqzxU8W4jmg/11GYk9POPMPUaPtvTZ9oyqXdLY/q+TCcpds7
FXYq7FXYq7FW1VmYKoLMxoqjcknsMVJTMeVvMzQmZdIvTCNjILeXiPp4075Lgl3OP+bw3XHG/eEu
lilikaOVGjkU0ZGBBB9wci3ggiwsxS7FXYq7FXYq9r/5xd1jWYvNd9pECmXSrm2NxdgmgikiIWOQ
e7c+JHfr+zmZo5Hiro8p7V4MZwxmfrBoed8x+n+17ZqH5dw3WpTXEd19Xt5W5+kE5EMftAGoAFcy
JaazdvERz0EZZ/l/oEFDKsly3/FjUH3JxycdPEMTnkU9tNPsbNeNrbxwDvwULX5kdctEQOTUZE83
5/5o32l2KuxV2KvS/wArfyS1vzoV1C7ZtN8vhqfWytZJ6GjLAp2PgXOw99xmRh05nvyDoe1u3cel
9MfVk7u73/qfR3l/8vfy98lWXr21lbWxgFZNUvCjTe7NNJ9j5LQe2bGOKEA8Nqe0tVqpUZE3/COX
yCldfnN+V9tKY5PMVszDvFzmX/go1dfxwHUQHVlDsTWSFjHL7vvTCz8xfl/5wgazgvdP1mNgedm5
jlaniYX+Kn+xyQnCfcWiem1OmPEYzh57j7Vvlb8ufK3lXVL+/wBCt2s/0ksa3FqHLQgxliGRWqy1
5morTwAxhijEkjqy1faebUQjHIeLh5HqybLHXuxV2KuxV2KsP82flL5C80B31HS447x9/r1r+4nq
e7Mmzn/XDZTPBCXMOz0fbGp0/wBMvT3Hcfs+DxnzX/zi5rlrzn8s6hHqMQqVtLqkE/squP3bn3PD
MSejI+kvU6P2sxy2zR4T3jcfr+95Dr3ljzDoFz9W1rTp7CU/ZEyFVb3R/suPdTmJKBjzD02n1eLM
LxyEh5JXkXIdir2z8r/+cdr3W4ItY81NJYabIA9vp6fDcyr2Zyw/doflyPtsczMOlJ3lyeU7V9pY
4iYYalLqeg/X9z3ey0f8vvI1gHhhsNDtwOJuZWjid6fzTSHm5+bHM4RhAdA8fPNqdXLcyyHu5/YO
SXN+dX5WrN6R8w2/KtKhZSu/+WE4/jkfzEO9vHYesq/DP2frTJl/L/zzYkH9H69bKKEgxztHX3FX
jP3HJeiY6FxwdTpJfx4z8R/a8Y/Mv/nG02sEuqeSzJMiAvLo0hLyUHX6u5+Jv9Rt/AnpmHm0lbxe
q7L9p+IiGo2/pfr/AFvBHR43ZHUo6EqysKEEbEEHMF7EG1uKXYq7FX09/wA46eX4tA8hX3mi7Sk2
pszxk7H6vbVRAK9OcnL5/Dmx0seGBkXz72o1Xi6gYhyh95/ZX2s/0z8w9KuWCXkbWbn9onnH94AI
+7LIakHns87LARyZTFLFLGssTrJG4qrqQQR4gjMkG2ghdir8980L7U7FXYqzv8nvy7bzt5rS2uAw
0exAuNTkWoqlaJECOjSkU+XI9suwYuOXk6ftrtL8rhsfXLaP6/h+p9NfmF590L8u/LMUvoqZiv1f
SdMjogYotAKD7McYpyPbYdSM2WXKMcXgezez8mtzEXtzlL8dS+SvN/nrzP5tvzd63evMAxMNqpKw
RA9o460G21ep7k5qsmSUzu+k6Ls/Dpo8OMV59T7yx/IOavilkikWWJ2jkQhkdSQykbggjocUEAii
94/Jv8/L5Ly38veb7g3FtMRHZatKf3kbk0VLhz9pD/Odx3qPs52n1JupPH9t+z0TE5cAojnHv936
vwetfmxB5v8A8JXF/wCVNQkstT04G4aKNUcTwqKyJR1f4gPiWny75lZxLhuJ3eb7Hlg8YRzREoS2
9x6ftfMn/K8PzV/6mCb/AJFQf9U81v5nJ3vffyDo/wDUx8z+t3/K8PzV/wCpgm/5FQf9U8fzOTvX
+QdH/qY+Z/W7/leH5q/9TBN/yKg/6p4/mcnev8g6P/Ux8z+t3/K8PzV/6mCb/kVB/wBU8fzOTvX+
QdH/AKmPmf1ro/zz/NVHVxr8hKmtGityDTxBjw/mcneg9gaM/wCTHzP63ovlP/nKaVeEHmrSw42B
vbDY/NoZDQ+5Dj5ZfDW/zg6PWeyQ54ZfCX6x+p67ovnT8vvPFm1rZ3lpqaSj97p1wo9TpU8oJgGN
PGlPfMuOSEx3vNZ9DqdJK5CUa/iH6wxDzX/zjb5G1fnNpLSaHdtuPR/e29fEwuaj/YOoyqekieWz
s9H7T6jHtOskfPY/P9YLH/y5/wCccrrRvNh1HzLLbX+n2IElhFEWKyy1+FpUdRQR0rx3qabkA5Xi
0lSuTm9p+0wy4eHEDGUufkPL3sq/OX844fJVsumaYEuPMV0nJFf4o7eM7epIO7H9hfpO2xt1Go4N
hzdd2J2KdUeOe2Ifb5D9L5V1rXtZ1y/e/wBXvJb68k+1LMxYgdeKjoqjso2GauUjI2X0TBp8eKPD
ACMfJAYG5Fadqeo6ZeR3unXMtndxGsc8DtG4+TKQcIJG4a8uKOSPDICUT3vpz8lPzubzQ6+X/MLI
mvKpNrdKAiXSqKsCo2EoArtsR4UzZafUcWx5vA9u9hfl/wB7i/u+o/m/sY3/AM5I/llDEh866VFw
5Oqa1Eg2q54pcU7EtRH8TQ+JyvV4f4h8XO9mO1Sf8Hmf6v6v0j+x8+5gPaOxVH6Fo91rWtWOk2gr
c388dvF3AMjBeR9lrU4Yxs006jMMWOU5coi3195vFronlvTfLliOEEcccSoP9826hVr82AP0ZstQ
eGIiHycTllySyS5k382D5htyc+XvM97o0wCky2bH97bk7fNfA5bjymPua8mMSepafqFpqFol1avz
if7we6sOxGbCMhIWHClEg0X5/wCaN9pdirsVfWn/ADjf5ej0z8u49QKgXOsTyXEjftenGxhjX5fA
WH+tm00kKhfe+b+0+pOTVcPSAA/Sfx5PB/zs82z+Y/zB1J+ZNlp0jWNklaqEgYq7D/Xk5NXwp4Zh
aifFM+T2HYWjGDTR/nS9R+P6gwPKHcOxV2KuxV9k/kd5tm8zfl7ZzXb+pe6ezWF05NS5hClGNe5i
Zanua5t9NPig+X9vaMYNSRH6ZeofH9r5e/M7y9H5e8+63pMShLeG4L26DosM4E0aj/VSQDNZmhwz
IfQOytSc2mhM8yN/eNixfK3YOxV2KuxV2Krkd43V0Yo6EMrKaEEbggjFBFvQvKn58/mH5f4RNejV
bJNvq1+DKQOnwygrKPb4iPbL4amcfN0us9n9Lm34eCXfHb7OT60Greh5fGr6hGLf0rT63eRKeQj4
x+pIoY0rx33za8W1l838G8vBHf1UPnQfDXmXX77zDr19rV83K5vpWlcVqFB2VF/yUWij2GaWcjI2
X1vS6eOHHHHHlEJZkXIdirsVRFhf3en31vfWcrQ3drIs0Ey7FXQ8lI+RGINGwwyY4ziYyFg7F9uW
E9j55/L+KSdALbXrCk8Y34NNHxcA+KPWh9s3QInD3h8nyRlpNSQOeOX3H9L4hu7Wa0u5rWYcZreR
opB4MhKn8RmlIp9ZhMSiCORUcWT2n/nGHyp9f803fmKdKwaRF6dux6G4uAVqPHjFyr/rDMzRwuV9
zyvtVrODCMQ5zO/uH7XofmvU/wBI65cTKaxRn0ofDgm1R8zU45p8UnjsUailGVM3YqnXlTV9SsNT
jjtFM63DBJLbs/uPAjxy3DMg7NeWII3ek/4d8v8A/VstP+REf/NObDgHc0fmcv8AOl8y7/Dvl/8A
6tlp/wAiI/8AmnHgHcv5nL/Ol8y7/Dvl/wD6tlp/yIj/AOaceAdy/mcv86XzKNhhhgiWKGNYokFE
jQBVA8ABsMk0ykSbPN8C6t6g1W89WvqevJz5deXM1rXvmilzfZcNcAruCEwNjsVdirsVfS//ADin
6n+GtbrX0/rqcfDl6Q5U9+mbHRfSXgva6vFh/V/S9ludG0i6lM1zY288zUDSSRI7Gmw3YE5mGILy
8c84ihIge9T/AMO+X/8Aq2Wn/IiP/mnBwDuZfmcv86XzLv8ADvl//q2Wn/IiP/mnHgHcv5nL/Ol8
y7/Dvl//AKtlp/yIj/5px4B3L+Zy/wA6XzLv8O+X/wDq2Wn/ACIj/wCaceAdy/mcv86XzLv8O+X/
APq2Wn/IiP8A5px4B3L+Zy/zpfMu/wAO+X/+rZaf8iI/+aceAdy/mcv86XzLv8O+X/8Aq2Wn/IiP
/mnHgHcv5nL/ADpfMpR+Z/qf8q68y+nXl+jbqvHrx9JuX0ca1yOb6D7nJ7Kr81jv+fH73w9mlfWn
Yq7FXYq7FX2T+Qnqf8ql0H1K8uNzTl14/W5uP0caUzb6b+7D5f7Q1+dyV/R/3IfKnn70j568x+jT
0v0pe+nTpx+sPT8M1eX6j730Ts6/y+O+fBH7gkOQcx9efl3oEvk38ooQIiNSu4jeXXEHkstzTjyp
09KPiD7jNpCPBi83y/tjVfmNXI/wx9I+H6zbGMwnHdiruuKvTPJfln9G24vbpP8ATp12U9Y0Pb/W
Pf7sz8GLhFnm4ebJew5MnzIaXYq7FXYq+Nfzu8pT+XPzB1EemVstTdr+yf8AZKzEtIo/1JOQp4U8
c1GohwzPm+o9g6wZ9NH+dH0n4fsYDlDuXYq7FXYq+yvyP8pT+Wfy9sre6jMV9fs1/dxnYq0wARSD
0IiVKjsa5t9NDhg+XdvawZ9TIj6Y+kfD9ts+y90zsVdirsVdirsVdirsVUL+yt7+xubG4HK3uong
mUd0kUqw+44CLFM8eQwkJDmDb4S8zeX77y9r99ot8vG5sZWiY0oGUbo61/ZdSGHsc0k4mJovr+l1
Mc+OOSPKQ/HySvIuQ7FXYqidO0+81G/t7CyiM13dSLDBEvVnc8VH3nCBZoMMmSMImUjURu+6vKuh
RaB5b0zRYiGWwt44C4/aZVHNv9k1Tm7hHhAD5Bq9Qc2WWQ/xEl5R50/5xl0bVLi4v9A1GSwvJ3aV
7e5rPAzuamjikibnvyzFyaMHcF6PQ+1OTGBHJESiOo2P6j9jzzy9+QfnS389aVYa5p/LRzOJLq+h
YS27Qw1dlLCjJ6nHgOQB3zHhpZcQBGzu9T7Q4JaeUscvXWwOxs/q57Pq+gpxpt0p2pm0fOXnfnPy
kLJm1GwT/RGP7+If7rJ7j/JP4Zg58Nbjk5eLLexYhmM3s18i+WPVZdWvE/dIa2sZ/aYftn2HbMvT
4r9RcfNk6Bn2ZjiuxV2KuxV2KsT/ADJ/LvSvPOgnT7tvQu4SZNPvgvJoZD12qOSNSjLX8QMqzYhM
U7LsztKekycUd4nmO98i+cfIHmnyhfNba1ZNFGWKw3iAtbyjxjkAp034mjDuM1WTFKB3fStF2jh1
Mbxn4dR8GO5W5y+GGaeVIYI2lmkIWONAWZmOwAA3JxRKQAs8nvv5OfkFfLe2/mLzfB6EUJEtlpEg
+NnG6vcD9kL1CHcn7VOhztPpTdyeN7a9oY8JxYDd85fq/X8n0RmweIdirsVdirsVdirsVdirsVdi
rzL84/yet/O1qmoaeyW3mG0ThFI+0dxGKkRSEdCCfhb6Dt0xtRg49xzd/wBidtHSHhlviP2eY/SH
ytr3l3XNAv3sNZspbK6Qn93KtAwBpyRvsuv+UpIzWSgYmi+iafU480eLHISCW5FvRel6TqmrXsdj
plrLeXkpokECF2PvQdh3PbDGJJoNWXNDHHimRGPm+nvyW/JL/Cjrr2vcJdfdCLe3WjJaqwIb4hs0
jA0JGw3Ar1zZafT8O55vAdudu/mP3eP+76n+d+x6/mW8y7FXYq7FWpI0kRo5FDI4KspFQQdiDiVY
Mv5fH9PkE/7iB+9Br8XX+68fp8PfMP8ALer+i5Pj+nzZyiIiKiKFRQFVRsABsABmY4zeKuxV2Kux
V2KuxVDal+jfqUv6T9H6jT9/9Z4elx/y+fw0+eA1W7Zi4+IcF8XlzeW3/wD0LV9af1/0J6tfi9Cn
Cvt6PwfdmMfB8noMf8q1t4lef7d2WeSf+VWV/wCdP/RXrcfi+pej9Y4/5dP3v/BZbj4P4addr/zn
+X46/pXX6mX5a6x2KuxV2KuxV2KuxV2KuxV2KuxV2KpV5k/wt+jj/iX6l+jq7/pH0vRrT/i74a0y
M+GvU5Ol8bi/dcXF/Ru/seaP/wBCy+q1f0Py5GvHlxrXtT4afLbMb9z5O+H8rV/lHoflL/BP1E/4
T/R31Pbn+jfR417c/S/a/wBbfMiHDXpp0ms/McX77j4v6V/pT3JuG7FXYq7FXYq7FXYq7FXYq//Z</xmpGImg:image> | |
</rdf:li> | |
</rdf:Alt> | |
</xmp:Thumbnails> | |
</rdf:Description> | |
<rdf:Description rdf:about="" | |
xmlns:xmpTPg="http://ns.adobe.com/xap/1.0/t/pg/" | |
xmlns:stDim="http://ns.adobe.com/xap/1.0/sType/Dimensions#" | |
xmlns:xmpG="http://ns.adobe.com/xap/1.0/g/"> | |
<xmpTPg:NPages>1</xmpTPg:NPages> | |
<xmpTPg:HasVisibleTransparency>False</xmpTPg:HasVisibleTransparency> | |
<xmpTPg:HasVisibleOverprint>False</xmpTPg:HasVisibleOverprint> | |
<xmpTPg:MaxPageSize rdf:parseType="Resource"> | |
<stDim:w>100.000000</stDim:w> | |
<stDim:h>100.000000</stDim:h> | |
<stDim:unit>Points</stDim:unit> | |
</xmpTPg:MaxPageSize> | |
<xmpTPg:PlateNames> | |
<rdf:Seq> | |
<rdf:li>Cyan</rdf:li> | |
<rdf:li>Magenta</rdf:li> | |
<rdf:li>Yellow</rdf:li> | |
<rdf:li>Black</rdf:li> | |
</rdf:Seq> | |
</xmpTPg:PlateNames> | |
<xmpTPg:SwatchGroups> | |
<rdf:Seq> | |
<rdf:li rdf:parseType="Resource"> | |
<xmpG:groupName>Default Swatch Group</xmpG:groupName> | |
<xmpG:groupType>0</xmpG:groupType> | |
</rdf:li> | |
</rdf:Seq> | |
</xmpTPg:SwatchGroups> | |
</rdf:Description> | |
<rdf:Description rdf:about="" | |
xmlns:illustrator="http://ns.adobe.com/illustrator/1.0/"> | |
<illustrator:Type>Document</illustrator:Type> | |
</rdf:Description> | |
<rdf:Description rdf:about="" | |
xmlns:dc="http://purl.org/dc/elements/1.1/"> | |
<dc:format>application/pdf</dc:format> | |
<dc:title> | |
<rdf:Alt> | |
<rdf:li xml:lang="x-default">logo</rdf:li> | |
</rdf:Alt> | |
</dc:title> | |
</rdf:Description> | |
<rdf:Description rdf:about="" | |
xmlns:xmpMM="http://ns.adobe.com/xap/1.0/mm/"> | |
<xmpMM:RenditionClass>proof:pdf</xmpMM:RenditionClass> | |
<xmpMM:DocumentID>uuid:4fae17bc-59c6-9342-b6da-8cb94c264d4a</xmpMM:DocumentID> | |
<xmpMM:InstanceID>uuid:ecb84851-93a3-9547-b0ae-1dc3fa2c0ab0</xmpMM:InstanceID> | |
</rdf:Description> | |
<rdf:Description rdf:about="" | |
xmlns:pdf="http://ns.adobe.com/pdf/1.3/"> | |
<pdf:Producer>Adobe PDF library 9.90</pdf:Producer> | |
</rdf:Description> | |
</rdf:RDF> | |
</x:xmpmeta> | |
<?xpacket end="w"?> endstream endobj 3 0 obj <</Count 1/Kids[8 0 R]/Type/Pages>> endobj 8 0 obj <</ArtBox[3.0 13.543 96.5889 84.2764]/BleedBox[0.0 0.0 100.0 100.0]/Contents 9 0 R/LastModified(D:20130622155827+10'00')/MediaBox[0.0 0.0 100.0 100.0]/Parent 3 0 R/PieceInfo<</Illustrator 10 0 R>>/Resources<</ExtGState<</GS0 11 0 R>>/Properties<</MC0 5 0 R/MC1 6 0 R>>>>/Thumb 12 0 R/TrimBox[0.0 0.0 100.0 100.0]/Type/Page>> endobj 9 0 obj <</Filter/FlateDecode/Length 1114>>stream | |
8;VGJ_$pAe$j%5g#Y/8.G]f[^A#Q9pU$sJ@-BS$94,G-<WGQ8j'TDl!%+#tGRN2bX | |
mZsjSF>#9J"p"S6@ES~> endstream endobj 13 0 obj [/Indexed/DeviceRGB 255 14 0 R] endobj 14 0 obj <</Filter[/ASCII85Decode/FlateDecode]/Length 428>>stream | |
8;X]O>EqN@%''O_@%e@?J;%+8(9e>X=MR6S?i^YgA3=].HDXF.R$lIL@"pJ+EP(%0 | |
b]6ajmNZn*!='OQZeQ^Y*,=]?C.B+\Ulg9dhD*"iC[;*=3`oP1[!S^)?1)IZ4dup` | |
E1r!/,*0[*9.aFIR2&b-C#s<Xl5FH@[<=!#6V)uDBXnIr.F>oRZ7Dl%MLY\.?d>Mn | |
6%Q2oYfNRF$$+ON<+]RUJmC0I<jlL.oXisZ;SYU[/7#<&37rclQKqeJe#,UF7Rgb1 | |
VNWFKf>nDZ4OTs0S!saG>GGKUlQ*Q?45:CI&4J'_2j<etJICj7e7nPMb=O6S7UOH< | |
PO7r\I.Hu&e0d&E<.')fERr/l+*W,)q^D*ai5<uuLX.7g/>$XKrcYp0n+Xl_nU*O( | |
l[$6Nn+Z_Nq0]s7hs]`XX1nZ8&94a\~> endstream endobj 5 0 obj <</Intent 15 0 R/Name(docs)/Type/OCG/Usage 16 0 R>> endobj 6 0 obj <</Intent 17 0 R/Name(trunk)/Type/OCG/Usage 18 0 R>> endobj 17 0 obj [/View/Design] endobj 18 0 obj <</CreatorInfo<</Creator(Adobe Illustrator 15.0)/Subtype/Artwork>>>> endobj 15 0 obj [/View/Design] endobj 16 0 obj <</CreatorInfo<</Creator(Adobe Illustrator 15.0)/Subtype/Artwork>>>> endobj 11 0 obj <</AIS false/BM/Normal/CA 1.0/OP false/OPM 1/SA true/SMask/None/Type/ExtGState/ca 1.0/op false>> endobj 10 0 obj <</LastModified(D:20130622155827+10'00')/Private 19 0 R>> endobj 19 0 obj <</AIMetaData 20 0 R/AIPrivateData1 21 0 R/AIPrivateData2 22 0 R/ContainerVersion 11/CreatorVersion 15/NumBlock 2/RoundtripStreamType 1/RoundtripVersion 15>> endobj 20 0 obj <</Length 970>>stream | |
%!PS-Adobe-3.0 %%Creator: Adobe Illustrator(R) 15.0 %%AI8_CreatorVersion: 15.0.0 %%For: (Alex) () %%Title: (noun_project_15207.svg) %%CreationDate: 22/06/13 3:58 PM %%Canvassize: 16383 %%BoundingBox: 258 373 353 444 %%HiResBoundingBox: 258.5 373.0879 352.0889 443.8213 %%DocumentProcessColors: Cyan Magenta Yellow Black %AI5_FileFormat 11.0 %AI12_BuildNumber: 399 %AI3_ColorUsage: Color %AI7_ImageSettings: 0 %%RGBProcessColor: 0 0 0 ([Registration]) %AI3_Cropmarks: 255.5 359.5449 355.5 459.5449 %AI3_TemplateBox: 305.5 396.5 305.5 396.5 %AI3_TileBox: 26 29.5449 585 812.5449 %AI3_DocumentPreview: None %AI5_ArtSize: 14400 14400 %AI5_RulerUnits: 2 %AI9_ColorModel: 1 %AI5_ArtFlags: 0 0 0 1 0 0 1 0 0 %AI5_TargetResolution: 800 %AI5_NumLayers: 2 %AI9_OpenToView: -491 815 1 1595 821 18 0 0 43 134 0 0 0 1 1 0 1 1 0 1 %AI5_OpenViewLayers: 77 %%PageOrigin:0 0 %AI7_GridSettings: 72 8 72 8 1 0 0.8 0.8 0.8 0.9 0.9 0.9 %AI9_Flatten: 1 %AI12_CMSettings: 00.MS %%EndComments endstream endobj 21 0 obj <</Length 16577>>stream | |
%%BoundingBox: 258 373 353 444 %%HiResBoundingBox: 258.5 373.0879 352.0889 443.8213 %AI7_Thumbnail: 128 100 8 %%BeginData: 16428 Hex Bytes %0000330000660000990000CC0033000033330033660033990033CC0033FF %0066000066330066660066990066CC0066FF009900009933009966009999 %0099CC0099FF00CC0000CC3300CC6600CC9900CCCC00CCFF00FF3300FF66 %00FF9900FFCC3300003300333300663300993300CC3300FF333300333333 %3333663333993333CC3333FF3366003366333366663366993366CC3366FF %3399003399333399663399993399CC3399FF33CC0033CC3333CC6633CC99 %33CCCC33CCFF33FF0033FF3333FF6633FF9933FFCC33FFFF660000660033 %6600666600996600CC6600FF6633006633336633666633996633CC6633FF %6666006666336666666666996666CC6666FF669900669933669966669999 %6699CC6699FF66CC0066CC3366CC6666CC9966CCCC66CCFF66FF0066FF33 %66FF6666FF9966FFCC66FFFF9900009900339900669900999900CC9900FF %9933009933339933669933999933CC9933FF996600996633996666996699 %9966CC9966FF9999009999339999669999999999CC9999FF99CC0099CC33 %99CC6699CC9999CCCC99CCFF99FF0099FF3399FF6699FF9999FFCC99FFFF %CC0000CC0033CC0066CC0099CC00CCCC00FFCC3300CC3333CC3366CC3399 %CC33CCCC33FFCC6600CC6633CC6666CC6699CC66CCCC66FFCC9900CC9933 %CC9966CC9999CC99CCCC99FFCCCC00CCCC33CCCC66CCCC99CCCCCCCCCCFF %CCFF00CCFF33CCFF66CCFF99CCFFCCCCFFFFFF0033FF0066FF0099FF00CC %FF3300FF3333FF3366FF3399FF33CCFF33FFFF6600FF6633FF6666FF6699 %FF66CCFF66FFFF9900FF9933FF9966FF9999FF99CCFF99FFFFCC00FFCC33 %FFCC66FFCC99FFCCCCFFCCFFFFFF33FFFF66FFFF99FFFFCC110000001100 %000011111111220000002200000022222222440000004400000044444444 %550000005500000055555555770000007700000077777777880000008800 %000088888888AA000000AA000000AAAAAAAABB000000BB000000BBBBBBBB %DD000000DD000000DDDDDDDDEE000000EE000000EEEEEEEE0000000000FF %00FF0000FFFFFF0000FF00FFFFFF00FFFFFF %524C45FD6BFFA8CAA7A7A0A7A1FD76FFA8A77CA17CA17CA17CA1A7FD74FF %A1A17CA7A1A17CA7A1A17CA7FD72FFA77CA17CA17CA17CA17CA17CA17CA8 %FD6FFFCFA77CA7A1A7A0A7A1A7A0A7A1A7A0A7A1FD6EFFA8A17CA17CA17C %A17CA17CA17CA17CA17CA1A7FD6CFFA8A17CA7A1A17CA7A1A17CA7A1A17C %A7A1A17CA7FD6BFFA1A17CA17CA17CA17CA17CA17CA17CA17CA17CA17CA8 %FD69FFA7A1A1A7A0A7A1A7A0A7A1A7A0A7A1A7A0A7A1A7A0A7A1FD68FFA7 %A07CA17CA17CA17CA17CA17CA17CA17CA17CA17CA17CA1A7FD67FFCAA0A1 %7CA7A1A17CA7A1A17CA7A1A17CA7A1A17CA7A1A17CA7CFFD67FFA77CA17C %A17CA17CA17CA17CA17CA17CA17CA17CA17CA17CA7FD68FFA7A0A7A1A7A0 %A7A1A7A0A7A1A7A0A7A1A7A0A7A1A7A0A7A0FD68FFA8A07CA17CA17CA17C %A17CA17CA17CA17CA17CA17CA17CA1A7FD4EFFA8FD19FFA7A17CA7A1A17C %A7A1A17CA7A1A17CA7A1A17CA7A1CAA8FD4FFFA17CA7A1A8A8FD15FF7CA1 %7CA17CA17CA17CA17CA17CA17CA17CA7A8FD52FFA0A7A0A17CA7A1A7A7FF %CFFD10FFCF7CA7A1A7A0A7A1A7A0A7A1A7A0A7A7FD54FFA8A17CA17CA17C %A17CA07CA1A1A7A7FFCFFD0CFFA77CA17CA17CA17CA17CA17CA1A8FD55FF %CAA0A17CA7A1A17CA7A1A17CA7A0A17CA7A1A8A8FD0AFFA77CA7A1A17CA7 %A1A17CA7CAFD56FFA1A17CA17CA17CA17CA17CA17CA17CA17CA17CA17CA8 %FD08FFA8A07CA17CA17CA17CA1A8FD4CFFCFFFA8CFA7A8A1FD04FFA7A1A7 %A0A7A1A7A0A7A1A7A0A7A1A7A0A7A1A7A0A1A8FD09FFA7A7A1A7A0A7A0A7 %CFFD44FFA8FFA8A8A7A8A1A17CA17CA17CA17CA0A7FFFFCF7CA17CA17CA1 %7CA17CA17CA17CA17CA17CA17CA17CCFFD0AFF7CA17CA17CA1A8FD42FFA8 %A7A1A17CA1A0A17CA1A0A17CA7A1A17CA7A0A7FFFFA8A17CA7A1A17CA7A1 %A17CA7A1A17CA7A1A17CA7A0A7CFFD0BFF7CA7A0A1A8FD43FFCF7CA17CA1 %7CA17CA17CA17CA17CA17CA17CA17CCFFFA87CA17CA17CA17CA17CA17CA1 %A1A77CA17CA17CA1A1FD0CFFA87CA1A8FD45FFCAA0A7A0A7A1A7A0A7A1A7 %A0A7A1A7A0A7A1A1A8FFA1A1A0A7A1A7A0A7A1A77CA7A8FFA8A7A1A7A0A7 %A0CAFD0DFFA7A8FD46FFA8A17CA17CA17CA17CA17CA17CA17CA17CA17CA7 %A8A77CA17CA17CA17CA17CA1A8FFFFFF7CA17CA17CA1A8FD0DFFA8FD48FF %A0A7A1A17CA7A1A17CA7A1A17CA7A1A17CA7A1CA7CA7A1A17CA7A1A17CCA %FD05FFA77CA7A1A17CFD57FFA77CA17CA17CA17CA17CA17CA17CA17CA17C %A1A1A17CA17CA17CA1A1FD07FFA8A07CA17CA1A8CFA8CAA8CACAFFFFFFA8 %FD4CFFA8A7A0A1A0A7A1A7A0A7A1A7A0A7A1A7A0A7A1A7A0A7A1A17CA7A8 %FD09FFA0A7A1A7A0A7A0A1A0A1A7FD04FFA7A0FD4CFFCACFA1A17CA17CA1 %7CA17CA17CA17CA17CA17CA17CA1A1A8FD06FFA1A8FFFFFFA77CA17CA17C %A17CA17CFD04FFA77CA17CA8FD4DFFCFCAA1A1A0A17CA1A0A17CA1A0A17C %A7A0A7A8FD06FFCFA1A1A1FFFFFFA8A17CA7A1A17CA7A0FD04FFCA7CA7A1 %A17CA7FD4FFFA8A8A1A7A1A7A1A7A1A7A1A8A8FD07FFA8A77CA17CA1A8FF %FFFF7CA17CA17CA17CCAFD04FF7CA17CA17CA17CA1A8FD47FFBD9ACBFD16 %FFA7A7A0A7A0A7A0CAFFFFFFA7A0A7A1A77CCAFD04FFA1A1A0A7A1A7A0A7 %A0A1A7FD46FF688C6894A1FD11FFA8A87CA07CA17CA17CA1A1FFFFFFA7A0 %7CA17CA1A8FFFFFFA1A17CA17CA17CA17CA17CA0A1FD44FFA18C8C8C688C %8DBDA8FD0BFFA8CFA7A17CA7A1A17CA7A1A17CA7FD04FFA0A17CA1A8FFFF %FFA8A17CA7A1A17CA7A1A17CA7A1A1A0FD43FF93688C688C688C688C6FC3 %CAFD05FFA8A87CA076A17CA17CA17CA17CA17CA17CCAFFFFA8A77CA0A1FF %FFFFA8A17CA17CA17CA17CA17CA17CA17CA17CA8FD40FFCA688D8C8D688D %8C8D688C68939ACAFD06FFA8A7A0A1A0A7A1A7A0A7A1A7A0A7A7FFFFFFA7 %A1A1FD04FFA7A0A7A0A7A1A7A0A7A1A7A0A7A1A7A0A7A0A7FD3FFF9A8C68 %8C688C688C688C688C688C6893A1FD07FFA7A17CA17CA17CA17CA17CA1CF %FFFFCF7CFD04FFA87CA17CA17CA17CA17CA17CA17CA17CA17CA17CA1A8FD %3DFF938C8C688D8C8C688D8C8C688D8C8C688C8C9AA1FD06FFCAA1A7A0A1 %7CA7A1A17CCFFFFFCFCFFD04FFA0A17CA7A1A17CA7A1A17CA7A0A17CA7A1 %A17CA7A0A1A7FD3BFFC4688C688C688C688C688C688C688C688C688C688C %69C3A8FD05FFA8A87CA07CA17CA0A1FD07FF7CA17CA17CA17CA17CA17CA1 %7CA7A8A87CA17CA17CA17CA0A1FD3AFF948C8C8D688D8C8D688D8C8D688D %8C8D688D8C8D688C689394CAFD06FFA8A8A1A17CA7FD06FFA7A7A1A7A0A7 %A1A7A0A7A0A1A1FD04FFCAA0A7A0A7A1A77CA7FD39FFCA8C688C688C688C %688C688C688C688C688C688C688C688C688C68939ACAFD06FFA7A17CCFFD %04FFA8A07CA17CA17CA17CA17CA1A8FD06FFA77CA17CA17CA1A8FD39FFC3 %688C688D8C8C688D8C8C688D8C8C688D8C8C688D8C8C688D8C8C688C8C9A %A1FD06FFCAA8FFFFFFA8A7A0A17CA7A1A17CA1A0A8CFFD07FFCFA7A0A17C %A1A8FD3AFF6F8C688C688C688C688C688C688C688C688C688C688C688C68 %8C688C688C688C689BA8FD08FFA17CA17CA17CA17CA07CCFFD05FFCAA8FF %FFFFA8A17CA0A1FD3AFFCA8D8C8D688D8C8D688D8C8D688D8C8D688D8C8D %688D8C8D688D8C8D688D8C8D688C688D93CAFD06FFA8CAA1A1A0A7A0A1A7 %FD06FFCA7CA7FD04FFA8A1A1FD3BFF9A688C688C688C688C688C688C688C %688C688C688C688C688C688C688C688C688C688C688C68939ACAFD06FFA7 %A17CA7A8FD05FFA8A77CA17CA7FD04FFA7CAFD3BFF8D8C688D8C8C688D8C %8C688D8C8C688D8C8C688D8C8C688D8C8C688D8C8C688D8C8C688D8C8C68 %8C8C94A1FD06FFCFFD06FFA1A17CA7A1A17CA8FD3FFFA18C688C688C688C %688C688C688C688C688C688C688C688C688C688C688C688C688C688C688C %688C688C688C689AA8FD08FFA87CA17CA17CA17CA17CA8FD0EFFA8FD2FFF %BC8C8D688D8C8D688D8C8D688D8C8D688D8C8D688D8C8D688D8C8D688D8C %8D688D8C8D688D8C8D688D8C8D688C688C93C3FD06FFA8CAA1A1A0A7A1A7 %A0A7A0CFFD0BFFCAA1A7FD2EFFCA688C688C688C688C688C688C688C688C %688C688C688C688C688C688C688C688C688C688C688C688C688C688C688C %688C68939ACAFD06FFA8A77CA17CA17CA17CCFFD07FFCAA1A17CA17CFD2E %FFA18C688D8C8C688D8C8C688D8C8C688D8C8C688D8C8C688D8C8C688D8C %8C688D8C8C688D8C8C688D8C8C688D8C8C688D8C8C688C8C93A1FD07FFFD %04A17CA1A1FD05FFA7A7A0A17CA7A1A1A8FD2DFF93688C688C688C688C68 %8C688C688C688C688C688C688C688C688C688C688C688C688C688C688C68 %8C688C688C688C688C688C688C688C689AA1FD05FF7CA17CA17CA1A1FD04 %FFA77CA17CA17CA17CA8FD0FFFA1FD14CAFD08FFCA8C8C688D8C8D688D8C %8D688D8C8D688D8C8D688D8C8D688D8C8D688D8C8D688D8C8D688D8C8D68 %8D8C8D688D8C8D688D8C8D688D8C8D688D688C93C3CAFFFFCFA0A7A0A7A1 %A1A7FD04FFA8A0A7A0A7A1A1A7FD0DFFCA688C688C688C688C688C688C68 %8C688C688C688C688CA8FD07FF9A8C688C688C688C688C688C688C688C68 %8C688C688C688C688C688C688C688C688C688C688C688C688C688C688C68 %8C688C688C688C688C688C688C688C9BFFFFA77CA17CA17CA0A7FFFFFFA8 %A77CA17CA17CA7FD0CFFCA688C8C8C688C8C8C688C8C8C688C8C8C688C8C %8C688C8CCAFD07FF93688D8C8C688D8C8C688D8C8C688D8C8C688D8C8C68 %8D8C8C688D8C8C688D8C8C688D8C8C688D8C8C688D8C8C688D8C8C688D8C %8C688D8C8C688D8C8C688CCAFFA8A1A1A17CA7A0A1A8FFFFFFA8A7A0A17C %A7A1FD0BFFCA688C688C688C688C688C688C688C688C688C688C688C688C %A8FD06FFC3688C688C688C688C688C688C688C688C688C688C688C688C68 %8C688C688C688C688C688C688C688C688C688C688C688C688C688C688C68 %8C688C688C688C689AFFFFA1A17CA17CA17CA1A8FFFFFFA8A17CA17CA1CA %FD0AFF8C8C688D8C8D688D8C8D688D8C8D688D8C8D688D8C8D688D8CFD07 %FF938C688D8C8D688D8C8D688D8C8D688D8C8D688D8C8D688D8C8D688D8C %8D688D8C8D688D8C8D688D8C8D688D8C8D688D8C8D688D8C8D688D8C8D68 %8D8C8D688D8CCAFFFFA0A7A0A7A1A77CA7FD04FFA8A1A0A7A1FD09FFCA68 %8C688C688C688C688C688C688C688C688C688C688C688C688CCAFD05FFA8 %8C688C688C688C688C688C688C688C688C688C688C688C688C688C688C68 %8C688C688C688C688C688C688C688C688C688C688C688C688C688C688C68 %8C688C688C93FFFFA87CA17CA17CA7A8FD05FFA7A17CA1A8FD08FF8C8C68 %8D8C8C688D8C8C688D8C8C688D8C8C688D8C8C688D8C8C68FD06FFC3688D %8C8C688D8C8C688D8C8C688D8C8C688D8C8C688D8C8C688D8C8C688D8C8C %688D8C8C688D8C8C688D8C8C688D8C8C688D8C8C688D8C8C688D8C8C688D %8C8C688CA1FFCFA7A0A1A1FD09FFA1A17CCAFD06FFCB688C688C688C688C %688C688C688C688C688C688C688C688C688C688CA8FD05FF938C688C688C %688C688C688C688C688C688C688C688C688C688C688C688C688C688C688C %688C688C688C688C688C688C688C688C688C688C688C688C688C688C688C %6893CAFFA7A7A8FD07FFA8A77CA17CA0A1FD06FF8D8C8C8D688D8C8D688D %8C8D688D8C8D688D8C8D688D8C8D688D8C8C8CFD05FFCA8C688D8C8D688D %8C8D688D8C8D688D8C8D688D8C8D688D8C8D688D8C8D688D8C8D688D8C8D %688D8C8D688D8C8D688D8C8D688D8C8D688D8C8D688D8C8D688D8C8D688D %68C3FD08FFA8CAA1A17CA7A1A77CA8FD05FF8C8C688C688C688C688C688C %688C688C688C688C688C688C688C688C688CA8FD04FF9A688C688C688C68 %8C688C688C688C688C688C688C688C688C688C688C688C688C688C688C68 %8C688C688C688C688C688C688C688C688C688C688C688C688C688C688C68 %8C68CAFD05FFCAA1A17CA17CA17CA17CA17CFD04FF8D8C8C8C688D8C8C68 %8D8C8C688D8C8C688D8C8C688D8C8C688D8C8C688C8CCAFD04FFFD048C68 %8D8C8C688D8C8C688D8C8C688D8C8C688D8C8C688D8C8C688D8C8C688D8C %8C688D8C8C688D8C8C688D8C8C688D8C8C688D8C8C688D8C8C688D8C8C68 %8D8C8C688C9AFFFFFFCAA77CA1A1A17CA7A1A17CA7A0A1CFFFFF698C688C %688C688C688C688C688C688C688C688C688C688C688C688C688C688CA8FF %FFFFA18C688C688C688C688C688C688C688C688C688C688C688C688C688C %688C688C688C688C688C688C688C688C688C688C688C688C688C688C688C %688C688C688C688C688C688C688CA1FFA8A17CA17CA17CA17CA17CA17CA1 %7CCFFF938C688D8C8D688D8C8D688D8C8D688D8C8D688D8C8D688D8C8D68 %8D8C8D688D8CFD04FF9A688D8C8D688D8C8D688D8C8D688D8C8D688D8C8D %688D8C8D688D8C8D688D8C8D688D8C8D688D8C8D688D8C8D688D8C8D688D %8C8D688D8C8D688D8C8D688D8C8D688D8C8D688D8C94FFFFA7A1A0A7A1A7 %A0A7A1A7A0A7A1A1A8698C688C688C688C688C688C688C688C688C688C68 %8C688C688C688C688C688C688CCAFFFFCA688C688C688C688C688C688C68 %8C688C688C688C688C688C688C688C688C688C688C688C688C688C688C68 %8C688C688C688C688C688C688C688C688C688C688C688C688C688C688C68 %C3FFFF7CA17CA17CA17CA17CA17CA17CA88C688D8C8C688D8C8C688D8C8C %688D8C8C688D8C8C688D8C8C688D8C8C688D8C8C68FFFFFFCAC3938C688C %8C8C688D8C8C688D8C8C688D8C8C688D8C8C688D8C8C688D8C8C688D8C8C %688D8C8C688D8C8C688D8C8C688D8C8C688D8C8C688D8C8C688D8C8C688D %8C8C688D8C8C688C93FFFFCAA0A17CA7A1A17CA7A1A17CA1A7688C688C68 %8C688C688C688C688C688C688C688C688C688C688C688C688C688C688CA8 %FD05FFA194688C688C688C688C688C688C688C688C688C688C688C688C68 %8C688C688C688C688C688C688C688C688C688C688C688C688C688C688C68 %8C688C688C688C688C688C688C688C9AFFA8A17CA17CA17CA17CA17CA17C %A78D688D8C8D688D8C8D688D8C8D688D8C8D688D8C8D688D8C8D688D8C8D %688D8C8C8CFD09FFA193688C8C8D688D8C8D688D8C8D688D8C8D688D8C8D %688D8C8D688D8C8D688D8C8D688D8C8D688D8C8D688D8C8D688D8C8D688D %8C8D688D8C8D688D8C8D688D8C8D688D8C8DCAFFA8A1A0A7A1A7A0A7A1A1 %7CA7A1688C688C688C688C688C688C688C688C688C688C688C688C688C68 %8C688C688C688CA8FD0AFFCA938D688C688C688C688C688C688C688C688C %688C688C688C688C688C688C688C688C688C688C688C688C688C688C688C %688C688C688C688C688C688C688C688C688C689AFFFFA1A17CA17CA17CA1 %A0A7A7FF8D8C8C688D8C8C688D8C8C688D8C8C688D8C8C688D8C8C688D8C %8C688D8C8C688C8CCAFD0CFFCAC3938C688C8C8C688D8C8C688D8C8C688D %8C8C688D8C8C688D8C8C688D8C8C688D8C8C688D8C8C688D8C8C688D8C8C %688D8C8C688D8C8C688D8C8C688D8C8C688D8CCAFFFFA0A17CA7A1CAA8FD %04FF688C688C688C688C688C688C688C688C688C688C688C688C688C688C %688C688C688CA8FD0FFFA19A688C688C688C688C688C688C688C688C688C %688C688C688C688C688C688C688C688C688C688C688C688C688C688C688C %688C688C688C688C688C688C688C93FFFFCAA7CFA8FD07FF8D8C8D688D8C %8D688D8C8D688D8C8D688D8C8D688D8C8C688C8C938C8D8C8C688D8CFD13 %FFA193688C8C8D688D8C8D688D8C8D688D8C8D688D8C8D688D8C8D688D8C %8C688D8C93688C688C688D8C8D688D8C8D688D8C8D688D8C8D688D8C8D68 %8D8C9AFD0DFF688C688C688C688C688C688C688C688C688C688C688C939A %A1CACACAA1C39A93688CCAFD14FFCA9493688C688C688C688C688C688C68 %8C688C688C688C688C68939AC3A1CACACAA1C3938C688C688C688C688C68 %8C688C688C688C688C688C688CA1FD0DFF8C688D8C8C688D8C8C688D8C8C %688D8C8C688C8C9AA8FD0AFFCA93CBFD16FFCAC3938C688C8C8C688D8C8C %688D8C8C688D8C8C688C93CAFD0AFFCA9A688C8C8C688D8C8C688D8C8C68 %8D8C8C688D8C8C93FD0EFF688C688C688C688C688C688C688C688C688C68 %CAFD0EFFA8FD19FFA89A688C688C688C688C688C688C688C688C9AFD0EFF %CA698C688C688C688C688C688C688C688C688C68A1FFFFA7CFFD0AFF8D68 %8D8C8D688D8C8D688D8C8D688D8C8C93FD07FFA1C4A1CACAFD21FFA19A68 %8C8C8D688D8C8D688D8C93CAFD05FFCBCAA1C4C3CBFD06FF9A8C688D8C8D %688D8C8D688D8C8D688D8C8DCAFFA8A1A0A8A8FD08FF688C688C688C688C %688C688C688C688C93FD05FFA193688C688C688C93C4FD21FFCA9A93688C %688C688C6893CAFD04FFCA9393688C688C68939AFD05FF9A8C688C688C68 %8C688C688C688C688C93FFFFA77CA17CA1A1FD07FF8D8C8C688D8C8C688D %8C8C688D8C8C8DFD04FFCA938C688C8C8C688D8C8C6894CAFD04FFC39AC3 %9ABD9AC39ABD9AC39ABD9AC39ABD9AC39ABD9BFD08FFCA938D688C8C8CA8 %FFFFFFCA9A688C688D8C8C688C688C8CCAFD04FF938C688D8C8C688D8C8C %688D8C8C68CAFFFF7CA7A1A17CA1A1CFFD05FF688C688C688C688C688C68 %8C688C68CAFFFFFFC3688C688C688C688C688C688C688CA1FFFFFFA16868 %8C688C688C688C688C688C688C688C688C688C689AA1FD07FFA8A1688CA1 %FFFFFFA88C688C688C688C688C688C688C68A1FD04FF688C688C688C688C %688C688C688DA8FFA7A07CA17CA17CA17CA8FD04FF8D8C8D688D8C8D688D %8C8D688D68C3FFFFFFCA688C8C8D688D8C8D688D8C8D688D8C8CCAFFFFFF %948C8C8D688D8C8D688D8C8D688D8C8D688D8C8D688D688C93C3CAFD07FF %CAFFFFFFCA93688D8C8D688D8C8D688D8C8D688D68C3FFFFFFC3688D8C8D %688D8C8D688D8C8C9AFFFFA7A0A7A1A7A0A7A1A17CCFFFFFFF688C688C68 %8C688C688C688C688DCAFFFFFF688C688C688C688C688C688C688C688C68 %93CAFFFFCA688C688C688C688C688C688C688C688C688C688C688C688C68 %8D93C4CAFD08FF93688C688C688C688C688C688C688C688C68CAFFFFFF93 %688C688C688C688C688C68CAFFCF7CA17CA17CA17CA17CA17CCAFFFF8C68 %8D8C8C688D8C8C688D8C8C9AFFFFFF948C8C8C688D8C8C688D8C8C688D8C %8C688D68C3FFFFFF9A688D8C8C688D8C8C688D8C8C688D8C8C688D8C8C68 %8D8C8C688C68939ACAFD05FFC3688D8C8C688D8C8C688D8C8C688D8C8C68 %8C93FFFFFFA18C688D8C8C688D8C8C6893FFFFFD04A17CA7A1A17CA7A1A1 %7CCFFF688C688C688C688C688C688C68CAFFFFA88C688C688C688C688C68 %8C688C688C688C688C68FFFFFFA18C688C688C688C688C688C688C688C68 %8C688C688C688C688C688C688C6894A8FFFFFF8D8C688C688C688C688C68 %8C688C688C688C688CA1FFFFCA688C688C688C688C688C9AFFA8A17CA17C %A17CA17CA17CA17CA17CCA8C688C8C8C688C8C8C688C8C93FFFFFFC3688D %8C8D688D8C8D688D8C8D688D8C8D688D8C8CA1FFFFFF688C8C8C688C8C8C %688C8C8C688C8C8C688C8C8C688C8C8C688C8C8C688C68CAFFFFCA8C688D %8C8D688D8C8D688D8C8D688D8C8D688D68BDFFFFFF93688D8C8D688D8C8C %8CFFFFCA7CA7A1A7A0A7A1A7A0A7A1A7A0A7A18D9368938D9368938D9368 %9393FFFFFF938C688C688C688C688C688C688C688C688C688C689AFFFFFF %9368938D9368938D9368938D9368938D9368938D9368938D9368938D9368 %93CAFFFFC3688C688C688C688C688C688C688C688C688C688C93FFFFFF94 %8C688C688C688C689AFFFFA1A17CA17CA17CA17CA17CA17CA17CFD04FFCA %FFFFFFCAFFFFFFCAFD04FF938C8C688D8C8C688D8C8C688D8C8C688D8C8C %688C93FD05FFCAFFFFFFCAFFFFFFCAFFFFFFCAFFFFFFCAFFFFFFCAFFFFFF %CAFFFFFFCAFFFFFF9A8C8C8C688D8C8C688D8C8C688D8C8C688D8C8C6893 %FFFFFFC3688C688D8C8C688CA1FFA8A1A1A17CA7A1A17CA7A1A17CA7A0CF %FD11FF688C688C688C688C688C688C688C688C688C688C6894FD25FF9A68 %8C688C688C688C688C688C688C688C688C688C68FFFFFF9A8C688C688C68 %8C68FFFFA77CA17CA17CA17CA17CA17CA17CA8FD12FF938C8D688D8C8D68 %8D8C8D688D8C8D688D8C8D688C93FD25FF9A8C8C8D688D8C8D688D8C8D68 %8D8C8D688D8C8D6893FFFFFFC3688D688D8C8D68C3FFFFA0A7A1A7A0A7A1 %A7A0A7A1A77CCAFD13FF938C688C688C688C688C688C688C688C688C688C %689AFD25FF9B688C688C688C688C688C688C688C688C688C688C68FFFFFF %938C688C688C688CA2FFFFA77CA17CA17CA17CA17CA17CA1CAFD13FFBD68 %8D8C8C688D8C8C688D8C8C688D8C8C688D8C8CA1FD25FFA18C688D8C8C68 %8D8C8C688D8C8C688D8C8C688D689AFFFFFF9A688C8C8C688C93FD04FFCA %7CA7A1A17CA7A1A17CA7CFFD14FFA18C688C688C688C688C688C688C688C %688C688C68CAFD26FF688C688C688C688C688C688C688C688C688C688CA1 %FFFFFFCAC3938C688C68C3FD05FFA87CA17CA17CA17CA0A7FD16FF938C8C %8D688D8C8D688D8C8D688D8C8D688D689AFD27FFBC688D8C8D688D8C8D68 %8D8C8D688D8C8D688C8CFD07FFCAC38D8DCAFD06FFCAA0A7A0A7A1A1A7FD %17FFC4688C688C688C688C688C688C688C688C688CA8FD27FFCA8C688C68 %8C688C688C688C688C688C688C68C3FD0AFFA8FD07FFCFA77CA17CA1A1FD %19FF9A688D8C8C688D8C8C688D8C8C688D8C8C9AFD29FFA18C688D8C8C68 %8D8C8C688D8C8C688D689AFD14FFCAA7A0A1A0FD1BFF93688C688C688C68 %8C688C688C688C94FD2BFF9A8C688C688C688C688C688C688C6893CAFD15 %FFA8A17CCAFD1CFFBD8C8C688D8C8D688D8C8C688C9BFD2DFFA18D688C68 %8D8C8D688D8C8C68BCFD18FFA7CAFD1EFFCA938C688C688C688C6893A1FD %2FFFA89A688C688C688C688C93C3FD3BFFCAC3939A939A9ACAFD33FFCBA1 %9A939A94C3A1FDA5FFFF %%EndData endstream endobj 22 0 obj <</Length 21057>>stream | |
0000000016 00000 n | |
0000000156 00000 n | |
0000022171 00000 n | |
0000000000 00000 f | |
0000024532 00000 n | |
0000024599 00000 n | |
0000064018 00000 n | |
0000022222 00000 n | |
0000022562 00000 n | |
0000025012 00000 n | |
0000024899 00000 n | |
0000023745 00000 n | |
0000023971 00000 n | |
0000024019 00000 n | |
0000024783 00000 n | |
0000024814 00000 n | |
0000024667 00000 n | |
0000024698 00000 n | |
0000025086 00000 n | |
0000025260 00000 n | |
0000026280 00000 n | |
0000042909 00000 n | |
0000064047 00000 n | |
trailer <</Size 24/Root 1 0 R/Info 23 0 R/ID[<A0B43D3BE9404E559B188985E5B021C8><89A09C775629479581326F0B9EF6E592>]>> startxref 64215 %%EOF |
<?xml version="1.0" encoding="utf-8"?> | |
<!-- Generator: Adobe Illustrator 15.0.0, SVG Export Plug-In . SVG Version: 6.00 Build 0) --> | |
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd"> | |
<svg version="1.1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" x="0px" y="0px" width="100px" | |
height="100px" viewBox="0 -25.635 100 100" enable-background="new 0 -25.635 100 100" xml:space="preserve"> | |
<g id="docs"> | |
<path fill="#C2A385" d="M86.108-9.909l5.229,9.952c0,0-1.832,0.083-5.297,1.95c-2.312,1.249-6.468,6.246-6.468,6.246L71.827-2.909 | |
c0,0,4.201-3.996,6.513-5.242C81.805-10.022,86.108-9.909,86.108-9.909"/> | |
<path fill="#C2A385" d="M65.604,20.731l-3.152-9.868c0,0-5.441,3.56-8.017,4.074c-1.008,0.202-1.93,0.335-2.749,0.425 | |
L65.604,20.731z"/> | |
<path fill="#C2A385" d="M72.326,23.321c0.268-0.226,0.537-0.44,0.804-0.616c3.104-2.054,6.139-3.685,6.268-3.755l1.882-1.005 | |
l1.369,1.634l2.864,3.417l3.198-4.334L76.68,9.783l-8.74,11.847L72.326,23.321z"/> | |
<path fill="#C2A385" d="M39.918,10.823l4.825,1.86l3.33,0.212c0.04,0.001,0.269,0.015,0.652,0.015c0.91,0,2.798-0.072,5.196-0.551 | |
c1.427-0.284,5.007-2.332,7.092-3.695l2.889-1.888l1.05,3.285l2.496,7.812l5.889-7.985l-4.625,0.163l1.348-6.225L55.133,0.593 | |
l-2.095,9.667c-0.531-2.599-1.841-5.727-1.841-5.727L37.709,6.055c0,0,0.885,2.206,1.586,4.529L39.918,10.823z"/> | |
<path fill="#C2A385" d="M91.233,45.562c-1.102-0.691-2.323-1.142-3.415-1.433l-3.779,9.804c1.932,1.246,5.197,5.738,5.197,5.738 | |
l7.336-9.206C96.572,50.466,93.162,46.771,91.233,45.562z"/> | |
<path fill="#C2A385" d="M93.192,32.166l-3.656,1.224c-0.019,0.007-1.779,0.613-4.117,2.069l2.817,4.868l0.626,1.08 | |
c3.306-0.562,7.727-1.922,7.727-1.922l-2.332-15.261c0,0-2.934,1.277-5.852,2.221l2.318,2.765L93.192,32.166z"/> | |
<path fill="#C2A385" d="M79.272,25.999l0.864,0.334l0.46,0.801l3.503,6.05c2.646-1.636,4.611-2.287,4.611-2.287l-8.075-9.632 | |
c0,0-2.584,1.391-5.376,3.188L79.272,25.999z"/> | |
</g> | |
<g id="trunk"> | |
<circle fill="#C00000" cx="66.019" cy="52.945" r="7.877"/> | |
<circle fill="#C00000" cx="22.693" cy="52.945" r="7.877"/> | |
<path fill="#C00000" d="M22.693,42.441c1.915,0,3.705,0.522,5.251,1.421V28.001H12.191L3,37.192v14.439h9.281 | |
C12.931,46.459,17.347,42.441,22.693,42.441z"/> | |
<path fill="#C00000" d="M55.925,50.075l-9.583-3.695H30.88c1.186,1.476,1.978,3.28,2.225,5.252h22.502 | |
C55.674,51.1,55.78,50.58,55.925,50.075z"/> | |
<path fill="#C00000" d="M79.312,28.328L39.961,13.149l-9.384,24.335l26.381,10.174c1.824-3.115,5.198-5.218,9.062-5.218 | |
c5.791,0,10.503,4.712,10.503,10.502c0,0.744-0.081,1.471-0.229,2.173l4.713,1.817L86.95,41.52L79.312,28.328z"/> | |
</g> | |
</svg> | |
<?php | <?php |
require_once '../include/common.inc.php'; | require_once '../include/common.inc.php'; |
//function createFOIDocumentsDesignDoc() { | //function createFOIDocumentsDesignDoc() { |
$foidb = $server->get_db('disclosr-foidocuments'); | $foidb = $server->get_db('disclosr-foidocuments'); |
$obj = new stdClass(); | $obj = new stdClass(); |
$obj->_id = "_design/" . urlencode("app"); | $obj->_id = "_design/" . urlencode("app"); |
$obj->language = "javascript"; | $obj->language = "javascript"; |
$obj->views->all->map = "function(doc) { emit(doc._id, doc); };"; | $obj->views->all->map = "function(doc) { emit(doc._id, doc); };"; |
$obj->views->byDate->map = "function(doc) { emit(doc.date, doc); };"; | $obj->views->byDate->map = "function(doc) { if (doc.title != \"Disclosure Log Updated\") { emit(doc.date, doc); } };"; |
$obj->views->byDateMonthYear->map = "function(doc) { emit(doc.date, doc); };"; | $obj->views->byDateMonthYear->map = "function(doc) { emit(doc.date, doc); };"; |
$obj->views->byDateMonthYear->reduce = "_count"; | $obj->views->byDateMonthYear->reduce = "_count"; |
$obj->views->byAgencyID->map = "function(doc) { emit(doc.agencyID, doc); };"; | $obj->views->byAgencyID->map = "function(doc) { emit(doc.agencyID, doc); };"; |
$obj->views->byAgencyID->reduce = "_count"; | $obj->views->byAgencyID->reduce = "_count"; |
$obj->views->fieldNames->map = ' | $obj->views->fieldNames->map = 'function(doc) { for(var propName in doc) { emit(propName, doc._id); }}'; |
function(doc) { | $obj->views->fieldNames->reduce = 'function (key, values, rereduce) { return values.length; }'; |
for(var propName in doc) { | |
emit(propName, doc._id); | |
} | |
}'; | |
$obj->views->fieldNames->reduce = 'function (key, values, rereduce) { | |
return values.length; | |
}'; | |
// allow safe updates (even if slightly slower due to extra: rev-detection check). | // allow safe updates (even if slightly slower due to extra: rev-detection check). |
$foidb->save($obj, true); | $foidb->save($obj, true); |
//function createDocumentsDesignDoc() { | //function createDocumentsDesignDoc() { |
$docdb = $server->get_db('disclosr-documents'); | $docdb = $server->get_db('disclosr-documents'); |
$obj = new stdClass(); | $obj = new stdClass(); |
$obj->_id = "_design/" . urlencode("app"); | $obj->_id = "_design/" . urlencode("app"); |
$obj->language = "javascript"; | $obj->language = "javascript"; |
$obj->views->web_server->map = "function(doc) {\n emit(doc.web_server, 1);\n}"; | $obj->views->web_server->map = "function(doc) {\n emit(doc.web_server, 1);\n}"; |
$obj->views->web_server->reduce = "_sum"; | $obj->views->web_server->reduce = "_sum"; |
$obj->views->byAgency->map = "function(doc) {\n emit(doc.agencyID, 1);\n}"; | $obj->views->byAgency->map = "function(doc) {\n emit(doc.agencyID, 1);\n}"; |
$obj->views->byAgency->reduce = "_sum"; | $obj->views->byAgency->reduce = "_sum"; |
$obj->views->byURL->map = "function(doc) {\n emit(doc.url, doc);\n}"; | $obj->views->byURL->map = "function(doc) {\n emit(doc.url, doc);\n}"; |
$obj->views->agency->map = "function(doc) {\n emit(doc.agencyID, doc);\n}"; | $obj->views->agency->map = "function(doc) {\n emit(doc.agencyID, doc);\n}"; |
$obj->views->byWebServer->map = "function(doc) {\n emit(doc.web_server, doc);\n}"; | $obj->views->byWebServer->map = "function(doc) {\n emit(doc.web_server, doc);\n}"; |
$obj->views->datasets->map = "function(doc) {\nif (doc.fieldName == \"data\") {\n emit(doc._id, doc);\n}\n}"; | $obj->views->datasets->map = "function(doc) {\nif (doc.fieldName == \"data\") {\n emit(doc._id, doc);\n}\n}"; |
$obj->views->datasetGroups->map = "function(doc) {\nif (doc.fieldName == \"data\") {\n doc.metadata[\"data.gov.au Category\"] && doc.metadata[\"data.gov.au Category\"].forEach(function(tag) {\n emit(tag, doc.url); \n });\n}\n}"; | |
$obj->views->getValidationRequired->map = "function(doc) {\nif (doc.mime_type == \"text/html\" \n&& typeof(doc.validation) == \"undefined\") {\n emit(doc._id, doc._attachments);\n}\n}"; | $obj->views->getValidationRequired->map = "function(doc) {\nif (doc.mime_type == \"text/html\" \n&& typeof(doc.validation) == \"undefined\") {\n emit(doc._id, doc._attachments);\n}\n}"; |
$docdb->save($obj, true); | $docdb->save($obj, true); |
//function createAgencyDesignDoc() { | //function createAgencyDesignDoc() { |
$db = $server->get_db('disclosr-agencies'); | $db = $server->get_db('disclosr-agencies'); |
$obj = new stdClass(); | $obj = new stdClass(); |
$obj->_id = "_design/" . urlencode("app"); | $obj->_id = "_design/" . urlencode("app"); |
$obj->language = "javascript"; | $obj->language = "javascript"; |
$obj->views->all->map = "function(doc) { emit(doc._id, doc); };"; | $obj->views->all->map = "function(doc) { emit(doc._id, doc); };"; |
$obj->views->byABN->map = "function(doc) { emit(doc.abn, doc); };"; | $obj->views->byABN->map = "function(doc) { emit(doc.abn, doc); };"; |
$obj->views->byCanonicalName->map = "function(doc) { | $obj->views->byCanonicalName->map = "function(doc) { |
if (doc.parentOrg || doc.orgType == 'FMA-DepartmentOfState') { | if (doc.parentOrg || doc.orgType == 'FMA-DepartmentOfState') { |
emit(doc.name, doc); | emit(doc.name, doc); |
} | } |
};"; | };"; |
$obj->views->byDeptStateName->map = "function(doc) { | $obj->views->byDeptStateName->map = "function(doc) { |
if (doc.orgType == 'FMA-DepartmentOfState') { | if (doc.orgType == 'FMA-DepartmentOfState') { |
emit(doc.name, doc._id); | emit(doc.name, doc._id); |
} | } |
};"; | };"; |
$obj->views->parentOrgs->map = "function(doc) { | $obj->views->parentOrgs->map = "function(doc) { |
if (doc.parentOrg) { | if (doc.parentOrg) { |
emit(doc._id, doc.parentOrg); | emit(doc._id, doc.parentOrg); |
} | } |
};"; | };"; |
$obj->views->byName->map = 'function(doc) { | $obj->views->byName->map = 'function(doc) { |
if (typeof(doc["status"]) == "undefined" || doc["status"] != "suspended") { | if (typeof(doc["status"]) == "undefined" || doc["status"] != "suspended") { |
emit(doc.name, doc._id); | emit(doc.name, doc._id); |
if (typeof(doc.shortName) != "undefined" && doc.shortName != doc.name) { | if (typeof(doc.shortName) != "undefined" && doc.shortName != doc.name) { |
emit(doc.shortName, doc._id); | emit(doc.shortName, doc._id); |
} | } |
for (name in doc.otherNames) { | for (name in doc.otherNames) { |
if (doc.otherNames[name] != "" && doc.otherNames[name] != doc.name) { | if (doc.otherNames[name] != "" && doc.otherNames[name] != doc.name) { |
emit(doc.otherNames[name], doc._id); | emit(doc.otherNames[name], doc._id); |
} | } |
} | } |
for (name in doc.foiBodies) { | for (name in doc.foiBodies) { |
if (doc.foiBodies[name] != "" && doc.foiBodies[name] != doc.name) { | if (doc.foiBodies[name] != "" && doc.foiBodies[name] != doc.name) { |
emit(doc.foiBodies[name], doc._id); | emit(doc.foiBodies[name], doc._id); |
} | } |
} | } |
for (name in doc.positions) { | for (name in doc.positions) { |
if (doc.positions[name] != "" && doc.positions[name] != doc.name) { | if (doc.positions[name] != "" && doc.positions[name] != doc.name) { |
emit(doc.positions[name], doc._id); | emit(doc.positions[name], doc._id); |
} | } |
} | } |
} | } |
};'; | };'; |
$obj->views->foiEmails->map = "function(doc) { | $obj->views->foiEmails->map = "function(doc) { |
emit(doc._id, doc.foiEmail); | emit(doc._id, doc.foiEmail); |
};"; | };"; |
$obj->views->byLastModified->map = "function(doc) { emit(doc.metadata.lastModified, doc); }"; | $obj->views->byLastModified->map = "function(doc) { emit(doc.metadata.lastModified, doc); }"; |
$obj->views->getActive->map = 'function(doc) { if (doc.status == "active") { emit(doc._id, doc); } };'; | $obj->views->getActive->map = 'function(doc) { if (doc.status == "active") { emit(doc._id, doc); } };'; |
$obj->views->getSuspended->map = 'function(doc) { if (doc.status == "suspended") { emit(doc._id, doc); } };'; | $obj->views->getSuspended->map = 'function(doc) { if (doc.status == "suspended") { emit(doc._id, doc); } };'; |
$obj->views->getScrapeRequired->map = "function(doc) { | $obj->views->getScrapeRequired->map = "function(doc) { |
var lastScrape = Date.parse(doc.metadata.lastScraped); | var lastScrape = Date.parse(doc.metadata.lastScraped); |
var today = new Date(); | var today = new Date(); |
if (!lastScrape || lastScrape.getTime() + 1000 != today.getTime()) { | if (!lastScrape || lastScrape.getTime() + 1000 != today.getTime()) { |
emit(doc._id, doc); | emit(doc._id, doc); |
} | } |
};"; | };"; |
$obj->views->showNamesABNs->map = "function(doc) { emit(doc._id, {name: doc.name, abn: doc.abn}); };"; | $obj->views->showNamesABNs->map = "function(doc) { emit(doc._id, {name: doc.name, abn: doc.abn}); };"; |
$obj->views->getConflicts->map = "function(doc) { | $obj->views->getConflicts->map = "function(doc) { |
if (doc._conflicts) { | if (doc._conflicts) { |
emit(null, [doc._rev].concat(doc._conflicts)); | emit(null, [doc._rev].concat(doc._conflicts)); |
} | } |
}"; | }"; |
$obj->views->getStatistics->map = | $obj->views->getStatistics->map = |
"function(doc) { | "function(doc) { |
if (doc.statistics) { | if (doc.statistics) { |
for (var statisticSet in doc.statistics) { | for (var statisticSet in doc.statistics) { |
for (var statisticPeriod in doc.statistics[statisticSet]) { | for (var statisticPeriod in doc.statistics[statisticSet]) { |
emit([statisticSet,statisticPeriod], doc.statistics[statisticSet][statisticPeriod]['value']); | emit([statisticSet,statisticPeriod], doc.statistics[statisticSet][statisticPeriod]['value']); |
} | } |
} | } |
} | } |
}"; | }"; |
$obj->views->getStatistics->reduce = '_sum'; | $obj->views->getStatistics->reduce = '_sum'; |
// http://stackoverflow.com/questions/646628/javascript-startswith | // http://stackoverflow.com/questions/646628/javascript-startswith |
$obj->views->score->map = 'if(!String.prototype.startsWith){ | $obj->views->score->map = 'if(!String.prototype.startsWith){ |
String.prototype.startsWith = function (str) { | String.prototype.startsWith = function (str) { |
return !this.indexOf(str); | return !this.indexOf(str); |
} | } |
} | } |
function(doc) { | function(doc) { |
count = 0; | count = 0; |
if (doc["status"] != "suspended") { | if (doc["status"] != "suspended") { |
for(var propName in doc) { | for(var propName in doc) { |
if(typeof(doc[propName]) != "undefined" && doc[propName] != "") { | if(typeof(doc[propName]) != "undefined" && doc[propName] != "") { |
count++; | count++; |
} | } |
} | } |
portfolio = doc.parentOrg; | portfolio = doc.parentOrg; |
if (doc.orgType == "FMA-DepartmentOfState") { | if (doc.orgType == "FMA-DepartmentOfState") { |
portfolio = doc._id; | portfolio = doc._id; |
} | } |
if (doc.orgType == "Court-Commonwealth" || doc.orgType == "FMA-DepartmentOfParliament") { | if (doc.orgType == "Court-Commonwealth" || doc.orgType == "FMA-DepartmentOfParliament") { |
portfolio = doc.orgType; | portfolio = doc.orgType; |
} | } |
emit(count+doc._id, {id:doc._id, name: doc.name, score:count, orgType: doc.orgType, portfolio:portfolio}); | emit(count+doc._id, {id:doc._id, name: doc.name, score:count, orgType: doc.orgType, portfolio:portfolio}); |
} | } |
}'; | }'; |
$obj->views->scoreHas->map = 'if(!String.prototype.startsWith){ | $obj->views->scoreHas->map = 'if(!String.prototype.startsWith){ |
String.prototype.startsWith = function (str) { | String.prototype.startsWith = function (str) { |
return !this.indexOf(str); | return !this.indexOf(str); |
} | } |
} | } |
if(!String.prototype.endsWith){ | if(!String.prototype.endsWith){ |
String.prototype.endsWith = function(suffix) { | String.prototype.endsWith = function(suffix) { |
return this.indexOf(suffix, this.length - suffix.length) !== -1; | return this.indexOf(suffix, this.length - suffix.length) !== -1; |
}; | }; |
} | } |
function(doc) { | function(doc) { |
if (typeof(doc["status"]) == "undefined" || doc["status"] != "suspended") { | if (typeof(doc["status"]) == "undefined" || doc["status"] != "suspended") { |
for(var propName in doc) { | for(var propName in doc) { |
if(typeof(doc[propName]) != "undefined" && (propName.startsWith("has") || propName.endsWith("URL"))) { | if(typeof(doc[propName]) != "undefined" && (propName.startsWith("has") || propName.endsWith("URL"))) { |
emit(propName, 1); | emit(propName, 1); |
} | } |
} | } |
emit("total", 1); | emit("total", 1); |
} | } |
}'; | }'; |
$obj->views->scoreHas->reduce = '_sum'; | $obj->views->scoreHas->reduce = '_sum'; |
$obj->views->fieldNames->map = ' | $obj->views->fieldNames->map = ' |
function(doc) { | function(doc) { |
for(var propName in doc) { | for(var propName in doc) { |
emit(propName, doc._id); | emit(propName, doc._id); |
} | } |
}'; | }'; |
$obj->views->fieldNames->reduce = '_count'; | $obj->views->fieldNames->reduce = '_count'; |
// allow safe updates (even if slightly slower due to extra: rev-detection check). | // allow safe updates (even if slightly slower due to extra: rev-detection check). |
$db->save($obj, true); | $db->save($obj, true); |
?> | ?> |
<?php | |
include_once('include/common.inc.php'); | |
include_header('Charts'); | |
$db = $server->get_db('disclosr-agencies'); | |
?> | |
<div class="foundation-header"> | |
<h1><a href="about.php">Charts</a></h1> | |
<h4 class="subheader">Lorem ipsum.</h4> | |
</div> | |
<div id="scores" style="width:900px;height:500px;"></div> | |
<div id="employees" style="width:1000px;height:900px;"></div> | |
<script id="source"> | |
window.onload = function() { | |
$(document).ready(function() { | |
var d1 = []; | |
var scorelabels = []; | |
<?php | |
try { | |
$rows = $db->get_view("app", "scoreHas?group=true", null, true)->rows; | |
$dataValues = Array(); | |
foreach ($rows as $row) { | |
$dataValues[$row->value] = $row->key; | |
} | |
$i = 0; | |
ksort($dataValues); | |
foreach ($dataValues as $value => $key) { | |
echo " d1.push([$i, $value]);" . PHP_EOL; | |
echo " scorelabels.push('$key');" . PHP_EOL; | |
$i++; | |
} | |
} catch (SetteeRestClientException $e) { | |
setteErrorHandler($e); | |
} | |
?> | |
function scoretrackformatter(obj) { | |
if (scorelabels[Math.floor(obj.x)]) { | |
return (scorelabels[Math.floor(obj.x)])+"="+obj.y; | |
} else { | |
return ""; | |
} | |
} | |
function scoretickformatter(val, axis) { | |
if (scorelabels[Math.floor(val)]) { | |
return '<p style="margin-top:8em;-webkit-transform:rotate(-90deg);">'+(scorelabels[Math.floor(val)])+"</b>"; | |
} else { | |
return ""; | |
} | |
} | |
Flotr.draw(document.getElementById("scores"), [ {data: d1}], { | |
HtmlText: true, | |
bars : { | |
show : true | |
}, | |
mouse : { | |
track : true, | |
relative : true, | |
trackFormatter: scoretrackformatter | |
},yaxis: { | |
autoscaling: true | |
}, | |
xaxis: { | |
autoscaling: true, | |
minorTickFreq: 0.6, | |
noTicks : scorelabels.length, | |
tickFormatter: scoretickformatter | |
} | |
}); | |
var emplabels = []; | |
function emptrackformatter(obj) { | |
return (obj.series.label)+" = "+obj.y+" in "+emplabels[Math.floor(obj.x)]; | |
} | |
function emptickformatter(val, axis) { | |
if (emplabels[Math.floor(val)]) { | |
return '<p style="margin-top:8em;-webkit-transform:rotate(-90deg);">'+(emplabels[Math.floor(val)])+"</b>"; | |
} else { | |
return ""; | |
} | |
} | |
function onDataReceived(series) { | |
emplabels = series.labels; | |
Flotr.draw(document.getElementById("employees"), series.data, { | |
mouse : { | |
track : true, | |
relative : true, | |
trackFormatter: emptrackformatter | |
},yaxis: { | |
max: 10000, | |
scaling: 'logarithmic' | |
}, | |
xaxis: { | |
minorTickFreq: 1, | |
noTicks: emplabels.length, | |
showMinorLabels: true, | |
tickFormatter: emptickformatter | |
}, | |
legend: { | |
show: false | |
} | |
}); | |
} | |
$.ajax({ | |
url: "admin/exportEmployees.csv.php?format=json", | |
method: 'GET', | |
dataType: 'json', | |
success: onDataReceived | |
}); | |
}); | |
}; | |
</script> | |
<?php | |
include_footer(); | |
?> |
<?xml version="1.0" encoding="UTF-8"?> | |
<module type="WEB_MODULE" version="4"> | |
<component name="FacetManager"> | |
<facet type="Python" name="Python"> | |
<configuration sdkName="" /> | |
</facet> | |
</component> | |
<component name="NewModuleRootManager" inherit-compiler-output="true"> | |
<exclude-output /> | |
<content url="file://$MODULE_DIR$" /> | |
<orderEntry type="inheritedJdk" /> | |
<orderEntry type="sourceFolder" forTests="false" /> | |
</component> | |
</module> | |
<?php | <?php |
include('template.inc.php'); | include('template.inc.php'); |
include_header_documents("About"); | include_header_documents("About"); |
include_once('../include/common.inc.php'); | include_once('../include/common.inc.php'); |
?> | ?> |
<h1>About</h1> | <h1>About</h1> |
Written and managed by Alex Sadleir (maxious [at] lambdacomplex.org) | |
<?php | <?php |
include_footer_documents(); | include_footer_documents(); |
?> | ?> |
<?php | <?php |
include('template.inc.php'); | include('template.inc.php'); |
include_once('../include/common.inc.php'); | include_once('../include/common.inc.php'); |
$agenciesdb = $server->get_db('disclosr-agencies'); | $agenciesdb = $server->get_db('disclosr-agencies'); |
$idtoname = Array(); | $idtoname = Array(); |
foreach ($agenciesdb->get_view("app", "byCanonicalName")->rows as $row) { | foreach ($agenciesdb->get_view("app", "byCanonicalName")->rows as $row) { |
$idtoname[$row->id] = trim($row->value->name); | $idtoname[$row->id] = trim($row->value->name); |
} | } |
$foidocsdb = $server->get_db('disclosr-foidocuments'); | $foidocsdb = $server->get_db('disclosr-foidocuments'); |
include_header_documents((isset($_REQUEST['id']) ? $idtoname[$_REQUEST['id']] : 'Entries by Agency')); | include_header_documents((isset($_REQUEST['id']) ? $idtoname[$_REQUEST['id']] : 'Entries by Agency')); |
$endkey = (isset($_REQUEST['end_key']) ? $_REQUEST['end_key'] : '9999-99-99'); | $endkey = (isset($_REQUEST['end_key']) ? $_REQUEST['end_key'] : '9999-99-99'); |
?> | ?> |
<div class="headline">Read all the information released by Australian Federal Government agencies under the FOI Act | <div class="headline">Read all the information released by Australian Federal Government agencies under the FOI Act |
in one place! | in one place! |
</div> | </div> |
<a style='float:right' href="rss.xml.php"><img src="img/feed-icon-14x14.png" alt="RSS Icon"/> All Agencies RSS Feed</a> | <a style='float:right' href="rss.xml.php"><img src="img/feed-icon-14x14.png" alt="RSS Icon"/> All Agencies RSS Feed</a> |
<br> | <br> |
<?php | <?php |
try { | try { |
if ($_REQUEST['id']) { | if (isset($_REQUEST['id'])) { |
$rows = $foidocsdb->get_view("app", "byAgencyID", $_REQUEST['id'], false, false, false)->rows; | $rows = $foidocsdb->get_view("app", "byAgencyID", $_REQUEST['id'], false, false, false)->rows; |
foreach ($rows as $row) { | foreach ($rows as $row) { |
//print_r($rows); | //print_r($rows); |
echo displayLogEntry($row, $idtoname); | echo displayLogEntry($row, $idtoname); |
if (!isset($startkey)) | if (!isset($startkey)) |
$startkey = $row->key; | $startkey = $row->key; |
$endkey = $row->key; | $endkey = $row->key; |
} | } |
} else { | } else { |
$rows = $foidocsdb->get_view("app", "byAgencyID?group=true", null, false, false, true)->rows; | $rows = $foidocsdb->get_view("app", "byAgencyID?group=true", null, false, false, true)->rows; |
if ($rows) { | if ($rows) { |
foreach ($rows as $row) { | foreach ($rows as $row) { |
echo '<a href="agency.php?id=' . $row->key . '">' . $idtoname[$row->key] . " (" . $row->value . " records)</a> <br>\n"; | echo '<a href="agency.php?id=' . $row->key . '">' . $idtoname[$row->key] . " (" . $row->value . " records)</a> <br>\n"; |
} | } |
} | } |
} | } |
} catch (SetteeRestClientException $e) { | } catch (SetteeRestClientException $e) { |
setteErrorHandler($e); | setteErrorHandler($e); |
} | } |
echo "<a class='btn btn-large btn-primary' href='?end_key=$endkey' style='float:right;'>next page <i class='icon-circle-arrow-right icon-white'></i></a>"; | echo "<a class='btn btn-large btn-primary' href='?end_key=$endkey' style='float:right;'>next page <i class='icon-circle-arrow-right icon-white'></i></a>"; |
include_footer_documents(); | include_footer_documents(); |
?> | ?> |
<?php | <?php |
include('template.inc.php'); | include('template.inc.php'); |
include_header_documents("Charts"); | include_header_documents("Charts"); |
include_once('../include/common.inc.php'); | include_once('../include/common.inc.php'); |
$agenciesdb = $server->get_db('disclosr-agencies'); | $agenciesdb = $server->get_db('disclosr-agencies'); |
$idtoname = Array(); | $idtoname = Array(); |
foreach ($agenciesdb->get_view("app", "byCanonicalName")->rows as $row) { | foreach ($agenciesdb->get_view("app", "byCanonicalName")->rows as $row) { |
$idtoname[$row->id] = trim($row->value->name); | $idtoname[$row->id] = trim($row->value->name); |
} | } |
$foidocsdb = $server->get_db('disclosr-foidocuments'); | $foidocsdb = $server->get_db('disclosr-foidocuments'); |
?> | ?> |
<div class="foundation-header"> | <div class="foundation-header"> |
<h1><a href="about.php">Charts</a></h1> | <h1><a href="about.php">Charts</a></h1> |
<h4 class="subheader">Lorem ipsum.</h4> | <h4 class="subheader"></h4> |
</div> | </div> |
<div id="bydate" style="width:1000px;height:300px;"></div> | <div id="bydate" style="width:1000px;height:300px;"></div> |
<div id="byagency" style="width:1200px;height:300px;"></div> | <div id="byagency" style="width:1000px;height:1400px;"></div> |
<script id="source"> | <script id="source"> |
window.onload = function () { | window.onload = function () { |
$(document).ready(function () { | $(document).ready(function () { |
var | var |
d1 = [], | d1 = [], |
options1, | options1, |
o1; | o1; |
<?php | <?php |
try { | try { |
$rows = $foidocsdb->get_view("app", "byDateMonthYear?group=true",null, false,false,true)->rows; | $rows = $foidocsdb->get_view("app", "byDateMonthYear?group=true",null, false,false,true)->rows; |
$dataValues = Array(); | $dataValues = Array(); |
foreach ($rows as $row) { | foreach ($rows as $row) { |
$dataValues[$row->key] = $row->value; | $dataValues[$row->key] = $row->value; |
} | } |
$i = 0; | $i = 0; |
ksort($dataValues); | ksort($dataValues); |
foreach ($dataValues as $key => $value) { | foreach ($dataValues as $key => $value) { |
$date = date_create_from_format('Y-m-d', $key); | $date = date_create_from_format('Y-m-d', $key); |
if (date_format($date, 'U') != "") { | if (date_format($date, 'U') != "") { |
echo " d1.push([".date_format($date, 'U')."000, $value]);" . PHP_EOL; | echo " d1.push([".date_format($date, 'U')."000, $value]);" . PHP_EOL; |
// echo " emplabels.push('$key');" . PHP_EOL; | // echo " emplabels.push('$key');" . PHP_EOL; |
$i++; | $i++; |
} | } |
} | } |
} catch (SetteeRestClientException $e) { | } catch (SetteeRestClientException $e) { |
setteErrorHandler($e); | setteErrorHandler($e); |
} | } |
?> | ?> |
options1 = { | options1 = { |
xaxis: { | xaxis: { |
mode: 'time', | mode: 'time', |
labelsAngle: 45 | labelsAngle: 45 |
}, | }, |
selection: { | selection: { |
mode: 'x' | mode: 'x' |
}, | }, |
HtmlText: false, | HtmlText: false, |
title: 'Time' | title: 'Disclosure Log entries added by Date' |
}; | }; |
// Draw graph with default options, overwriting with passed options | // Draw graph with default options, overwriting with passed options |
function drawGraph(opts) { | function drawGraph(opts) { |
// Clone the options, so the 'options' variable always keeps intact. | // Clone the options, so the 'options' variable always keeps intact. |
o1 = Flotr._.extend(Flotr._.clone(options1), opts || {}); | o1 = Flotr._.extend(Flotr._.clone(options1), opts || {}); |
// Return a new graph. | // Return a new graph. |
return Flotr.draw( | return Flotr.draw( |
document.getElementById("bydate"), | document.getElementById("bydate"), |
[ d1 ], | [ d1 ], |
o1 | o1 |
); | ); |
} | } |
graph = drawGraph(); | graph = drawGraph(); |
Flotr.EventAdapter.observe(document.getElementById("bydate"), 'flotr:select', function (area) { | Flotr.EventAdapter.observe(document.getElementById("bydate"), 'flotr:select', function (area) { |
// Draw selected area | // Draw selected area |
graph = drawGraph({ | graph = drawGraph({ |
xaxis: { min: area.x1, max: area.x2, mode: 'time', labelsAngle: 45 }, | xaxis: { min: area.x1, max: area.x2, mode: 'time', labelsAngle: 45 }, |
yaxis: { min: area.y1, max: area.y2 } | yaxis: { min: area.y1, max: area.y2 } |
}); | }); |
}); | }); |
// When graph is clicked, draw the graph with default area. | // When graph is clicked, draw the graph with default area. |
Flotr.EventAdapter.observe(document.getElementById("bydate"), 'flotr:click', function () { | Flotr.EventAdapter.observe(document.getElementById("bydate"), 'flotr:click', function () { |
graph = drawGraph(); | graph = drawGraph(); |
}); | }); |
}); | }); |
}; | }; |
var d2 = []; | var d2 = []; |
var agencylabels = []; | var agencylabels = []; |
function agencytrackformatter(obj) { | function agencytrackformatter(obj) { |
return agencylabels[Math.floor(obj.x)] + " = " + obj.y; | return agencylabels[Math.floor(obj.y)] + " = " + obj.x; |
} | } |
function agencytickformatter(val, axis) { | function agencytickformatter(val, axis) { |
if (agencylabels[Math.floor(val)]) { | if (agencylabels[Math.floor(val)]) { |
return '<p style="margin-top:8em;-webkit-transform:rotate(-90deg);">' + (agencylabels[Math.floor(val)]) + "</b>"; | return (agencylabels[Math.floor(val)]) ; |
} else { | } else { |
return ""; | return ""; |
} | } |
} | } |
<?php | <?php |
try { | try { |
$rows = $foidocsdb->get_view("app", "byAgencyID?group=true",null, false,false,true)->rows; | $rows = $foidocsdb->get_view("app", "byAgencyID?group=true",null, false,false,true)->rows; |
$dataValues = Array(); | $dataValues = Array(); |
$i = 0; | $i = 0; |
foreach ($rows as $row) { | foreach ($rows as $row) { |
echo " d2.push([".$i.", $row->value]);" . PHP_EOL; | echo " d2.push([ $row->value,$i]);" . PHP_EOL; |
echo " agencylabels.push(['".str_replace("'","",$idtoname[$row->key])."']);" . PHP_EOL; | echo " agencylabels.push(['".str_replace("'","",$idtoname[$row->key])."']);" . PHP_EOL; |
$i++; | $i++; |
} | } |
} catch (SetteeRestClientException $e) { | } catch (SetteeRestClientException $e) { |
setteErrorHandler($e); | setteErrorHandler($e); |
} | } |
?> | ?> |
// Draw the graph | // Draw the graph |
Flotr.draw( | Flotr.draw( |
document.getElementById("byagency"), | document.getElementById("byagency"), |
[d2], | [d2], |
{ | { |
title: "Disclosure Log entries by Agency", | |
bars: { | bars: { |
show: true, | show: true, |
horizontal: false, | horizontal: true, |
shadowSize: 0, | shadowSize: 0, |
barWidth: 0.5 | barWidth: 0.5 |
}, | }, |
mouse: { | mouse: { |
track: true, | track: true, |
relative: true, | relative: true, |
trackFormatter: agencytrackformatter | trackFormatter: agencytrackformatter |
}, | }, |
yaxis: { | yaxis: { |
min: 0, | |
autoscaleMargin: 1 | |
}, | |
xaxis: { | |
minorTickFreq: 1, | minorTickFreq: 1, |
noTicks: agencylabels.length, | noTicks: agencylabels.length, |
showMinorLabels: true, | showMinorLabels: true, |
tickFormatter: agencytickformatter | tickFormatter: agencytickformatter |
}, | |
xaxis: { | |
min: 0, | |
autoscaleMargin: 1 | |
}, | }, |
legend: { | legend: { |
show: false | show: false |
} | } |
} | } |
); | ); |
</script> | </script> |
<?php | <?php |
include_footer_documents(); | include_footer_documents(); |
?> | ?> |
import ckanclient | |
import couchdb | |
from ckanclient import CkanApiError | |
import re | |
class LoaderError(Exception): | |
pass | |
# Instantiate the CKAN client. | |
#ckan = ckanclient.CkanClient(base_location='http://localhost:5000/api', api_key='b47b24cd-591d-40c1-8677-d73101d56d1b') | |
api_key = 'ff34526e-f794-4068-8235-fcbba38cd8bc' | |
server = 'data.disclosurelo.gs' | |
api_key = 'c30eb6f5-0f90-47e0-bf05-9b1b4e3a461a' | |
server = 'ckan.data.gov.au' | |
ckan = ckanclient.CkanClient(base_location='http://' + server + '/api', | |
api_key=api_key) | |
couch = couchdb.Server('http://127.0.0.1:5984/') | |
#couch = couchdb.Server('http://192.168.1.113:5984/') | |
# https://github.com/okfn/ckanext-importlib | |
def munge(name): | |
# convert spaces to underscores | |
name = re.sub(' ', '_', name).lower() | |
# convert symbols to dashes | |
name = re.sub('[:]', '_-', name).lower() | |
name = re.sub('[/]', '-', name).lower() | |
# take out not-allowed characters | |
name = re.sub('[^a-zA-Z0-9-_]', '', name).lower() | |
# remove double underscores | |
name = re.sub('__', '_', name).lower() | |
return name | |
def name_munge(input_name): | |
return munge(input_name.replace(' ', '').replace('.', '_').replace('&', 'and')) | |
docsdb = couch['disclosr-documents'] | |
if __name__ == "__main__": | |
groups = {} | |
for doc in docsdb.view('app/datasetGroups'): | |
group_name = doc.key | |
if group_name != "Not specified": | |
pkg_name = filter(lambda x: x in '0123456789abcdefghijklmnopqrstuvwxyz-_', | |
doc.value.replace("http://data.gov.au/dataset/", '').replace('/', '')[:100]); | |
if group_name in groups.keys(): | |
groups[group_name] = list(set(groups[group_name] + [pkg_name])) | |
else: | |
groups[group_name] = [pkg_name] | |
# add dataset to group(s) | |
for group_name in groups.keys(): | |
if group_name != "Not specified": | |
group_url = name_munge(group_name[:100]) | |
print group_name | |
print groups[group_name] | |
try: | |
# Update the group details | |
group_entity = ckan.group_entity_get(group_url) | |
print "group "+group_name+" exists" | |
if 'packages' in group_entity.keys(): | |
group_entity['packages'] = list(set(group_entity['packages'] + groups[group_name])) | |
else: | |
group_entity['packages'] = groups[group_name] | |
ckan.group_entity_put(group_entity) | |
except CkanApiError, e: | |
if ckan.last_status == 404: | |
print "group "+group_name+" does not exist, creating" | |
group_entity = { | |
'name': group_url, | |
'title': group_name, | |
'description': group_name, | |
'packages': groups[group_name] | |
} | |
#print group_entity | |
ckan.group_register_post(group_entity) | |
elif ckan.last_status == 409: | |
print "group already exists" | |
else: | |
raise LoaderError('Unexpected status %s adding to group under \'%s\': %r' % ( | |
ckan.last_status, pkg_name, e.args)) | |
# coding=utf-8 | |
import ckanclient | import ckanclient |
import couchdb | import couchdb |
from ckanclient import CkanApiError | from ckanclient import CkanApiError |
import re | import re |
import html2text # aaronsw :( | import html2text # aaronsw :( |
import ckanapi # https://github.com/open-data/ckanapi | import ckanapi # https://github.com/open-data/ckanapi |
import scrape | |
import datetime, os, hashlib | |
import urllib2 | |
class LoaderError(Exception): | class LoaderError(Exception): |
pass | pass |
import tempfile | |
def add_package_resource_cachedurl(ckan, package_name, url, name, format, license_id, size,**kwargs): | |
excluded_apis = "recent-earthquakes,sentinel-hotspots,abc-local-stations,action-bus-service-gtfs-feed-act,current-and-future-road-reports-traffic-restrictions-in-south-australia,current-and-future-road-reports-traffic-restrictions-in-south-australia,current-and-future-road-reports-traffic-restrictions-in-south-australia,current-and-future-road-reports-traffic-restrictions-in-south-australia,current-and-future-road-reports-traffic-restrictions-in-south-australia,current-and-future-road-reports-traffic-restrictions-in-south-australia,journey-planner-data-act,live-traffic-cameras-nsw,precis-forecast-national,precis-forecast-national,precis-forecast-new-south-wales,precis-forecast-new-south-wales,precis-forecast-northern-territory,precis-forecast-northern-territory,precis-forecast-queensland,precis-forecast-queensland,precis-forecast-south-australia,precis-forecast-south-australia,precis-forecast-tasmania,precis-forecast-tasmania,precis-forecast-victoria,precis-forecast-victoria,precis-forecast-western-australia,precis-forecast-western-australia,register-of-penalty-notices-nsw,sentinel-hotspots,trove-people-and-organisations-data,weather-data-services-radar,abc-local-stations,act-emergency-services-agency-esa-28093-current-incidents,act-emergency-services-agency-esa-news-alerts,act-government-news-and-events,act-government-summaries-of-cabinet-outcomes,act-magistrates-court-judgements,act-supreme-court-judgements,act-supreme-court-sentences,actpla-latest-news,all-vacant-act-government-jobs,community-engagement-current-engagements,community-engagement-news,edd-media-releases,edd-news-and-events,freedom-of-information-foi-summaries,libraries-act-announcements,nsw-rural-fire-service-current-incidents,nsw-rural-fire-service-major-updates,precis-forecast-new-south-wales,precis-forecast-south-australia,precis-forecast-tasmania,precis-forecast-victoria,sentinel-hotspots,south-australian-road-crash-statistics,trove-people-and-organisations-data,weather-warnings-for-new-south-wales-australian-capital-territory,weather-warnings-for-northern-territory,weather-warnings-for-queensland,weather-warnings-for-south-australia,weather-warnings-for-tasmania,weather-warnings-for-victoria,weather-warnings-for-western-australia".split(",") | |
if "xls" in url: | |
format = "xls" | |
if "pdf" in url: | |
format = "pdf" | |
if "xlsx" in url: | |
format = "xlsx" | |
(returned_url, mime_type, content) = scrape.fetchURL(scrape.docsdb, | |
url, "dataset_resource", "AGIMO", False) | |
if mime_type in ["application/vnd.ms-excel","application/msexcel","application/x-msexcel","application/x-ms-excel","application/x-excel","application/x-dos_ms_excel","application/xls","application/x-xls"]: | |
format = "xls" | |
if mime_type in ["application/xlsx","application/x-xlsx","application/vnd.openxmlformats-officedocument.spreadsheetml.sheet"]: | |
format = "xlsx" | |
#put file extensions on for windows users downloading files | |
suffix = name.encode("ascii","ignore").replace("/","") | |
if len(suffix) < 5 or (suffix[-4] != "." and suffix[-5] != "."): | |
suffix = suffix + "." + format | |
if content != None and package_name not in excluded_apis: | |
tf = tempfile.NamedTemporaryFile(suffix=suffix) | |
tfName = os.path.abspath(tf.name) | |
print tfName | |
tf.seek(0) | |
tf.write(content) | |
tf.flush() | |
ckan.add_package_resource (package_name, tfName, name=name, format=format, license_id=license_id) | |
else: | |
print "fetch error" | |
return ckan.add_package_resource(package_name, url, name=name, resource_type='data', | |
format=format, | |
size=size, mimetype=mime_type, license_id=license_id) | |
# Instantiate the CKAN client. | # Instantiate the CKAN client. |
#ckan = ckanclient.CkanClient(base_location='http://localhost:5000/api', api_key='b47b24cd-591d-40c1-8677-d73101d56d1b') | api_key = 'ff34526e-f794-4068-8235-fcbba38cd8bc' |
api_key = 'b3ab75e4-afbb-465b-a09d-8171c8c69a7a' | server = 'data.disclosurelo.gs' |
ckan = ckanclient.CkanClient(base_location='http://data.disclosurelo.gs/api', | |
ckan = ckanclient.CkanClient(base_location='http://' + server + '/api', | |
api_key=api_key) | api_key=api_key) |
ckandirect = ckanapi.RemoteCKAN('http://data.disclosurelo.gs', api_key=api_key) | ckandirect = ckanapi.RemoteCKAN('http://' + server, api_key=api_key) |
couch = couchdb.Server('http://127.0.0.1:5984/') | couch = couchdb.Server('http://127.0.0.1:5984/') |
#couch = couchdb.Server('http://192.168.1.113:5984/') | #couch = couchdb.Server('http://192.168.1.113:5984/') |
import urllib | |
import urlparse | |
def url_fix(s, charset='utf-8'): | |
"""Sometimes you get an URL by a user that just isn't a real | |
URL because it contains unsafe characters like ' ' and so on. This | |
function can fix some of the problems in a similar way browsers | |
handle data entered by the user: | |
>>> url_fix(u'http://de.wikipedia.org/wiki/Elf (Begriffsklärung)') | |
'http://de.wikipedia.org/wiki/Elf%20%28Begriffskl%C3%A4rung%29' | |
:param charset: The target charset for the URL if the url was | |
given as unicode string. | |
""" | |
if isinstance(s, unicode): | |
s = s.encode(charset, 'ignore') | |
if not urlparse.urlparse(s).scheme: | |
s = "http://" + s | |
scheme, netloc, path, qs, anchor = urlparse.urlsplit(s) | |
path = urllib.quote(path, '/%') | |
qs = urllib.quote_plus(qs, ':&=') | |
return urlparse.urlunsplit((scheme, netloc, path, qs, anchor)) | |
# http://code.activestate.com/recipes/578019-bytes-to-human-human-to-bytes-converter/ | # http://code.activestate.com/recipes/578019-bytes-to-human-human-to-bytes-converter/ |
SYMBOLS = { | SYMBOLS = { |
'customary': ('B', 'KB', 'MB', 'GB', 'T', 'P', 'E', 'Z', 'Y'), | 'customary': ('B', 'KB', 'MB', 'GB', 'T', 'P', 'E', 'Z', 'Y'), |
'customary_ext': ('byte', 'kilo', 'mega', 'giga', 'tera', 'peta', 'exa', | 'customary_ext': ('byte', 'kilo', 'mega', 'giga', 'tera', 'peta', 'exa', |
'zetta', 'iotta'), | 'zetta', 'iotta'), |
'iec': ('Bi', 'Ki', 'Mi', 'Gi', 'Ti', 'Pi', 'Ei', 'Zi', 'Yi'), | 'iec': ('Bi', 'Ki', 'Mi', 'Gi', 'Ti', 'Pi', 'Ei', 'Zi', 'Yi'), |
'iec_ext': ('byte', 'kibi', 'mebi', 'gibi', 'tebi', 'pebi', 'exbi', | 'iec_ext': ('byte', 'kibi', 'mebi', 'gibi', 'tebi', 'pebi', 'exbi', |
'zebi', 'yobi'), | 'zebi', 'yobi'), |
} | } |
def human2bytes(s): | def human2bytes(s): |
""" | """ |
Attempts to guess the string format based on default symbols | Attempts to guess the string format based on default symbols |
set and return the corresponding bytes as an integer. | set and return the corresponding bytes as an integer. |
When unable to recognize the format ValueError is raised. | When unable to recognize the format ValueError is raised. |
>>> human2bytes('0 B') | >>> human2bytes('0 B') |
0 | 0 |
>>> human2bytes('1 K') | >>> human2bytes('1 K') |
1024 | 1024 |
>>> human2bytes('1 M') | >>> human2bytes('1 M') |
1048576 | 1048576 |
>>> human2bytes('1 Gi') | >>> human2bytes('1 Gi') |
1073741824 | 1073741824 |
>>> human2bytes('1 tera') | >>> human2bytes('1 tera') |
1099511627776 | 1099511627776 |
>>> human2bytes('0.5kilo') | >>> human2bytes('0.5kilo') |
512 | 512 |
>>> human2bytes('0.1 byte') | >>> human2bytes('0.1 byte') |
0 | 0 |
>>> human2bytes('1 k') # k is an alias for K | >>> human2bytes('1 k') # k is an alias for K |
1024 | 1024 |
>>> human2bytes('12 foo') | >>> human2bytes('12 foo') |
Traceback (most recent call last): | Traceback (most recent call last): |
... | ... |
ValueError: can't interpret '12 foo' | ValueError: can't interpret '12 foo' |
""" | """ |
if s == None: | |
return 0 | |
s = s.replace(',', '') | |
init = s | init = s |
num = "" | num = "" |
while s and s[0:1].isdigit() or s[0:1] == '.': | while s and s[0:1].isdigit() or s[0:1] == '.': |
num += s[0] | num += s[0] |
s = s[1:] | s = s[1:] |
num = float(num) | num = float(num) |
letter = s.strip() | letter = s.strip() |
for name, sset in SYMBOLS.items(): | for name, sset in SYMBOLS.items(): |
if letter in sset: | if letter in sset: |
break | break |
else: | else: |
if letter == 'k': | if letter == 'k': |
# treat 'k' as an alias for 'K' as per: http://goo.gl/kTQMs | # treat 'k' as an alias for 'K' as per: http://goo.gl/kTQMs |
sset = SYMBOLS['customary'] | sset = SYMBOLS['customary'] |
letter = letter.upper() | letter = letter.upper() |
else: | else: |
raise ValueError("can't interpret %r" % init) | raise ValueError("can't interpret %r" % init) |
prefix = {sset[0]: 1} | prefix = {sset[0]: 1} |
for i, s in enumerate(sset[1:]): | for i, s in enumerate(sset[1:]): |
prefix[s] = 1 << (i + 1) * 10 | prefix[s] = 1 << (i + 1) * 10 |
return int(num * prefix[letter]) | return int(num * prefix[letter]) |
# https://github.com/okfn/ckanext-importlib | # https://github.com/okfn/ckanext-importlib |
def munge(name): | def munge(name): |
# convert spaces to underscores | # convert spaces to underscores |
name = re.sub(' ', '_', name).lower() | name = re.sub(' ', '_', name).lower() |
# convert symbols to dashes | # convert symbols to dashes |
name = re.sub('[:]', '_-', name).lower() | name = re.sub('[:]', '_-', name).lower() |
name = re.sub('[/]', '-', name).lower() | name = re.sub('[/]', '-', name).lower() |
# take out not-allowed characters | # take out not-allowed characters |
name = re.sub('[^a-zA-Z0-9-_]', '', name).lower() | name = re.sub('[^a-zA-Z0-9-_]', '', name).lower() |
# remove double underscores | # remove double underscores |
name = re.sub('__', '_', name).lower() | name = re.sub('__', '_', name).lower() |
return name | return name |
#todo "{'name': [u'Url must be purely lowercase alphanumeric (ascii) characters and these symbols: -_']}" | |
# http://data.gov.au/dataset/australian-domestic-regional-and-international-airline-activity-%E2%80%93-time-series/ | |
def name_munge(input_name): | def name_munge(input_name): |
return munge(input_name.replace(' ', '').replace('.', '_').replace('&', 'and')) | return munge(input_name.replace(' ', '').replace('.', '_').replace('&', 'and')) |
#[:100] | |
#return input_name.replace(' ', '').replace('.', '_').replace('&', 'and') | |
def get_license_id(licencename): | |
def get_licence_id(licencename): | |
map = { | map = { |
"Creative Commons - Attribution-Share Alike 2.0 Australia (CC-SA)\nThe downloadable version of the database is licensed under CC-BY-SA Creative Commons Attribution Share Alike and contains only the database fields that are released under that license. These fields are object title, object number, object description as well as temporal, spatial and dimension details. It also contains a persistent URL for each record.": 'cc-by-sa', | "Creative Commons - Attribution-Share Alike 2.0 Australia (CC-SA)\nThe downloadable version of the database is licensed under CC-BY-SA Creative Commons Attribution Share Alike and contains only the database fields that are released under that license. These fields are object title, object number, object description as well as temporal, spatial and dimension details. It also contains a persistent URL for each record.": 'cc-by-sa', |
"CreativeCommonsAttributionNonCommercial30AustraliaCCBYNC30": 'cc-nc', | "CreativeCommonsAttributionNonCommercial30AustraliaCCBYNC30": 'cc-nc', |
'Otherpleasespecify': 'notspecified', | 'Otherpleasespecify': 'notspecified', |
'': 'notspecified', | '': 'notspecified', |
"Publicly available data": 'notspecified', | "Publicly available data": 'notspecified', |
"CreativeCommonsAttributionNoDerivativeWorks30AustraliaCCBYND30": "other-closed", | "CreativeCommonsAttributionNoDerivativeWorks30AustraliaCCBYND30": "cc-by-nd", |
"CreativeCommonsAttributionNonCommercialNoDerivs30AustraliaCCBYNCND30": "other-closed", | "CreativeCommonsAttributionNonCommercialNoDerivs30AustraliaCCBYNCND30": "cc-nc-nd", |
'CreativeCommonsAttribution30AustraliaCCBY30': 'cc-by', | 'CreativeCommonsAttribution30AustraliaCCBY30': 'cc-by', |
"Creative Commons - Attribution 2.5 Australia (CC-BY)": 'cc-by', | "Creative Commons - Attribution 2.5 Australia (CC-BY)": 'cc-by', |
'CreativeCommonsAttributionCCBY25': 'cc-by', | 'CreativeCommonsAttributionCCBY25': 'cc-by', |
"PublicDomain": 'other-pd', | "PublicDomain": 'other-pd', |
} | } |
if licencename not in map.keys(): | if licencename not in map.keys(): |
raise Exception(licencename + " not found"); | raise Exception(licencename + " not found"); |
return map[licencename]; | return map[licencename]; |
goodcsvdata = "afl-in-victoria,annual-budget-initiatives-by-suburb-brisbane-city-council,athletics-in-victoria-gfyl,bicycle-racks-mosman-municipal-council,boat-ramps-brisbane-city-council,brisbane-access-ratings-database,bus-stops-brisbane-city-council,cemeteries-brisbane-city-council,cfa-locations,citycycle-stations-brisbane-city-council,community-gardens-brisbane-city-council,community-halls-brisbane-city-council,cooking-classes-gfyl,court-locations-victoria,customer-service-centres-brisbane-city-council,dance-in-victoria-gfyl,disability-activity-gfyl,dog-parks-brisbane-city-council,ferry-terminals-brisbane-city-council,fishing-club-in-victoria-gfyl,fitness-centres-in-victoria-gfyl,gardens-reserves-gfyl,golf-courses-brisbane-city-council,gymnastics-in-victoria-gfyl,historic-cemeteries-brisbane-city-council,ice-skating-centres-gfyl,immunisation-clinics-brisbane-city-council,libraries-brisbane-city-council,licenced-venues-victoria,lifesaving-locations-victoria,loading-zones-brisbane-city-council,major-projects-victoria,markets-in-victoria,martial-arts-in-victoria-gfyl,melbourne-water-use-by-postcode,members-of-parliament-both-houses-nsw,members-of-the-legislative-assembly-nsw,members-of-the-legislative-council-nsw,mfb-locations-vic,ministers-of-the-nsw-parliament,mosman-local-government-area,mosman-rider-route,mosman-wwii-honour-roll,neighbourhood-houses-gfyl,news-feeds-mosman-municipal-council,off-street-car-parks-mosman-municipal-council,orienteering-clubs-gfyl,parking-meter-areas-brisbane-city-council,parks-and-reserves-mosman-municipal-council,parks-brisbane-city-council,personal-training-gfyl,picnic-areas-brisbane-city-council,playgrounds-brisbane-city-council,playgrounds-mosman-municipal-council,police-region-crime-statistics-victoria,police-service-area-crime-statistics-victoria,pony-clubs-in-victoria-gfyl,prison-locations-victoria,public-amenities-maintained-by-mosman-council,public-art-brisbane-city-council,public-internet-locations-vic,public-toilets-brisbane-city-council,racecourse-locations-victoria,recent-development-applications-mosman-municipal-council,recreation-groups-gfyl,recreational-fishing-spots,regional-business-centres-brisbane-city-council,reports-of-swooping-birds-mosman-municipal-council,restricted-parking-areas-brisbane-city-council,rollerskating-centres-in-victoria-gfyl,sailing-clubs-gfyl,school-locations-victoria,shadow-ministers-of-the-nsw-parliament,skate-parks-gfyl,sporting-clubs-and-organisations-gfyl,stakeboard-parks-brisbane-city-council,state-bodies-gfyl,street-names-brisbane-city-council,suburbs-and-adjoining-suburbs-brisbane-city-council,swimming-pools-brisbane-city-council,swimming-pools-gfyl,tennis-courts-brisbane-city-council,top-40-book-club-reads-brisbane-city-council,tracks-and-trails-gfyl,triathlon-clubs-gfyl,urban-water-restrictions-victoria,veterinary-services-in-mosman,victorian-microbreweries,volunteering-centres-services-and-groups-victoria,walking-groups-gfyl,ward-offices-brisbane-city-council,waste-collection-days-brisbane-city-council,waste-transfer-stations-brisbane-city-council,water-consumption-in-melbourne,water-sports-in-victoria-gfyl,wifi-hot-spots-brisbane-city-council,yoga-pilates-and-tai-chi-in-victoria-gfyl,2809cycling-in-new-south-wales-what-the-data-tells-us2809-and-related-data,act-barbecue-bbq-locations,act-tafe-locations,ausindustry-locations,austender-contract-notice-export,austender-contract-notice-export,austender-contract-notice-export,austender-contract-notice-export,austender-contract-notice-export,austender-contract-notice-export,austender-contract-notice-export,austender-contract-notice-export,australian-gas-light-company-maps,australian-gas-light-company-maps,australian-ports,australian-public-service-statistical-bulletin-2011-12,australian-public-service-statistical-bulletin-snapshot-at-december-31-2011,australian-public-service-statistical-bulletin-tables-0910,austrics-timetable-set,capital-works-call-tender-schedule,collection-item-usage-state-library-of-victoria,country-and-commodity-trade-data-spreadsheet,country-and-commodity-trade-data-spreadsheet-2,country-by-level-of-processing-trade-data-spreadsheet,crime-incident-type-and-frequency-by-capital-city-and-nationally,csiro-locations,data-from-the-oaic-public-sector-information-survey-2012,data-from-the-oaic-public-sector-information-survey-2012,data-from-the-oaic-public-sector-information-survey-2012,department-of-finance-and-deregulation-office-locations,digitised-maps,diisr-division-locations-excluding-ausindustry-enterprise-connect-and-nmi,diisr-locations,diisr-portfolio-agency-locations-excluding-csiro,distance-to-legal-service-providers-from-disadvantaged-suburbs,enterprise-connect-locations,fire-insurance-maps-sydney-block-plans-1919-1940,fire-insurance-maps-sydney-block-plans-1919-1940,first-fleet-collection,first-fleet-collection,first-fleet-maps,first-fleet-maps,freedom-of-information-annual-estimated-costs-and-staff-time-statistical-data-2011-12,freedom-of-information-quarterly-request-and-review-statistical-data-2011-12,freedom-of-information-requests-estimated-costs-and-charges-collected-1982-83-to-2011-12,higher-education-course-completions,higher-education-enrolments,historical-australian-government-contract-data,historical-australian-government-contract-data,historical-australian-government-contract-data,historical-australian-government-contract-data,historical-australian-government-contract-data,historical-australian-government-contract-data,historical-australian-government-contract-data,historical-australian-government-contract-data,historical-australian-government-contract-data,historical-australian-government-contract-data,historical-australian-government-contract-data,historical-australian-government-contract-data,historical-australian-government-contract-data,historical-australian-government-contract-data,historical-australian-government-contract-data,journey-planner-data-nt,library-catalogue-search-terms-state-library-of-victoria,location-of-act-schools,location-of-centrelink-offices,location-of-european-wasps-nests,location-of-lawyers-and-legal-service-providers-by-town,location-of-legal-assistance-service-providers,location-of-medicare-offices,location-of-medicare-offices,maps-of-the-southern-hemisphere-16th-18th-centuries,maps-of-the-southern-hemisphere-16th-18th-centuries,music-queensland,national-measurement-institute-locations,new-south-wales-officers-and-men-of-the-australian-imperial-force-a-i-f-and-the-australian-naval-for,new-south-wales-officers-and-men-of-the-australian-imperial-force-a-i-f-and-the-australian-naval-for,photographs-of-nsw-life-pre-1955,photographs-of-nsw-life-pre-1955,photographs-of-sydney-before-1885,photographs-of-sydney-before-1885,picture-queensland,plgr-28093-playgrounds-act,police-station-locations,queensland-public-libraries,rare-printed-books,rare-printed-books,real-estate-maps,regional-australia-funding-projects,sa-memory-state-library-of-south-australia,search-engine-terms-state-library-of-victoria,south-australian-photographs-state-library-of-south-australia,south-australian-sheet-music-state-library-of-south-australia,sydney-bond-store-maps-1894,sydney-bond-store-maps-1894,sydney-maps-1917,sydney-maps-1917,tafe-institute-locations-victoria,tafe-sa-campus-locations,tolt-public-toilets-act,victorian-public-library-branches-state-library-of-victoria,western-australia-public-library-network,world-war-one-photographs-by-frank-hurley,world-war-one-photographs-by-frank-hurley,citycat-timetables-brisbane-city-council,cityferry-timetables-brisbane-city-council,cost-of-salinity-to-local-infrastructure-1996-97-summary-of-component-costs-of-salinity-by-reporting,cost-of-salinity-to-local-infrastructure-1996-97-summary-of-component-costs-of-salinity-by-reporting,downstream-cost-calculator-model-and-data-for-199697-or-2001-prices,economics-of-australian-soil-conditions-199697-limiting-factor-or-relative-yield-min-of-ry_salt2000-,geographical-names-register-gnr-of-nsw,victorian-dryland-salinity-assessment-2000-d01cac_ramsar_final-xls,victorian-dryland-salinity-assessment-2000-d02cac_fauna_final-xls,victorian-dryland-salinity-assessment-2000-d03cac_fauna_dist_final-xls,victorian-dryland-salinity-assessment-2000-dc04cac_hydrol_final-xls,victorian-dryland-salinity-assessment-2000-dc05cac_wetland_final-xls,victorian-dryland-salinity-assessment-2000-dc06cac_util_final-xls,victorian-dryland-salinity-assessment-2000-dc07cac_road_final-xls,victorian-dryland-salinity-assessment-2000-dc08cac_towns_final-xls,victorian-dryland-salinity-assessment-2000-dc09cac_flora_final-xls,victorian-dryland-salinity-assessment-2000-dc10cac_flora_dist_final-xls,victorian-dryland-salinity-assessment-2000-dc12cac_infrastructure-xls,victorian-dryland-salinity-assessment-2000-dc13cac_natural_envt-xls,victorian-dryland-salinity-assessment-2000-dc14cac_agriculture-xls,victorian-dryland-salinity-assessment-2000-dc16cac_agric_cost-xls,victorian-dryland-salinity-assessment-2000-dc17cac_shallow_wt-xls,victorian-dryland-salinity-assessment-2000-dc18cac_agric_cost_time-xls,victorian-dryland-salinity-assessment-2000-dc21cac_water_resources_new-xls,victorian-dryland-salinity-assessment-2000-dc22cac_risk-xls,licensed-broadcasting-transmitter-data,nsw-crime-data,recorded-crime-dataset-nsw,crime-statistics-in-nsw-by-month,2001-02-to-2007-08-local-government-survey-victoria,2009-green-light-report,annual-statistical-reports-fire-brigades-nsw-200304,annual-statistical-reports-fire-brigades-nsw-200405,annual-statistical-reports-fire-brigades-nsw-200506,annual-statistical-reports-fire-brigades-nsw-200607,arts-on-the-map,assets-and-liabilities-of-australian-located-operations,assets-of-australian-located-operations,assets-of-australian-located-operations-by-country,assets-of-financial-institutions,back-issues-of-monthly-banking-statistics,banks-assets,banks-consolidated-group-capital,banks-consolidated-group-impaired-assets,banks-consolidated-group-off-balance-sheet-business,banks-liabilities,building-societies-selected-assets-and-liabilities,byteback2842-locations-vic,cash-management-trusts,city-of-melbourne-street-furniture-database,community-services-nsw,consolidated-exposures-immediate-and-ultimate-risk-basis,consolidated-exposures-immediate-risk-basis-foreign-claims-by-country,consolidated-exposures-immediate-risk-basis-international-claims-by-country,consolidated-exposures-ultimate-risk-basis,consolidated-exposures-ultimate-risk-basis-foreign-claims-by-country,cosolidated-exposures-immediate-risk-basis,credit-unions-selected-assets-and-liabilities,daily-net-foreign-exchange-transactions,detox-your-home,education-national-assessment-program-literacy-and-numeracy-nsw,employment-data-by-nsw-regions,excise-beer-clearance-data-updated-each-month-beer-clearance-summary-data,finance-companies-and-general-financiers-selected-assets-and-liabilities,foreign-exchange-transactions-and-holdings-of-official-reserve-assets,half-yearly-life-insurance-bulletin-december-2010,health-behaviours-in-nsw,international-liabilities-by-country-of-the-australian-located-operations-of-banks-and-rfcs,liabilities-and-assets-monthly,liabilities-and-assets-weekly,liabilities-of-australian-located-operations,life-insurance-offices-statutory-funds,managed-funds,monetary-policy-changes,money-market-corporations-selected-assets-and-liabilities,monthly-airport-traffic-data-for-top-ten-airports-january-1985-to-december-2008,monthly-banking-statistics-april-2011,monthly-banking-statistics-june-2011,monthly-banking-statistics-may-2011,open-market-operations-2009-to-current,projected-households-vic-rvic-msd-2006-2056,projected-population-by-age-and-sex-vic-rvic-msd-2006-2056,public-unit-trust,quarterly-bank-performance-statistics,quarterly-general-insurance-performance-statistics-march-2011,quarterly-superannuation-performance-march-2011,recorded-crime-dataset-nsw,residential-land-bulletin,resourcesmart-retailers,resourcesmart-retailers-vic,road-fatalities-nsw,securitisation-vehicles,selected-asset-and-liabilities-of-the-private-non-financial-sectors,seperannuation-funds-outside-life-offices,solar-report-vic,towns-in-time-victoria,vif2008-projected-population-by-5-year-age-groups-and-sex-sla-lga-ssd-sd-2006-2026,vif2008-projected-population-totals-and-components-vic-rvic-msd-2006-2056,vif2008-projected-population-totals-sla-lga-ssd-sd-2006-2026,arts-festivals-victoria,arts-organisations-victoria,arts-spaces-and-places-victoria,ausgrid-average-electricity-use,collecting-institutions-victoria,indigenous-arts-organisations-victoria,latest-coastal-weather-observations-for-coolangatta-qld,top-10-fiction-books-brisbane-city-council".split(",") | |
goodotherdata = "abc-local-stations,abc-local-stations,abc-local-stations,act-emergency-services-agency-esa-28093-current-incidents,act-government-news-and-events,act-government-summaries-of-cabinet-outcomes,act-magistrates-court-judgements,act-supreme-court-judgements,act-supreme-court-sentences,action-bus-service-gtfs-feed-act,actpla-latest-news,agricultural-commodities-for-199697-linked-to-profit-function-surfaces,agricultural-structure-classification,agricultural-structure-classification,all-vacant-act-government-jobs,annual-family-income-1996-1997-to-1998-1999-three-year-average,apvma-pubcris-dataset-for-registered-agricultural-and-veterinary-chemical-products-and-approved-acti,argus-newspaper-collection-of-photographs-state-library-of-victoria,assessment-of-terrestrial-biodiversity-2002-biodiversity-audit-data-entry-system-bades,assessment-of-terrestrial-biodiversity-2002-database,assisted-immigration-1848-1912-index,ausgrid-average-electricity-use,ausgrid-average-electricity-use-2011,ausindustry-locations,ausindustry-locations,austender-contract-notice-export,australian-broadband-guarantee,australian-broadband-guarantee,australian-data-access,australian-dryland-salinity-assessment-spatial-data-12500000-nlwra-2001,australian-dryland-salinity-assessment-spatial-data-12500000-nlwra-2001,australian-groundwater-flow-systems-national-land-and-water-resources-audit-january-2000,australian-groundwater-flow-systems-national-land-and-water-resources-audit-january-2000,australian-irrigation-areas-raster-version-1a-national-land-and-water-resources-audit,australian-irrigation-areas-raster-version-1a-national-land-and-water-resources-audit,australian-irrigation-areas-vector-version-1a-national-land-and-water-resources-audit,australian-irrigation-areas-vector-version-1a-national-land-and-water-resources-audit,australian-public-service-statistical-bulletin-2010-11,australian-water-resources-assessment-2000-database,australiana-index-state-library-of-victoria,available-water-capacity-for-australian-areas-of-intensive-agriculture-of-layer-1-a-horizon-top-soil,bicycle-racks-mosman-municipal-council,bikeways-briisbane-city-council,bikeways-briisbane-city-council,boreholes-in-the-murray-basin-southeastern-australia,boreholes-in-the-murray-basin-southeastern-australia,british-convict-transportation-registers,calculated-annual-and-monthly-potential-evaporation-mm,calculated-annual-and-monthly-potential-evaporation-mm,canberra-suburb-boundaries,catchment-and-subcatchments-grid,cemeteries-brisbane-city-council,cemeteries-brisbane-city-council,coal-fields-in-the-murray-basin-southeastern-australia,coal-fields-in-the-murray-basin-southeastern-australia,commonwealth-agencies,commonwealth-electoral-boundaries-archive-2009,commonwealth-electoral-boundaries-archive-2009,cost-of-salinity-to-local-infrastructure-1996-97-total-cost-all-infrastructure-buildings-road-rail-a,cost-of-salinity-to-local-infrastructure-1996-97-total-cost-all-infrastructure-buildings-road-rail-a,cost-of-salinity-to-local-infrastructure-1996-97-total-cost-increase-to-local-infrastructure-based-o,cost-of-salinity-to-local-infrastructure-1996-97-total-cost-of-all-infrastructure-buildings-road-rai,cost-of-salinity-to-local-infrastructure-1996-97-total-cost-of-all-infrastructure-buildings-road-rai,cost-of-salinity-to-local-infrastructure-1996-97-total-cost-of-all-infrastructure-buildings-road-rai,cost-of-salinity-to-local-infrastructure-1996-97-total-cost-of-all-infrastructure-buildings-road-rai,cost-of-salinity-to-local-infrastructure-1996-97-total-cost-of-the-general-infrastructure-component-,cost-of-salinity-to-local-infrastructure-1996-97-total-cost-of-the-rail-component-of-infrastructure-,cost-of-salinity-to-local-infrastructure-1996-97-total-cost-the-general-infrastructure-component-bui,cost-of-salinity-to-local-infrastructure-1996-97-total-cost-the-road-component-of-infrastructure-bas,cost-of-salinity-to-local-infrastructure-1996-97-total-cost-the-road-component-of-infrastructure-bas,cost-of-salinity-to-local-infrastructure-1996-97-total-cost-to-the-bridge-component-of-infrastructur,cost-of-salinity-to-local-infrastructure-1996-97-total-cost-to-the-bridge-component-of-infrastructur,country-by-level-of-processing-trade-data-spreadsheet-2,country-by-level-of-processing-trade-data-spreadsheet-2011-12,crime-incidents-data-2004-international-crime-victimisation-survey-icvs-australian-component,cropping-management-practices-1998-1999,csiro-locations,csiro-locations,current-and-future-road-reports-traffic-restrictions-in-south-australia,current-and-future-road-reports-traffic-restrictions-in-south-australia,current-and-future-road-reports-traffic-restrictions-in-south-australia,current-and-future-road-reports-traffic-restrictions-in-south-australia,current-and-future-road-reports-traffic-restrictions-in-south-australia,current-and-future-road-reports-traffic-restrictions-in-south-australia,cybersafety-outreach-program,cybersafety-outreach-program,data-source-for-polygonal-data-used-by-the-asris-project-in-generation-of-modelled-surfaces,department-of-finance-and-deregulation-office-locations,department-of-finance-and-deregulation-office-locations,depositional-path-length,digital-enterprise,digital-hubs,digitised-maps,diisr-division-locations-excluding-ausindustry-enterprise-connect-and-nmi,diisr-division-locations-excluding-ausindustry-enterprise-connect-and-nmi,diisr-locations,diisr-portfolio-agency-locations-excluding-csiro,diisr-portfolio-agency-locations-excluding-csiro,directory-gov-au-full-data-export,distance-to-ridges,economics-of-australian-soil-conditions-199697-factor-most-limiting-yield-aciditysodicitysalinity,economics-of-australian-soil-conditions-199697-gross-benefit-acidity-hayr,economics-of-australian-soil-conditions-199697-gross-benefit-of-the-limiting-factor-hayr,economics-of-australian-soil-conditions-199697-gross-benefit-salinity-hayr,economics-of-australian-soil-conditions-199697-gross-benefit-sodicity-hayr,economics-of-australian-soil-conditions-199697-impact-cost-of-salinity-2000-2020-hayr,economics-of-australian-soil-conditions-199697-relative-yield-from-acidity,economics-of-australian-soil-conditions-199697-relative-yield-from-salinity-in-2000,economics-of-australian-soil-conditions-199697-relative-yield-from-salinity-in-2020,economics-of-australian-soil-conditions-199697-relative-yield-from-sodicity,edd-media-releases,edd-news-and-events,egovernment-resource-centre-website-analytics,elevation-of-the-pre-tertiary-basement-in-the-murray-basin,elevation-of-the-pre-tertiary-basement-in-the-murray-basin,enterprise-connect-locations,enterprise-connect-locations,equivalent-fresh-water-head-difference-between-the-shallowest-and-deepest-aquifers,equivalent-fresh-water-head-difference-between-the-shallowest-and-deepest-aquifers,erosion-gully-density,erosion-path-length,estimated-proportion-of-farms-carrying-out-landcare-related-work-1998-1999,estimated-value-of-agricultural-operations-evao-1996-1997,farm-equity-ratio-1996-1997-to-1998-1999-three-year-average,farm-family-cash-income-1196-1997-to-1998-1999-three-year-average,farmer-population-1996,farms-with-significant-degradation-problems-irrigation-salinity-1998-1999,farms-with-significant-degradation-problems-irrigation-salinity-1998-1999-2,farms-with-significant-degradation-problems-soil-acidity-1998-1999,forests-of-australia-2003,freedom-of-information-foi-summaries,geology-lithology-12-500-000-scale,glenorchy-city-council-building-footprints,glenorchy-city-council-building-footprints,glenorchy-city-council-building-footprints,glenorchy-city-council-kerbs,glenorchy-city-council-kerbs,glenorchy-city-council-kerbs,glenorchy-city-council-stormwater-pipes,glenorchy-city-council-stormwater-pipes,glenorchy-city-council-stormwater-pipes,glenorchy-city-council-stormwater-pits,glenorchy-city-council-stormwater-pits,glenorchy-city-council-stormwater-pits,groundwater-sdl-resource-units,groundwater-sdl-resource-units,groundwater-sdl-resource-units,higher-qualifications-of-farmers-and-farm-managers-1996,historical-australian-government-contract-data,historical-australian-government-contract-data,hydrologic-indicator-sites,hydrologic-indicator-sites,immigration-land-orders-1861-1874,indicators-of-catchment-condition-in-the-intensive-land-use-zone-of-australia-biota-condition-sub-in,indicators-of-catchment-condition-in-the-intensive-land-use-zone-of-australia-catchment-condition-in,indicators-of-catchment-condition-in-the-intensive-land-use-zone-of-australia-feral-animal-density,indicators-of-catchment-condition-in-the-intensive-land-use-zone-of-australia-human-population-densi,indicators-of-catchment-condition-in-the-intensive-land-use-zone-of-australia-impoundment-density,indicators-of-catchment-condition-in-the-intensive-land-use-zone-of-australia-industrial-point-sourc,indicators-of-catchment-condition-in-the-intensive-land-use-zone-of-australia-intensive-agricultural,indicators-of-catchment-condition-in-the-intensive-land-use-zone-of-australia-land-condition-sub-ind,indicators-of-catchment-condition-in-the-intensive-land-use-zone-of-australia-native-vegetation-frag,indicators-of-catchment-condition-in-the-intensive-land-use-zone-of-australia-nutrient-point-source-,indicators-of-catchment-condition-in-the-intensive-land-use-zone-of-australia-pesticide-hazard,indicators-of-catchment-condition-in-the-intensive-land-use-zone-of-australia-predicted-2050-salinit,indicators-of-catchment-condition-in-the-intensive-land-use-zone-of-australia-protected-areas,indicators-of-catchment-condition-in-the-intensive-land-use-zone-of-australia-rivers-in-acidificatio,indicators-of-catchment-condition-in-the-intensive-land-use-zone-of-australia-rivers-in-salt-hazard,indicators-of-catchment-condition-in-the-intensive-land-use-zone-of-australia-rivers-through-forests,indicators-of-catchment-condition-in-the-intensive-land-use-zone-of-australia-soil-acidification-haz,indicators-of-catchment-condition-in-the-intensive-land-use-zone-of-australia-soil-degradation-hazar,indicators-of-catchment-condition-in-the-intensive-land-use-zone-of-australia-suspended-sediment-loa,indicators-of-catchment-condition-in-the-intensive-land-use-zone-of-australia-weed-density,integrated-vegetation-cover-2003-version-1,john-t-collins-collection-state-library-of-victoria,journal-of-the-h-m-s-endeavour-1768-1771,journey-planner-data-act,krantz-sheldon-architectural-images,land-use-of-australia-version-3-28093-20012002,lands-surveys-historic-map-series-western-australia,latest-coastal-weather-observations-for-coolangatta-qld,launceston-city-council-addresses,launceston-city-council-building-footprints,launceston-city-council-contours,launceston-city-council-detail-survey-drawing-file,launceston-city-council-drainage,launceston-city-council-fences,launceston-city-council-pavement,launceston-city-council-railway,launceston-city-council-roads,libraries-act-announcements,licensed-broadcasting-transmitter-data,linc-tasmania,look-up-table-of-auslig-river-basins-of-australia-1997,major-water-resources-infrastructure-part-of-the-australian-water-resources-assessment-2000-database,mean-annual-concentration-of-mineral-nitrogen-in-soil-water-mgn-kgh20-in-the-pre-1788-scenario,mean-annual-concentration-of-mineral-nitrogen-in-soil-water-mgn-kgh20-in-the-pre-1788-scenario,mean-annual-concentration-of-mineral-nitrogen-in-soil-water-mgn-kgh20-in-the-present-day-scenario,mean-annual-concentration-of-mineral-nitrogen-in-soil-water-mgn-kgh20-in-the-present-day-scenario,mean-annual-deep-drainage-mm-y-in-the-pre-1788-scenario,mean-annual-deep-drainage-mm-y-in-the-pre-1788-scenario,mean-annual-deep-drainage-mm-y-in-the-present-day-scenario,mean-annual-deep-drainage-mm-y-in-the-present-day-scenario,mean-annual-transpiration-from-the-plant-canopy-for-the-pre-1788-scenario,mean-annual-transpiration-from-the-plant-canopy-for-the-pre-1788-scenario,mean-annual-transpiration-from-the-plant-canopy-for-the-present-day-scenario,mean-annual-transpiration-from-the-plant-canopy-for-the-present-day-scenario,mean-transpiration-in-april-from-the-plant-canopy-for-the-pre-1788-scenario,mean-transpiration-in-april-from-the-plant-canopy-for-the-pre-1788-scenario,mean-transpiration-in-august-from-the-plant-canopy-for-the-pre-1788-scenario,mean-transpiration-in-august-from-the-plant-canopy-for-the-pre-1788-scenario,mean-transpiration-in-august-from-the-plant-canopy-for-the-present-day-scenario,mean-transpiration-in-august-from-the-plant-canopy-for-the-present-day-scenario,mean-transpiration-in-december-from-the-plant-canopy-for-the-pre-1788-scenario,mean-transpiration-in-december-from-the-plant-canopy-for-the-pre-1788-scenario,mean-transpiration-in-december-from-the-plant-canopy-for-the-present-day-scenario,mean-transpiration-in-december-from-the-plant-canopy-for-the-present-day-scenario,mean-transpiration-in-february-from-the-plant-canopy-for-the-pre-1788-scenario,mean-transpiration-in-february-from-the-plant-canopy-for-the-pre-1788-scenario,mean-transpiration-in-january-from-the-plant-canopy-for-the-pre-1788-scenario,mean-transpiration-in-january-from-the-plant-canopy-for-the-pre-1788-scenario,mean-transpiration-in-january-from-the-plant-canopy-for-the-present-day-scenario,mean-transpiration-in-january-from-the-plant-canopy-for-the-present-day-scenario,mean-transpiration-in-july-from-the-plant-canopy-for-the-pre-1788-scenario,mean-transpiration-in-july-from-the-plant-canopy-for-the-pre-1788-scenario,mean-transpiration-in-july-from-the-plant-canopy-for-the-present-day-scenario,mean-transpiration-in-july-from-the-plant-canopy-for-the-present-day-scenario,mean-transpiration-in-june-from-the-plant-canopy-for-the-pre-1788-scenario,mean-transpiration-in-june-from-the-plant-canopy-for-the-pre-1788-scenario,mean-transpiration-in-june-from-the-plant-canopy-for-the-present-day-scenario,mean-transpiration-in-june-from-the-plant-canopy-for-the-present-day-scenario,mean-transpiration-in-march-from-the-plant-canopy-for-the-pre-1788-scenario,mean-transpiration-in-march-from-the-plant-canopy-for-the-pre-1788-scenario,mean-transpiration-in-march-from-the-plant-canopy-for-the-present-day-scenario,mean-transpiration-in-march-from-the-plant-canopy-for-the-present-day-scenario,mean-transpiration-in-march-from-the-plant-canopy-for-the-present-day-scenario-2,mean-transpiration-in-march-from-the-plant-canopy-for-the-present-day-scenario-2,mean-transpiration-in-may-from-the-plant-canopy-for-the-pre-1788-scenario,mean-transpiration-in-may-from-the-plant-canopy-for-the-pre-1788-scenario,mean-transpiration-in-may-from-the-plant-canopy-for-the-present-day-scenario,mean-transpiration-in-may-from-the-plant-canopy-for-the-present-day-scenario,mean-transpiration-in-november-from-the-plant-canopy-for-the-pre-1788-scenario,mean-transpiration-in-november-from-the-plant-canopy-for-the-pre-1788-scenario,mean-transpiration-in-november-from-the-plant-canopy-for-the-present-day-scenario,mean-transpiration-in-november-from-the-plant-canopy-for-the-present-day-scenario,mean-transpiration-in-october-from-the-plant-canopy-for-the-pre-1788-scenario,mean-transpiration-in-october-from-the-plant-canopy-for-the-pre-1788-scenario,mean-transpiration-in-october-from-the-plant-canopy-for-the-present-day-scenario,mean-transpiration-in-october-from-the-plant-canopy-for-the-present-day-scenario,mean-transpiration-in-september-from-the-plant-canopy-for-the-pre-1788-scenario,mean-transpiration-in-september-from-the-plant-canopy-for-the-pre-1788-scenario,mean-transpiration-in-september-from-the-plant-canopy-for-the-present-day-scenario,mean-transpiration-in-september-from-the-plant-canopy-for-the-present-day-scenario,mildenhall-photographs-of-early-canberra,mobility-map-brisbane-city,mobility-map-mt-coot-tha,mosman-local-government-area,mosman-rider-route,mosman-wwii-honour-roll,mosman-wwii-honour-roll,murray-darling-basin-water-resource-plan-areas-groundwater,murray-darling-basin-water-resource-plan-areas-groundwater,murray-darling-basin-water-resource-plan-areas-surface-water,murray-darling-basin-water-resource-plan-areas-surface-water,music-queensland,national-broadband-network,national-broadband-network,national-broadband-network-2011-10,national-broadband-network-2011-10,national-broadband-network-2011-12,national-broadband-network-2011-12,national-broadband-network-2012,national-broadband-network-28093-august-2011,national-broadband-network-28093-august-2011,national-broadband-network-28093-july-2011,national-broadband-network-28093-july-2011,national-broadband-network-february-2012,national-broadband-network-february-2012,national-broadband-network-september-2011,national-broadband-network-september-2011,national-library-of-australia-sheet-music-collection,national-measurement-institute-locations,national-parks-and-asset-locations-south-australia,national-public-toilet-map,new-south-wales-dryland-salinity-assessment-2000-assessmet-of-dryland-salinity-extent-2000,new-south-wales-dryland-salinity-assessment-2000-assessmet-of-dryland-salinity-extent-2000,new-south-wales-dryland-salinity-assessment-2000-assessmet-of-dryland-salinity-extent-2000,new-south-wales-dryland-salinity-assessment-2000-assessmet-of-dryland-salinity-extent-2000,new-south-wales-dryland-salinity-assessment-2000-assessmet-of-dryland-salinity-extent-2000,new-south-wales-dryland-salinity-assessment-2000-assessmet-of-dryland-salinity-extent-2000,new-south-wales-dryland-salinity-assessment-2000-assessmet-of-dryland-salinity-extent-2000,new-south-wales-dryland-salinity-assessment-2000-assessmet-of-dryland-salinity-extent-2020,new-south-wales-dryland-salinity-assessment-2000-assessmet-of-dryland-salinity-extent-2020,new-south-wales-dryland-salinity-assessment-2000-assessmet-of-dryland-salinity-extent-2020,new-south-wales-dryland-salinity-assessment-2000-assessmet-of-dryland-salinity-extent-2020,new-south-wales-dryland-salinity-assessment-2000-assessmet-of-dryland-salinity-extent-2020,new-south-wales-dryland-salinity-assessment-2000-assessmet-of-dryland-salinity-extent-2020,new-south-wales-dryland-salinity-assessment-2000-assessmet-of-dryland-salinity-extent-2020,new-south-wales-dryland-salinity-assessment-2000-assessmet-of-dryland-salinity-extent-2050,new-south-wales-dryland-salinity-assessment-2000-assessmet-of-dryland-salinity-extent-2050,new-south-wales-dryland-salinity-assessment-2000-assessmet-of-dryland-salinity-extent-2050,new-south-wales-dryland-salinity-assessment-2000-assessmet-of-dryland-salinity-extent-2050,new-south-wales-dryland-salinity-assessment-2000-assessmet-of-dryland-salinity-extent-2050,new-south-wales-dryland-salinity-assessment-2000-assessmet-of-dryland-salinity-extent-2050,new-south-wales-dryland-salinity-assessment-2000-assessmet-of-dryland-salinity-extent-2050,nsw-newspapers-catalogue-data,nsw-rural-fire-service-current-incidents,nsw-rural-fire-service-major-updates,off-street-car-parks-mosman-municipal-council,open-database-brisbane-city-council,ost-of-salinity-to-local-infrastructure-1996-97-total-cost-of-the-rail-component-of-infrastructure-b,parking-areas-brisbane-city-council,parking-areas-brisbane-city-council,parks-and-reserves-mosman-municipal-council,parks-brisbane-city-council,parks-brisbane-city-council,picture-australia-metadata,picture-queensland,picture-queensland,playgrounds-mosman-municipal-council,police-station-locations,police-station-locations,port-phillip-papers-state-library-of-victoria,precis-forecast-national,precis-forecast-national,precis-forecast-new-south-wales,precis-forecast-new-south-wales,precis-forecast-new-south-wales,precis-forecast-northern-territory,precis-forecast-northern-territory,precis-forecast-queensland,precis-forecast-queensland,precis-forecast-south-australia,precis-forecast-south-australia,precis-forecast-south-australia,precis-forecast-tasmania,precis-forecast-tasmania,precis-forecast-tasmania,precis-forecast-victoria,precis-forecast-victoria,precis-forecast-victoria,precis-forecast-western-australia,precis-forecast-western-australia,public-amenities-maintained-by-mosman-council,radio-and-television-broadcasting-stations-book-internet-edition,real-estate-maps,recent-earthquakes,regional-development-australia,regional-development-australia-2011-september-2011,regional-development-australia-may-2012,reports-of-swooping-birds-mosman-municipal-council,sentinel-hotspots,sentinel-hotspots,slq-catalogue-searches,slq-catalogue-searches,slv-rural-water,slv-shipping,slwa-digital-photographic-collection,south-australian-boat-ramp-locator,south-australian-road-crash-statistics,state-library-of-victoria-online-image-collection,state-library-of-victoria-online-image-collection-inc-high-res,state-of-the-service-report-2010-11-australian-public-service-employee-survey-results,state-of-the-service-report-2010-11-australian-public-service-employee-survey-results,statistical-local-areas-1996-for-agricultural-structure-classification,surface-water-gauging-stations-part-of-the-australian-water-resources-assessment-2000-database,surface-water-gauging-stations-part-of-the-australian-water-resources-assessment-2000-database,surface-water-sdl-resource-units,surface-water-sdl-resource-units,tasmanian-herbarium,tasmanian-museum-and-art-gallery-faunal-collection".split(",") | |
docsdb = couch['disclosr-documents'] | docsdb = couch['disclosr-documents'] |
if __name__ == "__main__": | if __name__ == "__main__": |
orgs_list = [] | orgs_list = [] |
orgs_ids = {} | |
for doc in docsdb.view('app/datasets'): | for doc in docsdb.view('app/datasets'): |
print " --- " | |
print doc.id | print doc.id |
if doc.value['url'] != "http://data.gov.au/data/" and doc.value['agencyID'] != "qld": | if doc.value['url'] != "http://data.gov.au/data/" and doc.value['agencyID'] != "qld": |
# Collect the package metadata. | # Collect the package metadata. |
pkg_name = filter(lambda x: x in '0123456789abcdefghijklmnopqrstuvwxyz-_', | pkg_name = filter(lambda x: x in '0123456789abcdefghijklmnopqrstuvwxyz-_', |
doc.value['url'].replace("http://data.gov.au/dataset/", '').replace('/', '')[:100]); | doc.value['url'].replace("http://data.gov.au/dataset/", '').replace('/', '')[:100]); |
print pkg_name | print pkg_name |
#add to or create organization using direct API | if pkg_name != "" : |
org_name = name_munge(doc.value['metadata']["Agency"][:100]) | |
if org_name not in orgs_list: | #add to or create organization using direct API |
orgs_list = ckandirect.action.organization_list()['result'] | agency = doc.value['metadata']["Agency"] |
#print orgs_list | if agency == "APS": |
agency = "Australian Public Service Commission" | |
if agency == "Department of Broadband, Communications and the Digital Ecomomy": | |
agency = "Department of Broadband, Communications and the Digital Economy" | |
if agency == "Shared Services, Treasury Directorate": | |
agency = "Shared Services Procurement, Treasury Directorate" | |
if agency == "Treasury - Shared Services": | |
agency = "Shared Services Procurement, Treasury Directorate" | |
if agency == "Territory and Municipal Services (TAMS)": | |
agency = "Territory and Municipal Services Directorate" | |
if agency == "State Library of NSW": | |
agency = "State Library of New South Wales" | |
org_name = name_munge(agency[:100]) | |
if org_name not in orgs_list: | if org_name not in orgs_list: |
try: | orgs_list = ckandirect.action.organization_list()['result'] |
print "org not found, creating " + org_name | #print orgs_list |
ckandirect.action.organization_create(name=org_name, title=doc.value['metadata']["Agency"], | if org_name not in orgs_list: |
description=doc.value['metadata']["Agency"]) | try: |
orgs_list.append(org_name) | print "org not found, creating " + org_name |
except ckanapi.ValidationError, e: | ckandirect.action.organization_create(name=org_name, title=agency, |
print e | description=agency) |
raise LoaderError('Unexpected status') | orgs_list.append(org_name) |
else: | except ckanapi.ValidationError, e: |
print "org found, adding dataset to " + org_name | print e |
raise LoaderError('Unexpected status') | |
org = ckandirect.action.organization_show(id=org_name) | |
# todo cache org names -> id mapping | |
tags = [] | |
if doc.value['agencyID'] == "AGIMO": | |
if len(doc.value['metadata']["Keywords / Tags"]) > 0: | |
if hasattr(doc.value['metadata']["Keywords / Tags"], '__iter__'): | |
tags = tags + doc.value['metadata']["Keywords / Tags"] | |
else: | else: |
tags = tags + [doc.value['metadata']["Keywords / Tags"]] | print "org found, adding dataset to " + org_name |
tags = [re.sub('[^a-zA-Z0-9-_.]', '', tag.replace('&', 'and')).lower() for tag in tags if tag] | # cache org names -> id mapping |
#print tags | if org_name not in orgs_ids: |
package_entity = { | org = ckandirect.action.organization_show(id=org_name) |
'name': pkg_name, | orgs_ids[org_name] = org["result"]["id"] |
'title': doc.value['metadata']['DCTERMS.Title'], | org_id = orgs_ids[org_name] |
'url': doc.value['metadata']['DCTERMS.Source.URI'], | print "org id is " + org_id |
'tags': tags, #tags are mandatory? | tags = [] |
'author': doc.value['metadata']["DCTERMS.Creator"], | creator = doc.value['metadata']["DCTERMS.Creator"] |
'maintainer': doc.value['metadata']["DCTERMS.Creator"], | if doc.value['agencyID'] == "AGIMO": |
'licence_id': get_licence_id(doc.value['metadata']['DCTERMS.License']), | if len(doc.value['metadata']["Keywords / Tags"]) > 0: |
'notes': html2text.html2text(doc.value['metadata']['Description']), | if hasattr(doc.value['metadata']["Keywords / Tags"], '__iter__'): |
'owner_org': org["result"]["id"] | tags = tags + doc.value['metadata']["Keywords / Tags"] |
#todo add missing key values like jurasdiction | else: |
} | tags = tags + [doc.value['metadata']["Keywords / Tags"]] |
if doc.value['agencyID'] == "qld": | |
package_entity = doc.value['metadata'] | tags = [re.sub('[^a-zA-Z0-9-_.]', '', tag.replace('&', 'and')).lower() for tag in tags if tag] |
#print tags | |
try: | extras = [] |
#print package_entity | |
ckan.package_register_post(package_entity) | for extra_key in doc.value['metadata'].keys(): |
except CkanApiError, e: | if extra_key not in ["Description", "Content-Language", "DCTERMS.Description", |
if ckan.last_message == "{\"name\": [\"That URL is already in use.\"]}": | "Keywords / Tags", |
print "package already exists" | "data.gov.au Category", "Download", "Permalink", "DCTERMS.Identifier"]: |
else: | if doc.value['metadata'][extra_key] != None and doc.value['metadata'][extra_key] != "": |
print ckan.last_message | extras.append([extra_key, doc.value['metadata'][extra_key]]) |
raise LoaderError('Unexpected status %s checking for package under \'%s\': %r' % ( | |
ckan.last_status, pkg_name, e.args)) | package_entity = { |
pkg = ckan.package_entity_get(pkg_name) | 'name': pkg_name, |
'title': doc.value['metadata']['DCTERMS.Title'], | |
# add dataset to group(s) | 'url': doc.value['metadata']['DCTERMS.Source.URI'], |
groups = [] | 'tags': tags, #tags are mandatory? |
if 'data.gov.au Category' in doc.value['metadata'].keys() and len( | 'author': creator, |
doc.value['metadata']['data.gov.au Category']) > 0: | 'maintainer': creator, |
if hasattr(doc.value['metadata']['data.gov.au Category'], '__iter__'): | 'license_id': get_license_id(doc.value['metadata']['DCTERMS.License']), |
groups = groups + doc.value['metadata']['data.gov.au Category'] | 'notes': html2text.html2text(doc.value['metadata']['Description']).replace('AC/a!a','-').replace('AC/a!aC/',"'").replace("AC/a!E",":").replace("A "," "), |
else: | 'owner_org': org_id, |
groups = groups + [doc.value['metadata']['data.gov.au Category']] | 'extras': extras, |
'private': (pkg_name not in goodcsvdata and pkg_name not in goodotherdata) | |
for group_name in groups: | } |
group_url = name_munge(group_name[:100]) | |
try: | try: |
print ckan.group_entity_get(group_url) | #print package_entity |
ckan.package_register_post(package_entity) | |
# Update the group details | except CkanApiError, e: |
group_entity = ckan.last_message | if ckan.last_message == "{\"name\": [\"That URL is already in use.\"]}": |
print "group "+group_name+" exists" | print "package already exists" |
if 'packages' in group_entity.keys(): | |
group_entity['packages'] = list(set(group_entity['packages'] + [pkg_name])) | |
else: | else: |
group_entity['packages'] = [pkg_name] | print ckan.last_message |
ckan.group_entity_put(group_entity) | |
except CkanApiError, e: | |
if ckan.last_status == 404: | |
print "group "+group_name+" does not exist, creating" | |
group_entity = { | |
'name': group_url, | |
'title': group_name, | |
'description': group_name, | |
'packages': [pkg_name] | |
} | |
#print group_entity | |
ckan.group_register_post(group_entity) | |
elif ckan.last_status == 409: | |
print "group already exists" | |
else: | |
raise LoaderError('Unexpected status %s adding to group under \'%s\': %r' % ( | |
ckan.last_status, pkg_name, e.args)) | |
# add resources (downloadable data files) | |
if 'Download' in doc.value['metadata'].keys(): | |
try: | |
resources = pkg.get('resources', []) | |
if len(resources) < len(doc.value['metadata']['Download']): | |
for resource in doc.value['metadata']['Download']: | |
# http://docs.ckan.org/en/ckan-1.7/domain-model-resource.html | |
# (KML/KMZ) / (Shapefile) /(Other) | |
format = "plain" | |
if resource['format'] == '(XML)': | |
format = 'xml' | |
if resource['format'] == '(CSV/XLS)': | |
format = 'csv' | |
name = resource['href'] | |
if 'name' in resource.keys(): | |
name = resource['name'] | |
print resource | |
ckan.add_package_resource(pkg_name, resource['href'], name=name, resource_type='data', | |
format=format, | |
size=human2bytes(resource['size'].replace(',', ''))) | |
else: | |
print "resources already exist" | |
except CkanApiError, e: | |
if ckan.last_status == 404: | |
print "parent dataset does not exist" | |
else: | |
raise LoaderError('Unexpected status %s checking for package under \'%s\': %r' % ( | raise LoaderError('Unexpected status %s checking for package under \'%s\': %r' % ( |
ckan.last_status, pkg_name, e.args)) | ckan.last_status, pkg_name, e.args)) |
pkg = ckan.package_entity_get(pkg_name) | |
# add resources (downloadable data files) | |
if 'Download' in doc.value['metadata'].keys(): | |
try: | |
resources = pkg.get('resources', []) | |
if len(resources) < len(doc.value['metadata']['Download']): | |
for resource in doc.value['metadata']['Download']: | |
# http://docs.ckan.org/en/ckan-1.7/domain-model-resource.html | |
# (KML/KMZ) / (Shapefile) /(Other) | |
format = "plain" | |
if resource['format'] == '(XML)': | |
format = 'xml' | |
if resource['format'] == '(CSV/XLS)': | |
format = 'csv' | |
if resource['format'] == '(Shapefile)': | |
format = 'shp' | |
if resource['format'] == '(KML/KMZ)': | |
format = 'kml' | |
name = resource['href'] | |
if 'name' in resource.keys(): | |
name = resource['name'] | |
print resource | |
add_package_resource_cachedurl(ckan, pkg_name, url_fix(resource['href']), name, | |
format, get_license_id(doc.value['metadata']['DCTERMS.License']), | |
human2bytes(resource.get('size', '0B'))) | |
else: | |
print "resources already exist" | |
except CkanApiError, e: | |
if ckan.last_status == 404: | |
print "parent dataset does not exist" | |
else: | |
raise LoaderError('Unexpected status %s checking for package under \'%s\': %r' % ( | |
ckan.last_status, pkg_name, e.args)) | |
import couchdb | |
couch = couchdb.Server('http://127.0.0.1:5984/') | |
#couch = couchdb.Server('http://192.168.1.113:5984/') | |
import urllib | |
import urlparse | |
import httplib2 | |
import httplib | |
import csv | |
def url_fix(s, charset='utf-8'): | |
"""Sometimes you get an URL by a user that just isn't a real | |
URL because it contains unsafe characters like ' ' and so on. This | |
function can fix some of the problems in a similar way browsers | |
handle data entered by the user: | |
:param charset: The target charset for the URL if the url was | |
given as unicode string. | |
""" | |
if isinstance(s, unicode): | |
s = s.encode(charset, 'ignore') | |
if not urlparse.urlparse(s).scheme: | |
s = "http://"+s | |
scheme, netloc, path, qs, anchor = urlparse.urlsplit(s) | |
path = urllib.quote(path, '/%') | |
qs = urllib.quote_plus(qs, ':&=') | |
return urlparse.urlunsplit((scheme, netloc, path, qs, anchor)) | |
# http://code.activestate.com/recipes/578019-bytes-to-human-human-to-bytes-converter/ | |
SYMBOLS = { | |
'customary': ('B', 'KB', 'MB', 'GB', 'T', 'P', 'E', 'Z', 'Y'), | |
'customary_ext': ('byte', 'kilo', 'mega', 'giga', 'tera', 'peta', 'exa', | |
'zetta', 'iotta'), | |
'iec': ('Bi', 'Ki', 'Mi', 'Gi', 'Ti', 'Pi', 'Ei', 'Zi', 'Yi'), | |
'iec_ext': ('byte', 'kibi', 'mebi', 'gibi', 'tebi', 'pebi', 'exbi', | |
'zebi', 'yobi'), | |
} | |
docsdb = couch['disclosr-documents'] | |
out = csv.writer(open("output.csv","w"), delimiter=',',quoting=csv.QUOTE_ALL) | |
if __name__ == "__main__": | |
for doc in docsdb.view('app/datasets'): | |
if doc.value['url'] != "http://data.gov.au/data/" and doc.value['agencyID'] != "qld": | |
# Collect the package metadata. | |
pkg_name = filter(lambda x: x in '0123456789abcdefghijklmnopqrstuvwxyz-_', | |
doc.value['url'].replace("http://data.gov.au/dataset/", '').replace('/', '')[:100]); | |
if 'Download' in doc.value['metadata'].keys() and len(doc.value['metadata']['Download']) > 0: | |
for resource in doc.value['metadata']['Download']: | |
# http://docs.ckan.org/en/ckan-1.7/domain-model-resource.html | |
# (KML/KMZ) / (Shapefile) /(Other) | |
format = "plain" | |
if resource['format'] == '(XML)': | |
format = 'xml' | |
if resource['format'] == '(CSV/XLS)': | |
format = 'csv' | |
if resource['format'] == '(Shapefile)': | |
format = 'shp' | |
if resource['format'] == '(KML/KMZ)': | |
format = 'kml' | |
name = resource['href'] | |
if 'name' in resource.keys(): | |
name = resource['name'] | |
if resource['href'].startswith("ftp"): | |
out.writerow([pkg_name, url_fix(resource['href']), name,format, "ftp", ""]) | |
else: | |
try: | |
h = httplib2.Http(disable_ssl_certificate_validation=True) | |
resp = h.request(url_fix(resource['href']), 'HEAD') | |
content_type = resp[0]['content-type'] if 'content-type' in resp[0].keys() else "" | |
out.writerow([pkg_name.encode('ascii', 'ignore'), url_fix(resource['href']).encode('ascii', 'ignore'), name.encode('ascii', 'ignore'),format, resp[0]['status'], content_type]) | |
except httplib2.ServerNotFoundError: | |
out.writerow([pkg_name.encode('ascii', 'ignore'), url_fix(resource['href']).encode('ascii', 'ignore'), name.encode('ascii', 'ignore'),format, "500","badurl"]) | |
except httplib.InvalidURL: | |
out.writerow([pkg_name.encode('ascii', 'ignore'), url_fix(resource['href']).encode('ascii', 'ignore'), name.encode('ascii', 'ignore'),format, "500","badurl"]) | |
except httplib2.RelativeURIError: | |
out.writerow([pkg_name.encode('ascii', 'ignore'), url_fix(resource['href']).encode('ascii', 'ignore'), name.encode('ascii', 'ignore'),format, "500","badurl"]) | |
else: | |
out.writerow([pkg_name.encode('ascii', 'ignore')]) | |
import sys, os | import sys, os |
import time | import time |
import scrape | import scrape |
from bs4 import BeautifulSoup | from bs4 import BeautifulSoup |
from unidecode import unidecode | from unidecode import unidecode |
listurl = "http://data.gov.au/data/" | listurl = "http://data.gov.au/data/" |
(url, mime_type, datasetlisthtml) = scrape.fetchURL(scrape.docsdb, | (url, mime_type, datasetlisthtml) = scrape.fetchURL(scrape.docsdb, |
listurl, "data", "AGIMO") | listurl, "data", "AGIMO") |
soup = BeautifulSoup(datasetlisthtml) | soup = BeautifulSoup(datasetlisthtml) |
for atag in soup.find_all(class_='result-title'): | for atag in soup.find_all(class_='result-title'): |
if atag.has_key('href'): | if atag.has_key('href'): |
url = scrape.fullurl(listurl, atag['href']) | url = scrape.fullurl(listurl, atag['href']) |
(url, mime_type, html) = scrape.fetchURL(scrape.docsdb, | (url, mime_type, html) = scrape.fetchURL(scrape.docsdb, |
url, "data", "AGIMO", False) | url, "data", "AGIMO", False) |
hash = scrape.mkhash(scrape.canonurl(url)) | hash = scrape.mkhash(scrape.canonurl(url)) |
doc = scrape.docsdb.get(hash) | doc = scrape.docsdb.get(hash) |
if "metadata" not in doc.keys() or True: | if "metadata" not in doc.keys() or True: |
doc['type'] = "dataset" | doc['type'] = "dataset" |
doc['metadata'] = {} | doc['metadata'] = {} |
soup = BeautifulSoup(html) | soup = BeautifulSoup(html) |
for metatag in soup.find_all('meta'): | for metatag in soup.find_all('meta'): |
if metatag.has_key('name'): | if metatag.has_key('name'): |
doc['metadata'][metatag['name']] = metatag['content'] | doc['metadata'][metatag['name']] = metatag['content'] |
for list in soup.find_all('dl'): | for list in soup.find_all('dl'): |
last_title = "" | last_title = "" |
for child in list.children: | for child in list.children: |
if str(type(child)) != "<class 'bs4.element.NavigableString'>": | if str(type(child)) != "<class 'bs4.element.NavigableString'>": |
if child.name == 'dt' and child.string != None: | if child.name == 'dt' and child.string != None: |
last_title = child.string.strip() | last_title = child.string.strip() |
if child.name == 'dd': | if child.name == 'dd': |
#print last_title | #print last_title |
if last_title == "Description": | if last_title == "Description": |
doc['metadata'][last_title] = unidecode(str(child)).encode('ascii', 'ignore') | doc['metadata'][last_title] = unidecode(str(child)).encode('ascii', 'ignore') |
elif last_title == "Download": | elif last_title == "Download": |
doc['metadata'][last_title] = [] | doc['metadata'][last_title] = [] |
for item in child.find_all("li"): | for item in child.find_all("li"): |
link = item.find("a") | link = item.find("a") |
format = item.find(property="dc:format") | format = item.find(property="dc:format") |
linkobj = {"href":link['href'].replace("/bye?","").strip(), | linkobj = {"href":link['href'].replace("/bye?","").strip(), |
"format": format.string.strip(), "size": format.next_sibling.string.strip()} | "format": format.string.strip()} |
if format.next_sibling.string != None: | |
linkobj["size"] = format.next_sibling.string.strip() | |
if link.string != None: | if link.string != None: |
linkobj["name"] = link.string.strip() | linkobj["name"] = link.string.strip() |
doc['metadata'][last_title].append(linkobj) | doc['metadata'][last_title].append(linkobj) |
else: | else: |
atags = child.find_all('a') | atags = child.find_all('a') |
if len(atags) < 2: | if len(atags) < 2: |
[s.extract() for s in child(class_='viewAll')] | [s.extract() for s in child(class_='viewAll')] |
doc['metadata'][last_title] = ''.join(child.stripped_strings).strip() | doc['metadata'][last_title] = ''.join(child.stripped_strings).strip() |
else: | else: |
doc['metadata'][last_title] = [item.string.replace(",","").strip() for item in atags] | doc['metadata'][last_title] = [item.string.replace(",","").strip() for item in atags] |
print doc['metadata'] | print doc['metadata'] |
scrape.docsdb.save(doc) | scrape.docsdb.save(doc) |
#time.sleep(2) | #time.sleep(2) |
import sys | import sys |
import os | import os |
sys.path.insert(0, os.path.join(os.path.dirname(__file__) or '.', '../')) | sys.path.insert(0, os.path.join(os.path.dirname(__file__) or '.', '../')) |
import scrape | import scrape |
from bs4 import BeautifulSoup | from bs4 import BeautifulSoup |
from time import mktime | from time import mktime |
import feedparser | import feedparser |
import abc | import abc |
import unicodedata | import unicodedata |
import re | import re |
import dateutil | import dateutil |
from dateutil.parser import * | from dateutil.parser import * |
from datetime import * | from datetime import * |
import codecs | import codecs |
import difflib | import difflib |
from StringIO import StringIO | from StringIO import StringIO |
from pdfminer.pdfparser import PDFDocument, PDFParser | from pdfminer.pdfparser import PDFDocument, PDFParser |
from pdfminer.pdfinterp import PDFResourceManager, PDFPageInterpreter, process_pdf | from pdfminer.pdfinterp import PDFResourceManager, PDFPageInterpreter, process_pdf |
from pdfminer.pdfdevice import PDFDevice, TagExtractor | from pdfminer.pdfdevice import PDFDevice, TagExtractor |
from pdfminer.converter import TextConverter | from pdfminer.converter import TextConverter |
from pdfminer.cmapdb import CMapDB | from pdfminer.cmapdb import CMapDB |
from pdfminer.layout import LAParams | from pdfminer.layout import LAParams |
class GenericDisclogScraper(object): | class GenericDisclogScraper(object): |
__metaclass__ = abc.ABCMeta | __metaclass__ = abc.ABCMeta |
agencyID = None | agencyID = None |
disclogURL = None | disclogURL = None |
def remove_control_chars(self, input): | def remove_control_chars(self, input): |
return "".join([i for i in input if ord(i) in range(32, 127)]) | return "".join([i for i in input if ord(i) in range(32, 127)]) |
def getAgencyID(self): | def getAgencyID(self): |
""" disclosr agency id """ | """ disclosr agency id """ |
if self.agencyID is None: | if self.agencyID is None: |
self.agencyID = os.path.basename(sys.argv[0]).replace(".py", "") | self.agencyID = os.path.basename(sys.argv[0]).replace(".py", "") |
return self.agencyID | return self.agencyID |
def getURL(self): | def getURL(self): |
""" disclog URL""" | """ disclog URL""" |
if self.disclogURL is None: | if self.disclogURL is None: |
agency = scrape.agencydb.get(self.getAgencyID()) | agency = scrape.agencydb.get(self.getAgencyID()) |
self.disclogURL = agency['FOIDocumentsURL'] | self.disclogURL = agency['FOIDocumentsURL'] |
return self.disclogURL | return self.disclogURL |
@abc.abstractmethod | @abc.abstractmethod |
def doScrape(self): | def doScrape(self): |
""" do the scraping """ | """ do the scraping """ |
return | return |
class GenericHTMLDisclogScraper(GenericDisclogScraper): | class GenericHTMLDisclogScraper(GenericDisclogScraper): |
def doScrape(self): | def doScrape(self): |
foidocsdb = scrape.couch['disclosr-foidocuments'] | foidocsdb = scrape.couch['disclosr-foidocuments'] |
(url, mime_type, rcontent) = scrape.fetchURL(scrape.docsdb, | (url, mime_type, rcontent) = scrape.fetchURL(scrape.docsdb, |
self.getURL(), "foidocuments", self.getAgencyID()) | self.getURL(), "foidocuments", self.getAgencyID()) |
content = rcontent | content = rcontent |
dochash = scrape.mkhash(content) | dochash = scrape.mkhash(content) |
doc = foidocsdb.get(dochash) | doc = foidocsdb.get(dochash) |
if doc is None: | if doc is None: |
print "saving " + dochash | print "saving " + dochash |
description = "This log may have updated but as it was not in a table last time we viewed it, we cannot extract what has changed. Please refer to the agency's website Disclosure Log to see the most recent entries" | description = "This log may have updated but as it was not in a table last time we viewed it, we cannot extract what has changed. Please refer to the agency's website Disclosure Log to see the most recent entries" |
last_attach = scrape.getLastAttachment(scrape.docsdb, self.getURL()) | last_attach = scrape.getLastAttachment(scrape.docsdb, self.getURL()) |
if last_attach != None: | if last_attach != None: |
html_diff = difflib.HtmlDiff() | html_diff = difflib.HtmlDiff() |
diff = html_diff.make_table(last_attach.read().split('\n'), | diff = html_diff.make_table(last_attach.read().split('\n'), |
content.split('\n')) | content.split('\n')) |
edate = date.today().strftime("%Y-%m-%d") | edate = date.today().strftime("%Y-%m-%d") |
doc = {'_id': dochash, 'agencyID': self.getAgencyID() | doc = {'_id': dochash, 'agencyID': self.getAgencyID() |
, 'url': self.getURL(), 'docID': dochash, | , 'url': self.getURL(), 'docID': dochash, |
"date": edate, "title": "Disclosure Log Updated", | "date": edate, "title": "Disclosure Log Updated", |
"description": self.remove_control_chars(description), "diff": self.remove_control_chars(diff)} | "description": self.remove_control_chars(description), "diff": self.remove_control_chars(diff)} |
foidocsdb.save(doc) | foidocsdb.save(doc) |
else: | else: |
print "already saved" | print "already saved" |
class GenericPDFDisclogScraper(GenericDisclogScraper): | class GenericPDFDisclogScraper(GenericDisclogScraper): |
def doScrape(self): | def doScrape(self): |
foidocsdb = scrape.couch['disclosr-foidocuments'] | foidocsdb = scrape.couch['disclosr-foidocuments'] |
(url, mime_type, content) = scrape.fetchURL(scrape.docsdb, | (url, mime_type, content) = scrape.fetchURL(scrape.docsdb, |
self.getURL(), "foidocuments", self.getAgencyID()) | self.getURL(), "foidocuments", self.getAgencyID()) |
laparams = LAParams() | laparams = LAParams() |
rsrcmgr = PDFResourceManager(caching=True) | rsrcmgr = PDFResourceManager(caching=True) |
outfp = StringIO() | outfp = StringIO() |
device = TextConverter(rsrcmgr, outfp, codec='utf-8', | device = TextConverter(rsrcmgr, outfp, codec='utf-8', |
laparams=laparams) | laparams=laparams) |
fp = StringIO() | fp = StringIO() |
fp.write(content) | fp.write(content) |
process_pdf(rsrcmgr, device, fp, set(), caching=True, | process_pdf(rsrcmgr, device, fp, set(), caching=True, |
check_extractable=True) | check_extractable=True) |
description = outfp.getvalue() | description = outfp.getvalue() |
fp.close() | fp.close() |
device.close() | device.close() |
outfp.close() | outfp.close() |
dochash = scrape.mkhash(description) | dochash = scrape.mkhash(description) |
doc = foidocsdb.get(dochash) | doc = foidocsdb.get(dochash) |
if doc is None: | if doc is None: |
print "saving " + dochash | print "saving " + dochash |
edate = date.today().strftime("%Y-%m-%d") | edate = date.today().strftime("%Y-%m-%d") |
doc = {'_id': dochash, 'agencyID': self.getAgencyID() | doc = {'_id': dochash, 'agencyID': self.getAgencyID() |
, 'url': self.getURL(), 'docID': dochash, | , 'url': self.getURL(), 'docID': dochash, |
"date": edate, "title": "Disclosure Log Updated", "description": self.remove_control_chars(description)} | "date": edate, "title": "Disclosure Log Updated", "description": self.remove_control_chars(description)} |
foidocsdb.save(doc) | foidocsdb.save(doc) |
else: | else: |
print "already saved" | print "already saved" |
class GenericDOCXDisclogScraper(GenericDisclogScraper): | class GenericDOCXDisclogScraper(GenericDisclogScraper): |
def doScrape(self): | def doScrape(self): |
foidocsdb = scrape.couch['disclosr-foidocuments'] | foidocsdb = scrape.couch['disclosr-foidocuments'] |
(url, mime_type, content) = scrape.fetchURL(scrape.docsdb | (url, mime_type, content) = scrape.fetchURL(scrape.docsdb |
, self.getURL(), "foidocuments", self.getAgencyID()) | , self.getURL(), "foidocuments", self.getAgencyID()) |
mydoc = zipfile.ZipFile(file) | mydoc = zipfile.ZipFile(file) |
xmlcontent = mydoc.read('word/document.xml') | xmlcontent = mydoc.read('word/document.xml') |
document = etree.fromstring(xmlcontent) | document = etree.fromstring(xmlcontent) |
## Fetch all the text out of the document we just created | ## Fetch all the text out of the document we just created |
paratextlist = getdocumenttext(document) | paratextlist = getdocumenttext(document) |
# Make explicit unicode version | # Make explicit unicode version |
newparatextlist = [] | newparatextlist = [] |
for paratext in paratextlist: | for paratext in paratextlist: |
newparatextlist.append(paratext.encode("utf-8")) | newparatextlist.append(paratext.encode("utf-8")) |
## Print our documnts test with two newlines under each paragraph | ## Print our documnts test with two newlines under each paragraph |
description = '\n\n'.join(newparatextlist).strip(' \t\n\r') | description = '\n\n'.join(newparatextlist).strip(' \t\n\r') |
dochash = scrape.mkhash(description) | dochash = scrape.mkhash(description) |
doc = foidocsdb.get(dochash) | doc = foidocsdb.get(dochash) |
if doc is None: | if doc is None: |
print "saving " + dochash | print "saving " + dochash |
edate = time().strftime("%Y-%m-%d") | edate = time().strftime("%Y-%m-%d") |
doc = {'_id': dochash, 'agencyID': self.getAgencyID() | doc = {'_id': dochash, 'agencyID': self.getAgencyID() |
, 'url': self.getURL(), 'docID': dochash, | , 'url': self.getURL(), 'docID': dochash, |
"date": edate, "title": "Disclosure Log Updated", "description": description} | "date": edate, "title": "Disclosure Log Updated", "description": description} |
foidocsdb.save(doc) | foidocsdb.save(doc) |
else: | else: |
print "already saved" | print "already saved" |
class GenericRSSDisclogScraper(GenericDisclogScraper): | class GenericRSSDisclogScraper(GenericDisclogScraper): |
def doScrape(self): | def doScrape(self): |
foidocsdb = scrape.couch['disclosr-foidocuments'] | foidocsdb = scrape.couch['disclosr-foidocuments'] |
(url, mime_type, content) = scrape.fetchURL(scrape.docsdb, | (url, mime_type, content) = scrape.fetchURL(scrape.docsdb, |
self.getURL(), "foidocuments", self.getAgencyID()) | self.getURL(), "foidocuments", self.getAgencyID()) |
feed = feedparser.parse(content) | feed = feedparser.parse(content) |
for entry in feed.entries: | for entry in feed.entries: |
#print entry | #print entry |
print entry.id | print entry.id |
dochash = scrape.mkhash(entry.id) | dochash = scrape.mkhash(entry.id) |
doc = foidocsdb.get(dochash) | doc = foidocsdb.get(dochash) |
#print doc | #print doc |
if doc is None: | if doc is None: |
print "saving " + dochash | print "saving " + dochash |
edate = datetime.fromtimestamp( | edate = datetime.fromtimestamp( |
mktime(entry.published_parsed)).strftime("%Y-%m-%d") | mktime(entry.published_parsed)).strftime("%Y-%m-%d") |
doc = {'_id': dochash, 'agencyID': self.getAgencyID(), | doc = {'_id': dochash, 'agencyID': self.getAgencyID(), |
'url': entry.link, 'docID': entry.id, | 'url': entry.link, 'docID': entry.id, |
"date": edate, "title": entry.title} | "date": edate, "title": entry.title} |
self.getDescription(entry, entry, doc) | self.getDescription(entry, entry, doc) |
foidocsdb.save(doc) | foidocsdb.save(doc) |
else: | else: |
print "already saved" | print "already saved" |
def getDescription(self, content, entry, doc): | def getDescription(self, content, entry, doc): |
""" get description from rss entry""" | """ get description from rss entry""" |
doc.update({'description': content.summary}) | doc.update({'description': content.summary}) |
return | return |
class GenericOAICDisclogScraper(GenericDisclogScraper): | class GenericOAICDisclogScraper(GenericDisclogScraper): |
__metaclass__ = abc.ABCMeta | __metaclass__ = abc.ABCMeta |
@abc.abstractmethod | @abc.abstractmethod |
def getColumns(self, columns): | def getColumns(self, columns): |
""" rearranges columns if required """ | """ rearranges columns if required """ |
return | return |
def getColumnCount(self): | def getColumnCount(self): |
return 5 | return 5 |
def getDescription(self, content, entry, doc): | def getDescription(self, content, entry, doc): |
""" get description from rss entry""" | """ get description from rss entry""" |
descriptiontxt = "" | descriptiontxt = "" |
for string in content.stripped_strings: | for string in content.stripped_strings: |
descriptiontxt = descriptiontxt + " \n" + string | descriptiontxt = descriptiontxt + " \n" + string |
doc.update({'description': descriptiontxt}) | doc.update({'description': descriptiontxt}) |
def getTitle(self, content, entry, doc): | def getTitle(self, content, entry, doc): |
doc.update({'title': (''.join(content.stripped_strings))}) | doc.update({'title': (''.join(content.stripped_strings))}) |
def getTable(self, soup): | def getTable(self, soup): |
return soup.table | return soup.table |
def getRows(self, table): | def getRows(self, table): |
return table.find_all('tr') | return table.find_all('tr') |
def getDocHash(self, id,date, url): | |
if id.string is None: | |
print "no id, using date as hash" | |
return scrape.mkhash( | |
self.remove_control_chars( | |
url + (''.join(date.stripped_strings)))) | |
else: | |
return scrape.mkhash( | |
self.remove_control_chars( | |
url + (''.join(id.stripped_strings)))) | |
def getDate(self, content, entry, doc): | def getDate(self, content, entry, doc): |
strdate = ''.join(content.stripped_strings).strip() | strdate = ''.join(content.stripped_strings).strip() |
(a, b, c) = strdate.partition("(") | (a, b, c) = strdate.partition("(") |
strdate = self.remove_control_chars(a.replace("Octber", "October").replace("1012","2012")replace("Janrurary", "January").replace("1012","2012")) | strdate = self.remove_control_chars(a.replace("Octber", "October").replace("1012","2012").replace("Janrurary", "January").replace("1012","2012")) |
print strdate | print strdate |
try: | try: |
edate = parse(strdate, dayfirst=True, fuzzy=True).strftime("%Y-%m-%d") | edate = parse(strdate, dayfirst=True, fuzzy=True).strftime("%Y-%m-%d") |
except ValueError: | except ValueError: |
print >> sys.stderr, "ERROR date invalid %s " % strdate | print >> sys.stderr, "ERROR date invalid %s " % strdate |
print >> sys.stderr, "ERROR date originally %s " % ''.join(content.stripped_strings).strip() | print >> sys.stderr, "ERROR date originally %s " % ''.join(content.stripped_strings).strip() |
edate = date.today().strftime("%Y-%m-%d") | edate = date.today().strftime("%Y-%m-%d") |
print edate | print edate |
doc.update({'date': edate}) | doc.update({'date': edate}) |
return | return |
def getLinks(self, content, entry, doc): | def getLinks(self, content, entry, doc): |
links = [] | links = [] |
for atag in entry.find_all("a"): | for atag in entry.find_all("a"): |
if atag.has_key('href'): | if atag.has_key('href'): |
links.append(scrape.fullurl(content, atag['href'])) | links.append(scrape.fullurl(content, atag['href'])) |
if links != []: | if links != []: |
doc.update({'links': links}) | doc.update({'links': links}) |
return | return |
def doScrape(self): | def doScrape(self): |
foidocsdb = scrape.couch['disclosr-foidocuments'] | foidocsdb = scrape.couch['disclosr-foidocuments'] |
(url, mime_type, content) = scrape.fetchURL(scrape.docsdb, | (url, mime_type, content) = scrape.fetchURL(scrape.docsdb, |
self.getURL(), "foidocuments", self.getAgencyID()) | self.getURL(), "foidocuments", self.getAgencyID()) |
if content is not None: | if content is not None: |
if mime_type == "text/html" or mime_type == "application/xhtml+xml" or mime_type == "application/xml": | if mime_type == "text/html" or mime_type == "application/xhtml+xml" or mime_type == "application/xml": |
# http://www.crummy.com/software/BeautifulSoup/documentation.html | # http://www.crummy.com/software/BeautifulSoup/documentation.html |
print "parsing" | print "parsing" |
soup = BeautifulSoup(content) | soup = BeautifulSoup(content) |
table = self.getTable(soup) | table = self.getTable(soup) |
for row in self.getRows(table): | for row in self.getRows(table): |
columns = row.find_all('td') | columns = row.find_all('td') |
if len(columns) is self.getColumnCount(): | if len(columns) is self.getColumnCount(): |
(id, date, title, | (id, date, title, |
description, notes) = self.getColumns(columns) | description, notes) = self.getColumns(columns) |
print self.remove_control_chars( | print self.remove_control_chars( |
''.join(id.stripped_strings)) | ''.join(id.stripped_strings)) |
if id.string is None: | dochash = self.getDocHash(id,date,url) |
dochash = scrape.mkhash( | |
self.remove_control_chars( | |
url + (''.join(date.stripped_strings)))) | |
else: | |
dochash = scrape.mkhash( | |
self.remove_control_chars( | |
url + (''.join(id.stripped_strings)))) | |
doc = foidocsdb.get(dochash) | doc = foidocsdb.get(dochash) |
if doc is None: | if doc is None: |
print "saving " + dochash | print "saving " + dochash |
doc = {'_id': dochash, | doc = {'_id': dochash, |
'agencyID': self.getAgencyID(), | 'agencyID': self.getAgencyID(), |
'url': self.getURL(), | 'url': self.getURL(), |
'docID': (''.join(id.stripped_strings))} | 'docID': (''.join(id.stripped_strings))} |
self.getLinks(self.getURL(), row, doc) | self.getLinks(self.getURL(), row, doc) |
self.getTitle(title, row, doc) | self.getTitle(title, row, doc) |
self.getDate(date, row, doc) | self.getDate(date, row, doc) |
self.getDescription(description, row, doc) | self.getDescription(description, row, doc) |
if notes is not None: | if notes is not None: |
doc.update({'notes': ( | doc.update({'notes': ( |
''.join(notes.stripped_strings))}) | ''.join(notes.stripped_strings))}) |
badtitles = ['-', 'Summary of FOI Request' | badtitles = ['-', 'Summary of FOI Request' |
, 'FOI request(in summary form)' | , 'FOI request(in summary form)' |
, 'Summary of FOI request received by the ASC', | , 'Summary of FOI request received by the ASC', |
'Summary of FOI request received by agency/minister', | 'Summary of FOI request received by agency/minister', |
'Description of Documents Requested', 'FOI request', | 'Description of Documents Requested', 'FOI request', |
'Description of FOI Request', 'Summary of request', 'Description', 'Summary', | 'Description of FOI Request', 'Summary of request', 'Description', 'Summary', |
'Summary of FOIrequest received by agency/minister', | 'Summary of FOIrequest received by agency/minister', |
'Summary of FOI request received', 'Description of FOI Request', | 'Summary of FOI request received', 'Description of FOI Request', |
"FOI request", 'Results 1 to 67 of 67'] | "FOI request", 'Results 1 to 67 of 67'] |
if doc['title'] not in badtitles and 'description' in doc.keys() and doc['description'] != '': | if doc['title'] not in badtitles and 'description' in doc.keys() and doc['description'] != '': |
print "saving" | print "saving" |
foidocsdb.save(doc) | foidocsdb.save(doc) |
else: | else: |
print "already saved " + dochash | print "already saved " + dochash |
elif len(row.find_all('th')) is self.getColumnCount(): | elif len(row.find_all('th')) is self.getColumnCount(): |
print "header row" | print "header row" |
else: | else: |
print >> sys.stderr, "ERROR number of columns incorrect" | print >> sys.stderr, "ERROR number of columns incorrect" |
print row | print row |
# www.robotstxt.org/ | # www.robotstxt.org/ |
# http://code.google.com/web/controlcrawlindex/ | # http://code.google.com/web/controlcrawlindex/ |
User-agent: * | User-agent: * |
Disallow: /admin/ | Disallow: /admin/ |
Disallow: /viewDocument.php | |
Sitemap: http://disclosurelo.gs/sitemap.xml.php | Sitemap: http://disclosurelo.gs/sitemap.xml.php |
#!/bin/bash | |
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" | DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" |
echo $DIR | |
cd $DIR | cd $DIR |
echo "" > /tmp/disclosr-error | echo "" > /tmp/disclosr-error |
for f in scrapers/*.py; do | for f in $DIR/scrapers/*.py; do |
echo "Processing $f file.."; | echo "Processing $f file.."; |
md5=`md5sum /tmp/disclosr-error` | md5=`md5sum /tmp/disclosr-error` |
python $f 3>&1 1>&2 2>&3 | tee --append /tmp/disclosr-error; | python $f 3>&1 1>&2 2>&3 | tee --append /tmp/disclosr-error; |
md52=`md5sum /tmp/disclosr-error` | md52=`md5sum /tmp/disclosr-error` |
if [ "$md5" != "$md52" ]; then | if [ "$md5" != "$md52" ]; then |
echo "^^^^^^^^^^^^^^ $f" >> /tmp/disclosr-error; | echo "^^^^^^^^^^^^^^ $f" >> /tmp/disclosr-error; |
fi | fi |
if [ "$?" -ne "0" ]; then | if [ "$?" -ne "0" ]; then |
echo "error"; | echo "error"; |
sleep 1; | sleep 1; |
fi | fi |
done | done |
if [ -s /tmp/disclosr-error ] ; then | if [ -s /tmp/disclosr-error ] ; then |
echo "emailling logs.."; | echo "emailling logs.."; |
mail -E -s "Disclosr errors" maxious@lambdacomplex.org < /tmp/disclosr-error ; | mail -E -s "Disclosr errors" maxious@lambdacomplex.org < /tmp/disclosr-error ; |
fi | fi |
#http://packages.python.org/CouchDB/client.html | #http://packages.python.org/CouchDB/client.html |
import couchdb | import couchdb |
import urllib2 | import urllib2 |
from BeautifulSoup import BeautifulSoup | from BeautifulSoup import BeautifulSoup |
import re | import re |
import hashlib | import hashlib |
from urlparse import urljoin | from urlparse import urljoin |
import time | import time |
import os | import os |
import sys | import sys |
import mimetypes | import mimetypes |
import urllib | import urllib |
import urlparse | import urlparse |
import socket | import socket |
#couch = couchdb.Server('http://192.168.1.148:5984/') | #couch = couchdb.Server('http://192.168.1.148:5984/') |
#couch = couchdb.Server('http://192.168.1.113:5984/') | #couch = couchdb.Server('http://192.168.1.113:5984/') |
couch = couchdb.Server('http://127.0.0.1:5984/') | couch = couchdb.Server('http://127.0.0.1:5984/') |
def mkhash(input): | def mkhash(input): |
return hashlib.md5(input).hexdigest().encode("utf-8") | return hashlib.md5(input).hexdigest().encode("utf-8") |
def canonurl(url): | def canonurl(url): |
r"""Return the canonical, ASCII-encoded form of a UTF-8 encoded URL, or '' | r"""Return the canonical, ASCII-encoded form of a UTF-8 encoded URL, or '' |
if the URL looks invalid. | if the URL looks invalid. |
>>> canonurl('\xe2\x9e\xa1.ws') # tinyarro.ws | >>> canonurl('\xe2\x9e\xa1.ws') # tinyarro.ws |
'http://xn--hgi.ws/' | 'http://xn--hgi.ws/' |
""" | """ |
# strip spaces at the ends and ensure it's prefixed with 'scheme://' | # strip spaces at the ends and ensure it's prefixed with 'scheme://' |
url = url.strip() | url = url.strip() |
if not url: | if not url: |
return '' | return '' |
if not urlparse.urlsplit(url).scheme: | if not urlparse.urlsplit(url).scheme: |
url = 'http://' + url | url = 'http://' + url |
# turn it into Unicode | # turn it into Unicode |
#try: | #try: |
# url = unicode(url, 'utf-8') | # url = unicode(url, 'utf-8') |
#except UnicodeDecodeError: | #except UnicodeDecodeError: |
# return '' # bad UTF-8 chars in URL | # return '' # bad UTF-8 chars in URL |
# parse the URL into its components | # parse the URL into its components |
parsed = urlparse.urlsplit(url) | parsed = urlparse.urlsplit(url) |
scheme, netloc, path, query, fragment = parsed | scheme, netloc, path, query, fragment = parsed |
# ensure scheme is a letter followed by letters, digits, and '+-.' chars | # ensure scheme is a letter followed by letters, digits, and '+-.' chars |
if not re.match(r'[a-z][-+.a-z0-9]*$', scheme, flags=re.I): | if not re.match(r'[a-z][-+.a-z0-9]*$', scheme, flags=re.I): |
return '' | return '' |
scheme = str(scheme) | scheme = str(scheme) |
# ensure domain and port are valid, eg: sub.domain.<1-to-6-TLD-chars>[:port] | # ensure domain and port are valid, eg: sub.domain.<1-to-6-TLD-chars>[:port] |
match = re.match(r'(.+\.[a-z0-9]{1,6})(:\d{1,5})?$', netloc, flags=re.I) | match = re.match(r'(.+\.[a-z0-9]{1,6})(:\d{1,5})?$', netloc, flags=re.I) |
if not match: | if not match: |
return '' | return '' |
domain, port = match.groups() | domain, port = match.groups() |
netloc = domain + (port if port else '') | netloc = domain + (port if port else '') |
netloc = netloc.encode('idna') | netloc = netloc.encode('idna') |
# ensure path is valid and convert Unicode chars to %-encoded | # ensure path is valid and convert Unicode chars to %-encoded |
if not path: | if not path: |
path = '/' # eg: 'http://google.com' -> 'http://google.com/' | path = '/' # eg: 'http://google.com' -> 'http://google.com/' |
path = urllib.quote(urllib.unquote(path.encode('utf-8')), safe='/;') | path = urllib.quote(urllib.unquote(path.encode('utf-8')), safe='/;') |
# ensure query is valid | # ensure query is valid |
query = urllib.quote(urllib.unquote(query.encode('utf-8')), safe='=&?/') | query = urllib.quote(urllib.unquote(query.encode('utf-8')), safe='=&?/') |
# ensure fragment is valid | # ensure fragment is valid |
fragment = urllib.quote(urllib.unquote(fragment.encode('utf-8'))) | fragment = urllib.quote(urllib.unquote(fragment.encode('utf-8'))) |
# piece it all back together, truncating it to a maximum of 4KB | # piece it all back together, truncating it to a maximum of 4KB |
url = urlparse.urlunsplit((scheme, netloc, path, query, fragment)) | url = urlparse.urlunsplit((scheme, netloc, path, query, fragment)) |
return url[:4096] | return url[:4096] |
def fullurl(url, href): | def fullurl(url, href): |
href = href.replace(" ", "%20") | href = href.replace(" ", "%20") |
href = re.sub('#.*$', '', href) | href = re.sub('#.*$', '', href) |
return urljoin(url, href) | return urljoin(url, href) |
#http://diveintopython.org/http_web_services/etags.html | #http://diveintopython.org/http_web_services/etags.html |
class NotModifiedHandler(urllib2.BaseHandler): | class NotModifiedHandler(urllib2.BaseHandler): |
def http_error_304(self, req, fp, code, message, headers): | def http_error_304(self, req, fp, code, message, headers): |
addinfourl = urllib2.addinfourl(fp, headers, req.get_full_url()) | addinfourl = urllib2.addinfourl(fp, headers, req.get_full_url()) |
addinfourl.code = code | addinfourl.code = code |
return addinfourl | return addinfourl |
def getLastAttachment(docsdb, url): | def getLastAttachment(docsdb, url): |
hash = mkhash(url) | hash = mkhash(url) |
doc = docsdb.get(hash) | doc = docsdb.get(hash) |
if doc != None and "_attachments" in doc.keys(): | if doc != None and "_attachments" in doc.keys(): |
last_attachment_fname = doc["_attachments"].keys()[-1] | last_attachment_fname = doc["_attachments"].keys()[-1] |
last_attachment = docsdb.get_attachment(doc, last_attachment_fname) | last_attachment = docsdb.get_attachment(doc, last_attachment_fname) |
return last_attachment | return last_attachment |
else: | else: |
return None | return None |
def fetchURL(docsdb, url, fieldName, agencyID, scrape_again=True): | def fetchURL(docsdb, url, fieldName, agencyID, scrape_again=True): |
url = canonurl(url) | url = canonurl(url) |
hash = mkhash(url) | hash = mkhash(url) |
req = urllib2.Request(url) | req = urllib2.Request(url) |
print "Fetching %s (%s)" % (url, hash) | print "Fetching %s (%s)" % (url, hash) |
if url.startswith("mailto") or url.startswith("javascript") or url.startswith("#") or url == None or url == "": | if url.startswith("mailto") or url.startswith("javascript") or url.startswith("#") or url == None or url == "": |
print >> sys.stderr, "Not a valid HTTP url" | print >> sys.stderr, "Not a valid HTTP url" |
return (None, None, None) | return (None, None, None) |
doc = docsdb.get(hash) | doc = docsdb.get(hash) |
if doc == None: | if doc == None: |
doc = {'_id': hash, 'agencyID': agencyID, 'url': url, 'fieldName': fieldName, 'type': 'website'} | doc = {'_id': hash, 'agencyID': agencyID, 'url': url, 'fieldName': fieldName, 'type': 'website'} |
else: | else: |
if (('page_scraped' in doc) and ((time.time() - doc['page_scraped']) < 60 * 24 * 14) or (scrape_again == False)): | if (('page_scraped' in doc) and ((time.time() - doc['page_scraped']) < 60 * 24 * 14) or (scrape_again == False)): |
print "Uh oh, trying to scrape URL again too soon!" + hash | print "Uh oh, trying to scrape URL again too soon!" + hash |
if "_attachments" in doc.keys(): | if (not doc.has_key('file_size') or doc["file_size"] != "0") and "_attachments" in doc.keys(): |
last_attachment_fname = doc["_attachments"].keys()[-1] | last_attachment_fname = doc["_attachments"].keys()[-1] |
last_attachment = docsdb.get_attachment(doc, last_attachment_fname) | last_attachment = docsdb.get_attachment(doc, last_attachment_fname) |
content = last_attachment.read() | content = last_attachment.read() |
mime_type = doc['mime_type'] | mime_type = doc['mime_type'] |
else: | else: |
content = None | content = None |
mime_type = None | mime_type = None |
return (doc['url'], mime_type, content) | return (doc['url'], mime_type, content) |
req.add_header("User-Agent", "Mozilla/4.0 (compatible; Prometheus webspider; owner maxious@lambdacomplex.org)") | req.add_header("User-Agent", "Mozilla/4.0 (compatible; Prometheus webspider; owner maxious@lambdacomplex.org)") |
#if there is a previous version stored in couchdb, load caching helper tags | #if there is a previous version stored in couchdb, load caching helper tags |
if doc.has_key('etag'): | if doc.has_key('etag'): |
req.add_header("If-None-Match", doc['etag']) | req.add_header("If-None-Match", doc['etag']) |
if doc.has_key('last_modified'): | if doc.has_key('last_modified'): |
req.add_header("If-Modified-Since", doc['last_modified']) | req.add_header("If-Modified-Since", doc['last_modified']) |
opener = urllib2.build_opener(NotModifiedHandler()) | opener = urllib2.build_opener(NotModifiedHandler()) |
try: | try: |
url_handle = opener.open(req, None, 20) | url_handle = opener.open(req, None, 20) |
doc['url'] = url_handle.geturl() # may have followed a redirect to a new url | doc['url'] = url_handle.geturl() # may have followed a redirect to a new url |
headers = url_handle.info() # the addinfourls have the .info() too | headers = url_handle.info() # the addinfourls have the .info() too |
doc['etag'] = headers.getheader("ETag") | doc['etag'] = headers.getheader("ETag") |
doc['last_modified'] = headers.getheader("Last-Modified") | doc['last_modified'] = headers.getheader("Last-Modified") |
doc['date'] = headers.getheader("Date") | doc['date'] = headers.getheader("Date") |
doc['page_scraped'] = time.time() | doc['page_scraped'] = time.time() |
doc['web_server'] = headers.getheader("Server") | doc['web_server'] = headers.getheader("Server") |
doc['via'] = headers.getheader("Via") | doc['via'] = headers.getheader("Via") |
doc['powered_by'] = headers.getheader("X-Powered-By") | doc['powered_by'] = headers.getheader("X-Powered-By") |
doc['file_size'] = headers.getheader("Content-Length") | doc['file_size'] = headers.getheader("Content-Length") |
content_type = headers.getheader("Content-Type") | content_type = headers.getheader("Content-Type") |
if content_type != None: | if content_type != None: |
doc['mime_type'] = content_type.split(";")[0] | doc['mime_type'] = content_type.split(";")[0] |
else: | else: |
(type, encoding) = mimetypes.guess_type(url) | (type, encoding) = mimetypes.guess_type(url) |
doc['mime_type'] = type | doc['mime_type'] = type |
if hasattr(url_handle, 'code'): | if hasattr(url_handle, 'code'): |
if url_handle.code == 304: | if url_handle.code == 304: |
print "the web page has not been modified" + hash | print "the web page has not been modified" + hash |
last_attachment_fname = doc["_attachments"].keys()[-1] | last_attachment_fname = doc["_attachments"].keys()[-1] |
last_attachment = docsdb.get_attachment(doc, last_attachment_fname) | last_attachment = docsdb.get_attachment(doc, last_attachment_fname) |
content = last_attachment | content = last_attachment |
return (doc['url'], doc['mime_type'], content.read()) | return (doc['url'], doc['mime_type'], content.read()) |
else: | else: |
print "new webpage loaded" | print "new webpage loaded" |
content = url_handle.read() | content = url_handle.read() |
docsdb.save(doc) | docsdb.save(doc) |
doc = docsdb.get(hash) # need to get a _rev | doc = docsdb.get(hash) # need to get a _rev |
docsdb.put_attachment(doc, content, str(time.time()) + "-" + os.path.basename(url), doc['mime_type']) | docsdb.put_attachment(doc, content, str(time.time()) + "-" + os.path.basename(url), doc['mime_type']) |
return (doc['url'], doc['mime_type'], content) | return (doc['url'], doc['mime_type'], content) |
#store as attachment epoch-filename | #store as attachment epoch-filename |
except (urllib2.URLError, socket.timeout) as e: | except (urllib2.URLError, socket.timeout) as e: |
print >> sys.stderr,"error!" | print >> sys.stderr,"error!" |
error = "" | error = "" |
if hasattr(e, 'reason'): | if hasattr(e, 'reason'): |
error = "error %s in downloading %s" % (str(e.reason), url) | error = "error %s in downloading %s" % (str(e.reason), url) |
elif hasattr(e, 'code'): | elif hasattr(e, 'code'): |
error = "error %s in downloading %s" % (e.code, url) | error = "error %s in downloading %s" % (e.code, url) |
print >> sys.stderr, error | print >> sys.stderr, error |
doc['error'] = error | doc['error'] = error |
docsdb.save(doc) | docsdb.save(doc) |
return (None, None, None) | return (None, None, None) |
def scrapeAndStore(docsdb, url, depth, fieldName, agencyID): | def scrapeAndStore(docsdb, url, depth, fieldName, agencyID): |
(url, mime_type, content) = fetchURL(docsdb, url, fieldName, agencyID) | (url, mime_type, content) = fetchURL(docsdb, url, fieldName, agencyID) |
badURLs = ["http://www.ausport.gov.au/supporting/funding/grants_and_scholarships/grant_funding_report"] | badURLs = ["http://www.ausport.gov.au/supporting/funding/grants_and_scholarships/grant_funding_report"] |
if content != None and depth > 0 and url != "http://www.ausport.gov.au/supporting/funding/grants_and_scholarships/grant_funding_report": | if content != None and depth > 0 and url != "http://www.ausport.gov.au/supporting/funding/grants_and_scholarships/grant_funding_report": |
if mime_type == "text/html" or mime_type == "application/xhtml+xml" or mime_type == "application/xml": | if mime_type == "text/html" or mime_type == "application/xhtml+xml" or mime_type == "application/xml": |
# http://www.crummy.com/software/BeautifulSoup/documentation.html | # http://www.crummy.com/software/BeautifulSoup/documentation.html |
soup = BeautifulSoup(content) | soup = BeautifulSoup(content) |
navIDs = soup.findAll( | navIDs = soup.findAll( |
id=re.compile('nav|Nav|menu|bar|left|right|sidebar|more-links|breadcrumb|footer|header')) | id=re.compile('nav|Nav|menu|bar|left|right|sidebar|more-links|breadcrumb|footer|header')) |
for nav in navIDs: | for nav in navIDs: |
print "Removing element", nav['id'] | print "Removing element", nav['id'] |
nav.extract() | nav.extract() |
navClasses = soup.findAll( | navClasses = soup.findAll( |
attrs={'class': re.compile('nav|menu|bar|left|right|sidebar|more-links|breadcrumb|footer|header')}) | attrs={'class': re.compile('nav|menu|bar|left|right|sidebar|more-links|breadcrumb|footer|header')}) |
for nav in navClasses: | for nav in navClasses: |
print "Removing element", nav['class'] | print "Removing element", nav['class'] |
nav.extract() | nav.extract() |
links = soup.findAll('a') # soup.findAll('a', id=re.compile("^p-")) | links = soup.findAll('a') # soup.findAll('a', id=re.compile("^p-")) |
linkurls = set([]) | linkurls = set([]) |
for link in links: | for link in links: |
if link.has_key("href"): | if link.has_attr("href"): |
if link['href'].startswith("http"): | if link['href'].startswith("http"): |
# lets not do external links for now | # lets not do external links for now |
# linkurls.add(link['href']) | # linkurls.add(link['href']) |
None | None |
if link['href'].startswith("mailto"): | if link['href'].startswith("mailto"): |
# not http | # not http |
None | None |
if link['href'].startswith("javascript"): | if link['href'].startswith("javascript"): |
# not http | # not http |
None | None |
else: | else: |
# remove anchors and spaces in urls | # remove anchors and spaces in urls |
linkurls.add(fullurl(url, link['href'])) | linkurls.add(fullurl(url, link['href'])) |
for linkurl in linkurls: | for linkurl in linkurls: |
#print linkurl | #print linkurl |
scrapeAndStore(docsdb, linkurl, depth - 1, fieldName, agencyID) | scrapeAndStore(docsdb, linkurl, depth - 1, fieldName, agencyID) |
# select database | # select database |
agencydb = couch['disclosr-agencies'] | agencydb = couch['disclosr-agencies'] |
docsdb = couch['disclosr-documents'] | docsdb = couch['disclosr-documents'] |
if __name__ == "__main__": | if __name__ == "__main__": |
for row in agencydb.view('app/all'): #not recently scraped agencies view? | for row in agencydb.view('app/all'): #not recently scraped agencies view? |
agency = agencydb.get(row.id) | agency = agencydb.get(row.id) |
print agency['name'] | print agency['name'] |
for key in agency.keys(): | for key in agency.keys(): |
if key == "FOIDocumentsURL" and "status" not in agency.keys() and False: | if key == "FOIDocumentsURL" and "status" not in agency.keys() and False: |
scrapeAndStore(docsdb, agency[key], 0, key, agency['_id']) | scrapeAndStore(docsdb, agency[key], 0, key, agency['_id']) |
if key == 'website' and True: | if key == 'website' and True: |
scrapeAndStore(docsdb, agency[key], 0, key, agency['_id']) | scrapeAndStore(docsdb, agency[key], 0, key, agency['_id']) |
if "metadata" not in agency.keys(): | if "metadata" not in agency.keys(): |
agency['metadata'] = {} | agency['metadata'] = {} |
agency['metadata']['lastScraped'] = time.time() | agency['metadata']['lastScraped'] = time.time() |
if key.endswith('URL') and False: | if key.endswith('URL') and False: |
print key | print key |
depth = 1 | depth = 1 |
if 'scrapeDepth' in agency.keys(): | if 'scrapeDepth' in agency.keys(): |
depth = agency['scrapeDepth'] | depth = agency['scrapeDepth'] |
scrapeAndStore(docsdb, agency[key], depth, key, agency['_id']) | scrapeAndStore(docsdb, agency[key], depth, key, agency['_id']) |
agencydb.save(agency) | agencydb.save(agency) |
import sys,os | import sys,os |
sys.path.insert(0, os.path.join(os.path.dirname(__file__) or '.', '../')) | sys.path.insert(0, os.path.join(os.path.dirname(__file__) or '.', '../')) |
import genericScrapers | import genericScrapers |
import scrape | import scrape |
from bs4 import BeautifulSoup | from bs4 import BeautifulSoup |
#http://www.doughellmann.com/PyMOTW/abc/ | #http://www.doughellmann.com/PyMOTW/abc/ |
class ScraperImplementation(genericScrapers.GenericOAICDisclogScraper): | class ScraperImplementation(genericScrapers.GenericOAICDisclogScraper): |
def getTable(self,soup): | def getTable(self,soup): |
return soup.find(id = "maincontentcontainer").table | return soup.find(class_ = "contentcontainer").table |
def getColumnCount(self): | def getColumnCount(self): |
return 5 | return 5 |
def getColumns(self,columns): | def getColumns(self,columns): |
(date, disclogdate, title, description, notes) = columns | (date, disclogdate, title, description, notes) = columns |
return (date, date, title, description, notes) | return (date, date, title, description, notes) |
if __name__ == '__main__': | if __name__ == '__main__': |
print 'Subclass:', issubclass(ScraperImplementation, genericScrapers.GenericOAICDisclogScraper) | print 'Subclass:', issubclass(ScraperImplementation, genericScrapers.GenericOAICDisclogScraper) |
print 'Instance:', isinstance(ScraperImplementation(), genericScrapers.GenericOAICDisclogScraper) | print 'Instance:', isinstance(ScraperImplementation(), genericScrapers.GenericOAICDisclogScraper) |
ScraperImplementation().doScrape() | ScraperImplementation().doScrape() |
import sys,os | import sys,os |
sys.path.insert(0, os.path.join(os.path.dirname(__file__) or '.', '../')) | sys.path.insert(0, os.path.join(os.path.dirname(__file__) or '.', '../')) |
import genericScrapers | import genericScrapers |
import dateutil | import dateutil |
from dateutil.parser import * | from dateutil.parser import * |
from datetime import * | from datetime import * |
import scrape | import scrape |
from bs4 import BeautifulSoup | from bs4 import BeautifulSoup |
class ScraperImplementation(genericScrapers.GenericOAICDisclogScraper): | class ScraperImplementation(genericScrapers.GenericOAICDisclogScraper): |
def __init__(self): | def __init__(self): |
super(ScraperImplementation, self).__init__() | super(ScraperImplementation, self).__init__() |
def getDescription(self,content, entry,doc): | def getDescription(self,content, entry,doc): |
link = None | link = None |
links = [] | links = [] |
description = "" | description = "" |
for atag in entry.find_all('a'): | for atag in entry.find_all('a'): |
if atag.has_key('href'): | if atag.has_attr('href'): |
link = scrape.fullurl(self.getURL(), atag['href']) | link = scrape.fullurl(self.getURL(), atag['href']) |
(url, mime_type, htcontent) = scrape.fetchURL(scrape.docsdb, link, "foidocuments", self.getAgencyID(), False) | (url, mime_type, htcontent) = scrape.fetchURL(scrape.docsdb, link, "foidocuments", self.getAgencyID(), False) |
if htcontent != None: | if htcontent != None: |
if mime_type == "text/html" or mime_type == "application/xhtml+xml" or mime_type =="application/xml": | if mime_type == "text/html" or mime_type == "application/xhtml+xml" or mime_type =="application/xml": |
soup = BeautifulSoup(htcontent) | soup = BeautifulSoup(htcontent) |
row = soup.find(id="content_div_148050") | row = soup.find(id="content_div_148050") |
description = ''.join(row.stripped_strings) | description = ''.join(row.stripped_strings) |
for atag in row.find_all("a"): | for atag in row.find_all("a"): |
if atag.has_key('href'): | if atag.has_attr('href'): |
links.append(scrape.fullurl(link, atag['href'])) | links.append(scrape.fullurl(link, atag['href'])) |
if links != []: | if links != []: |
doc.update({'links': links}) | doc.update({'links': links}) |
if description != "": | if description != "": |
doc.update({ 'description': description}) | doc.update({ 'description': description}) |
def getColumnCount(self): | def getColumnCount(self): |
return 4 | return 4 |
def getColumns(self, columns): | def getColumns(self, columns): |
(id, date, datepub, title) = columns | (id, date, datepub, title) = columns |
return (id, date, title, title, None) | return (id, date, title, title, None) |
if __name__ == '__main__': | if __name__ == '__main__': |
print 'Subclass:', issubclass(ScraperImplementation, genericScrapers.GenericOAICDisclogScraper) | print 'Subclass:', issubclass(ScraperImplementation, genericScrapers.GenericOAICDisclogScraper) |
print 'Instance:', isinstance(ScraperImplementation(), genericScrapers.GenericOAICDisclogScraper) | print 'Instance:', isinstance(ScraperImplementation(), genericScrapers.GenericOAICDisclogScraper) |
nsi = ScraperImplementation() | nsi = ScraperImplementation() |
nsi.disclogURL = "http://www.dbcde.gov.au/about_us/freedom_of_information_disclosure_log/foi_list?result_146858_result_page=1" | nsi.disclogURL = "http://www.dbcde.gov.au/about_us/freedom_of_information_disclosure_log/foi_list?result_146858_result_page=1" |
nsi.doScrape() | nsi.doScrape() |
nsi.disclogURL = "http://www.dbcde.gov.au/about_us/freedom_of_information_disclosure_log/foi_list?result_146858_result_page=2" | nsi.disclogURL = "http://www.dbcde.gov.au/about_us/freedom_of_information_disclosure_log/foi_list?result_146858_result_page=2" |
nsi.doScrape() | nsi.doScrape() |
nsi.disclogURL = "http://www.dbcde.gov.au/about_us/freedom_of_information_disclosure_log/foi_list?result_146858_result_page=3" | nsi.disclogURL = "http://www.dbcde.gov.au/about_us/freedom_of_information_disclosure_log/foi_list?result_146858_result_page=3" |
nsi.doScrape() | nsi.doScrape() |
nsi.disclogURL = "http://www.dbcde.gov.au/about_us/freedom_of_information_disclosure_log/foi_list?result_146858_result_page=4" | nsi.disclogURL = "http://www.dbcde.gov.au/about_us/freedom_of_information_disclosure_log/foi_list?result_146858_result_page=4" |
nsi.doScrape() | nsi.doScrape() |
nsi.disclogURL = "http://www.dbcde.gov.au/about_us/freedom_of_information_disclosure_log/foi_list?result_146858_result_page=5" | nsi.disclogURL = "http://www.dbcde.gov.au/about_us/freedom_of_information_disclosure_log/foi_list?result_146858_result_page=5" |
nsi.doScrape() | nsi.doScrape() |
import sys,os | import sys,os |
sys.path.insert(0, os.path.join(os.path.dirname(__file__) or '.', '../')) | sys.path.insert(0, os.path.join(os.path.dirname(__file__) or '.', '../')) |
import genericScrapers | import genericScrapers |
import scrape | import scrape |
from bs4 import BeautifulSoup | from bs4 import BeautifulSoup |
#http://www.doughellmann.com/PyMOTW/abc/ | #http://www.doughellmann.com/PyMOTW/abc/ |
class ScraperImplementation(genericScrapers.GenericOAICDisclogScraper): | class ScraperImplementation(genericScrapers.GenericOAICDisclogScraper): |
def getDescription(self,content, entry,doc): | def getDescription(self,content, entry,doc): |
link = None | link = None |
links = [] | links = [] |
description = "" | description = "" |
for atag in entry.find_all('a'): | for atag in entry.find_all('a'): |
if atag.has_key('href'): | if atag.has_attr('href'): |
link = scrape.fullurl(self.getURL(),atag['href']) | link = scrape.fullurl(self.getURL(),atag['href']) |
(url,mime_type,htcontent) = scrape.fetchURL(scrape.docsdb, link, "foidocuments", self.getAgencyID(), False) | (url,mime_type,htcontent) = scrape.fetchURL(scrape.docsdb, link, "foidocuments", self.getAgencyID(), False) |
if htcontent != None: | if htcontent != None: |
if mime_type == "text/html" or mime_type == "application/xhtml+xml" or mime_type =="application/xml": | if mime_type == "text/html" or mime_type == "application/xhtml+xml" or mime_type =="application/xml": |
# http://www.crummy.com/software/BeautifulSoup/documentation.html | # http://www.crummy.com/software/BeautifulSoup/documentation.html |
soup = BeautifulSoup(htcontent) | soup = BeautifulSoup(htcontent) |
rowtitle = soup.find(class_ = "wc-title").find("h1").string | rowtitle = soup.find(class_ = "wc-title").find("h1").string |
if rowtitle != None: | if rowtitle != None: |
description = rowtitle + ": " | description = rowtitle + ": " |
for row in soup.find(class_ ="wc-content").find_all('td'): | for row in soup.find(class_ ="wc-content").find_all('td'): |
if row != None: | if row != None: |
for text in row.stripped_strings: | for text in row.stripped_strings: |
description = description + text + "\n" | description = description + text + "\n" |
for atag in row.find_all("a"): | for atag in row.find_all("a"): |
if atag.has_key('href'): | if atag.has_attr('href'): |
links.append(scrape.fullurl(link,atag['href'])) | links.append(scrape.fullurl(link,atag['href'])) |
if links != []: | if links != []: |
doc.update({'links': links}) | doc.update({'links': links}) |
if description != "": | if description != "": |
doc.update({ 'description': description}) | doc.update({ 'description': description}) |
def getColumnCount(self): | def getColumnCount(self): |
return 2 | return 2 |
def getTable(self,soup): | def getTable(self,soup): |
return soup.find(class_ = "ms-rteTable-default") | return soup.find(class_ = "ms-rteTable-default") |
def getColumns(self,columns): | def getColumns(self,columns): |
(date, title) = columns | (date, title) = columns |
return (title, date, title, title, None) | return (title, date, title, title, None) |
if __name__ == '__main__': | if __name__ == '__main__': |
print 'Subclass:', issubclass(ScraperImplementation, genericScrapers.GenericOAICDisclogScraper) | print 'Subclass:', issubclass(ScraperImplementation, genericScrapers.GenericOAICDisclogScraper) |
print 'Instance:', isinstance(ScraperImplementation(), genericScrapers.GenericOAICDisclogScraper) | print 'Instance:', isinstance(ScraperImplementation(), genericScrapers.GenericOAICDisclogScraper) |
ScraperImplementation().doScrape() | ScraperImplementation().doScrape() |
import sys,os | import sys,os |
sys.path.insert(0, os.path.join(os.path.dirname(__file__) or '.', '../')) | sys.path.insert(0, os.path.join(os.path.dirname(__file__) or '.', '../')) |
import genericScrapers | import genericScrapers |
import scrape | import scrape |
from bs4 import BeautifulSoup | from bs4 import BeautifulSoup |
#http://www.doughellmann.com/PyMOTW/abc/ | #http://www.doughellmann.com/PyMOTW/abc/ |
class ScraperImplementation(genericScrapers.GenericOAICDisclogScraper): | class ScraperImplementation(genericScrapers.GenericOAICDisclogScraper): |
def getTable(self,soup): | def getTable(self,soup): |
return soup.find(class_ = "ms-rtestate-field").table | return soup.find(class_ = "ms-rtestate-field").table |
def getColumns(self,columns): | def getColumns(self,columns): |
(id, date, title, description, notes) = columns | (id, date, title, description, notes) = columns |
return (id, date, title, description, notes) | return (id, date, title, description, notes) |
def getLinks(self, content, entry, doc): | def getLinks(self, content, entry, doc): |
link = None | link = None |
links = [] | links = [] |
for atag in entry.find_all('a'): | for atag in entry.find_all('a'): |
if atag.has_key('href'): | if atag.has_attr('href'): |
link = scrape.fullurl(self.getURL(),atag['href']) | link = scrape.fullurl(self.getURL(),atag['href']) |
(url,mime_type,htcontent) = scrape.fetchURL(scrape.docsdb, link, "foidocuments", self.getAgencyID(), False) | (url,mime_type,htcontent) = scrape.fetchURL(scrape.docsdb, link, "foidocuments", self.getAgencyID(), False) |
if htcontent != None: | if htcontent != None: |
if mime_type == "text/html" or mime_type == "application/xhtml+xml" or mime_type =="application/xml": | if mime_type == "text/html" or mime_type == "application/xhtml+xml" or mime_type =="application/xml": |
# http://www.crummy.com/software/BeautifulSoup/documentation.html | # http://www.crummy.com/software/BeautifulSoup/documentation.html |
soup = BeautifulSoup(htcontent) | soup = BeautifulSoup(htcontent) |
for atag in soup.find(class_ = "article-content").find_all('a'): | for atag in soup.find(class_ = "article-content").find_all('a'): |
if atag.has_key('href'): | if atag.has_attr('href'): |
links.append(scrape.fullurl(link,atag['href'])) | links.append(scrape.fullurl(link,atag['href'])) |
if links != []: | if links != []: |
doc.update({'links': links}) | doc.update({'links': links}) |
doc.update({'url': link}) | doc.update({'url': link}) |
return | return |
if __name__ == '__main__': | if __name__ == '__main__': |
print 'Subclass:', issubclass(ScraperImplementation, genericScrapers.GenericOAICDisclogScraper) | print 'Subclass:', issubclass(ScraperImplementation, genericScrapers.GenericOAICDisclogScraper) |
print 'Instance:', isinstance(ScraperImplementation(), genericScrapers.GenericOAICDisclogScraper) | print 'Instance:', isinstance(ScraperImplementation(), genericScrapers.GenericOAICDisclogScraper) |
ScraperImplementation().doScrape() | ScraperImplementation().doScrape() |
import sys,os | import sys,os |
sys.path.insert(0, os.path.join(os.path.dirname(__file__) or '.', '../')) | sys.path.insert(0, os.path.join(os.path.dirname(__file__) or '.', '../')) |
import genericScrapers | import genericScrapers |
import scrape | import scrape |
from bs4 import BeautifulSoup | from bs4 import BeautifulSoup |
#http://www.doughellmann.com/PyMOTW/abc/ | #http://www.doughellmann.com/PyMOTW/abc/ |
class ScraperImplementation(genericScrapers.GenericOAICDisclogScraper): | class ScraperImplementation(genericScrapers.GenericOAICDisclogScraper): |
def getDocHash(self, id,date, url): | |
''' url changes on ever request so ignore for hash ''' | |
return scrape.mkhash( | |
self.remove_control_chars( | |
''.join(id.stripped_strings))) | |
def getColumnCount(self): | def getColumnCount(self): |
return 4 | return 4 |
def getColumns(self,columns): | def getColumns(self,columns): |
(date, id, title, description) = columns | (date, id, title, description) = columns |
return (id, date, title, description, None) | return (id, date, title, description, None) |
if __name__ == '__main__': | if __name__ == '__main__': |
print 'Subclass:', issubclass(ScraperImplementation, genericScrapers.GenericOAICDisclogScraper) | print 'Subclass:', issubclass(ScraperImplementation, genericScrapers.GenericOAICDisclogScraper) |
print 'Instance:', isinstance(ScraperImplementation(), genericScrapers.GenericOAICDisclogScraper) | print 'Instance:', isinstance(ScraperImplementation(), genericScrapers.GenericOAICDisclogScraper) |
ScraperImplementation().doScrape() | ScraperImplementation().doScrape() |
import sys,os | import sys,os |
sys.path.insert(0, os.path.join(os.path.dirname(__file__) or '.', '../')) | sys.path.insert(0, os.path.join(os.path.dirname(__file__) or '.', '../')) |
import genericScrapers | import genericScrapers |
import scrape | import scrape |
from bs4 import BeautifulSoup | from bs4 import BeautifulSoup |
#http://www.doughellmann.com/PyMOTW/abc/ | #http://www.doughellmann.com/PyMOTW/abc/ |
class ScraperImplementation(genericScrapers.GenericOAICDisclogScraper): | class ScraperImplementation(genericScrapers.GenericOAICDisclogScraper): |
def getTable(self,soup): | |
return soup.find(id = "centercontent").table | |
def getColumnCount(self): | def getColumnCount(self): |
return 5 | return 5 |
def getColumns(self,columns): | def getColumns(self,columns): |
(id, date, title, description,notes) = columns | (id, date, title, description,notes) = columns |
return (id, date, title, description, notes) | return (id, date, title, description, notes) |
if __name__ == '__main__': | if __name__ == '__main__': |
print 'Subclass:', issubclass(ScraperImplementation, genericScrapers.GenericOAICDisclogScraper) | print 'Subclass:', issubclass(ScraperImplementation, genericScrapers.GenericOAICDisclogScraper) |
print 'Instance:', isinstance(ScraperImplementation(), genericScrapers.GenericOAICDisclogScraper) | print 'Instance:', isinstance(ScraperImplementation(), genericScrapers.GenericOAICDisclogScraper) |
ScraperImplementation().doScrape() | ScraperImplementation().doScrape() |
import sys,os | import sys,os |
sys.path.insert(0, os.path.join(os.path.dirname(__file__) or '.', '../')) | sys.path.insert(0, os.path.join(os.path.dirname(__file__) or '.', '../')) |
import genericScrapers | import genericScrapers |
import scrape | import scrape |
from bs4 import BeautifulSoup | from bs4 import BeautifulSoup |
import codecs | import codecs |
#http://www.doughellmann.com/PyMOTW/abc/ | #http://www.doughellmann.com/PyMOTW/abc/ |
class NewScraperImplementation(genericScrapers.GenericOAICDisclogScraper): | class NewScraperImplementation(genericScrapers.GenericOAICDisclogScraper): |
def getDescription(self,content, entry,doc): | def getDescription(self,content, entry,doc): |
link = None | link = None |
links = [] | links = [] |
description = "" | description = "" |
for atag in entry.find_all('a'): | for atag in entry.find_all('a'): |
if atag.has_key('href'): | if atag.has_attr('href'): |
link = scrape.fullurl(self.getURL(),atag['href']) | link = scrape.fullurl(self.getURL(),atag['href']) |
(url,mime_type,htcontent) = scrape.fetchURL(scrape.docsdb, link, "foidocuments", self.getAgencyID(), False) | (url,mime_type,htcontent) = scrape.fetchURL(scrape.docsdb, link, "foidocuments", self.getAgencyID(), False) |
if htcontent != None: | if htcontent != None: |
if mime_type == "text/html" or mime_type == "application/xhtml+xml" or mime_type =="application/xml": | if mime_type == "text/html" or mime_type == "application/xhtml+xml" or mime_type =="application/xml": |
# http://www.crummy.com/software/BeautifulSoup/documentation.html | # http://www.crummy.com/software/BeautifulSoup/documentation.html |
soup = BeautifulSoup(htcontent) | soup = BeautifulSoup(htcontent) |
for text in soup.find(class_ = "mainContent").stripped_strings: | for text in soup.find(class_ = "mainContent").stripped_strings: |
description = description + text.encode('ascii', 'ignore') | description = description + text.encode('ascii', 'ignore') |
for atag in soup.find(id="SortingTable").find_all("a"): | for atag in soup.find(id="SortingTable").find_all("a"): |
if atag.has_key('href'): | if atag.has_attr('href'): |
links.append(scrape.fullurl(link,atag['href'])) | links.append(scrape.fullurl(link,atag['href'])) |
if links != []: | if links != []: |
doc.update({'links': links}) | doc.update({'links': links}) |
if description != "": | if description != "": |
doc.update({ 'description': description}) | doc.update({ 'description': description}) |
def getColumnCount(self): | def getColumnCount(self): |
return 2 | return 2 |
def getTable(self,soup): | def getTable(self,soup): |
return soup.find(id = "TwoColumnSorting") | return soup.find(id = "TwoColumnSorting") |
def getColumns(self,columns): | def getColumns(self,columns): |
( title, date) = columns | ( title, date) = columns |
return (title, date, title, title, None) | return (title, date, title, title, None) |
class OldScraperImplementation(genericScrapers.GenericOAICDisclogScraper): | class OldScraperImplementation(genericScrapers.GenericOAICDisclogScraper): |
def getDescription(self,content, entry,doc): | def getDescription(self,content, entry,doc): |
link = None | link = None |
links = [] | links = [] |
description = "" | description = "" |
for atag in entry.find_all('a'): | for atag in entry.find_all('a'): |
if atag.has_key('href'): | if atag.has_attr('href'): |
link = scrape.fullurl(self.getURL(),atag['href']) | link = scrape.fullurl(self.getURL(),atag['href']) |
(url,mime_type,htcontent) = scrape.fetchURL(scrape.docsdb, link, "foidocuments", self.getAgencyID(), False) | (url,mime_type,htcontent) = scrape.fetchURL(scrape.docsdb, link, "foidocuments", self.getAgencyID(), False) |
if htcontent != None: | if htcontent != None: |
if mime_type == "text/html" or mime_type == "application/xhtml+xml" or mime_type =="application/xml": | if mime_type == "text/html" or mime_type == "application/xhtml+xml" or mime_type =="application/xml": |
# http://www.crummy.com/software/BeautifulSoup/documentation.html | # http://www.crummy.com/software/BeautifulSoup/documentation.html |
soup = BeautifulSoup(htcontent) | soup = BeautifulSoup(htcontent) |
for text in soup.find(id="content-item").stripped_strings: | for text in soup.find(id="content-item").stripped_strings: |
description = description + text + " \n" | description = description + text + " \n" |
for atag in soup.find(id="content-item").find_all("a"): | for atag in soup.find(id="content-item").find_all("a"): |
if atag.has_key('href'): | if atag.has_attr('href'): |
links.append(scrape.fullurl(link,atag['href'])) | links.append(scrape.fullurl(link,atag['href'])) |
if links != []: | if links != []: |
doc.update({'links': links}) | doc.update({'links': links}) |
if description != "": | if description != "": |
doc.update({ 'description': description}) | doc.update({ 'description': description}) |
if links != []: | if links != []: |
doc.update({'links': links}) | doc.update({'links': links}) |
if description != "": | if description != "": |
doc.update({ 'description': description}) | doc.update({ 'description': description}) |
def getColumnCount(self): | def getColumnCount(self): |
return 2 | return 2 |
def getTable(self,soup): | def getTable(self,soup): |
return soup.find(class_ = "doc-list") | return soup.find(class_ = "doc-list") |
def getColumns(self,columns): | def getColumns(self,columns): |
(date, title) = columns | (date, title) = columns |
return (title, date, title, title, None) | return (title, date, title, title, None) |
if __name__ == '__main__': | if __name__ == '__main__': |
print 'Subclass:', issubclass(NewScraperImplementation, genericScrapers.GenericOAICDisclogScraper) | print 'Subclass:', issubclass(NewScraperImplementation, genericScrapers.GenericOAICDisclogScraper) |
print 'Instance:', isinstance(NewScraperImplementation(), genericScrapers.GenericOAICDisclogScraper) | print 'Instance:', isinstance(NewScraperImplementation(), genericScrapers.GenericOAICDisclogScraper) |
NewScraperImplementation().doScrape() | NewScraperImplementation().doScrape() |
print 'Subclass:', issubclass(OldScraperImplementation, genericScrapers.GenericOAICDisclogScraper) | print 'Subclass:', issubclass(OldScraperImplementation, genericScrapers.GenericOAICDisclogScraper) |
print 'Instance:', isinstance(OldScraperImplementation(), genericScrapers.GenericOAICDisclogScraper) | print 'Instance:', isinstance(OldScraperImplementation(), genericScrapers.GenericOAICDisclogScraper) |
osi = OldScraperImplementation() | osi = OldScraperImplementation() |
osi.disclogURL = "http://archive.treasury.gov.au/content/foi_publications.asp?year=-1&abstract=0&classification=&=&titl=Disclosure+Log+-+Documents+Released+Under+FOI" | osi.disclogURL = "http://archive.treasury.gov.au/content/foi_publications.asp?year=-1&abstract=0&classification=&=&titl=Disclosure+Log+-+Documents+Released+Under+FOI" |
osi.doScrape() | osi.doScrape() |
import sys,os | import sys,os |
sys.path.insert(0, os.path.join(os.path.dirname(__file__) or '.', '../')) | sys.path.insert(0, os.path.join(os.path.dirname(__file__) or '.', '../')) |
import genericScrapers | import genericScrapers |
#RSS feed not detailed | #RSS feed not detailed |
import scrape | import scrape |
from bs4 import BeautifulSoup | from bs4 import BeautifulSoup |
#http://www.doughellmann.com/PyMOTW/abc/ | #http://www.doughellmann.com/PyMOTW/abc/ |
class ScraperImplementation(genericScrapers.GenericRSSDisclogScraper): | class ScraperImplementation(genericScrapers.GenericRSSDisclogScraper): |
def getDescription(self,content, entry,doc): | def getDescription(self,content, entry,doc): |
(url,mime_type,htcontent) = scrape.fetchURL(scrape.docsdb, entry.link, "foidocuments", self.getAgencyID(), False) | (url,mime_type,htcontent) = scrape.fetchURL(scrape.docsdb, entry.link, "foidocuments", self.getAgencyID(), False) |
if htcontent != None: | if htcontent != None: |
if mime_type == "text/html" or mime_type == "application/xhtml+xml" or mime_type =="application/xml": | if mime_type == "text/html" or mime_type == "application/xhtml+xml" or mime_type =="application/xml": |
# http://www.crummy.com/software/BeautifulSoup/documentation.html | # http://www.crummy.com/software/BeautifulSoup/documentation.html |
soup = BeautifulSoup(content) | soup = BeautifulSoup(content) |
links = [] | links = [] |
description = "" | description = "" |
dldivs = soup.find('div',class_="download") | dldivs = soup.find('div',class_="download") |
if dldivs != None: | if dldivs != None: |
for atag in dldivs.find_all("a"): | for atag in dldivs.find_all("a"): |
if atag.has_key('href'): | if atag.has_attr('href'): |
links.append(scrape.fullurl(url,atag['href'])) | links.append(scrape.fullurl(url,atag['href'])) |
nodldivs = soup.find('div',class_="incompleteNotification") | nodldivs = soup.find('div',class_="incompleteNotification") |
if nodldivs != None and nodldivs.stripped_strings != None: | if nodldivs != None and nodldivs.stripped_strings != None: |
for text in nodldivs.stripped_strings: | for text in nodldivs.stripped_strings: |
description = description + text | description = description + text |
for row in soup.table.find_all('tr'): | for row in soup.table.find_all('tr'): |
if row != None: | if row != None: |
description = description + "\n" + row.find('th').string + ": " | description = description + "\n" + row.find('th').string + ": " |
for text in row.find('div').stripped_strings: | for text in row.find('div').stripped_strings: |
description = description + text | description = description + text |
if links != []: | if links != []: |
doc.update({'links': links}) | doc.update({'links': links}) |
if description != "": | if description != "": |
doc.update({ 'description': description}) | doc.update({ 'description': description}) |
if __name__ == '__main__': | if __name__ == '__main__': |
print 'Subclass:', issubclass(ScraperImplementation, genericScrapers.GenericRSSDisclogScraper) | print 'Subclass:', issubclass(ScraperImplementation, genericScrapers.GenericRSSDisclogScraper) |
print 'Instance:', isinstance(ScraperImplementation(), genericScrapers.GenericRSSDisclogScraper) | print 'Instance:', isinstance(ScraperImplementation(), genericScrapers.GenericRSSDisclogScraper) |
ScraperImplementation().doScrape() | ScraperImplementation().doScrape() |
import sys,os | import sys,os |
sys.path.insert(0, os.path.join(os.path.dirname(__file__) or '.', '../')) | sys.path.insert(0, os.path.join(os.path.dirname(__file__) or '.', '../')) |
import genericScrapers | import genericScrapers |
import scrape | import scrape |
from bs4 import BeautifulSoup | from bs4 import BeautifulSoup |
#http://www.doughellmann.com/PyMOTW/abc/ | #http://www.doughellmann.com/PyMOTW/abc/ |
class ScraperImplementation(genericScrapers.GenericOAICDisclogScraper): | class ScraperImplementation(genericScrapers.GenericOAICDisclogScraper): |
def getTable(self,soup): | |
return soup.find(id="main").table | |
def getColumnCount(self): | def getColumnCount(self): |
return 7 | return 7 |
def getColumns(self,columns): | def getColumns(self,columns): |
(id, date, title, description,link,deldate,notes) = columns | (id, date, title, description,link,deldate,notes) = columns |
return (id, date, title, description, notes) | return (id, date, title, description, notes) |
if __name__ == '__main__': | if __name__ == '__main__': |
print 'Subclass:', issubclass(ScraperImplementation, genericScrapers.GenericOAICDisclogScraper) | print 'Subclass:', issubclass(ScraperImplementation, genericScrapers.GenericOAICDisclogScraper) |
print 'Instance:', isinstance(ScraperImplementation(), genericScrapers.GenericOAICDisclogScraper) | print 'Instance:', isinstance(ScraperImplementation(), genericScrapers.GenericOAICDisclogScraper) |
ScraperImplementation().doScrape() | ScraperImplementation().doScrape() |
import sys,os | import sys,os |
sys.path.insert(0, os.path.join(os.path.dirname(__file__) or '.', '../')) | sys.path.insert(0, os.path.join(os.path.dirname(__file__) or '.', '../')) |
import genericScrapers | import genericScrapers |
import scrape | import scrape |
from bs4 import BeautifulSoup | from bs4 import BeautifulSoup |
#http://www.doughellmann.com/PyMOTW/abc/ | #http://www.doughellmann.com/PyMOTW/abc/ |
class ScraperImplementation(genericScrapers.GenericOAICDisclogScraper): | class ScraperImplementation(genericScrapers.GenericOAICDisclogScraper): |
def getTable(self,soup): | def getTable(self,soup): |
return soup.find(id = "ctl00_PlaceHolderMain_PublishingPageContent__ControlWrapper_RichHtmlField").table | return soup.find(id = "block-system-main").table |
def getColumnCount(self): | def getColumnCount(self): |
return 7 | return 2 |
def getColumns(self,columns): | def getColumns(self,columns): |
(id, date, title, description,link,deldate, notes) = columns | (date, title) = columns |
return (id, date, title, description, notes) | return (date, date, title, title, None) |
if __name__ == '__main__': | if __name__ == '__main__': |
print 'Subclass:', issubclass(ScraperImplementation, genericScrapers.GenericOAICDisclogScraper) | print 'Subclass:', issubclass(ScraperImplementation, genericScrapers.GenericOAICDisclogScraper) |
print 'Instance:', isinstance(ScraperImplementation(), genericScrapers.GenericOAICDisclogScraper) | print 'Instance:', isinstance(ScraperImplementation(), genericScrapers.GenericOAICDisclogScraper) |
ScraperImplementation().doScrape() | ScraperImplementation().doScrape() |
<?php | <?php |
include ('../include/common.inc.php'); | include ('../include/common.inc.php'); |
$last_updated = date('Y-m-d', @filemtime('cbrfeed.zip')); | |
header("Content-Type: text/xml"); | header("Content-Type: text/xml"); |
echo "<?xml version='1.0' encoding='UTF-8'?>"; | echo "<?xml version='1.0' encoding='UTF-8'?>"; |
echo '<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">' . "\n"; | echo '<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">' . "\n"; |
echo " <url><loc>" . local_url() . "index.php</loc><priority>1.0</priority></url>\n"; | echo " <url><loc>" . local_url() . "index.php</loc><priority>1.0</priority></url>\n"; |
foreach (scandir("./") as $file) { | foreach (scandir("./") as $file) { |
if (strpos($file, ".php") !== false && $file != "index.php" && $file != "sitemap.xml.php") | if (strpos($file, ".php") !== false && ($file != "index.php" && $file != "sitemap.xml.php"&& $file != "viewDocument.php")) { |
echo " <url><loc>" . local_url() . "$file</loc><priority>0.6</priority></url>\n"; | echo " <url><loc>" . local_url() . "$file</loc><priority>0.6</priority></url>\n"; |
} | |
} | } |
$agenciesdb = $server->get_db('disclosr-agencies'); | $agenciesdb = $server->get_db('disclosr-agencies'); |
$foidocsdb = $server->get_db('disclosr-foidocuments'); | |
try { | try { |
$rows = $agenciesdb->get_view("app", "byCanonicalName")->rows; | $rows = $agenciesdb->get_view("app", "byCanonicalName")->rows; |
foreach ($rows as $row) { | foreach ($rows as $row) { |
echo '<url><loc>' . local_url() . 'agency.php?id=' . $row->value->_id . "</loc><priority>0.3</priority></url>\n"; | echo '<url><loc>' . local_url() . 'agency.php?id=' . $row->value->_id . "</loc><priority>0.3</priority></url>\n"; |
} | } |
unset($rows); | |
$rows = null; | |
} catch (SetteeRestClientException $e) { | } catch (SetteeRestClientException $e) { |
setteErrorHandler($e); | setteErrorHandler($e); |
} | } |
$foidocsdb = $server->get_db('disclosr-foidocuments'); | |
foreach (range(0, 8) as $number) { | |
try { | try { |
$rows = $foidocsdb->get_view("app", "all")->rows; | $rows = $foidocsdb->get_view("app", "all", Array($number,$number+1))->rows; |
foreach ($rows as $row) { | foreach ($rows as $row) { |
echo '<url><loc>' . local_url() . 'view.php?id=' . $row->value->_id . "</loc><priority>0.3</priority></url>\n"; | echo '<url><loc>' . local_url() . 'view.php?id=' . $row->value->_id . "</loc><priority>0.3</priority></url>\n"; |
} | } |
unset($rows); | |
$rows = null; | |
} catch (SetteeRestClientException $e) { | |
setteErrorHandler($e); | |
} | |
} | |
try { | |
$rows = $foidocsdb->get_view("app", "all", Array('9','fffffffff'))->rows; | |
foreach ($rows as $row) { | |
echo '<url><loc>' . local_url() . 'view.php?id=' . $row->value->_id . "</loc><priority>0.3</priority></url>\n"; | |
} | |
unset($rows); | |
$rows = null; | |
} catch (SetteeRestClientException $e) { | } catch (SetteeRestClientException $e) { |
setteErrorHandler($e); | setteErrorHandler($e); |
} | } |
echo '</urlset>'; | echo '</urlset>'; |
?> | ?> |
<?php | <?php |
function include_header_documents($title) | function include_header_documents($title) |
{ | { |
header('X-UA-Compatible: IE=edge,chrome=1'); | header('X-UA-Compatible: IE=edge,chrome=1'); |
?> | ?> |
<!doctype html> | <!doctype html> |
<!-- paulirish.com/2008/conditional-stylesheets-vs-css-hacks-answer-neither/ --> | <!-- paulirish.com/2008/conditional-stylesheets-vs-css-hacks-answer-neither/ --> |
<!--[if lt IE 7]> | <!--[if lt IE 7]> |
<html class="no-js lt-ie9 lt-ie8 lt-ie7" lang="en"> <![endif]--> | <html class="no-js lt-ie9 lt-ie8 lt-ie7" lang="en"> <![endif]--> |
<!--[if IE 7]> | <!--[if IE 7]> |
<html class="no-js lt-ie9 lt-ie8" lang="en"> <![endif]--> | <html class="no-js lt-ie9 lt-ie8" lang="en"> <![endif]--> |
<!--[if IE 8]> | <!--[if IE 8]> |
<html class="no-js lt-ie9" lang="en"> <![endif]--> | <html class="no-js lt-ie9" lang="en"> <![endif]--> |
<!-- Consider adding a manifest.appcache: h5bp.com/d/Offline --> | <!-- Consider adding a manifest.appcache: h5bp.com/d/Offline --> |
<!--[if gt IE 8]><!--> <html class="no-js" lang="en"> <!--<![endif]--> | <!--[if gt IE 8]><!--> <html class="no-js" lang="en"> <!--<![endif]--> |
<head> | <head> |
<meta charset="utf-8"> | <meta charset="utf-8"> |
<title>Australian Disclosure Logs<?php if ($title != "") echo " - $title"; ?></title> | <title>Australian Disclosure Logs<?php if ($title != "") echo " - $title"; ?></title> |
<meta name="description" content=""> | <meta name="description" content=""> |
<!-- Mobile viewport optimized: h5bp.com/viewport --> | <!-- Mobile viewport optimized: h5bp.com/viewport --> |
<meta name="viewport" content="width=device-width"> | <meta name="viewport" content="width=device-width"> |
<link rel="alternate" type="application/rss+xml" title="Latest Disclosure Log Entries" href="rss.xml.php"/> | <link rel="alternate" type="application/rss+xml" title="Latest Disclosure Log Entries" href="rss.xml.php"/> |
<!-- Place favicon.ico and apple-touch-icon.png in the root directory: mathiasbynens.be/notes/touch-icons --> | <!-- Place favicon.ico and apple-touch-icon.png in the root directory: mathiasbynens.be/notes/touch-icons --> |
<meta name="google-site-verification" content="jkknX5g2FCpQvrW030b1Nq2hyoa6mb3EDiA7kCoHNj8"/> | <meta name="google-site-verification" content="jkknX5g2FCpQvrW030b1Nq2hyoa6mb3EDiA7kCoHNj8"/> |
<!-- Le styles --> | <!-- Le styles --> |
<link href="css/bootstrap.min.css" rel="stylesheet"> | <link href="css/bootstrap.min.css" rel="stylesheet"> |
<style type="text/css"> | <style type="text/css"> |
body { | body { |
padding-top: 60px; | padding-top: 60px; |
padding-bottom: 40px; | padding-bottom: 40px; |
} | } |
.sidebar-nav { | .sidebar-nav { |
padding: 9px 0; | padding: 9px 0; |
} | } |
</style> | </style> |
<link href="css/bootstrap-responsive.min.css" rel="stylesheet"> | <link href="css/bootstrap-responsive.min.css" rel="stylesheet"> |
<!-- HTML5 shim, for IE6-8 support of HTML5 elements --> | <!-- HTML5 shim, for IE6-8 support of HTML5 elements --> |
<!--[if lt IE 9]> | <!--[if lt IE 9]> |
<script src="http://html5shim.googlecode.com/svn/trunk/html5.js"></script> | <script src="http://html5shim.googlecode.com/svn/trunk/html5.js"></script> |
<![endif]--> | <![endif]--> |
<!-- More ideas for your <head> here: h5bp.com/d/head-Tips --> | <!-- More ideas for your <head> here: h5bp.com/d/head-Tips --> |
<!-- All JavaScript at the bottom, except this Modernizr build. | <!-- All JavaScript at the bottom, except this Modernizr build. |
Modernizr enables HTML5 elements & feature detects for optimal performance. | Modernizr enables HTML5 elements & feature detects for optimal performance. |
Create your own custom Modernizr build: www.modernizr.com/download/ | Create your own custom Modernizr build: www.modernizr.com/download/ |
<script src="js/libs/modernizr-2.5.3.min.js"></script>--> | <script src="js/libs/modernizr-2.5.3.min.js"></script>--> |
<script src="js/jquery.js"></script> | <script src="js/jquery.js"></script> |
<script type="text/javascript" src="js/flotr2.min.js"></script> | <script type="text/javascript" src="js/flotr2.min.js"></script> |
</head> | </head> |
<body> | <body> |
<div class="navbar navbar-inverse navbar-fixed-top"> | <div class="navbar navbar-inverse navbar-fixed-top"> |
<div class="navbar-inner"> | <div class="navbar-inner"> |
<div class="container-fluid"> | <div class="container-fluid"> |
<a class="btn btn-navbar" data-toggle="collapse" data-target=".nav-collapse"> | <!--<a class="btn btn-navbar" data-toggle="collapse" data-target=".nav-collapse"> |
<span class="icon-bar"></span> | <span class="icon-bar"></span> |
<span class="icon-bar"></span> | <span class="icon-bar"></span> |
<span class="icon-bar"></span> | <span class="icon-bar"></span> |
</a> | </a> --> |
<a class="brand" href="#">Australian Disclosure Logs</a> | <a class="brand" href="#">Australian Disclosure Logs</a> |
<div class="nav-collapse collapse"> | <div class="nav-collapse collapse"> |
<p class="navbar-text pull-right"> | <p class="navbar-text pull-right"> |
<small> | <small> |
Subsites on: | Subsites on: |
</small> | </small> |
<a href="http://orgs.disclosurelo.gs">Government Agencies</a> | <a href="http://orgs.disclosurelo.gs">Government Agencies</a> |
• <a href="http://lobbyists.disclosurelo.gs">Political Lobbyists</a> | • <a href="http://lobbyists.disclosurelo.gs">Political Lobbyists</a> |
• <a href="http://contracts.disclosurelo.gs">Government Contracts and Spending</a> | • <a href="http://contracts.disclosurelo.gs">Government Contracts and Spending</a> |
</p> | </p> |
<ul class="nav"> | <ul class="nav"> |
<li><a href="agency.php">By Agency</a></li> | <li><a href="agency.php">By Agency</a></li> |
<li><a href="date.php">By Date</a></li> | <li><a href="date.php">By Date</a></li> |
<li><a href="disclogsList.php">List of Disclosure Logs</a></li> | <li><a href="disclogsList.php">List of Disclosure Logs</a></li> |
<li><a href="charts.php">Charts</a></li> | |
<li><a href="about.php">About</a></li> | <li><a href="about.php">About</a></li> |
</ul> | </ul> |
</div> | </div> |
<!--/.nav-collapse --> | <!--/.nav-collapse --> |
</div> | </div> |
</div> | </div> |
</div> | </div> |
<div class="container"> | <div class="container"> |
<?php | <?php |
} | } |
function include_footer_documents() | function include_footer_documents() |
{ | { |
global $ENV; | global $ENV; |
?> | ?> |
</div> <!-- /container --> | </div> <!-- /container --> |
<hr> | <hr> |
<footer> | <footer> |
<p>Not affiliated with or endorsed by any government agency.</p> | <p>Not affiliated with or endorsed by any government agency.</p> |
</footer> | </footer> |
<?php | <?php |
if ($ENV != "DEV") { | if ($ENV != "DEV") { |
echo "<script type='text/javascript'> | echo "<script type='text/javascript'> |
var _gaq = _gaq || []; | var _gaq = _gaq || []; |
_gaq.push(['_setAccount', 'UA-12341040-4']); | _gaq.push(['_setAccount', 'UA-12341040-4']); |
_gaq.push(['_setDomainName', 'disclosurelo.gs']); | _gaq.push(['_setDomainName', 'disclosurelo.gs']); |
_gaq.push(['_setAllowLinker', true]); | _gaq.push(['_setAllowLinker', true]); |
_gaq.push(['_trackPageview']); | _gaq.push(['_trackPageview']); |
(function() { | (function() { |
var ga = document.createElement('script'); | var ga = document.createElement('script'); |
ga.type = 'text/javascript'; | ga.type = 'text/javascript'; |
ga.async = true; | ga.async = true; |
ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js'; | ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js'; |
var s = document.getElementsByTagName('script')[0]; | var s = document.getElementsByTagName('script')[0]; |
s.parentNode.insertBefore(ga, s); | s.parentNode.insertBefore(ga, s); |
})(); | })(); |
</script>"; | </script>"; |
} | } |
?> | ?> |
<!-- Le javascript | <!-- Le javascript |
================================================== --> | ================================================== --> |
<!-- Placed at the end of the document so the pages load faster --> | <!-- Placed at the end of the document so the pages load faster --> |
<!-- | <!-- |
<script src="js/bootstrap-transition.js"></script> | <script src="js/bootstrap-transition.js"></script> |
<script src="js/bootstrap-alert.js"></script> | <script src="js/bootstrap-alert.js"></script> |
<script src="js/bootstrap-modal.js"></script> | <script src="js/bootstrap-modal.js"></script> |
<script src="js/bootstrap-dropdown.js"></script> | <script src="js/bootstrap-dropdown.js"></script> |
<script src="js/bootstrap-scrollspy.js"></script> | <script src="js/bootstrap-scrollspy.js"></script> |
<script src="js/bootstrap-tab.js"></script> | <script src="js/bootstrap-tab.js"></script> |
<script src="js/bootstrap-tooltip.js"></script> | <script src="js/bootstrap-tooltip.js"></script> |
<script src="js/bootstrap-popover.js"></script> | <script src="js/bootstrap-popover.js"></script> |
<script src="js/bootstrap-button.js"></script> | <script src="js/bootstrap-button.js"></script> |
<script src="js/bootstrap-collapse.js"></script> | <script src="js/bootstrap-collapse.js"></script> |
<script src="js/bootstrap-carousel.js"></script> | <script src="js/bootstrap-carousel.js"></script> |
<script src="js/bootstrap-typeahead.js"></script>--> | <script src="js/bootstrap-typeahead.js"></script>--> |
</body> | </body> |
</html> | </html> |
<?php | <?php |
} | } |
function truncate($string, $length, $stopanywhere = false) | function truncate($string, $length, $stopanywhere = false) |
{ | { |
//truncates a string to a certain char length, stopping on a word if not specified otherwise. | //truncates a string to a certain char length, stopping on a word if not specified otherwise. |
if (strlen($string) > $length) { | if (strlen($string) > $length) { |
//limit hit! | //limit hit! |
$string = substr($string, 0, ($length - 3)); | $string = substr($string, 0, ($length - 3)); |
if ($stopanywhere) { | if ($stopanywhere) { |
//stop anywhere | //stop anywhere |
$string .= '...'; | $string .= '...'; |
} else { | } else { |
//stop on a word. | //stop on a word. |
$string = substr($string, 0, strrpos($string, ' ')) . '...'; | $string = substr($string, 0, strrpos($string, ' ')) . '...'; |
} | } |
} | } |
return $string; | return $string; |
} | } |
function displayLogEntry($row, $idtoname) | function displayLogEntry($row, $idtoname) |
{ | { |
$result = ""; | $result = ""; |
$result .= '<div itemscope itemtype="http://schema.org/Article">'; | $result .= '<div itemscope itemtype="http://schema.org/Article">'; |
$result .= '<h2><a href="http://disclosurelo.gs/view.php?id=' . $row->value->_id . '"> <span itemprop="datePublished">' . $row->value->date . "</span>: <span itemprop='name headline'>" . truncate($row->value->title, 120) . "</span>"; | $result .= '<h2><a href="http://disclosurelo.gs/view.php?id=' . $row->value->_id . '"> <span itemprop="datePublished">' . $row->value->date . "</span>: <span itemprop='name headline'>" . truncate($row->value->title, 120) . "</span>"; |
$result .= ' (<span itemprop="author publisher creator">' . $idtoname[$row->value->agencyID] . '</span>)</a></h2>'; | $result .= ' (<span itemprop="author publisher creator">' . $idtoname[$row->value->agencyID] . '</span>)</a></h2>'; |
$result .= "<p itemprop='description articleBody text'> Title: " . $row->value->title . "<br/>"; | $result .= "<p itemprop='description articleBody text'> Title: " . $row->value->title . "<br/>"; |
if (isset($row->value->description)) { | if (isset($row->value->description)) { |
$result .= str_replace("\n", "<br>", preg_replace("/(^[\r\n]*|[\r\n]+)[\s\t]*[\r\n]+/", "", trim($row->value->description))); | $result .= str_replace("\n", "<br>", preg_replace("/(^[\r\n]*|[\r\n]+)[\s\t]*[\r\n]+/", "", trim($row->value->description))); |
} | } |
if (isset($row->value->notes)) { | if (isset($row->value->notes)) { |
$result .= " <br>Note: " . $row->value->notes; | $result .= " <br>Note: " . $row->value->notes; |
} | } |
$result .= "</p>"; | $result .= "</p>"; |
if (isset($row->value->links)) { | if (isset($row->value->links)) { |
$result .= '<h3>Links/Documents</h3><ul itemprop="associatedMedia">'; | $result .= '<h3>Links/Documents</h3><ul itemprop="associatedMedia">'; |
foreach ($row->value->links as $link) { | foreach ($row->value->links as $link) { |
$result .= '<li itemscope itemtype="http://schema.org/MediaObject"><a href="' . htmlspecialchars($link) . '" itemprop="url contentURL">' . htmlspecialchars($link) . "</a></li>"; | $result .= '<li itemscope itemtype="http://schema.org/MediaObject"><a href="' . htmlspecialchars($link) . '" itemprop="url contentURL">' . htmlspecialchars($link) . "</a></li>"; |
} | } |
$result .= "</ul>"; | $result .= "</ul>"; |
} | } |
$result .= "<small><A itemprop='url' href='" . $row->value->url . "'>View original source...</a> ID: " . strip_tags($row->value->docID) . "</small>"; | $result .= "<small><A itemprop='url' href='" . $row->value->url . "'>View original source...</a> ID: " . strip_tags($row->value->docID) . "</small>"; |
$result .= "</div>\n"; | $result .= "</div>\n"; |
return $result; | return $result; |
} | } |
<?php | |
include_once('include/common.inc.php'); | |
include_header('Charts'); | |
$db = $server->get_db('disclosr-agencies'); | |
?> | |
<div class="foundation-header"> | |
<h1><a href="about.php">Charts</a></h1> | |
<h4 class="subheader">Lorem ipsum.</h4> | |
</div> | |
<div id="scores" style="width:900px;height:500px;"></div> | |
<script id="source"> | |
window.onload = function() { | |
$(document).ready(function() { | |
var d1 = []; | |
var scorelabels = []; | |
<?php | |
try { | |
$rows = $db->get_view("app", "scoreHas?group=true", null, true)->rows; | |
$dataValues = Array(); | |
foreach ($rows as $row) { | |
$dataValues[$row->value] = $row->key; | |
} | |
$i = 0; | |
ksort($dataValues); | |
foreach ($dataValues as $value => $key) { | |
echo " d1.push([$i, $value]);" . PHP_EOL; | |
echo " scorelabels.push('$key');" . PHP_EOL; | |
$i++; | |
} | |
} catch (SetteeRestClientException $e) { | |
setteErrorHandler($e); | |
} | |
?> | |
function scoretrackformatter(obj) { | |
if (scorelabels[Math.floor(obj.x)]) { | |
return (scorelabels[Math.floor(obj.x)])+"="+obj.y; | |
} else { | |
return ""; | |
} | |
} | |
function scoretickformatter(val, axis) { | |
if (scorelabels[Math.floor(val)]) { | |
return '<p style="margin-top:8em;-webkit-transform:rotate(-90deg);">'+(scorelabels[Math.floor(val)])+"</b>"; | |
} else { | |
return ""; | |
} | |
} | |
Flotr.draw(document.getElementById("scores"), [ {data: d1}], { | |
HtmlText: true, | |
bars : { | |
show : true | |
}, | |
mouse : { | |
track : true, | |
relative : true, | |
trackFormatter: scoretrackformatter | |
},yaxis: { | |
autoscaling: true | |
}, | |
xaxis: { | |
autoscaling: true, | |
minorTickFreq: 0.6, | |
noTicks : scorelabels.length, | |
tickFormatter: scoretickformatter | |
} | |
}); | |
<div id="employees" style="width:1000px;height:900px;"></div> | |
var emplabels = []; | |
function emptrackformatter(obj) { | |
return (obj.series.label)+" = "+obj.y+" in "+emplabels[Math.floor(obj.x)]; | |
} | |
function emptickformatter(val, axis) { | |
if (emplabels[Math.floor(val)]) { | |
return '<p style="margin-top:8em;-webkit-transform:rotate(-90deg);">'+(emplabels[Math.floor(val)])+"</b>"; | |
} else { | |
return ""; | |
} | |
} | |
function onDataReceived(series) { | |
emplabels = series.labels; | |
Flotr.draw(document.getElementById("employees"), series.data, { | |
mouse : { | |
track : true, | |
relative : true, | |
trackFormatter: emptrackformatter | |
},yaxis: { | |
max: 10000, | |
scaling: 'logarithmic' | |
}, | |
xaxis: { | |
minorTickFreq: 1, | |
noTicks: emplabels.length, | |
showMinorLabels: true, | |
tickFormatter: emptickformatter | |
}, | |
legend: { | |
show: false | |
} | |
}); | |
} | |
$.ajax({ | |
url: "admin/exportEmployees.csv.php?format=json", | |
method: 'GET', | |
dataType: 'json', | |
success: onDataReceived | |
}); | |
}); | |
}; | |
</script> | |
<?php | |
include_footer(); | |
?> | |
<?php | |
// use https://github.com/okfn/publicbodies/blob/master/data/nz.csv format | |
include_once("include/common.inc.php"); | |
setlocale(LC_CTYPE, 'C'); | |
$headers = Array("title","abbr","key","category","parent","parent_key","description","url","jurisdiction","jurisdiction_code","source","source_url","address","contact","email","tags","created_at","updated_at"); | |
$db = $server->get_db('disclosr-agencies'); | |
$foiEmail = Array(); | |
try { | |
$rows = $db->get_view("app", "foiEmails", null, true)->rows; | |
//print_r($rows); | |
foreach ($rows as $row) { | |
$foiEmail[$row->key] = $row->value; | |
} | |
} catch (SetteeRestClientException $e) { | |
setteErrorHandler($e); | |
die(); | |
} | |
$fp = fopen('php://output', 'w'); | |
if ($fp && $db) { | |
header('Content-Type: text/csv; charset=utf-8'); | |
header('Content-Disposition: attachment; filename="export.' . date("c") . '.csv"'); | |
header('Pragma: no-cache'); | |
header('Expires: 0'); | |
fputcsv($fp, $headers); | |
try { | |
$agencies = $db->get_view("app", "byCanonicalName", null, true)->rows; | |
//print_r($rows); | |
foreach ($agencies as $agency) { | |
// print_r($agency); | |
if (isset($agency->value->foiEmail) && $agency->value->foiEmail != "null" && !isset($agency->value->status)) { | |
$row = Array(); | |
$row["title"] = trim($agency->value->name); | |
$row["abbr"] = (isset($agency->value->shortName) ? $agency->value->shortName : ""); | |
$row["key"] = (isset($agency->value->shortName) ? "au/".strtolower($agency->value->shortName) : ""); | |
$row["category"] =""; | |
$row["parent"] =""; | |
$row["parentkey"] =""; | |
$row["description"] = (isset($agency->value->description) ? $agency->value->description : ""); | |
$row["url"] = (isset($agency->value->website) ? $agency->value->website : ""); | |
$row["jurisdiction"] = "Australia"; | |
$row["jurisdiction_code"] = "au"; | |
$row["source"] =""; | |
$row["source_url"] =""; | |
$row["address"] =""; | |
$row["contact"] =""; | |
$row["email"] = (isset($agency->value->foiEmail) ? $agency->value->foiEmail : ""); | |
$row["tags"] =""; | |
$row["created_at"] =""; | |
$row["updated_at"] =""; | |
$otherBodies = Array(); | |
if (isset($agency->value->foiBodies)) { | |
$otherBodies = array_merge($otherBodies, $agency->value->foiBodies); | |
} | |
if (isset($agency->value->positions)) { | |
$positions = Array(); | |
foreach ($agency->value->positions as $position) { | |
$positions[] = "Office of the ".$position; | |
} | |
$otherBodies = array_merge($otherBodies, $positions); | |
} | |
sort($otherBodies); | |
if (count($otherBodies) > 0) { | |
$row["description"] .= "<br/> This department also responds to requests for information held by " . implode(", ", $otherBodies); | |
} | |
fputcsv($fp, array_values($row)); | |
} | |
} | |
} catch (SetteeRestClientException $e) { | |
setteErrorHandler($e); | |
} | |
die; | |
} | |
?> | |
<?php | <?php |
include_once('include/common.inc.php'); | include_once('include/common.inc.php'); |
function displayValue($key, $value, $mode) { | function displayValue($key, $value, $mode) |
{ | |
global $db, $schemas; | global $db, $schemas; |
$ignoreKeys = Array("metadata" ,"metaTags", "statistics","rtkURLs","rtkDescriptions"); | $ignoreKeys = Array("metadata", "metaTags", "statistics", "rtkURLs", "rtkDescriptions"); |
if ($mode == "view") { | if ($mode == "view") { |
if (strpos($key, "_") === 0 || in_array($key,$ignoreKeys)) | if (strpos($key, "_") === 0 || in_array($key, $ignoreKeys)) |
return; | return; |
echo "<tr>"; | echo "<tr>"; |
echo "<td class='$key'>"; | echo "<td class='$key'>"; |
if (isset($schemas['agency']["properties"][$key])) { | if (isset($schemas['agency']["properties"][$key])) { |
echo $schemas['agency']["properties"][$key]['x-title'] . "<br><small>" . $schemas['agency']["properties"][$key]['description'] . "</small>"; | echo $schemas['agency']["properties"][$key]['x-title'] . "<br><small>" . $schemas['agency']["properties"][$key]['description'] . "</small>"; |
} | } |
echo "</td><td>"; | echo "</td><td>"; |
if (is_array($value)) { | if (is_array($value)) { |
echo "<ol>"; | echo "<ol>"; |
foreach ($value as $subkey => $subvalue) { | foreach ($value as $subkey => $subvalue) { |
echo "<li "; | echo "<li "; |
if (isset($schemas['agency']["properties"][$key]['x-property'])) { | if (isset($schemas['agency']["properties"][$key]['x-property'])) { |
echo ' property="' . $schemas['agency']["properties"][$key]['x-property'] . '" '; | echo ' property="' . $schemas['agency']["properties"][$key]['x-property'] . '" '; |
} if (isset($schemas['agency']["properties"][$key]['x-itemprop'])) { | } |
if (isset($schemas['agency']["properties"][$key]['x-itemprop'])) { | |
echo ' itemprop="' . $schemas['agency']["properties"][$key]['x-itemprop'] . '" '; | echo ' itemprop="' . $schemas['agency']["properties"][$key]['x-itemprop'] . '" '; |
} | } |
echo " >"; | echo " >"; |
echo "$subvalue</li>"; | echo "$subvalue</li>"; |
} | } |
echo "</ol></td></tr>"; | echo "</ol></td></tr>"; |
} else { | } else { |
if (isset($schemas['agency']["properties"][$key]['x-property'])) { | if (isset($schemas['agency']["properties"][$key]['x-property'])) { |
echo '<span property="' . $schemas['agency']["properties"][$key]['x-property'] . '">'; | echo '<span property="' . $schemas['agency']["properties"][$key]['x-property'] . '">'; |
} else { | } else { |
echo "<span>"; | echo "<span>"; |
} | } |
if ((strpos($key, "URL") > 0 || $key == 'website') && $value != "") { | if ((strpos($key, "URL") > 0 || $key == 'website') && $value != "") { |
echo "<a " . ($key == 'website' ? 'itemprop="url"' : '') . " href='$value'>$value</a>"; | echo "<a " . ($key == 'website' ? 'itemprop="url"' : '') . " href='$value'>$value</a>"; |
} else if ($key == 'abn') { | } else if ($key == 'abn') { |
echo "<a href='http://www.abr.business.gov.au/SearchByAbn.aspx?SearchText=$value'>$value</a>"; | echo "<a href='http://www.abr.business.gov.au/SearchByAbn.aspx?SearchText=$value'>$value</a>"; |
} else { | } else { |
echo "$value"; | echo "$value"; |
} | } |
echo "</span>"; | echo "</span>"; |
} | } |
echo "</td></tr>"; | echo "</td></tr>"; |
} | } |
if ($mode == "edit") { | if ($mode == "edit") { |
if (is_array($value)) { | if (is_array($value)) { |
echo '<div class="row"> | echo '<div class="row"> |
<div class="seven columns"> | <div class="seven columns"> |
<fieldset> | <fieldset> |
<h5>' . $key . '</h5>'; | <h5>' . $key . '</h5>'; |
foreach ($value as $subkey => $subvalue) { | foreach ($value as $subkey => $subvalue) { |
echo "<label>$subkey</label><input class='input-text' type='text' id='$key$subkey' name='$key" . '[' . $subkey . "]' value='$subvalue'/></tr>"; | echo "<label>$subkey</label><input class='input-text' type='text' id='$key$subkey' name='$key" . '[' . $subkey . "]' value='$subvalue'/></tr>"; |
} | } |
echo "</fieldset> | echo "</fieldset> |
</div> | </div> |
</div>"; | </div>"; |
} else { | } else { |
if (strpos($key, "_") === 0) { | if (strpos($key, "_") === 0) { |
echo"<input type='hidden' id='$key' name='$key' value='$value'/>"; | echo "<input type='hidden' id='$key' name='$key' value='$value'/>"; |
} else if ($key == "parentOrg") { | } else if ($key == "parentOrg") { |
echo "<label for='$key'>$key</label><select id='$key' name='$key'><option value=''> Select... </option>"; | echo "<label for='$key'>$key</label><select id='$key' name='$key'><option value=''> Select... </option>"; |
$rows = $db->get_view("app", "byDeptStateName")->rows; | $rows = $db->get_view("app", "byDeptStateName")->rows; |
//print_r($rows); | //print_r($rows); |
foreach ($rows as $row) { | foreach ($rows as $row) { |
echo "<option value='{$row->value}'" . (($row->value == $value) ? "SELECTED" : "") . " >" . str_replace("Department of ", "", $row->key) . "</option>"; | echo "<option value='{$row->value}'" . (($row->value == $value) ? "SELECTED" : "") . " >" . str_replace("Department of ", "", $row->key) . "</option>"; |
} | } |
echo" </select>"; | echo " </select>"; |
} else { | } else { |
echo "<label>$key</label><input class='input-text' type='text' id='$key' name='$key' value='$value'/>"; | echo "<label>$key</label><input class='input-text' type='text' id='$key' name='$key' value='$value'/>"; |
if ((strpos($key, "URL") > 0 || $key == 'website') && $value != "") { | if ((strpos($key, "URL") > 0 || $key == 'website') && $value != "") { |
echo "<a " . ($key == 'website' ? 'itemprop="url"' : '') . " href='$value'>view</a>"; | echo "<a " . ($key == 'website' ? 'itemprop="url"' : '') . " href='$value'>view</a>"; |
} | } |
if ($key == 'abn') { | if ($key == 'abn') { |
echo "<a href='http://www.abr.business.gov.au/SearchByAbn.aspx?SearchText=$value'>view abn</a>"; | echo "<a href='http://www.abr.business.gov.au/SearchByAbn.aspx?SearchText=$value'>view abn</a>"; |
} | } |
} | } |
} | } |
} | } |
// | // |
} | } |
function addDefaultFields($row) { | function addDefaultFields($row) |
{ | |
global $schemas; | global $schemas; |
$defaultFields = array_keys($schemas['agency']['properties']); | $defaultFields = array_keys($schemas['agency']['properties']); |
foreach ($defaultFields as $defaultField) { | foreach ($defaultFields as $defaultField) { |
if (!isset($row[$defaultField])) { | if (!isset($row[$defaultField])) { |
if ($schemas['agency']['properties'][$defaultField]['type'] == "string") { | if ($schemas['agency']['properties'][$defaultField]['type'] == "string") { |
$row[$defaultField] = ""; | $row[$defaultField] = ""; |
} | } |
if ($schemas['agency']['properties'][$defaultField]['type'] == "array") { | if ($schemas['agency']['properties'][$defaultField]['type'] == "array") { |
$row[$defaultField] = Array(""); | $row[$defaultField] = Array(""); |
} | } |
} else if ($schemas['agency']['properties'][$defaultField]['type'] == "array") { | } else if ($schemas['agency']['properties'][$defaultField]['type'] == "array") { |
if (is_array($row[$defaultField])) { | if (is_array($row[$defaultField])) { |
$row[$defaultField][] = ""; | $row[$defaultField][] = ""; |
$row[$defaultField][] = ""; | $row[$defaultField][] = ""; |
$row[$defaultField][] = ""; | $row[$defaultField][] = ""; |
} else { | } else { |
$value = $row[$defaultField]; | $value = $row[$defaultField]; |
$row[$defaultField] = Array($value); | $row[$defaultField] = Array($value); |
$row[$defaultField][] = ""; | $row[$defaultField][] = ""; |
$row[$defaultField][] = ""; | $row[$defaultField][] = ""; |
} | } |
} | } |
} | } |
return $row; | return $row; |
} | } |
$db = $server->get_db('disclosr-agencies'); | $db = $server->get_db('disclosr-agencies'); |
if (isset($_REQUEST['id'])) { | if (isset($_REQUEST['id'])) { |
//get an agency record as json/html, search by name/abn/id | //get an agency record as json/html, search by name/abn/id |
// by name = startkey="Ham"&endkey="Ham\ufff0" | // by name = startkey="Ham"&endkey="Ham\ufff0" |
// edit? | // edit? |
$obj = $db->get($_REQUEST['id']); | $obj = $db->get($_REQUEST['id']); |
include_header(isset($obj->name) ? $obj->name : ""); | include_header(isset($obj->name) ? $obj->name : ""); |
//print_r($row); | //print_r($row); |
if (sizeof($_POST) > 0) { | if (sizeof($_POST) > 0) { |
//print_r($_POST); | //print_r($_POST); |
foreach ($_POST as $postkey => $postvalue) { | foreach ($_POST as $postkey => $postvalue) { |
if ($postvalue == "") { | if ($postvalue == "") { |
unset($_POST[$postkey]); | |
} | |
if (is_array($postvalue)) { | |
if (count($postvalue) == 1 && $postvalue[0] == "") { | |
unset($_POST[$postkey]); | unset($_POST[$postkey]); |
} | } else { |
if (is_array($postvalue)) { | foreach ($_POST[$postkey] as $key => &$value) { |
if (count($postvalue) == 1 && $postvalue[0] == "") { | if ($value == "") { |
unset($_POST[$postkey]); | unset($_POST[$postkey][$key]); |
} else { | |
foreach ($_POST[$postkey] as $key => &$value) { | |
if ($value == "") { | |
unset($_POST[$postkey][$key]); | |
} | |
} | } |
} | } |
} | } |
} | } |
if (isset($_POST['_id']) && $db->get_rev($_POST['_id']) == $_POST['_rev']) { | } |
echo "Edited version was latest version, continue saving"; | if (isset($_POST['_id']) && $db->get_rev($_POST['_id']) == $_POST['_rev']) { |
$newdoc = $_POST; | echo "Edited version was latest version, continue saving"; |
$newdoc['metadata']['lastModified'] = time(); | $newdoc = $_POST; |
$obj = $db->save($newdoc); | $newdoc['metadata']['lastModified'] = time(); |
} else { | $obj = $db->save($newdoc); |
echo "ALERT doc revised by someone else while editing. Document not saved."; | |
} | |
} | |
$mode = "view"; | |
$rowArray = object_to_array($obj); | |
ksort($rowArray); | |
if ($mode == "edit") { | |
$row = addDefaultFields($rowArray); | |
} else { | } else { |
$row = $rowArray; | echo "ALERT doc revised by someone else while editing. Document not saved."; |
} | } |
} | |
if ($mode == "view") { | |
echo ' <div class="container-fluid"> | $mode = "view"; |
$rowArray = object_to_array($obj); | |
ksort($rowArray); | |
if ($mode == "edit") { | |
$row = addDefaultFields($rowArray); | |
} else { | |
$row = $rowArray; | |
} | |
if ($mode == "view") { | |
echo ' <div class="container-fluid"> | |
<div class="row-fluid"> | <div class="row-fluid"> |
<div class="span3"> | <div class="span3"> |
<div class="well sidebar-nav"> | <div class="well sidebar-nav"> |
<ul class="nav nav-list"> | <ul class="nav nav-list"> |
<li class="nav-header">Statistics</li>'; | <li class="nav-header">Statistics</li>'; |
if (isset($row['statistics']['employees'])) { | if (isset($row['statistics']['employees'])) { |
echo '<div><i class="icon-user" style="float:left"></i><p style="margin-left:16px;">'; | echo '<div><i class="icon-user" style="float:left"></i><p style="margin-left:16px;">'; |
$keys = array_keys($row['statistics']['employees']); | $keys = array_keys($row['statistics']['employees']); |
$lastkey = $keys[count($keys)-1]; | $lastkey = $keys[count($keys) - 1]; |
echo $row['statistics']['employees'][$lastkey]['value'].' employees <small>('.$lastkey.')</small>'; | echo $row['statistics']['employees'][$lastkey]['value'] . ' employees <small>(' . $lastkey . ')</small>'; |
echo '</div>'; | echo '</div>'; |
} | } |
if (isset($row['statistics']['budget'])) { | if (isset($row['statistics']['budget'])) { |
echo '<div><i class="icon-shopping-cart" style="float:left"></i><p style="margin-left:16px;">'; | echo '<div><i class="icon-shopping-cart" style="float:left"></i><p style="margin-left:16px;">'; |
$keys = array_keys($row['statistics']['budget']); | $keys = array_keys($row['statistics']['budget']); |
$lastkey = $keys[count($keys)-1]; | $lastkey = $keys[count($keys) - 1]; |
echo "$".number_format(floatval($row['statistics']['budget'][$lastkey]['value'])).' <small>('.$lastkey.' budget)</small>'; | echo "$" . number_format(floatval($row['statistics']['budget'][$lastkey]['value'])) . ' <small>(' . $lastkey . ' budget)</small>'; |
echo '</div>'; | echo '</div>'; |
} | } |
echo ' </ul> | echo ' </ul> |
</div><!--/.well --> | </div><!--/.well --> |
</div><!--/span--> | </div><!--/span--> |
<div class="span9">'; | <div class="span9">'; |
echo '<div itemscope itemtype="http://schema.org/GovernmentOrganization" typeof="schema:GovernmentOrganization" about="#' . $row['_id'] . '">'; | echo '<div itemscope itemtype="http://schema.org/GovernmentOrganization" typeof="schema:GovernmentOrganization org:Organization" about="#' . $row['_id'] . '">'; |
echo '<div class="hero-unit"> | echo '<div class="hero-unit"> |
<h1 itemprop="name">' . $row['name'] . '</h1>'; | <h1 itemprop="name">' . $row['name'] . '</h1>'; |
if (isset($row['description'])) { | if (isset($row['description'])) { |
echo '<p>'.$row['description'].'</p>'; | echo '<p>' . $row['description'] . '</p>'; |
} | } |
echo '</div><table width="100%">'; | echo '</div><table width="100%">'; |
echo "<tr><th>Field Name</th><th>Field Value</th></tr>"; | echo "<tr><th>Field Name</th><th>Field Value</th></tr>"; |
} | } |
if ($mode == "edit") { | if ($mode == "edit") { |
?> | ?> |
<input id="addfield" type="button" value="Add Field"/> | <input id="addfield" type="button" value="Add Field"/> |
<script> | <script> |
window.onload = function() { | window.onload = function () { |
$(document).ready(function() { | $(document).ready(function () { |
// put all your jQuery goodness in here. | // put all your jQuery goodness in here. |
// http://charlie.griefer.com/blog/2009/09/17/jquery-dynamically-adding-form-elements/ | // http://charlie.griefer.com/blog/2009/09/17/jquery-dynamically-adding-form-elements/ |
$('#addfield').click(function() { | $('#addfield').click(function () { |
var field_name=window.prompt("fieldname?",""); | var field_name = window.prompt("fieldname?", ""); |
if (field_name !="") { | if (field_name != "") { |
$('#submitbutton').before($('<span></span>') | $('#submitbutton').before($('<span></span>') |
.append("<label>"+field_name+"</label>") | .append("<label>" + field_name + "</label>") |
.append("<input class='input-text' type='text' id='"+field_name+"' name='"+field_name+"'/>") | .append("<input class='input-text' type='text' id='" + field_name + "' name='" + field_name + "'/>") |
); | ); |
} | } |
}); | }); |
}); | }); |
}; | }; |
</script> | </script> |
<form id="editform" class="nice" method="post"> | <form id="editform" class="nice" method="post"> |
<?php | <?php |
} | } |
foreach ($row as $key => $value) { | foreach ($row as $key => $value) { |
echo displayValue($key, $value, $mode); | echo displayValue($key, $value, $mode); |
} | } |
if ($mode == "view") { | if ($mode == "view") { |
echo "</table></div>"; | echo "</table></div>"; |
echo ' </div><!--/span--> | echo ' </div><!--/span--> |
</div><!--/row--> | </div><!--/row--> |
</div><!--/span--> | </div><!--/span--> |
</div><!--/row-->'; | </div><!--/row-->'; |
} | } |
if ($mode == "edit") { | if ($mode == "edit") { |
echo '<input id="submitbutton" type="submit"/></form>'; | echo '<input id="submitbutton" type="submit"/></form>'; |
} | } |
} else { | } else { |
// show all list | // show all list |
include_header('Agencies'); | include_header('Agencies'); |
echo ' <div class="container-fluid"> | echo ' <div class="container-fluid"> |
<div class="row-fluid"> | <div class="row-fluid"> |
<div class="span3"> | <div class="span3"> |
<div class="well sidebar-nav"> | <div class="well sidebar-nav"> |
<ul class="nav nav-list"> | <ul class="nav nav-list"> |
<li class="nav-header">Sidebar</li>'; | <li class="nav-header">Sidebar</li>'; |
echo ' </ul> | echo ' </ul> |
</div><!--/.well --> | </div><!--/.well --> |
</div><!--/span--> | </div><!--/span--> |
<div class="span9"> | <div class="span9"> |
<div class="hero-unit"> | <div class="hero-unit"> |
<h1>Australian Government Agencies</h1> | <h1>Australian Government Agencies</h1> |
<p>Explore collected information about Australian Government Agencies below.</p> | <p>Explore collected information about Australian Government Agencies below.</p> |
</div> | </div> |
<div class="row-fluid"> | <div class="row-fluid"> |
<div class="span4">'; | <div class="span4">'; |
try { | try { |
$rows = $db->get_view("app", "byCanonicalName")->rows; | $rows = $db->get_view("app", "byCanonicalName")->rows; |
//print_r($rows); | //print_r($rows); |
$rowCount = count($rows); | $rowCount = count($rows); |
foreach ($rows as $i => $row) { | foreach ($rows as $i => $row) { |
if ($i % ($rowCount/3) == 0 && $i != 0 && $i != $rowCount -2 ) echo '</div><div class="span4">'; | if ($i % ($rowCount / 3) == 0 && $i != 0 && $i != $rowCount - 2) echo '</div><div class="span4">'; |
// print_r($row); | // print_r($row); |
echo '<span itemscope itemtype="http://schema.org/GovernmentOrganization" typeof="schema:GovernmentOrganization foaf:Organization" about="getAgency.php?id=' . $row->value->_id . '"> | echo '<span itemscope itemtype="http://schema.org/GovernmentOrganization" typeof="schema:GovernmentOrganization foaf:Organization" about="getAgency.php?id=' . $row->value->_id . '"> |
<a href="getAgency.php?id=' . $row->value->_id . '" rel="schema:url foaf:page" property="schema:name foaf:name" itemprop="url"><span itemprop="name">' . | <a href="getAgency.php?id=' . $row->value->_id . '" rel="schema:url foaf:page" property="schema:name foaf:name" itemprop="url"><span itemprop="name">' . |
(isset($row->value->name) ? $row->value->name : "ERROR NAME MISSING") | (isset($row->value->name) ? $row->value->name : "ERROR NAME MISSING") |
. '</span></a></span><br><br>'; | . '</span></a></span><br><br>'; |
} | } |
} catch (SetteeRestClientException $e) { | } catch (SetteeRestClientException $e) { |
setteErrorHandler($e); | setteErrorHandler($e); |
} | } |
echo ' </div><!--/span--> | echo ' </div><!--/span--> |
</div><!--/row--> | </div><!--/row--> |
</div><!--/span--> | </div><!--/span--> |
</div><!--/row-->'; | </div><!--/row-->'; |
} | } |
include_footer(); | include_footer(); |
?> | ?> |
<?php | <?php |
include_once('include/common.inc.php'); | include_once('include/common.inc.php'); |
//include_header(); | //include_header(); |
$format = "html"; | $format = "html"; |
if (isset($_REQUEST['format'])) { | if (isset($_REQUEST['format'])) { |
$format = $_REQUEST['format']; | $format = $_REQUEST['format']; |
} | } |
function add_node($id, $label, $parent="") { | function add_node($id, $label, $parent="") { |
global $format; | global $format; |
if ($format == "html") { | if ($format == "html") { |
// echo "nodes[\"$id\"] = graph.newNode({label: \"$label\"});" . PHP_EOL; | // echo "nodes[\"$id\"] = graph.newNode({label: \"$label\"});" . PHP_EOL; |
} | } |
if ($format == "dot" && $label != "") { | if ($format == "dot" && $label != "") { |
echo "$id [label=\"$label\"];". PHP_EOL; | echo "\"$id\" [label=\"$label\", shape=plaintext];". PHP_EOL; |
} | } |
if ($format == "gexf") { | if ($format == "gexf") { |
echo "<node id='$id' label=\"".htmlentities($label,ENT_XML1)."\" ".($parent != ""? "pid='$parent'><viz:size value='1'/>":"><viz:size value='2'/>") | echo "<node id='$id' label=\"".htmlentities($label)."\" ".($parent != ""? "pid='$parent'><viz:size value='1'/>":"><viz:size value='2'/>") |
."<viz:color b='".rand(0,255)."' g='".rand(0,255)."' r='".rand(0,255)."'/>" | ."<viz:color b='".rand(0,255)."' g='".rand(0,255)."' r='".rand(0,255)."'/>" |
."</node>". PHP_EOL; | ."</node>". PHP_EOL; |
} | } |
} | } |
function add_edge($from, $to, $color) { | function add_edge($from, $to, $color) { |
global $format; | global $format; |
if ($format == "html") { | if ($format == "html") { |
// echo "graph.newEdge(nodes[\"$from\"], nodes['$to'], {color: '$color'});" . PHP_EOL; | // echo "graph.newEdge(nodes[\"$from\"], nodes['$to'], {color: '$color'});" . PHP_EOL; |
} | } |
if ($format == "dot") { | if ($format == "dot") { |
echo "$from -> $to ".($color != ""? "[color=$color]":"").";". PHP_EOL; | echo "\"$from\" -> \"$to\" ".($color != ""? "[color=$color]":"").";". PHP_EOL; |
} | } |
if ($format == "gexf") { | if ($format == "gexf") { |
echo "<edge id='$from$to' source='$from' target='$to' />". PHP_EOL; | echo "<edge id='$from$to' source='$from' target='$to' />". PHP_EOL; |
} | } |
} | } |
if ($format == "gexf") { | if ($format == "gexf") { |
//header('Content-Type: text/xml'); | //header('Content-Type: text/xml'); |
header('Content-Type: application/gexf+xml'); | header('Content-Type: application/gexf+xml'); |
echo '<?xml version="1.0" encoding="UTF-8"?> | echo '<?xml version="1.0" encoding="UTF-8"?> |
<gexf xmlns="http://www.gexf.net/1.2draft" xmlns:viz="http://www.gexf.net/1.2draft/viz" version="1.2"> | <gexf xmlns="http://www.gexf.net/1.2draft" xmlns:viz="http://www.gexf.net/1.2draft/viz" version="1.2"> |
<meta lastmodifieddate="2009-03-20"> | <meta lastmodifieddate="2009-03-20"> |
<creator>Gexf.net</creator> | <creator>Gexf.net</creator> |
<description>A hello world! file</description> | <description>A hello world! file</description> |
</meta> | </meta> |
<graph mode="static" defaultedgetype="directed"> | <graph mode="static" defaultedgetype="directed"> |
<nodes>'. PHP_EOL; | <nodes>'. PHP_EOL; |
} | } |
if ($format == "dot") { | if ($format == "dot") { |
echo 'digraph g {'. PHP_EOL; | echo 'digraph g {'. PHP_EOL; |
} | } |
$db = $server->get_db('disclosr-agencies'); | $db = $server->get_db('disclosr-agencies'); |
add_node("fedg","Federal Government - Commonwealth of Australia"); | add_node("fedg","Federal Government - Commonwealth of Australia"); |
try { | try { |
$rows = $db->get_view("app", "byCanonicalName", null, true)->rows; | $rows = $db->get_view("app", "byCanonicalName", null, true)->rows; |
//print_r($rows); | //print_r($rows); |
foreach ($rows as $row) { | foreach ($rows as $row) { |
add_node($row->id, $row->key); | add_node($row->id, $row->value->name); |
} | } |
} catch (SetteeRestClientException $e) { | } catch (SetteeRestClientException $e) { |
setteErrorHandler($e); | setteErrorHandler($e); |
} | } |
if ($format == "gexf") { | if ($format == "gexf") { |
echo '</nodes> | echo '</nodes> |
<edges>'. PHP_EOL; | <edges>'. PHP_EOL; |
} | } |
try { | try { |
$rows = $db->get_view("app", "byDeptStateName", null, true)->rows; | $rows = $db->get_view("app", "byDeptStateName", null, true)->rows; |
//print_r($rows); | //print_r($rows); |
foreach ($rows as $row) { | foreach ($rows as $row) { |
add_edge("fedg", $row->value, 'yellow'); | add_edge("fedg", $row->value, 'yellow'); |
} | } |
} catch (SetteeRestClientException $e) { | } catch (SetteeRestClientException $e) { |
setteErrorHandler($e); | setteErrorHandler($e); |
} | } |
try { | try { |
$rows = $db->get_view("app", "parentOrgs", null, true)->rows; | $rows = $db->get_view("app", "parentOrgs", null, true)->rows; |
// print_r($rows); | // print_r($rows); |
foreach ($rows as $row) { | foreach ($rows as $row) { |
add_edge($row->key, $row->value, 'blue'); | add_edge($row->key, $row->value, 'blue'); |
} | } |
} catch (SetteeRestClientException $e) { | } catch (SetteeRestClientException $e) { |
setteErrorHandler($e); | setteErrorHandler($e); |
} | } |
if ($format == "html") { | if ($format == "html") { |
?> | ?> |
<div id="sigma-example" width="960" style="min-height:800px;background-color: #333;"></div> | <div id="sigma-example" width="960" style="min-height:800px;background-color: #333;"></div> |
<script src="js/sigma.min.js"></script> | <script src="js/sigma.min.js"></script> |
<script src="js/sigma/plugins/sigma.parseGexf.js"></script> | <script src="js/sigma/plugins/sigma.parseGexf.js"></script> |
<script src="js/sigma/plugins/sigma.forceatlas2.js"></script> | <script src="js/sigma/plugins/sigma.forceatlas2.js"></script> |
<script type="text/javascript">function init() { | <script type="text/javascript">function init() { |
// Instanciate sigma.js and customize rendering : | // Instanciate sigma.js and customize rendering : |
var sigInst = sigma.init(document.getElementById('sigma-example')).drawingProperties({ | var sigInst = sigma.init(document.getElementById('sigma-example')).drawingProperties({ |
defaultLabelColor: '#fff', | defaultLabelColor: '#fff', |
defaultLabelSize: 14, | defaultLabelSize: 14, |
defaultLabelBGColor: '#fff', | defaultLabelBGColor: '#fff', |
defaultLabelHoverColor: '#000', | defaultLabelHoverColor: '#000', |
labelThreshold: 6, | labelThreshold: 6, |
defaultEdgeType: 'curve' | defaultEdgeType: 'curve' |
}).graphProperties({ | }).graphProperties({ |
minNodeSize: 0.5, | minNodeSize: 0.5, |
maxNodeSize: 5, | maxNodeSize: 5, |
minEdgeSize: 5, | minEdgeSize: 5, |
maxEdgeSize: 5 | maxEdgeSize: 5 |
}).mouseProperties({ | }).mouseProperties({ |
maxRatio: 32 | maxRatio: 32 |
}); | }); |
// Parse a GEXF encoded file to fill the graph | // Parse a GEXF encoded file to fill the graph |
// (requires "sigma.parseGexf.js" to be included) | // (requires "sigma.parseGexf.js" to be included) |
sigInst.parseGexf('graph.php?format=gexf'); | sigInst.parseGexf('graph.php?format=gexf'); |
sigInst.bind('downnodes',function(event){ | sigInst.bind('downnodes',function(event){ |
var nodes = event.content; | var nodes = event.content; |
}); | }); |
// Start the ForceAtlas2 algorithm | // Start the ForceAtlas2 algorithm |
// (requires "sigma.forceatlas2.js" to be included) | // (requires "sigma.forceatlas2.js" to be included) |
sigInst.startForceAtlas2(); | sigInst.startForceAtlas2(); |
// Draw the graph : | // Draw the graph : |
sigInst.draw(); | sigInst.draw(); |
} | } |
if (document.addEventListener) { | if (document.addEventListener) { |
document.addEventListener("DOMContentLoaded", init, false); | document.addEventListener("DOMContentLoaded", init, false); |
} else { | } else { |
window.onload = init; | window.onload = init; |
} | } |
</script> | </script> |
<?php | <?php |
} | } |
if ($format == "dot") { | if ($format == "dot") { |
echo "}"; | echo "}"; |
} | } |
if ($format == "gexf") { | if ($format == "gexf") { |
echo ' </edges> | echo ' </edges> |
</graph> | </graph> |
</gexf>'. PHP_EOL; | </gexf>'. PHP_EOL; |
} | } |
//include_footer(); | //include_footer(); |
?> | ?> |
<?php | <?php |
date_default_timezone_set("Australia/Sydney"); | date_default_timezone_set("Australia/Sydney"); |
$basePath = ""; | $basePath = ""; |
if (strstr($_SERVER['PHP_SELF'], "alaveteli/") | if (strstr($_SERVER['PHP_SELF'], "alaveteli/") |
|| strstr($_SERVER['PHP_SELF'], "admin/") | || strstr($_SERVER['PHP_SELF'], "admin/") |
|| strstr($_SERVER['PHP_SELF'], "lib/") | || strstr($_SERVER['PHP_SELF'], "lib/") |
|| strstr($_SERVER['PHP_SELF'], "include/") | || strstr($_SERVER['PHP_SELF'], "include/") |
|| strstr($_SERVER['PHP_SELF'], "documents/") | || strstr($_SERVER['PHP_SELF'], "documents/") |
|| $_SERVER['SERVER_NAME'] == "disclosurelo.gs" | || $_SERVER['SERVER_NAME'] == "disclosurelo.gs" |
|| $_SERVER['SERVER_NAME'] == "www.disclosurelo.gs" | || $_SERVER['SERVER_NAME'] == "www.disclosurelo.gs" |
|| $_SERVER['SERVER_NAME'] == "direct.disclosurelo.gs" | |
) | ) |
$basePath = "../"; | $basePath = "../"; |
include_once ('couchdb.inc.php'); | include_once ('couchdb.inc.php'); |
include_once ('template.inc.php'); | include_once ('template.inc.php'); |
require_once $basePath.'lib/Requests/library/Requests.php'; | require_once $basePath.'lib/Requests/library/Requests.php'; |
Requests::register_autoloader(); | Requests::register_autoloader(); |
$ENV = "DEV"; | $ENV = "DEV"; |
if (isset($_SERVER['SERVER_NAME']) && $_SERVER['SERVER_NAME'] != 'localhost') { | if (false && isset($_SERVER['SERVER_NAME']) && $_SERVER['SERVER_NAME'] != 'localhost') { |
require $basePath."lib/amon-php/amon.php"; | require $basePath."lib/amon-php/amon.php"; |
Amon::config(array('address'=> 'http://127.0.0.1:2464', | Amon::config(array('address'=> 'http://127.0.0.1:2464', |
'protocol' => 'http', | 'protocol' => 'http', |
'secret_key' => "I2LJ6dOMmlnXgVAkTPFXd5M3ejkga8Gd2FbBt6iqZdw")); | 'secret_key' => "I2LJ6dOMmlnXgVAkTPFXd5M3ejkga8Gd2FbBt6iqZdw")); |
Amon::setup_exception_handler(); | Amon::setup_exception_handler(); |
$ENV = "PROD"; | $ENV = "PROD"; |
} | } |
# Convert a stdClass to an Array. http://www.php.net/manual/en/language.types.object.php#102735 | # Convert a stdClass to an Array. http://www.php.net/manual/en/language.types.object.php#102735 |
function object_to_array(stdClass $Class) { | function object_to_array(stdClass $Class) { |
# Typecast to (array) automatically converts stdClass -> array. | # Typecast to (array) automatically converts stdClass -> array. |
$Class = (array) $Class; | $Class = (array) $Class; |
# Iterate through the former properties looking for any stdClass properties. | # Iterate through the former properties looking for any stdClass properties. |
# Recursively apply (array). | # Recursively apply (array). |
foreach ($Class as $key => $value) { | foreach ($Class as $key => $value) { |
if (is_object($value) && get_class($value) === 'stdClass') { | if (is_object($value) && get_class($value) === 'stdClass') { |
$Class[$key] = object_to_array($value); | $Class[$key] = object_to_array($value); |
} | } |
} | } |
return $Class; | return $Class; |
} | } |
# Convert an Array to stdClass. http://www.php.net/manual/en/language.types.object.php#102735 | # Convert an Array to stdClass. http://www.php.net/manual/en/language.types.object.php#102735 |
function array_to_object(array $array) { | function array_to_object(array $array) { |
# Iterate through our array looking for array values. | # Iterate through our array looking for array values. |
# If found recurvisely call itself. | # If found recurvisely call itself. |
foreach ($array as $key => $value) { | foreach ($array as $key => $value) { |
if (is_array($value)) { | if (is_array($value)) { |
$array[$key] = array_to_object($value); | $array[$key] = array_to_object($value); |
} | } |
} | } |
# Typecast to (object) will automatically convert array -> stdClass | # Typecast to (object) will automatically convert array -> stdClass |
return (object) $array; | return (object) $array; |
} | } |
function dept_to_portfolio($deptName) { | function dept_to_portfolio($deptName) { |
return trim(str_replace("Department of", "", str_replace("Department of the", "Department of", $deptName))); | return trim(str_replace("Department of", "", str_replace("Department of the", "Department of", $deptName))); |
} | } |
function phrase_to_tag ($phrase) { | function phrase_to_tag ($phrase) { |
return str_replace(" ","_",str_replace("'","",str_replace(",","",strtolower($phrase)))); | return str_replace(" ","_",str_replace("'","",str_replace(",","",strtolower($phrase)))); |
} | } |
function local_url() { | function local_url() { |
return "http://" . $_SERVER['HTTP_HOST'] . rtrim(dirname($_SERVER['PHP_SELF']), '/\\') . "/"; | return "http://" . $_SERVER['HTTP_HOST'] . rtrim(dirname($_SERVER['PHP_SELF']), '/\\') . "/"; |
} | } |
function GetDomain($url) | function GetDomain($url) |
{ | { |
$nowww = ereg_replace('www\.','',$url); | $nowww = ereg_replace('www\.','',$url); |
$domain = parse_url($nowww); | $domain = parse_url($nowww); |
if(!empty($domain["host"])) | if(!empty($domain["host"])) |
{ | { |
return $domain["host"]; | return $domain["host"]; |
} else | } else |
{ | { |
return $domain["path"]; | return $domain["path"]; |
} | } |
} | } |
<?php | <?php |
include $basePath . "schemas/schemas.inc.php"; | include $basePath . "schemas/schemas.inc.php"; |
require ($basePath . 'couchdb/settee/src/settee.php'); | require ($basePath . 'couchdb/settee/src/settee.php'); |
if (php_uname('n') == "vanille") { | if (php_uname('n') == "vanille") { |
$serverAddr = 'http://192.168.178.21:5984/'; | $serverAddr = 'http://192.168.178.21:5984/'; |
} else | } else |
if (php_uname('n') == "KYUUBEY") { | if (php_uname('n') == "KYUUBEY") { |
$serverAddr = 'http://192.168.1.148:5984/'; | $serverAddr = 'http://192.168.1.148:5984/'; |
$serverAddr = 'http://127.0.0.1:5984/'; | $serverAddr = 'http://127.0.0.1:5984/'; |
} else | } else |
if (php_uname('n') == "ikurt-20") { | if (php_uname('n') == "ikurt-20") { |
$serverAddr = 'http://192.168.1.113:5984/'; | //$serverAddr = 'http://192.168.1.113:5984/'; |
//$serverAddr = 'http://127.0.0.1:5984/'; | $serverAddr = 'http://127.0.0.1:5984/'; |
} else { | } else { |
$serverAddr = 'http://127.0.0.1:5984/'; | $serverAddr = 'http://127.0.0.1:5984/'; |
} | } |
$server = new SetteeServer($serverAddr); | $server = new SetteeServer($serverAddr); |
function setteErrorHandler($e) | function setteErrorHandler($e) |
{ | { |
if (class_exists('Amon')) { | if (class_exists('Amon')) { |
Amon::log($e->getMessage() . " " . print_r($_SERVER, true), array('error')); | Amon::log($e->getMessage() . " " . print_r($_SERVER, true), array('error')); |
} | } |
echo $e->getMessage() . "<br>" . PHP_EOL; | echo $e->getMessage() . "<br>" . PHP_EOL; |
} | } |
<?php | <?php |
function include_header($title) { | function include_header($title) { |
global $basePath; | global $basePath; |
?> | ?> |
<!DOCTYPE html> | <!DOCTYPE html> |
<!-- paulirish.com/2008/conditional-stylesheets-vs-css-hacks-answer-neither/ --> | <!-- paulirish.com/2008/conditional-stylesheets-vs-css-hacks-answer-neither/ --> |
<!--[if lt IE 7]> <html class="no-js lt-ie9 lt-ie8 lt-ie7" lang="en"> <![endif]--> | <!--[if lt IE 7]> <html class="no-js lt-ie9 lt-ie8 lt-ie7" lang="en"> <![endif]--> |
<!--[if IE 7]> <html class="no-js lt-ie9 lt-ie8" lang="en"> <![endif]--> | <!--[if IE 7]> <html class="no-js lt-ie9 lt-ie8" lang="en"> <![endif]--> |
<!--[if IE 8]> <html class="no-js lt-ie9" lang="en"> <![endif]--> | <!--[if IE 8]> <html class="no-js lt-ie9" lang="en"> <![endif]--> |
<!--[if gt IE 8]><!--> <html lang="en"> <!--<![endif]--> | <!--[if gt IE 8]><!--> <html lang="en"> <!--<![endif]--> |
<head> | <head> |
<meta charset="utf-8" /> | <meta charset="utf-8" /> |
<!-- Set the viewport width to device width for mobile --> | <!-- Set the viewport width to device width for mobile --> |
<meta name="viewport" content="width=device-width" /> | <meta name="viewport" content="width=device-width" /> |
<title><?php echo $title; ?> - Disclosr</title> | <title><?php echo $title; ?> - Disclosr</title> |
<!-- Included CSS Files --> | <!-- Included CSS Files --> |
<link href="<?php echo $basePath ?>css/bootstrap.min.css" rel="stylesheet"> | <link href="<?php echo $basePath ?>css/bootstrap.min.css" rel="stylesheet"> |
<style type="text/css"> | <style type="text/css"> |
body { | body { |
padding-top: 60px; | padding-top: 60px; |
padding-bottom: 40px; | padding-bottom: 40px; |
} | } |
.sidebar-nav { | .sidebar-nav { |
padding: 9px 0; | padding: 9px 0; |
} | } |
.flotr-dummy-div { | |
margin-left: -999px; | |
} | |
</style> | </style> |
<link href="<?php echo $basePath ?>css/bootstrap-responsive.min.css" rel="stylesheet"> | <link href="<?php echo $basePath ?>css/bootstrap-responsive.min.css" rel="stylesheet"> |
<!--[if lt IE 9]> | <!--[if lt IE 9]> |
<link rel="stylesheet" href="<?php echo $basePath ?>stylesheets/ie.css"> | <link rel="stylesheet" href="<?php echo $basePath ?>stylesheets/ie.css"> |
<![endif]--> | <![endif]--> |
<!-- IE Fix for HTML5 Tags --> | <!-- IE Fix for HTML5 Tags --> |
<!--[if lt IE 9]> | <!--[if lt IE 9]> |
<script src="http://html5shiv.googlecode.com/svn/trunk/html5.js"></script> | <script src="http://html5shiv.googlecode.com/svn/trunk/html5.js"></script> |
<![endif]--> | <![endif]--> |
</head> | </head> |
<body xmlns:schema="http://schema.org/" xmlns:foaf="http://xmlns.com/foaf/0.1/"> | <body xmlns:schema="http://schema.org/" xmlns:foaf="http://xmlns.com/foaf/0.1/" xmlns:org="http://www.w3.org/ns/org#" xmlns:skos="http://www.w3.org/2004/02/skos/core#"> |
<div class="navbar navbar-inverse navbar-fixed-top"> | <div class="navbar navbar-inverse navbar-fixed-top"> |
<div class="navbar-inner"> | <div class="navbar-inner"> |
<div class="container-fluid"> | <div class="container-fluid"> |
<a class="btn btn-navbar" data-toggle="collapse" data-target=".nav-collapse"> | <a class="btn btn-navbar" data-toggle="collapse" data-target=".nav-collapse"> |
<span class="icon-bar"></span> | <span class="icon-bar"></span> |
<span class="icon-bar"></span> | <span class="icon-bar"></span> |
<span class="icon-bar"></span> | <span class="icon-bar"></span> |
</a> | </a> |
<a class="brand" href="#">Disclosr</a> | <a class="brand" href="#">Disclosr</a> |
<div class="nav-collapse collapse"> | <div class="nav-collapse collapse"> |
<ul class="nav"> | <ul class="nav"> |
<li><a href="getAgency.php">Agencies</a></li> | <li><a href="getAgency.php">Agencies</a></li> |
<li><a href="ranking.php">Open Gov Ranking</a></li> | |
<li><a href="headcount.php">Employee Headcount Graph</a></li> | <li><a href="headcount.php">Employee Headcount Graph</a></li> |
<li><a href="budget.php">Budget Graph</a></li> | <li><a href="budget.php">Budget Graph</a></li> |
<li><a href="about.php">About/FAQ</a></li> | <li><a href="about.php">About/FAQ</a></li> |
</ul> | </ul> |
</div><!--/.nav-collapse --> | </div><!--/.nav-collapse --> |
</div> | </div> |
</div> | </div> |
</div> | </div> |
<div class="container-fluid"> | <div class="container-fluid"> |
<?php } | <?php } |
function include_footer() { | function include_footer() { |
global $basePath; | global $basePath; |
?> | ?> |
</div> <!-- /container --> | </div> <!-- /container --> |
<hr> | <hr> |
<footer> | <footer> |
<p>Not affiliated with or endorsed by any government agency.</p> | <p>Not affiliated with or endorsed by any government agency.</p> |
</footer> | </footer> |
<!-- Included JS Files --> | <!-- Included JS Files --> |
<script src="http://code.jquery.com/jquery-1.7.1.min.js"></script> | <script src="http://code.jquery.com/jquery-1.7.1.min.js"></script> |
<script type="text/javascript" src="<?php echo $basePath ?>js/flotr2/flotr2.js"></script> | <script type="text/javascript" src="<?php echo $basePath ?>js/flotr2/flotr2.js"></script> |
<?php | <?php |
if (strpos($_SERVER['SERVER_NAME'], ".gs")) { | if (strpos($_SERVER['SERVER_NAME'], ".gs")) { |
?> | ?> |
<script type="text/javascript"> | <script type="text/javascript"> |
var _gaq = _gaq || []; | var _gaq = _gaq || []; |
_gaq.push(['_setAccount', 'UA-12341040-2']); | _gaq.push(['_setAccount', 'UA-12341040-2']); |
_gaq.push(['_trackPageview']); | _gaq.push(['_trackPageview']); |
(function() { | (function() { |
var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true; | var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true; |
ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js'; | ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js'; |
var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s); | var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s); |
})(); | })(); |
</script> | </script> |
</body> | </body> |
</html> | </html> |
<?php } | <?php } |
} | } |
<?php | |
/* | |
* Copyright (C) 2012 Michael Bemmerl <mail@mx-server.de> | |
* | |
* This file is part of the "Universal Feed Writer" project. | |
* | |
* This program is free software: you can redistribute it and/or modify | |
* it under the terms of the GNU General Public License as published by | |
* the Free Software Foundation, either version 3 of the License, or | |
* (at your option) any later version. | |
* | |
* This program is distributed in the hope that it will be useful, | |
* but WITHOUT ANY WARRANTY; without even the implied warranty of | |
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | |
* GNU General Public License for more details. | |
* | |
* You should have received a copy of the GNU General Public License | |
* along with this program. If not, see <http://www.gnu.org/licenses/>. | |
*/ | |
if (!class_exists('FeedWriter')) | |
require dirname(__FILE__) . '/FeedWriter.php'; | |
/** | |
* Wrapper for creating RSS1 feeds | |
* | |
* @package UniversalFeedWriter | |
*/ | |
class RSS1FeedWriter extends FeedWriter | |
{ | |
function __construct() | |
{ | |
parent::__construct(RSS1); | |
} | |
} | |
/** | |
* Wrapper for creating RSS2 feeds | |
* | |
* @package UniversalFeedWriter | |
*/ | |
class RSS2FeedWriter extends FeedWriter | |
{ | |
function __construct() | |
{ | |
parent::__construct(RSS2); | |
} | |
} | |
/** | |
* Wrapper for creating ATOM feeds | |
* | |
* @package UniversalFeedWriter | |
*/ | |
class ATOMFeedWriter extends FeedWriter | |
{ | |
function __construct() | |
{ | |
parent::__construct(ATOM); | |
} | |
} | |
This package can be used to generate feeds in either RSS 1.0, RSS 2.0 or ATOM | This package can be used to generate feeds in either RSS 1.0, RSS 2.0 or ATOM |
formats. | formats. |
There are three main classes that abstracts the feed information and another to | There are three main classes that abstract the feed information and another to |
encapsulate the feed items information. | encapsulate the feed items information. |
Applications can create feed writer object, several feed item objects, set | Applications can create feed writer object, several feed item objects, set |
several types of properties of either feeds and feed items, and add items to | several types of properties of either feeds and feed items, and add items to |
the feed. | the feed. |
Once a feed is fully composed with its items, the feed writer class can generate | Once a feed is fully composed with its items, the feed writer class can generate |
the necessary XML structure to describe the feed in the RSS or ATOM formats. | the necessary XML structure to describe the feed in the RSS or ATOM formats. |
The feed is generated as part of the current feed output. | The feed is generated as part of the current feed output. |
Requirements | Requirements |
============ | ============ |
PHP >= 5.0 | PHP >= 5.3 |
If you don't have 5.3 available on your system, there's a version supporting | |
PHP >= 5.0 in the "legacy-php-5.0" branch. | |
Documentation | |
============= | |
The documentation can be found in the "gh-pages" branch or on GitHub Pages: | |
http://mibe.github.io/FeedWriter/ | |
Authors | |
======= | |
(in chronological order) | |
Anis uddin Ahmad | |
Michael Bemmerl | |
Phil Freo | |
Paul Ferrett | |
Brennen Bearnes | |
Michael Robinson | |
<<<<<<< HEAD | |
<?php | <?php |
/* | /* |
* Copyright (C) 2008 Anis uddin Ahmad <anisniit@gmail.com> | * Copyright (C) 2008 Anis uddin Ahmad <anisniit@gmail.com> |
* | * |
* This file is part of the "Universal Feed Writer" project. | * This file is part of the "Universal Feed Writer" project. |
* | * |
* This program is free software: you can redistribute it and/or modify | * This program is free software: you can redistribute it and/or modify |
* it under the terms of the GNU General Public License as published by | * it under the terms of the GNU General Public License as published by |
* the Free Software Foundation, either version 3 of the License, or | * the Free Software Foundation, either version 3 of the License, or |
* (at your option) any later version. | * (at your option) any later version. |
* | * |
* This program is distributed in the hope that it will be useful, | * This program is distributed in the hope that it will be useful, |
* but WITHOUT ANY WARRANTY; without even the implied warranty of | * but WITHOUT ANY WARRANTY; without even the implied warranty of |
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the |
* GNU General Public License for more details. | * GNU General Public License for more details. |
* | * |
* You should have received a copy of the GNU General Public License | * You should have received a copy of the GNU General Public License |
* along with this program. If not, see <http://www.gnu.org/licenses/>. | * along with this program. If not, see <http://www.gnu.org/licenses/>. |
*/ | */ |
include("../FeedTypes.php"); | include("../FeedTypes.php"); |
// IMPORTANT : No need to add id for feed or channel. It will be automatically created from link. | // IMPORTANT : No need to add id for feed or channel. It will be automatically created from link. |
//Creating an instance of ATOMFeedWriter class. | //Creating an instance of ATOMFeedWriter class. |
//The constant ATOM is passed to mention the version | //The constant ATOM is passed to mention the version |
$TestFeed = new ATOMFeedWriter(); | $TestFeed = new ATOMFeedWriter(); |
//Setting the channel elements | //Setting the channel elements |
//Use wrapper functions for common elements | //Use wrapper functions for common elements |
$TestFeed->setTitle('Testing the RSS writer class'); | $TestFeed->setTitle('Testing the RSS writer class'); |
$TestFeed->setLink('http://www.ajaxray.com/rss2/channel/about'); | $TestFeed->setLink('http://www.ajaxray.com/rss2/channel/about'); |
//For other channel elements, use setChannelElement() function | //For other channel elements, use setChannelElement() function |
$TestFeed->setChannelElement('updated', date(DATE_ATOM , time())); | $TestFeed->setChannelElement('updated', date(DATE_ATOM , time())); |
$TestFeed->setChannelElement('author', array('name'=>'Anis uddin Ahmad')); | $TestFeed->setChannelElement('author', array('name'=>'Anis uddin Ahmad')); |
//Adding a feed. Genarally this protion will be in a loop and add all feeds. | //Adding a feed. Genarally this protion will be in a loop and add all feeds. |
//Create an empty FeedItem | //Create an empty FeedItem |
$newItem = $TestFeed->createNewItem(); | $newItem = $TestFeed->createNewItem(); |
//Add elements to the feed item | //Add elements to the feed item |
//Use wrapper functions to add common feed elements | //Use wrapper functions to add common feed elements |
$newItem->setTitle('The first feed'); | $newItem->setTitle('The first feed'); |
$newItem->setLink('http://www.yahoo.com'); | $newItem->setLink('http://www.yahoo.com'); |
$newItem->setDate(time()); | $newItem->setDate(time()); |
//Internally changed to "summary" tag for ATOM feed | //Internally changed to "summary" tag for ATOM feed |
$newItem->setDescription('This is a test of adding CDATA encoded description by the php <b>Universal Feed Writer</b> class'); | $newItem->setDescription('This is a test of adding CDATA encoded description by the php <b>Universal Feed Writer</b> class'); |
//Now add the feed item | //Now add the feed item |
$TestFeed->addItem($newItem); | $TestFeed->addItem($newItem); |
//OK. Everything is done. Now genarate the feed. | //OK. Everything is done. Now genarate the feed. |
$TestFeed->generateFeed(); | $TestFeed->generateFeed(); |
?> | ?> |
======= | |
<?php | |
// You should use an autoloader instead of including the files directly. | |
// This is done here only to make the examples work out of the box. | |
include('../Item.php'); | |
include('../Feed.php'); | |
include('../ATOM.php'); | |
date_default_timezone_set('UTC'); | |
use \FeedWriter\ATOM; | |
/* | |
* Copyright (C) 2008 Anis uddin Ahmad <anisniit@gmail.com> | |
* | |
* This file is part of the "Universal Feed Writer" project. | |
* | |
* This program is free software: you can redistribute it and/or modify | |
* it under the terms of the GNU General Public License as published by | |
* the Free Software Foundation, either version 3 of the License, or | |
* (at your option) any later version. | |
* | |
* This program is distributed in the hope that it will be useful, | |
* but WITHOUT ANY WARRANTY; without even the implied warranty of | |
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | |
* GNU General Public License for more details. | |
* | |
* You should have received a copy of the GNU General Public License | |
* along with this program. If not, see <http://www.gnu.org/licenses/>. | |
*/ | |
// IMPORTANT : No need to add id for feed or channel. It will be automatically created from link. | |
//Creating an instance of ATOM class. | |
$TestFeed = new ATOM; | |
//Setting the channel elements | |
//Use wrapper functions for common elements | |
$TestFeed->setTitle('Testing the RSS writer class'); | |
$TestFeed->setLink('http://www.ajaxray.com/rss2/channel/about'); | |
$TestFeed->setDate(new DateTime()); | |
//For other channel elements, use setChannelElement() function | |
$TestFeed->setChannelElement('author', array('name'=>'Anis uddin Ahmad')); | |
//You can add additional link elements, e.g. to a PubSubHubbub server with custom relations. | |
$TestFeed->setSelfLink('http://example.com/myfeed'); | |
$TestFeed->setAtomLink('http://pubsubhubbub.appspot.com', 'hub'); | |
//Adding a feed. Generally this portion will be in a loop and add all feeds. | |
//Create an empty Item | |
$newItem = $TestFeed->createNewItem(); | |
//Add elements to the feed item | |
//Use wrapper functions to add common feed elements | |
$newItem->setTitle('The first feed'); | |
$newItem->setLink('http://www.yahoo.com'); | |
$newItem->setDate(time()); | |
$newItem->setAuthor('Anis uddin Ahmad', 'anis@example.invalid'); | |
$newItem->setEnclosure('http://upload.wikimedia.org/wikipedia/commons/4/49/En-us-hello-1.ogg', 11779, 'audio/ogg'); | |
//Internally changed to "summary" tag for ATOM feed | |
$newItem->setDescription('This is a test of adding CDATA encoded description by the php <b>Universal Feed Writer</b> class'); | |
$newItem->setContent('<h1>hi.</h1> <p>This is the content for the entry.</p>'); | |
//Now add the feed item | |
$TestFeed->addItem($newItem); | |
//OK. Everything is done. Now generate the feed. | |
$TestFeed->printFeed(); | |
>>>>>>> origin/master | |
<<<<<<< HEAD | |
<?php | <?php |
/* | /* |
* Copyright (C) 2008 Anis uddin Ahmad <anisniit@gmail.com> | * Copyright (C) 2008 Anis uddin Ahmad <anisniit@gmail.com> |
* | * |
* This file is part of the "Universal Feed Writer" project. | * This file is part of the "Universal Feed Writer" project. |
* | * |
* This program is free software: you can redistribute it and/or modify | * This program is free software: you can redistribute it and/or modify |
* it under the terms of the GNU General Public License as published by | * it under the terms of the GNU General Public License as published by |
* the Free Software Foundation, either version 3 of the License, or | * the Free Software Foundation, either version 3 of the License, or |
* (at your option) any later version. | * (at your option) any later version. |
* | * |
* This program is distributed in the hope that it will be useful, | * This program is distributed in the hope that it will be useful, |
* but WITHOUT ANY WARRANTY; without even the implied warranty of | * but WITHOUT ANY WARRANTY; without even the implied warranty of |
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the |
* GNU General Public License for more details. | * GNU General Public License for more details. |
* | * |
* You should have received a copy of the GNU General Public License | * You should have received a copy of the GNU General Public License |
* along with this program. If not, see <http://www.gnu.org/licenses/>. | * along with this program. If not, see <http://www.gnu.org/licenses/>. |
*/ | */ |
// This is a minimum example of using the class | // This is a minimum example of using the class |
include("../FeedTypes.php"); | include("../FeedTypes.php"); |
//Creating an instance of RSS2FeedWriter class. | //Creating an instance of RSS2FeedWriter class. |
$TestFeed = new RSS2FeedWriter(); | $TestFeed = new RSS2FeedWriter(); |
//Setting the channel elements | //Setting the channel elements |
//Use wrapper functions for common channel elements | //Use wrapper functions for common channel elements |
$TestFeed->setTitle('Testing & Checking the RSS writer class'); | $TestFeed->setTitle('Testing & Checking the RSS writer class'); |
$TestFeed->setLink('http://www.ajaxray.com/projects/rss'); | $TestFeed->setLink('http://www.ajaxray.com/projects/rss'); |
$TestFeed->setDescription('This is a test of creating a RSS 2.0 feed Universal Feed Writer'); | $TestFeed->setDescription('This is a test of creating a RSS 2.0 feed Universal Feed Writer'); |
//Image title and link must match with the 'title' and 'link' channel elements for valid RSS 2.0 | //Image title and link must match with the 'title' and 'link' channel elements for valid RSS 2.0 |
$TestFeed->setImage('Testing the RSS writer class','http://www.ajaxray.com/projects/rss','http://www.rightbrainsolution.com/_resources/img/logo.png'); | $TestFeed->setImage('Testing the RSS writer class','http://www.ajaxray.com/projects/rss','http://www.rightbrainsolution.com/_resources/img/logo.png'); |
//Let's add some feed items: Create two empty FeedItem instances | //Let's add some feed items: Create two empty FeedItem instances |
$itemOne = $TestFeed->createNewItem(); | $itemOne = $TestFeed->createNewItem(); |
$itemTwo = $TestFeed->createNewItem(); | $itemTwo = $TestFeed->createNewItem(); |
//Add item details | //Add item details |
$itemOne->setTitle('The title of the first entry.'); | $itemOne->setTitle('The title of the first entry.'); |
$itemOne->setLink('http://www.google.de'); | $itemOne->setLink('http://www.google.de'); |
$itemOne->setDate(time()); | $itemOne->setDate(time()); |
$itemOne->setDescription('And here\'s the description of the entry.'); | $itemOne->setDescription('And here\'s the description of the entry.'); |
$itemTwo->setTitle('Lorem ipsum'); | $itemTwo->setTitle('Lorem ipsum'); |
$itemTwo->setLink('http://www.example.com'); | $itemTwo->setLink('http://www.example.com'); |
$itemTwo->setDate(1234567890); | $itemTwo->setDate(1234567890); |
$itemTwo->setDescription('Lorem ipsum dolor sit amet, consectetur, adipisci velit'); | $itemTwo->setDescription('Lorem ipsum dolor sit amet, consectetur, adipisci velit'); |
//Now add the feed item | //Now add the feed item |
$TestFeed->addItem($itemOne); | $TestFeed->addItem($itemOne); |
$TestFeed->addItem($itemTwo); | $TestFeed->addItem($itemTwo); |
//OK. Everything is done. Now genarate the feed. | //OK. Everything is done. Now genarate the feed. |
$TestFeed->generateFeed(); | $TestFeed->generateFeed(); |
?> | ?> |
======= | |
<?php | |
// You should use an autoloader instead of including the files directly. | |
// This is done here only to make the examples work out of the box. | |
include('../Item.php'); | |
include('../Feed.php'); | |
include('../RSS2.php'); | |
date_default_timezone_set('UTC'); | |
use \FeedWriter\RSS2; | |
/* | |
* Copyright (C) 2008 Anis uddin Ahmad <anisniit@gmail.com> | |
* | |
* This file is part of the "Universal Feed Writer" project. | |
* | |
* This program is free software: you can redistribute it and/or modify | |
* it under the terms of the GNU General Public License as published by | |
* the Free Software Foundation, either version 3 of the License, or | |
* (at your option) any later version. | |
* | |
* This program is distributed in the hope that it will be useful, | |
* but WITHOUT ANY WARRANTY; without even the implied warranty of | |
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | |
* GNU General Public License for more details. | |
* | |
* You should have received a copy of the GNU General Public License | |
* along with this program. If not, see <http://www.gnu.org/licenses/>. | |
*/ | |
//Creating an instance of RSS2 class. | |
$TestFeed = new RSS2; | |
//Setting the channel elements | |
//Use wrapper functions for common channel elements | |
$TestFeed->setTitle('Testing & Checking the RSS writer class'); | |
$TestFeed->setLink('http://www.ajaxray.com/projects/rss'); | |
$TestFeed->setDescription('This is a test of creating a RSS 2.0 feed Universal Feed Writer'); | |
//Image title and link must match with the 'title' and 'link' channel elements for valid RSS 2.0 | |
$TestFeed->setImage('Testing & Checking the RSS writer class','http://www.ajaxray.com/projects/rss','http://www.rightbrainsolution.com/_resources/img/logo.png'); | |
//Let's add some feed items: Create two empty Item instances | |
$itemOne = $TestFeed->createNewItem(); | |
$itemTwo = $TestFeed->createNewItem(); | |
//Add item details | |
$itemOne->setTitle('The title of the first entry.'); | |
$itemOne->setLink('http://www.google.de'); | |
$itemOne->setDate(time()); | |
$itemOne->setDescription('And here\'s the description of the entry.'); | |
$itemTwo->setTitle('Lorem ipsum'); | |
$itemTwo->setLink('http://www.example.com'); | |
$itemTwo->setDate(1234567890); | |
$itemTwo->setDescription('Lorem ipsum dolor sit amet, consectetur, adipisci velit'); | |
//Now add the feed item | |
$TestFeed->addItem($itemOne); | |
$TestFeed->addItem($itemTwo); | |
//OK. Everything is done. Now generate the feed. | |
$TestFeed->printFeed(); | |
>>>>>>> origin/master | |
<<<<<<< HEAD | |
<?php | <?php |
/* | /* |
* Copyright (C) 2008 Anis uddin Ahmad <anisniit@gmail.com> | * Copyright (C) 2008 Anis uddin Ahmad <anisniit@gmail.com> |
* | * |
* This file is part of the "Universal Feed Writer" project. | * This file is part of the "Universal Feed Writer" project. |
* | * |
* This program is free software: you can redistribute it and/or modify | * This program is free software: you can redistribute it and/or modify |
* it under the terms of the GNU General Public License as published by | * it under the terms of the GNU General Public License as published by |
* the Free Software Foundation, either version 3 of the License, or | * the Free Software Foundation, either version 3 of the License, or |
* (at your option) any later version. | * (at your option) any later version. |
* | * |
* This program is distributed in the hope that it will be useful, | * This program is distributed in the hope that it will be useful, |
* but WITHOUT ANY WARRANTY; without even the implied warranty of | * but WITHOUT ANY WARRANTY; without even the implied warranty of |
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the |
* GNU General Public License for more details. | * GNU General Public License for more details. |
* | * |
* You should have received a copy of the GNU General Public License | * You should have received a copy of the GNU General Public License |
* along with this program. If not, see <http://www.gnu.org/licenses/>. | * along with this program. If not, see <http://www.gnu.org/licenses/>. |
*/ | */ |
include("../FeedTypes.php"); | include("../FeedTypes.php"); |
//Creating an instance of RSS1FeedWriter class. | //Creating an instance of RSS1FeedWriter class. |
//The constant RSS1 is passed to mention the version | //The constant RSS1 is passed to mention the version |
$TestFeed = new RSS1FeedWriter(); | $TestFeed = new RSS1FeedWriter(); |
//Setting the channel elements | //Setting the channel elements |
//Use wrapper functions for common elements | //Use wrapper functions for common elements |
//For other optional channel elements, use setChannelElement() function | //For other optional channel elements, use setChannelElement() function |
$TestFeed->setTitle('Testing the RSS writer class'); | $TestFeed->setTitle('Testing the RSS writer class'); |
$TestFeed->setLink('http://www.ajaxray.com/rss2/channel/about'); | $TestFeed->setLink('http://www.ajaxray.com/rss2/channel/about'); |
$TestFeed->setDescription('This is test of creating a RSS 1.0 feed by Universal Feed Writer'); | $TestFeed->setDescription('This is test of creating a RSS 1.0 feed by Universal Feed Writer'); |
//It's important for RSS 1.0 | //It's important for RSS 1.0 |
$TestFeed->setChannelAbout('http://www.ajaxray.com/rss2/channel/about'); | $TestFeed->setChannelAbout('http://www.ajaxray.com/rss2/channel/about'); |
//Adding a feed. Genarally this protion will be in a loop and add all feeds. | //Adding a feed. Genarally this protion will be in a loop and add all feeds. |
//Create an empty FeedItem | //Create an empty FeedItem |
$newItem = $TestFeed->createNewItem(); | $newItem = $TestFeed->createNewItem(); |
//Add elements to the feed item | //Add elements to the feed item |
//Use wrapper functions to add common feed elements | //Use wrapper functions to add common feed elements |
$newItem->setTitle('The first feed'); | $newItem->setTitle('The first feed'); |
$newItem->setLink('http://www.yahoo.com'); | $newItem->setLink('http://www.yahoo.com'); |
//The parameter is a timestamp for setDate() function | //The parameter is a timestamp for setDate() function |
$newItem->setDate(time()); | $newItem->setDate(time()); |
$newItem->setDescription('This is test of adding CDATA encoded description by the php <b>Universal Feed Writer</b> class'); | $newItem->setDescription('This is test of adding CDATA encoded description by the php <b>Universal Feed Writer</b> class'); |
//Use core addElement() function for other supported optional elements | //Use core addElement() function for other supported optional elements |
$newItem->addElement('dc:subject', 'Nothing but test'); | $newItem->addElement('dc:subject', 'Nothing but test'); |
//Now add the feed item | //Now add the feed item |
$TestFeed->addItem($newItem); | $TestFeed->addItem($newItem); |
//Adding multiple elements from array | //Adding multiple elements from array |
//Elements which have an attribute cannot be added by this way | //Elements which have an attribute cannot be added by this way |
$newItem = $TestFeed->createNewItem(); | $newItem = $TestFeed->createNewItem(); |
$newItem->addElementArray(array('title'=>'The 2nd feed', 'link'=>'http://www.google.com', 'description'=>'This is a test of the FeedWriter class')); | $newItem->addElementArray(array('title'=>'The 2nd feed', 'link'=>'http://www.google.com', 'description'=>'This is a test of the FeedWriter class')); |
$TestFeed->addItem($newItem); | $TestFeed->addItem($newItem); |
//OK. Everything is done. Now genarate the feed. | //OK. Everything is done. Now genarate the feed. |
$TestFeed->generateFeed(); | $TestFeed->generateFeed(); |
?> | ?> |
======= | |
<?php | |
// You should use an autoloader instead of including the files directly. | |
// This is done here only to make the examples work out of the box. | |
include('../Item.php'); | |
include('../Feed.php'); | |
include('../RSS1.php'); | |
date_default_timezone_set('UTC'); | |
use \FeedWriter\RSS1; | |
/* | |
* Copyright (C) 2008 Anis uddin Ahmad <anisniit@gmail.com> | |
* | |
* This file is part of the "Universal Feed Writer" project. | |
* | |
* This program is free software: you can redistribute it and/or modify | |
* it under the terms of the GNU General Public License as published by | |
* the Free Software Foundation, either version 3 of the License, or | |
* (at your option) any later version. | |
* | |
* This program is distributed in the hope that it will be useful, | |
* but WITHOUT ANY WARRANTY; without even the implied warranty of | |
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | |
* GNU General Public License for more details. | |
* | |
* You should have received a copy of the GNU General Public License | |
* along with this program. If not, see <http://www.gnu.org/licenses/>. | |
*/ | |
//Creating an instance of RSS1 class. | |
$TestFeed = new RSS1; | |
//Setting the channel elements | |
//Use wrapper functions for common elements | |
//For other optional channel elements, use setChannelElement() function | |
$TestFeed->setTitle('Testing the RSS writer class'); | |
$TestFeed->setLink('http://www.ajaxray.com/rss2/channel/about'); | |
$TestFeed->setDescription('This is test of creating a RSS 1.0 feed by Universal Feed Writer'); | |
//It's important for RSS 1.0 | |
$TestFeed->setChannelAbout('http://www.ajaxray.com/rss2/channel/about'); | |
//Adding a feed. Generally this portion will be in a loop and add all feeds. | |
//Create an empty FeedItem | |
$newItem = $TestFeed->createNewItem(); | |
//Add elements to the feed item | |
//Use wrapper functions to add common feed elements | |
$newItem->setTitle('The first feed'); | |
$newItem->setLink('http://www.yahoo.com'); | |
//The parameter is a timestamp for setDate() function | |
$newItem->setDate(time()); | |
$newItem->setDescription('This is test of adding CDATA encoded description by the php <b>Universal Feed Writer</b> class'); | |
//Use core addElement() function for other supported optional elements | |
$newItem->addElement('dc:subject', 'Nothing but test'); | |
//Now add the feed item | |
$TestFeed->addItem($newItem); | |
//Adding multiple elements from array | |
//Elements which have an attribute cannot be added by this way | |
$newItem = $TestFeed->createNewItem(); | |
$newItem->addElementArray(array('title'=>'The 2nd feed', 'link'=>'http://www.google.com', 'description'=>'This is a test of the FeedWriter class')); | |
$TestFeed->addItem($newItem); | |
//OK. Everything is done. Now generate the feed. | |
$TestFeed->printFeed(); | |
>>>>>>> origin/master | |
<<<<<<< HEAD | |
<?php | <?php |
/* | /* |
* Copyright (C) 2008 Anis uddin Ahmad <anisniit@gmail.com> | * Copyright (C) 2008 Anis uddin Ahmad <anisniit@gmail.com> |
* | * |
* This file is part of the "Universal Feed Writer" project. | * This file is part of the "Universal Feed Writer" project. |
* | * |
* This program is free software: you can redistribute it and/or modify | * This program is free software: you can redistribute it and/or modify |
* it under the terms of the GNU General Public License as published by | * it under the terms of the GNU General Public License as published by |
* the Free Software Foundation, either version 3 of the License, or | * the Free Software Foundation, either version 3 of the License, or |
* (at your option) any later version. | * (at your option) any later version. |
* | * |
* This program is distributed in the hope that it will be useful, | * This program is distributed in the hope that it will be useful, |
* but WITHOUT ANY WARRANTY; without even the implied warranty of | * but WITHOUT ANY WARRANTY; without even the implied warranty of |
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the |
* GNU General Public License for more details. | * GNU General Public License for more details. |
* | * |
* You should have received a copy of the GNU General Public License | * You should have received a copy of the GNU General Public License |
* along with this program. If not, see <http://www.gnu.org/licenses/>. | * along with this program. If not, see <http://www.gnu.org/licenses/>. |
*/ | */ |
include("../FeedTypes.php"); | include("../FeedTypes.php"); |
//Creating an instance of RSS2FeedWriter class. | //Creating an instance of RSS2FeedWriter class. |
//The constant RSS2 is passed to mention the version | //The constant RSS2 is passed to mention the version |
$TestFeed = new RSS2FeedWriter(); | $TestFeed = new RSS2FeedWriter(); |
//Setting the channel elements | //Setting the channel elements |
//Use wrapper functions for common channel elements | //Use wrapper functions for common channel elements |
$TestFeed->setTitle('Testing & Checking the RSS writer class'); | $TestFeed->setTitle('Testing & Checking the RSS writer class'); |
$TestFeed->setLink('http://www.ajaxray.com/projects/rss'); | $TestFeed->setLink('http://www.ajaxray.com/projects/rss'); |
$TestFeed->setDescription('This is a test of creating a RSS 2.0 feed with Universal Feed Writer'); | $TestFeed->setDescription('This is a test of creating a RSS 2.0 feed with Universal Feed Writer'); |
//Image title and link must match with the 'title' and 'link' channel elements for RSS 2.0 | //Image title and link must match with the 'title' and 'link' channel elements for RSS 2.0 |
$TestFeed->setImage('Testing the RSS writer class','http://www.ajaxray.com/projects/rss','http://www.rightbrainsolution.com/_resources/img/logo.png'); | $TestFeed->setImage('Testing the RSS writer class','http://www.ajaxray.com/projects/rss','http://www.rightbrainsolution.com/_resources/img/logo.png'); |
//Use core setChannelElement() function for other optional channels | //Use core setChannelElement() function for other optional channels |
$TestFeed->setChannelElement('language', 'en-us'); | $TestFeed->setChannelElement('language', 'en-us'); |
$TestFeed->setChannelElement('pubDate', date(DATE_RSS, time())); | $TestFeed->setChannelElement('pubDate', date(DATE_RSS, time())); |
//Adding a feed. Genarally this portion will be in a loop and add all feeds. | //Adding a feed. Genarally this portion will be in a loop and add all feeds. |
//Create an empty FeedItem | //Create an empty FeedItem |
$newItem = $TestFeed->createNewItem(); | $newItem = $TestFeed->createNewItem(); |
//Add elements to the feed item | //Add elements to the feed item |
//Use wrapper functions to add common feed elements | //Use wrapper functions to add common feed elements |
$newItem->setTitle('The first feed'); | $newItem->setTitle('The first feed'); |
$newItem->setLink('http://www.yahoo.com'); | $newItem->setLink('http://www.yahoo.com'); |
//The parameter is a timestamp for setDate() function | //The parameter is a timestamp for setDate() function |
$newItem->setDate(time()); | $newItem->setDate(time()); |
$newItem->setDescription('This is a test of adding CDATA encoded description by the php <b>Universal Feed Writer</b> class'); | $newItem->setDescription('This is a test of adding CDATA encoded description by the php <b>Universal Feed Writer</b> class'); |
$newItem->setEncloser('http://www.attrtest.com', '1283629', 'audio/mpeg'); | $newItem->setEncloser('http://www.attrtest.com', '1283629', 'audio/mpeg'); |
//Use core addElement() function for other supported optional elements | //Use core addElement() function for other supported optional elements |
$newItem->addElement('author', 'admin@ajaxray.com (Anis uddin Ahmad)'); | $newItem->addElement('author', 'admin@ajaxray.com (Anis uddin Ahmad)'); |
//Attributes have to passed as array in 3rd parameter | //Attributes have to passed as array in 3rd parameter |
$newItem->addElement('guid', 'http://www.ajaxray.com',array('isPermaLink'=>'true')); | $newItem->addElement('guid', 'http://www.ajaxray.com',array('isPermaLink'=>'true')); |
//Now add the feed item | //Now add the feed item |
$TestFeed->addItem($newItem); | $TestFeed->addItem($newItem); |
//Another method to add feeds from array() | //Another method to add feeds from array() |
//Elements which have attribute cannot be added by this way | //Elements which have attribute cannot be added by this way |
$newItem = $TestFeed->createNewItem(); | $newItem = $TestFeed->createNewItem(); |
$newItem->addElementArray(array('title'=>'The 2nd feed', 'link'=>'http://www.google.com', 'description'=>'This is a test of the FeedWriter class')); | $newItem->addElementArray(array('title'=>'The 2nd feed', 'link'=>'http://www.google.com', 'description'=>'This is a test of the FeedWriter class')); |
$TestFeed->addItem($newItem); | $TestFeed->addItem($newItem); |
//OK. Everything is done. Now genarate the feed. | //OK. Everything is done. Now genarate the feed. |
$TestFeed->generateFeed(); | $TestFeed->generateFeed(); |
?> | ?> |
======= | |
<?php | |
// You should use an autoloader instead of including the files directly. | |
// This is done here only to make the examples work out of the box. | |
include('../Item.php'); | |
include('../Feed.php'); | |
include('../RSS2.php'); | |
date_default_timezone_set('UTC'); | |
use \FeedWriter\RSS2; | |
/** | |
* Copyright (C) 2008 Anis uddin Ahmad <anisniit@gmail.com> | |
* Copyright (C) 2013 Michael Bemmerl <mail@mx-server.de> | |
* | |
* This file is part of the "Universal Feed Writer" project. | |
* | |
* This program is free software: you can redistribute it and/or modify | |
* it under the terms of the GNU General Public License as published by | |
* the Free Software Foundation, either version 3 of the License, or | |
* (at your option) any later version. | |
* | |
* This program is distributed in the hope that it will be useful, | |
* but WITHOUT ANY WARRANTY; without even the implied warranty of | |
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | |
* GNU General Public License for more details. | |
* | |
* You should have received a copy of the GNU General Public License | |
* along with this program. If not, see <http://www.gnu.org/licenses/>. | |
*/ | |
// Creating an instance of RSS2 class. | |
$TestFeed = new RSS2; | |
// Setting some basic channel elements. These three elements are mandatory. | |
$TestFeed->setTitle('Testing & Checking the Feed Writer project'); | |
$TestFeed->setLink('https://github.com/mibe/FeedWriter'); | |
$TestFeed->setDescription('This is just an example how to use the Feed Writer project in your code.'); | |
// Image title and link must match with the 'title' and 'link' channel elements for RSS 2.0, | |
// which were set above. | |
$TestFeed->setImage('Testing & Checking the Feed Writer project', 'https://github.com/mibe/FeedWriter', 'https://upload.wikimedia.org/wikipedia/commons/thumb/d/d9/Rss-feed.svg/256px-Rss-feed.svg.png'); | |
// Use core setChannelElement() function for other optional channel elements. | |
// See http://www.rssboard.org/rss-specification#optionalChannelElements | |
// for other optional channel elements. Here the language code for American English and | |
$TestFeed->setChannelElement('language', 'en-US'); | |
// The date when this feed was lastly updated. The publication date is also set. | |
$TestFeed->setDate(date(DATE_RSS, time())); | |
$TestFeed->setChannelElement('pubDate', date(\DATE_RSS, strtotime('2013-04-06'))); | |
// You can add additional link elements, e.g. to a PubSubHubbub server with custom relations. | |
// It's recommended to provide a backlink to the feed URL. | |
$TestFeed->setSelfLink('http://example.com/myfeed'); | |
$TestFeed->setAtomLink('http://pubsubhubbub.appspot.com', 'hub'); | |
// You can add more XML namespaces for more custom channel elements which are not defined | |
// in the RSS 2 specification. Here the 'creativeCommons' element is used. There are much more | |
// available. Have a look at this list: http://feedvalidator.org/docs/howto/declare_namespaces.html | |
$TestFeed->addNamespace('creativeCommons', 'http://backend.userland.com/creativeCommonsRssModule'); | |
$TestFeed->setChannelElement('creativeCommons:license', 'http://www.creativecommons.org/licenses/by/1.0'); | |
// If you want you can also add a line to publicly announce that you used | |
// this fine piece of software to generate the feed. ;-) | |
$TestFeed->addGenerator(); | |
// Here we are done setting up the feed. What's next is adding some feed items. | |
// Create a new feed item. | |
$newItem = $TestFeed->createNewItem(); | |
// Add basic elements to the feed item | |
// These are again mandatory for a valid feed. | |
$newItem->setTitle('Hello World!'); | |
$newItem->setLink('http://www.example.com'); | |
$newItem->setDescription('This is a test of adding a description by the <b>Feed Writer</b> classes. It\'s automatically CDATA encoded.'); | |
// The following method calls add some optional elements to the feed item. | |
// Let's set the publication date of this item. You could also use a UNIX timestamp or | |
// an instance of PHP's DateTime class. | |
$newItem->setDate('2013-04-07 00:50:30'); | |
// You can also attach a media object to a feed item. You just need the URL, the byte length | |
// and the MIME type of the media. Here's a quirk: The RSS2 spec says "The url must be an http url.". | |
// Other schemes like ftp, https, etc. produce an error in feed validators. | |
$newItem->setEnclosure('http://upload.wikimedia.org/wikipedia/commons/4/49/En-us-hello-1.ogg', 11779, 'audio/ogg'); | |
// If you want you can set the name (and email address) of the author of this feed item. | |
$newItem->setAuthor('Anis uddin Ahmad', 'admin@ajaxray.com'); | |
// You can set a globally unique identifier. This can be a URL or any other string. | |
// If you set permaLink to true, the identifier must be an URL. The default of the | |
// permaLink parameter is false. | |
$newItem->setId('http://example.com/URL/to/article', true); | |
// Use the addElement() method for other optional elements. | |
// This here will add the 'source' element. The second parameter is the value of the element | |
// and the third is an array containing the element attributes. | |
$newItem->addElement('source', 'Mike\'s page', array('url' => 'http://www.example.com')); | |
// Now add the feed item to the main feed. | |
$TestFeed->addItem($newItem); | |
// Another method to add feeds items is by using an array which contains key-value pairs | |
// of every item element. Elements which have attributes cannot be added by this way. | |
$newItem = $TestFeed->createNewItem(); | |
$newItem->addElementArray(array('title'=> 'The 2nd item', 'link' => 'http://www.google.com', 'description' => 'Just another test.')); | |
$TestFeed->addItem($newItem); | |
// OK. Everything is done. Now generate the feed. | |
// If you want to send the feed directly to the browser, use the printFeed() method. | |
$myFeed = $TestFeed->generateFeed(); | |
// Do anything you want with the feed in $myFeed. Why not send it to the browser? ;-) | |
// You could also save it to a file if you don't want to invoke your script every time. | |
echo $myFeed; | |
>>>>>>> origin/master | |
<?php | |
include_once('include/common.inc.php'); | |
include_header('Open Gov Rankings'); | |
$db = $server->get_db('disclosr-agencies'); | |
?> | |
<div class="foundation-header"> | |
<h1><a href="about.php">Open Government Rankings</a></h1> | |
<h4 class="subheader"></h4> | |
</div> | |
<table> | |
<?php | |
$agenciesdb = $server->get_db('disclosr-agencies'); | |
//$docsdb = $server->get_db('disclosr-documents'); | |
$scoredagencies = Array(); | |
$scores = Array(); | |
$columnKeys = Array(); | |
try { | |
$rows = $agenciesdb->get_view("app", "all", null, true)->rows; | |
if ($rows) { | |
foreach ($rows as $row) { | |
$columns = Array(); | |
foreach ($row->value as $key => $value) { | |
if ((strstr($key, "has") || strstr($key, "URL")) && $key != "rtkURLs") { | |
//echo "$key<br>"; | |
$columns[$key] = $value; | |
} | |
} | |
//print_r(array_keys($columns)); | |
$columnKeys = array_unique(array_merge($columnKeys, array_keys($columns))); | |
//print_r($columnKeys); | |
$score = count($columns); | |
if (isset($scores[$score])){ | |
$scores[$score]++; | |
} else { | |
$scores[$score] =1; | |
} | |
$scoredagencies[] = Array("id"=> $row->key, "website"=> (isset($row->value->website)?$row->value->website:""), "name" => $row->value->name, "columns" => $columns, "score" => $score); | |
} | |
} | |
} catch (SetteeRestClientException $e) { | |
setteErrorHandler($e); | |
} | |
function cmp($a, $b) | |
{ | |
if ($a['score'] == $b['score']) { | |
return strcmp($a['name'], $b['name']); | |
} | |
return ($a['score'] > $b['score']) ? -1 : 1; | |
} | |
usort($scoredagencies, "cmp"); | |
echo "<tr>"; | |
echo "<th>Agency Name</th>"; | |
echo "<th>Score</th>"; | |
foreach ($columnKeys as $columnID) { | |
echo "<th>" . (isset($schemas['agency']["properties"][$columnID]['x-title']) ? $schemas['agency']["properties"][$columnID]['x-title'] : "<i>$columnID</i>") . "</th>"; | |
} | |
echo "</tr>"; | |
foreach ($scoredagencies as $scoredagency) { | |
echo "<tr>"; | |
echo "<td><b><a href='getAgency.php?id=" . $scoredagency['id'] . "'>". $scoredagency['name'] . "</a></b></td>"; | |
echo "<td><b>" . $scoredagency['score'] . "</b></td>"; | |
foreach ($columnKeys as $key) { | |
echo "<td style='text-align: center;'>"; | |
if (isset($scoredagency['columns'][$key])) { | |
$value = $scoredagency['columns'][$key]; | |
if (is_array($value)) { | |
if (count($value) == 1) { | |
$href = $value[0]; | |
} else { | |
$href = $value[0]; | |
} | |
} else { | |
$href = $value; | |
} | |
if (isset($href[0]) && $href[0] == "@") { | |
$href = str_replace("@","https://twitter.com/",$href); | |
} | |
//$href= urlencode($href); | |
echo "<font color='lightgreen'>"; | |
if (strstr($href, "http")) { | |
echo "<a title='Yes' href='$href' style='color:lightgreen;'>✓</a>"; | |
} else { | |
echo "✓"; | |
} | |
echo "</font>"; | |
} else { | |
echo "<font color='orange'><abbr title='No'>✘</abbr></font>"; | |
} | |
echo "</td>"; | |
} | |
echo "</tr>\n"; | |
} | |
?> | |
</table><br> | |
<div id="criteria" style="width:500px;height:900px;"></div> | |
<div id="scores" style="width:900px;height:500px;"></div> | |
<script id="source"> | |
window.onload = function () { | |
$(document).ready(function () { | |
var d1 = []; | |
var scorelabels = []; | |
<?php | |
try { | |
$rows = $db->get_view("app", "scoreHas?group=true", null, true)->rows; | |
$dataValues = Array(); | |
foreach ($rows as $row) { | |
$dataValues[$row->value] = $row->key; | |
} | |
$i = 0; | |
ksort($dataValues); | |
foreach ($dataValues as $value => $key) { | |
echo " d1.push([$value, $i]);" . PHP_EOL; | |
echo " scorelabels.push('$key');" . PHP_EOL; | |
$i++; | |
} | |
} catch (SetteeRestClientException $e) { | |
setteErrorHandler($e); | |
} | |
?> | |
function scoretrackformatter(obj) { | |
if (scorelabels[Math.floor(obj.y)]) { | |
return (scorelabels[Math.floor(obj.y)]) + "=" + obj.x; | |
} else { | |
return ""; | |
} | |
} | |
function scoretickformatter(val, axis) { | |
if (scorelabels[Math.floor(val)]) { | |
return (scorelabels[Math.floor(val)]) ; | |
} else { | |
return ""; | |
} | |
} | |
Flotr.draw(document.getElementById("criteria"), [ | |
{data: d1} | |
], { | |
title: 'Total count of agencies with criteria', | |
HtmlText: true, | |
bars: { | |
show: true, | |
horizontal: true | |
}, | |
mouse: { | |
track: true, | |
relative: true, | |
trackFormatter: scoretrackformatter | |
}, yaxis: { | |
autoscaling: true, | |
minorTickFreq: 0.6, | |
noTicks: scorelabels.length, | |
tickFormatter: scoretickformatter | |
}, | |
xaxis: { | |
autoscaling: true | |
} | |
}); | |
var d2 = []; | |
<?php | |
try { | |
ksort($scores); | |
foreach ($scores as $key => $value) { | |
echo " d2.push([$key,$value]);" . PHP_EOL; | |
$i++; | |
} | |
} catch (SetteeRestClientException $e) { | |
setteErrorHandler($e); | |
} | |
?> | |
Flotr.draw(document.getElementById("scores"), [ | |
{data: d2} | |
], { | |
title: 'Frequency distribution of Scores', | |
HtmlText: true, | |
bars: { | |
show: true | |
}, | |
mouse: { | |
track: true, | |
relative: true | |
}, yaxis: { | |
autoscaling: true | |
}, | |
xaxis: { | |
autoscaling: true | |
} | |
}); | |
}); | |
}; | |
</script> | |
<?php | |
include_footer(); | |
?> | |
# www.robotstxt.org/ | # www.robotstxt.org/ |
# www.google.com/support/webmasters/bin/answer.py?hl=en&answer=156449 | # www.google.com/support/webmasters/bin/answer.py?hl=en&answer=156449 |
User-agent: * | User-agent: * |
Disallow: /admin/ | Disallow: /admin/ |
Disallow: /viewDocument.php | |
Sitemap: http://orgs.disclosurelo.gs/sitemap.xml.php | Sitemap: http://orgs.disclosurelo.gs/sitemap.xml.php |
<?php | <?php |
$schemas['agency'] = Array( | $schemas['agency'] = Array( |
"description" => "Representation of government agency and online transparency measures", | "description" => "Representation of government agency and online transparency measures", |
"type" => "object", | "type" => "object", |
"properties" => Array( | "properties" => Array( |
"name" => Array("type" => "string", "required" => true, "x-itemprop" => "name", "x-property" => "schema:name foaf:name", "x-title" => "Name", "description" => "Name, most recent and broadest"), | "name" => Array("type" => "string", "required" => true, "x-itemprop" => "name", "x-property" => "schema:name foaf:name skos:prefLabel ", "x-title" => "Name", "description" => "Name, most recent and broadest"), |
"shortName" => Array("type" => "string", "required" => false, "x-title" => "Short Name", "description" => "Name shortened, usually to an acronym"), | "shortName" => Array("type" => "string", "required" => false, "x-title" => "Short Name", "description" => "Name shortened, usually to an acronym"), |
"description" => Array("type" => "string", "required" => false, "x-title" => "Description", "description" => "Description of roles and responsiblities of organisation"), | "description" => Array("type" => "string", "required" => false, "x-title" => "Description", "description" => "Description of roles and responsiblities of organisation"), |
"foiEmail" => Array("type" => "string", "required" => false, "x-title" => "FOI Contact Email", "x-itemprop" => "email", "description" => "FOI contact email if not foi@"), | "foiEmail" => Array("type" => "string", "required" => false, "x-title" => "FOI Contact Email", "x-itemprop" => "email", "description" => "FOI contact email if not foi@"), |
"sameAs" => Array("type" => "array", "required" => false, "x-property"=>"owl:sameAs","x-title" => "Same As", "description" => "Same as other URLs/URIs for this entity", | "sameAs" => Array("type" => "array", "required" => false, "x-property"=>"owl:sameAs","x-title" => "Same As", "description" => "Same as other URLs/URIs for this entity", |
"items" => Array("type" => "string")), | "items" => Array("type" => "string")), |
"otherNames" => Array("type" => "array", "required" => true, "x-title" => "Past/Other Names", "description" => "Other names for organisation", | "otherNames" => Array("type" => "array", "required" => true, "x-title" => "Past/Other Names", "description" => "Other names for organisation", |
"items" => Array("type" => "string")), | "items" => Array("type" => "string")), |
"positions" => Array("type" => "array", "required" => true, "x-title" => "Political Positions", "description" => "Ministers and Parliamentary Secretaries", | "positions" => Array("type" => "array", "required" => true, "x-title" => "Political Positions", "description" => "Ministers and Parliamentary Secretaries", |
"items" => Array("type" => "string")), | "items" => Array("type" => "string")), |
"foiBodies" => Array("type" => "array", "required" => true, "x-title" => "FOI Bodies","x-property"=>"schema:members foaf:knows", "description" => "Organisational units within this agency that are subject to FOI Act but are not autonomous", | "foiBodies" => Array("type" => "array", "required" => true, "x-title" => "FOI Bodies","x-property"=>"schema:members foaf:knows org:hasSubOrganization", "description" => "Organisational units within this agency that are subject to FOI Act but are not autonomous", |
"items" => Array("type" => "string")), | "items" => Array("type" => "string")), |
"legislation" => Array("type" => "array", "required" => true, "x-title" => "Legislation", "description" => "Legislation administered by or created for the establishment of this organisation", | "legislation" => Array("type" => "array", "required" => true, "x-title" => "Legislation", "description" => "Legislation administered by or created for the establishment of this organisation", |
"items" => Array("type" => "string")), | "items" => Array("type" => "string")), |
"orgType" => Array("type" => "string", "required" => true, "x-title" => "Organisation Type", "description" => "Org type based on legal formation via FMA/CAC legislation etc."), | "orgType" => Array("type" => "string", "required" => true, "x-title" => "Organisation Type", "x-property" => "org:classification", "description" => "Org type based on legal formation via FMA/CAC legislation etc."), |
"parentOrg" => Array("type" => "string", "required" => true, "x-title" => "Parent Organisation", "description" => "Parent organisation, usually a department of state"), | "parentOrg" => Array("type" => "string", "required" => true, "x-title" => "Parent Organisation", "x-property" => "org:subOrganizationOf", "description" => "Parent organisation, usually a department of state"), |
"website" => Array("type" => "string", "required" => true, "x-title" => "Website", "x-itemprop" => "url", "x-property" => "schema:url foaf:homepage", "description" => "Website URL"), | "website" => Array("type" => "string", "required" => true, "x-title" => "Website", "x-itemprop" => "url", "x-property" => "schema:url foaf:homepage", "description" => "Website URL"), |
"abn" => Array("type" => "string", "required" => true, "x-title" => "Australian Business Number", "description" => "ABN from business register"), | "abn" => Array("type" => "string", "required" => true, "x-title" => "Australian Business Number", "x-property" => "org:identifier", "description" => "ABN from business register"), |
"established" => Array("type" => "string", "required" => true, "x-title" => "Date established", "description" => "Date established"), | "established" => Array("type" => "string", "required" => true, "x-title" => "Date established", "x-property" => "schema:foundingDate", "description" => "Date established"), |
"employees" => Array("type" => "string", "required" => true, "x-title" => "2010-2011 employees", "description" => "2010-2011 employees"), | "employees" => Array("type" => "string", "required" => true, "x-title" => "2010-2011 employees", "description" => "2010-2011 employees"), |
"contractListURL" => Array("type" => "string", "required" => true, "x-title" => "Contract Listing", "description" => "Departmental and agency contracts, <a href='http://www.aph.gov.au/senate/pubs/standing_orders/d05.htm'>mandated by the Senate</a>" ), | "contractListURL" => Array("type" => "string", "required" => true, "x-title" => "Contract Listing", "description" => "Departmental and agency contracts, <a href='http://www.aph.gov.au/senate/pubs/standing_orders/d05.htm'>mandated by the Senate</a>" ), |
"budgetURL" => Array("type" => "string", "required" => true,"x-title" => "Budget", "description" => "Portfolio Budget Statements and Portfolio Additional Estimates Statements"), | "budgetURL" => Array("type" => "string", "required" => true,"x-title" => "Budget", "description" => "Portfolio Budget Statements and Portfolio Additional Estimates Statements"), |
"grantsReportingURL" => Array("type" => "string", "required" => true, "x-title" => "Grants Awarded", | "grantsReportingURL" => Array("type" => "string", "required" => true, "x-title" => "Grants Awarded", |
"description" => "Departmental and agency grants <a href='http://www.aph.gov.au/senate/pubs/standing_orders/d05.htm'>mandated by the Senate</a> and <a href='http://www.finance.gov.au/publications/fmg-series/23-commonwealth-grant-guidelines.html'>Commonwealth grants guidelines</a> "), | "description" => "Departmental and agency grants <a href='http://www.aph.gov.au/senate/pubs/standing_orders/d05.htm'>mandated by the Senate</a> and <a href='http://www.finance.gov.au/publications/fmg-series/23-commonwealth-grant-guidelines.html'>Commonwealth grants guidelines</a> "), |
"annualReportURL" => Array("type" => "string", "required" => true, "x-title" => "Annual Report(s)", "description" => ""), | "annualReportURL" => Array("type" => "string", "required" => true, "x-title" => "Annual Report(s)", "description" => ""), |
"consultanciesURL" => Array("type" => "string", "required" => true, "x-title" => "Consultants Hired", "description" => ""), | "consultanciesURL" => Array("type" => "string", "required" => true, "x-title" => "Consultants Hired", "description" => ""), |
"legalExpenditureURL" => Array("type" => "string", "required" => true, "x-title" => "Legal Services Expenditure", "description" => "Legal Services Expenditure mandated by Legal Services Directions 2005"), | "legalExpenditureURL" => Array("type" => "string", "required" => true, "x-title" => "Legal Services Expenditure", "description" => "Legal Services Expenditure mandated by Legal Services Directions 2005"), |
"recordsListURL" => Array("type" => "string", "required" => true, "x-title" => "Files/Records Held", "description" => "Indexed lists of departmental and agency files, <a href='http://www.aph.gov.au/senate/pubs/standing_orders/d05.htm'>mandated by the Senate</a>"), | "recordsListURL" => Array("type" => "string", "required" => true, "x-title" => "Files/Records Held", "description" => "Indexed lists of departmental and agency files, <a href='http://www.aph.gov.au/senate/pubs/standing_orders/d05.htm'>mandated by the Senate</a>"), |
"FOIDocumentsURL" => Array("type" => "string", "required" => true, "x-title" => "FOI Documents Released", "description" => "FOI Disclosure Log URL"), | "FOIDocumentsURL" => Array("type" => "string", "required" => true, "x-title" => "FOI Documents Released", "description" => "FOI Disclosure Log URL"), |
"FOIDocumentsRSSURL" => Array("type" => "string", "required" => false, "x-title" => "RSS Feed of FOI Documents Released", "description" => "FOI Disclosure Log in RSS format"), | "FOIDocumentsRSSURL" => Array("type" => "string", "required" => false, "x-title" => "RSS Feed of FOI Documents Released", "description" => "FOI Disclosure Log in RSS format"), |
"hasFOIPDF" => Array("type" => "array", "required" => false, "x-title" => "Has FOI Documents Released in PDF", "description" => "FOI Disclosure Log contains any PDFs", | "hasFOIPDF" => Array("type" => "array", "required" => false, "x-title" => "Has FOI Documents Released in PDF", "description" => "FOI Disclosure Log contains any PDFs", |
"items" => Array("type" => "string")), | "items" => Array("type" => "string")), |
"infoPublicationSchemeURL" => Array("type" => "string", "required" => true, "x-title" => "Information Publication Scheme", "description" => ""), | "infoPublicationSchemeURL" => Array("type" => "string", "required" => true, "x-title" => "Information Publication Scheme", "description" => ""), |
"appointmentsURL" => Array("type" => "string", "required" => true, "x-title" => "Agency Appointments/Boards", "description" => "Departmental and agency appointments and vacancies , <a href='http://www.aph.gov.au/senate/pubs/standing_orders/d05.htm'>mandated by the Senate</a>"), | "appointmentsURL" => Array("type" => "string", "required" => true, "x-title" => "Agency Appointments/Boards", "description" => "Departmental and agency appointments and vacancies , <a href='http://www.aph.gov.au/senate/pubs/standing_orders/d05.htm'>mandated by the Senate</a>"), |
"advertisingURL" => Array("type" => "string", "required" => true, "x-title" => "Approved Advertising Campaigns", "description" => " Agency advertising and public information projects, <a href='http://www.aph.gov.au/senate/pubs/standing_orders/d05.htm'>mandated by the Senate</a> "), | "advertisingURL" => Array("type" => "string", "required" => true, "x-title" => "Approved Advertising Campaigns", "description" => " Agency advertising and public information projects, <a href='http://www.aph.gov.au/senate/pubs/standing_orders/d05.htm'>mandated by the Senate</a> "), |
"hasRSS" => Array("type" => "array", "required" => true, "x-title" => "Has RSS", "description" => ""), | "hasRSS" => Array("type" => "array", "required" => true, "x-title" => "Has RSS", "description" => ""), |
"hasBlog" => Array("type" => "array", "required" => true, "x-title" => "Has Blog", "description" => ""), | "hasBlog" => Array("type" => "array", "required" => true, "x-title" => "Has Blog", "description" => ""), |
"hasMobileApp" => Array("type" => "array", "required" => true, "x-title" => "Has Mobile App", "description" => ""), | "hasMobileApp" => Array("type" => "array", "required" => true, "x-title" => "Has Mobile App", "description" => ""), |
"hasMailingList" => Array("type" => "array", "required" => true, "x-title" => "Has Mailing List", "description" => "", | "hasMailingList" => Array("type" => "array", "required" => true, "x-title" => "Has Mailing List", "description" => "", |
"items" => Array("type" => "string")), | "items" => Array("type" => "string")), |
"hasTwitter" => Array("type" => "array", "required" => true, "x-title" => "Has Twitter", "description" => "", | "hasTwitter" => Array("type" => "array", "required" => true, "x-title" => "Has Twitter", "description" => "", |
"items" => Array("type" => "string")), | "items" => Array("type" => "string")), |
"hasFacebook" => Array("type" => "array", "required" => true, "x-title" => "Has Facebook", "description" => "", | "hasFacebook" => Array("type" => "array", "required" => true, "x-title" => "Has Facebook", "description" => "", |
"items" => Array("type" => "string")), | "items" => Array("type" => "string")), |
"hasYouTube" => Array("type" => "array", "required" => true, "x-title" => "Has YouTube", "description" => "", | "hasYouTube" => Array("type" => "array", "required" => true, "x-title" => "Has YouTube", "description" => "", |
"items" => Array("type" => "string")), | "items" => Array("type" => "string")), |
"hasFlickr" => Array("type" => "array", "required" => true, "x-title" => "Has Flickr", "description" => "", | "hasFlickr" => Array("type" => "array", "required" => true, "x-title" => "Has Flickr", "description" => "", |
"items" => Array("type" => "string")), | "items" => Array("type" => "string")), |
"hasCCBY" => Array("type" => "array", "required" => true, "x-title" => "Has CC-BY", "description" => "Has any page licenced Creative Commons - Attribution", | "hasCCBY" => Array("type" => "array", "required" => true, "x-title" => "Has CC-BY", "description" => "Has any page licenced Creative Commons - Attribution", |
"items" => Array("type" => "string")), | "items" => Array("type" => "string")), |
"hasRestrictiveLicence" => Array("type" => "array","required" => true, "x-title" => "Has Restrictive Licence", "description" => "Has any page licenced under terms more restrictive than Crown Copyright", | "hasRestrictiveLicence" => Array("type" => "array","required" => true, "x-title" => "Has Restrictive Licence", "description" => "Has any page licenced under terms more restrictive than Crown Copyright", |
"items" => Array("type" => "string")), | "items" => Array("type" => "string")), |
"hasPermissiveLicence" => Array("type" => "array","required" => true, "x-title" => "Has Permissive Licence", "description" => "Has any page licenced under terms more permissive than Crown Copyright but not clear CCBY", | "hasPermissiveLicence" => Array("type" => "array","required" => true, "x-title" => "Has Permissive Licence", "description" => "Has any page licenced under terms more permissive than Crown Copyright but not clear CCBY", |
"items" => Array("type" => "string")), | "items" => Array("type" => "string")), |
"hasCrownCopyright" => Array("type" => "array", "required" => true, "x-title" => "Has Standard Crown Copyright licence", "description" => "Has any page still licenced under the former Commonwealth Copyright Administration", | "hasCrownCopyright" => Array("type" => "array", "required" => true, "x-title" => "Has Standard Crown Copyright licence", "description" => "Has any page still licenced under the former Commonwealth Copyright Administration", |
"items" => Array("type" => "string")), | "items" => Array("type" => "string")), |
), | ), |
); | ); |
?> | ?> |
<?php | <?php |
include ('include/common.inc.php'); | include ('include/common.inc.php'); |
$last_updated = date('Y-m-d', @filemtime('cbrfeed.zip')); | $last_updated = date('Y-m-d', @filemtime('cbrfeed.zip')); |
header("Content-Type: text/xml"); | header("Content-Type: text/xml"); |
echo "<?xml version='1.0' encoding='UTF-8'?>"; | echo "<?xml version='1.0' encoding='UTF-8'?>"; |
echo '<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">' . "\n"; | echo '<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">' . "\n"; |
echo " <url><loc>" . local_url() . "index.php</loc><priority>1.0</priority></url>\n"; | echo " <url><loc>" . local_url() . "index.php</loc><priority>1.0</priority></url>\n"; |
foreach (scandir("./") as $file) { | foreach (scandir("./") as $file) { |
if (strpos($file, ".php") !== false && $file != "index.php" && $file != "sitemap.xml.php") | if (strpos($file, ".php") !== false && $file != "index.php" && $file != "sitemap.xml.php"&& $file != "viewDocument.php") |
echo " <url><loc>" . local_url() . "$file</loc><priority>0.3</priority></url>\n"; | echo " <url><loc>" . local_url() . "$file</loc><priority>0.3</priority></url>\n"; |
} | } |
$db = $server->get_db('disclosr-agencies'); | $db = $server->get_db('disclosr-agencies'); |
try { | try { |
$rows = $db->get_view("app", "byCanonicalName")->rows; | $rows = $db->get_view("app", "byCanonicalName")->rows; |
foreach ($rows as $row) { | foreach ($rows as $row) { |
echo '<url><loc>' . local_url() . 'getAgency.php?id=' . $row->value->_id . "</loc><priority>0.6</priority></url>\n"; | echo '<url><loc>' . local_url() . 'getAgency.php?id=' . $row->value->_id . "</loc><priority>0.6</priority></url>\n"; |
} | } |
} catch (SetteeRestClientException $e) { | } catch (SetteeRestClientException $e) { |
setteErrorHandler($e); | setteErrorHandler($e); |
} | } |
echo '</urlset>'; | echo '</urlset>'; |
?> | ?> |