Export employee stats
Export employee stats


Former-commit-id: cffd8ae4e74fc751c71cdaf62195ce99303268a2

[submodule "couchdb/couchdb-lucene"] [submodule "couchdb/couchdb-lucene"]
path = couchdb/couchdb-lucene path = couchdb/couchdb-lucene
url = https://github.com/rnewson/couchdb-lucene.git url = https://github.com/rnewson/couchdb-lucene.git
[submodule "couchdb/settee"] [submodule "couchdb/settee"]
path = couchdb/settee path = couchdb/settee
url = https://github.com/inadarei/settee.git url = https://github.com/inadarei/settee.git
[submodule "lib/springy"] [submodule "lib/springy"]
path = lib/springy path = lib/springy
url = https://github.com/dhotson/springy.git url = https://github.com/dhotson/springy.git
  [submodule "lib/php-diff"]
  path = lib/php-diff
  url = https://github.com/chrisboulton/php-diff.git
  [submodule "lib/Requests"]
  path = lib/Requests
  url = https://github.com/rmccue/Requests.git
  [submodule "javascripts/flotr2"]
  path = javascripts/flotr2
  url = https://github.com/HumbleSoftware/Flotr2.git
   
file:a/about.php -> file:b/about.php
<?php <?php
include_once('include/common.inc.php'); include_once('include/common.inc.php');
include_header(); include_header();
?> ?>
<div class="foundation-header"> <div class="foundation-header">
<h1><a href="about.php">About/FAQ</a></h1> <h1><a href="about.php">About/FAQ</a></h1>
<h4 class="subheader">Lorem ipsum.</h4> <h4 class="subheader">Lorem ipsum.</h4>
</div> </div>
<h2> What is this? </h2> <h2> What is this? </h2>
Disclosr is a project to monitor Australian Federal Government agencies Disclosr is a project to monitor Australian Federal Government agencies
compliance with their <a href="http://www.oaic.gov.au/publications/other_operational/foi_policy_frequently_asked_questions.html#_Toc291837571">"proactive disclosure requirements"</a>. compliance with their <a href="http://www.oaic.gov.au/publications/other_operational/foi_policy_frequently_asked_questions.html#_Toc291837571">"proactive disclosure requirements"</a>.
OGRE (Open Government Realization Evaluation) is a ranking of compliance with these requirements. OGRE (Open Government Realization Evaluation) is a ranking of compliance with these requirements.
Prometheus is the agent which polls agency websites to assess compliance. Prometheus is the agent which polls agency websites to assess compliance.
   
<h2> Open everything </h2> <h2> Open everything </h2>
all documents released CC-BY 3 AU All documents released CC-BY 3 AU
Open source git @ Open source git @
   
<h2>Organisational Data Sources</h2> <h2>Organisational Data Sources</h2>
   
http://www.comlaw.gov.au/Browse/Results/ByTitle/AdministrativeArrangementsOrders/Current/Ad/0 defines departments http://www.comlaw.gov.au/Browse/Results/ByTitle/AdministrativeArrangementsOrders/Current/Ad/0 defines departments
Agencies can be found in the Schedule to an Appropriation Bill (budget), Schedule to FMA Regulations and/or Public Service Act. Agencies can be found in the Schedule to an Appropriation Bill (budget), Schedule to FMA Regulations and/or Public Service Act.<br>
   
http://www.finance.gov.au/publications/flipchart/docs/FMACACFlipchart.pdf summarises these. view-source:https://www.tenders.gov.au/?event=public.advancedsearch.home is great for the suspended/active status http://www.finance.gov.au/publications/flipchart/docs/FMACACFlipchart.pdf summarises these. view-source:https://www.tenders.gov.au/?event=public.advancedsearch.home is great for the suspended/active status<br>
   
  Fraud in gov depts by Fairfax Media http://www.smh.com.au/national/public-service-keeps-fraud-cases-private-20110923-1kpdr.html
   
When defining the hierachy, this system is designed towards monitoring accountablity. Thus large agencies that have registered their own ABN When defining the hierachy, this system is designed towards monitoring accountablity. Thus large agencies that have registered their own ABN
and have their own accountablity mechanisms/website receive a seperate record as a child of their department. and have their own accountablity mechanisms/website receive a seperate record as a child of their department.
Some small agencies will choose to simply rely on their parent department's accountablity measures. Some small agencies will choose to simply rely on their parent department's accountablity measures.<br>
   
This flows through to organisation name and other/past names. A department that completely accounts for an agency will list that agency as an other child name. This flows through to organisation name and other/past names. A department that completely accounts for an agency will list that agency as an other child name.
As agencies themselves shift between departments, there may be scope for providing time ranges but typically the newest hierarchy will be the one recorded. As agencies themselves shift between departments, there may be scope for providing time ranges but typically the newest hierarchy will be the one recorded.
A department/agency name will be the newest active name assigned to that ABN. A department/agency name will be the newest active name assigned to that ABN.<br>
   
ABN information is derived from the ABR. This is the definitive umpire about which former name should be linked to which current name. ABN information is derived from the ABR. This is the definitive umpire about which former name should be linked to which current name.
For example "Department of Transport and Regional Services" became "Department of Infrastructure, Transport, Regional Development and Local Government" (same ABN) For example "Department of Transport and Regional Services" became "Department of Infrastructure, Transport, Regional Development and Local Government" (same ABN)
however it later split into "Department of Infrastructure and Transport" (same ABN) however it later split into "Department of Infrastructure and Transport" (same ABN)
and "Department of Regional Australia, Regional Development and Local Government" (new ABN). and "Department of Regional Australia, Regional Development and Local Government" (new ABN).<br>
   
Statistical information from http://www.apsc.gov.au/stateoftheservice/1011/statsbulletin/section1.html#t2total https://www.apsedii.gov.au/apsedii/CustomQueryx33.shtml Statistical information from http://www.apsc.gov.au/stateoftheservice/1011/statsbulletin/section1.html#t2total https://www.apsedii.gov.au/apsedii/CustomQueryx33.shtml
and individual annual reports. and individual annual reports.<br>
   
Webpage Assessment <h2>Webpage Assessment</h2>
Much due care has been put into correctly recording disclosure URLs. Typically the "About", "Corporate", "Publications" and "Sitemap" sections are checked at the very least. Much due care has been put into correctly recording disclosure URLs. Typically the "About", "Corporate", "Publications" and "Sitemap" sections are checked at the very least.
Occasionally it is nessicary to use a site or Google search. In several rare cases, there is a secret "Disclosure" navigation menu you can find if you find one of the mandatory publishing obligations in that category (seriously). Occasionally it is nessicary to use a site or Google search. In several rare cases, there is a secret "Disclosure" navigation menu you can find if you find one of the mandatory publishing obligations in that category (seriously).<br>
Some rules about leniency: Some rules about leniency:<br>
An empty FOI disclosure log counts, a page outlining what the FOI Act is does not. <ul>
A disclosure log in PDF or Word format counts :( <li>An empty FOI disclosure log counts, a page outlining what the FOI Act is does not.</li>
An empty File/Record list counts (although that's very minimalistic that you have no files, electronic or paper) <li>A disclosure log in PDF or Word format counts :(</li>
Only a current information publication scheme page counts, not a s.9 FOI Act page or an organisation chart. <li>An empty File/Record list counts (although that's very minimalistic that you have no files, electronic or paper)</li>
If there isn't a page easily listing all current and past Annual Reports, the most current one (html, pdf) counts. <li>Only a current information publication scheme page counts, not a s.9 FOI Act page or an organisation chart.</li>
Consultancy contracts might not need it's own webpage (if in Annual Report), grants/appointments might not apply to all organisations but Legal Services Expenditure (and all other obligations) does need a webpage. <li>If there isn't a page easily listing all current and past Annual Reports, the most current one (html, pdf) counts.</li>
  <li>Consultancy contracts might not need it's own webpage (if in Annual Report), grants/appointments might not apply to all organisations but Legal Services Expenditure (and all other obligations) does need a webpage. </li>
   
<h2>Open Government Scoring</h2> <h2>Open Government Scoring</h2>
+1 point for every true Has... attribute +1 point for every true Has... attribute<br>
-1 point for every false Has... (ie. Has Not) attribute -1 point for every false Has... (ie. Has Not) attribute</br>
   
Don't like this? Make your own score, suggest a better scoring mechanism. Don't like this? Make your own score, suggest a better scoring mechanism.</br>
   
<?php <?php
include_footer(); include_footer();
?> ?>
  "Agriculture, Fisheries & Forestry",3530
  Attorney-Generals,708
  Administrative Appeals Tribunal,108
  AUSTRAC,60
  Australian Crime Commission,225
  Australian Customs Service,4934
  Australian Protective Service,1124
  CrimTrac Agency,34
  Family Court of Australia,574
  Federal Court of Australia,221
  Federal Magistrates Service,46
  Federal Privacy Commissioner,32
  HREOC,79
  ITSA,227
  National Native Title Tribunal,241
  Office of the D.P.P.,403
  Office of Film & Literature Class,39
  Office of the Parl. Counsel,45
  Communications IT & the Arts,776
  Aust. Broadcasting Authority,114
  Aust. Communications Authority,391
  Aust. National Maritime Museum,90
  National Archives of Australia,363
  National Library of Australia,443
  National Museum of Australia,185
  NOIE,160
  Defence,18231
  Defence Housing Authority,661
  DEST,1464
  Australian Research Council,50
  DEWR,2055
  EOWA,11
  Australian Industrial Registry,185
  Comcare,300
  NOHSC,89
  Environment and Heritage,1271
  Australian Greenhouse Office,167
  Bureau of Meteorology,1403
  GBRMPA,153
  National Oceans Office,31
  Family and Community Services,4832
  Aust. Institute of Family Studies,39
  Centrelink,25147
  Finance and Administration,771
  Australian Electoral Comm.,729
  ComSuper,338
  Foreign Affairs & Trade,2763
  ACIAR,44
  Health & Ageing,5137
  Aust. Institute of Health & Welfare,152
  ARPANSA,126
  FSANZ,110
  PSR,25
  DIMIA,4089
  Aboriginal Hostels Ltd.,341
  ATSIC,1072
  AIATSIS,62
  Torres Strait Regional Authority,30
  Industry Tourism & Resources,2691
  Prime Minister & Cabinet,344
  ANAO,278
  Commonwealth Ombudsman,78
  Office of National Assessments,47
  APS Commission,164
  Transport & Regional Services,945
  National Capital Authority,76
  Treasury,906
  Australian Bureau of Statistics,3024
  ACCC,446
  ASIC,1048
  Australian Taxation Office,20238
  Productivity Commission,197
  Veterans Affairs,2335
  Australian War Memorial,215
 
  "Agriculture, Fisheries and Forestry",3928
  Attorney-General's,781
  Administrative Appeals Tribunal,117
  AUSTRAC,61
  Australian Crime Commission,315
  Australian Customs Service,4736
  Australian Protective Service,1234
  CrimTrac Agency,44
  Family Court of Australia,608
  Federal Court of Australia,238
  Federal Magistrates Service,56
  Federal Privacy Commissioner,29
  HREOC,84
  ITSA,240
  National Native Title Tribunal,243
  Office of the D.P.P.,416
  Office of Film & Literature Class.,37
  Office of Parliamentary Counsel,46
  Communications IT & the Arts,607
  Aust. Broadcasting Authority,114
  Aust. Communications Authority,407
  Australian Film Commission,176
  AGIMO,95
  Aust. National Maritime Museum,93
  National Archives of Australia,386
  National Library of Australia,438
  National Museum of Australia,186
  Defence,18301
  Defence Housing Authority,679
  DEST,1684
  Australian Research Council,53
  DEWR,2171
  Australian Industrial Registry,188
  Comcare,325
  EOWA,10
  NOHSC,89
  Environment and Heritage,1276
  Australian Greenhouse Office,166
  Bureau of Meteorology,1371
  GBRMPA,162
  National Oceans Office,39
  Family and Community Services,4818
  Aust. Institute of Family Studies,38
  Centrelink,24904
  Finance and Administration,895
  Australian Electoral Comm.,765
  ComSuper,329
  Foreign Affairs and Trade,2809
  ACIAR,40
  Health and Ageing,5365
  Aust. Institute of Health & Welfare,157
  ARPANSA,117
  FSANZ,104
  National Blood Authority,33
  OPSR,27
  DIMIA,4487
  Aboriginal Hostels Ltd.,291
  AIATSIS,59
  ATSIC,24
  ATSIS,1109
  Torres Strait Regional Authority,34
  "Industry, Tourism & Resources",2878
  Prime Minister & Cabinet,372
  ANAO,287
  APS Commission,172
  Commonwealth Ombudsman,82
  Office of National Assessments,51
  Transport & Regional Services,885
  National Capital Authority,68
  Treasury,984
  Australian Bureau of Statistics,2866
  ACCC,450
  ASIC,1137
  Australian Taxation Office,20501
  Productivity Commission,183
  Veterans' Affairs,2356
  Australian War Memorial,196
 
  "Agriculture, Fisheries & Forestry ",1297
  AQIS,3028
  Attorney-Generals ,1011
  Administrative Appeals Tribunal,154
  AUSTRAC,137
  Australian Crime Commission,412
  Australian Customs Service,5054
  Commonwealth DPP,451
  CrimTrac Agency,51
  Family Court of Australia,737
  Federal Court of Australia,403
  Federal Magistrates Service,102
  Federal Privacy Commissioner,40
  HREOC,95
  ITSA,275
  National Native Title Tribunal,265
  Office of Film & Literature Class,41
  Office of the Parl. Counsel,43
  "Communications, IT & the Arts ",804
  Aust. Broadcasting Authority,106
  Aust. Communications Authority,416
  Australian Film Commission,167
  Aust. National Maritime Museum,114
  National Archives of Australia,514
  National Library of Australia,523
  National Museum of Australia,260
  Defence ,18638
  Defence Housing Authority,685
  DEST ,2001
  Australian Research Council,71
  Questacon,211
  DEWR ,3141
  Australian Industrial Registry,202
  Comcare,384
  EOWA,28
  Environment & Heritage ,1852
  Bureau of Meteorology,1442
  GBRMPA,170
  ORER,11
  Family & Community Services ,1949
  Aust. Institute of Family Studies,66
  Finance & Administration ,1297
  Australian Electoral Comm.,808
  Commonwealth Grants Commission,40
  ComSuper,393
  Foreign Affairs & Trade ,2468
  ACIAR,47
  AusAID,615
  Health & Ageing ,4220
  Aust. Institute of Health & Welfare,215
  ARPANSA,130
  FSANZ,121
  National Blood Authority,40
  Professional Services Review,18
  Human Services ,37
  Centrelink,25317
  Child Support Agency,3238
  CRS Australia,1998
  Immigration ,5769
  Aboriginal Hostels Ltd.,458
  AIATSIS,116
  Migration Review Tribunal,132
  Refugee Review Tribunal,176
  Torres Strait Regional Authority,48
  "Industry, Tourism & Resources ",1906
  Geoscience Australia,635
  IP Australia,821
  NOPSA,28
  Prime Minister & Cabinet ,402
  ANAO,313
  Australian Public Service Commission,211
  Commonwealth Ombudsman,113
  National Water Commission,27
  Office of National Assessments,93
  Transport & Regional Services ,1178
  National Capital Authority,91
  Treasury ,885
  Australian Bureau of Statistics,3095
  ACCC,509
  AOFM,33
  ASIC,1494
  Australian Taxation Office,23417
  National Competition Council,11
  Productivity Commission,200
  Royal Australian Mint,117
  Veterans Affairs ,2655
  Australian War Memorial,310
 
  "Agriculture, Fisheries & Forestry",1498
  AQIS,3289
  Attorney-Generals,1226
  Administrative Appeals Tribunal,156
  AUSTRAC,141
  Australian Crime Commission,425
  Australian Customs Service,5574
  CrimTrac Agency,53
  Family Court of Australia,765
  Federal Court of Australia,412
  Federal Magistrates Service,112
  Federal Privacy Commissioner,44
  HREOC,102
  ITSA,270
  National Native Title Tribunal,244
  Commonwealth DPP,469
  Office of Film & Literature Class,49
  Office of the Parl. Counsel,46
  "Communications, IT & the Arts",833
  Australian Communications & Media Authority,497
  Australian Film Commission,255
  Australian National Maritime Museum,107
  Australian Sports Anti-Doping Authority,46
  National Archives of Australia,467
  National Library of Australia,499
  National Museum of Australia,301
  Defence,19791
  Defence Housing Authority,699
  DEST,2359
  AIATSIS,128
  Australian Research Council,69
  Questacon,230
  DEWR,3853
  Australian Building and Construction Commission,85
  Australian Fair Pay Commission,11
  Australian Industrial Registry,175
  Comcare,379
  EOWA,30
  Office of Workplace Services,188
  Environment & Heritage,2060
  Bureau of Meteorology,1404
  GBRMPA,172
  ORER,11
  FaCSIA,2589
  Aboriginal Hostels Ltd.,489
  Australian Institute of Family Studies,53
  Torres Strait Regional Authority,72
  Finance & Administration,1409
  Australian Electoral Commission,790
  Commonwealth Grants Commission,41
  ComSuper,435
  Foreign Affairs & Trade,2551
  ACIAR,50
  AusAID,706
  Health & Ageing,4301
  Australian Institute of Health & Welfare,203
  ARPANSA,136
  FSANZ,125
  National Blood Authority,52
  Professional Services Review,14
  Human Services,5545
  Centrelink,26549
  Medicare Australia,5393
  Immigration,6341
  Migration Review Tribunal,103
  Refugee Review Tribunal,161
  "Industry, Tourism & Resources",1979
  Geoscience Australia,669
  IP AUSTRALIA,888
  NOPSA,34
  Prime Minister & Cabinet,590
  ANAO,308
  Australian Public Service Commission,253
  Commonwealth Ombudsman,141
  National Water Commission,45
  Office of National Assessments,122
  Transport & Regional Services,1327
  National Capital Authority,95
  Treasury,915
  Australian Bureau of Statistics,3511
  ACCC,582
  AOFM,39
  ASIC,1540
  Australian Taxation Office,22568
  National Competition Council,9
  Productivity Commission,205
  Royal Australian Mint,145
  Veterans Affairs,2521
  Australian War Memorial,316
 
  "Agriculture, Fisheries & Forestry",4951
  Attorney-Generals,1504
  Administrative Appeals Tribunal,162
  AUSTRAC,247
  ACLEI,6
  Australian Crime Commission,554
  Australian Customs Service,5904
  CrimTrac Agency,81
  Family Court of Australia,727
  Federal Court of Australia,401
  Federal Magistrates Court,160
  Federal Privacy Commissioner,58
  HREOC,116
  ITSA,285
  National Native Title Tribunal,244
  Commonwealth DPP,497
  Office of the Parl. Counsel,48
  "Communications, IT & the Arts",914
  ACMA,552
  Australian Film Commission,244
  Aust. National Maritime Museum,109
  ASADA,55
  National Archives of Australia,465
  National Library of Australia,521
  National Museum of Australia,282
  Defence,21177
  Defence Housing Australia,699
  DEST,2470
  AIATSIS,114
  Australian Research Council,73
  Questacon,231
  DEWR,3862
  ABCC,128
  Australian Fair Pay Commission,28
  Australian Industrial Registry,175
  Comcare,426
  EOWA,21
  Office of Workplace Services,303
  Environment & Water Resources,2292
  Bureau of Meteorology,1451
  GBRMPA,171
  ORER,11
  FaCSIA,3099
  Aboriginal Hostels Ltd.,491
  Aust. Institute of Family Studies,59
  Torres Strait Regional Authority,76
  Finance & Administration,1514
  Australian Electoral Commission,886
  Commonwealth Grants Commission,41
  ComSuper,595
  Future Fund Management Agency,12
  Foreign Affairs & Trade,2784
  ACIAR,52
  AusAID,775
  Austrade,515
  Health & Ageing,4830
  Aust. Institute of Health & Welfare,211
  ARPANSA,134
  Cancer Australia,26
  FSANZ,131
  National Blood Authority,50
  NHMRC,221
  Professional Services Review,16
  Human Services,6433
  Centrelink,27296
  Medicare Australia,5959
  Immigration,6788
  MRT/RRT,304
  "Industry, Tourism & Resources",2075
  Geoscience Australia,682
  IP AUSTRALIA,913
  NOPSA,42
  Prime Minister & Cabinet,711
  ANAO,330
  Australian Public Service Commission,250
  Commonwealth Ombudsman,153
  National Water Commission,32
  Office of National Assessments,133
  Transport & Regional Services,1330
  National Capital Authority,106
  Treasury,996
  Australian Bureau of Statistics,3167
  ACCC,647
  AOFM,33
  ASIC,1664
  Australian Taxation Office,23110
  National Competition Council,9
  Productivity Commission,207
  Royal Australian Mint,181
  Veterans' Affairs,2401
  Australian War Memorial,293
 
  "Agriculture, Fisheries & Forestry",4785
  APVMA,130
  Export Wheat Commission,15
  Attorney-General's,1420
  Administrative Appeals Tribunal,114
  ACLEI,11
  AUSTRAC,332
  Australian Crime Commission,495
  Australian Customs Service,6106
  CrimTrac Agency,138
  Family Court of Australia,558
  Federal Court of Australia,261
  Federal Magistrates Court,141
  HREOC,93
  ITSA,270
  National Capital Authority,47
  National Native Title Tribunal,217
  Commonwealth DPP,509
  Office of the Parl. Counsel,50
  DBCDE,559
  ACMA,569
  Climate Change,218
  ORER,10
  Defence,20819
  Defence Housing Australia,565
  DEEWR,5619
  ABCC,137
  Australian Fair Pay Commission,25
  Australian Industrial Registry,157
  AIATSIS,48
  Comcare,469
  Office of the Workplace Ombudsman,365
  Workplace Authority,610
  "Environment, Water, Heritage and the Arts",2333
  Australian Film Commission,256
  Aust. National Maritime Museum,105
  Bureau of Meteorology,1265
  GBRMPA,184
  National Library of Australia,444
  National Museum of Australia,226
  National Water Commission,29
  FaHCSIA,2810
  Aboriginal Hostels Ltd.,362
  EOWA,17
  Torres Strait Regional Authority,36
  Finance & Deregulation,1349
  Australian Electoral Comm.,697
  ComSuper,533
  Future Fund Management Agency,38
  National Archives of Australia,390
  Foreign Affairs & Trade,2665
  ACIAR,32
  AusAID,801
  AUSTRADE,485
  Health & Ageing,4445
  Aust. Institute of Health & Welfare,228
  Australian Sports Anti-Doping Authority,60
  ARPANSA,132
  Cancer Australia,22
  FSANZ,118
  National Blood Authority,44
  NHMRC,209
  Private Health Insurance Ombudsman,11
  Professional Services Review,19
  Human Services,5951
  Centrelink,25416
  Medicare Australia,5593
  Immigration,6495
  RRT and MRT,270
  Infrastructure,1225
  DIISR,1656
  Australian Research Council,87
  IP AUSTRALIA,968
  Questacon,82
  Prime Minister & Cabinet,483
  Aust. Institute of Family Studies,42
  ANAO,307
  Australian Public Service Commission,201
  Commonwealth Ombudsman,155
  Federal Privacy Commissioner,61
  OIGIS,9
  Office of National Assessments,96
  "Resources, Energy and Tourism",329
  Geoscience Australia,611
  NOPSA,49
  Treasury,973
  Australian Bureau of Statistics,2858
  ACCC,658
  AOFM,30
  ASIC,1451
  Australian Taxation Office,22486
  Commonwealth Grants Commission,39
  National Competition Council,7
  Productivity Commission,179
  Royal Australian Mint,146
  Veterans' Affairs,2250
  Australian War Memorial,228
 
  "Agriculture, Fisheries & Forestry",5168
  Australian Fisheries Management Authority,260
  APVMA,161
  Wheat Exports Australia,17
  Attorney-General's,1663
  Administrative Appeals Tribunal,155
  ACLEI,18
  AUSTRAC,343
  Australian Crime Commission,527
  Australian Customs and Border Protection Service,6031
  CrimTrac Agency,187
  Family Court of Australia,669
  Federal Court of Australia,376
  Federal Magistrates Court,213
  HREOC,122
  ITSA,294
  National Capital Authority,58
  National Native Title Tribunal,246
  Commonwealth DPP,614
  Office of Parliamentary Counsel,50
  DBCDE,651
  ACMA,624
  Climate Change,415
  ORER,21
  Defence,21152
  Defence Housing Australia,728
  DEEWR,5996
  ABCC,156
  Australian Fair Pay Commission Secretariat,29
  Australian Industrial Registry,203
  Comcare,570
  Office of the Workplace Ombudsman,484
  Workplace Authority,470
  "Environment, Water, Heritage and the Arts",3392
  Australian National Maritime Museum,119
  Bureau of Meteorology,1545
  GBRMPA,241
  Murray Darling Basin Authority,235
  National Library of Australia,513
  National Museum of Australia,295
  National Film and Sound Archive,235
  National Water Commission,48
  Screen Australia,78
  FaHCSIA,3455
  Aboriginal Hostels Ltd.,519
  EOWA,31
  Torres Strait Regional Authority,77
  Finance & Deregulation,1706
  Australian Electoral Commission,805
  ComSuper,574
  Future Fund Management Agency,57
  National Archives of Australia,506
  Foreign Affairs & Trade,2984
  ACIAR,48
  AusAID,981
  AUSTRADE,524
  Health & Ageing,4912
  Australian Institute of Health & Welfare,265
  AOTDTA,15
  Australian Sports Anti-Doping Authority,60
  ARPANSA,152
  Cancer Australia,25
  FSANZ,123
  National Blood Authority,42
  NHMRC,238
  Private Health Insurance Ombudsman,10
  Professional Services Review,30
  Human Services,6620
  Centrelink,27973
  Medicare Australia,5844
  Immigration & Citizenship,7052
  MRT & RRT,273
  Infrastructure,1279
  DIISR,1843
  AIATSIS,119
  Australian Research Council,107
  IP AUSTRALIA,1049
  Questacon,233
  Prime Minister & Cabinet,647
  Australian Institute of Family Studies,75
  ANAO,365
  Australian Public Service Commission,241
  Commonwealth Ombudsman,168
  Federal Privacy Commissioner,64
  OIGIS,13
  Office of National Assessments,148
  Old Parliament House,88
  "Resources, Energy and Tourism",401
  Geoscience Australia,745
  NOPSA,54
  Treasury,1103
  Australian Bureau of Statistics,2871
  ACCC,787
  AOFM,36
  ASIC,1882
  Australian Taxation Office,23505
  Commonwealth Grants Commission,46
  National Competition Council,9
  Productivity Commission,183
  Royal Australian Mint,197
  Veterans' Affairs,2165
  Australian War Memorial,313
 
  National Competition Council,9
  Private Health Insurance Ombudsman,12
  OIGIS,12
  Wheat Exports Australia,13
  Cancer Australia,19
  ACLEI,20
  ORER,25
  Professional Services Review,30
  AOTDTA,31
  AOFM,35
  EOWA,36
  Commonwealth Grants Commission,40
  National Blood Authority,46
  ACIAR,51
  NOPSA,51
  Office of the Parl. Counsel,54
  National Water Commission,57
  Office of the Privacy Commissioner,58
  National Capital Authority,60
  ASADA,69
  Future Fund Management Agency,70
  Aust. Institute of Family Studies,76
  Torres Strait Regional Authority,82
  Old Parliament House,95
  Screen Australia,102
  Australian Transport Safety Bureau,107
  Safe Work Australia,109
  Australian Research Council,109
  Human Rights Commission,124
  AIATSIS,126
  FSANZ,130
  Aust. National Maritime Museum,132
  Office of National Assessments,146
  ABCC,151
  ARPANSA,152
  APVMA,155
  Commonwealth Ombudsman,156
  Administrative Appeals Tribunal,165
  Royal Australian Mint,171
  CrimTrac Agency,185
  Federal Magistrates Court,186
  Productivity Commission,196
  Questacon,218
  National Native Title Tribunal,223
  AFMA,225
  National Film and Sound Archive,232
  GBRMPA,237
  Australian Public Service Commission,247
  NHMRC,249
  MRT & RRT,273
  Australian War Memorial,291
  Fair Work Australia,299
  National Museum of Australia,312
  Murray Darling Basin Authority,314
  AUSTRAC,318
  ITSA,321
  Federal Court of Australia,360
  ANAO,367
  Aust. Institute of Health & Welfare,383
  "Resources, Energy and Tourism",454
  National Archives of Australia,504
  National Library of Australia,540
  Australian Crime Commission,543
  Aboriginal Hostels Ltd.,556
  Commonwealth DPP,558
  AUSTRADE,563
  ComSuper,587
  Comcare,593
  Defence Housing Australia,630
  Family Court of Australia,660
  ACMA,663
  Prime Minister & Cabinet,680
  DBCDE,715
  Geoscience Australia,723
  ACCC,804
  Australian Electoral Commission,922
  Office of the Fair Work Ombudsman,969
  Climate Change and Energy Efficiency,1019
  AusAID,1037
  IP AUSTRALIA,1050
  Infrastructure,1105
  Treasury,1133
  Bureau of Meteorology,1656
  Attorney-General's,1707
  Finance & Deregulation,1841
  DIISR,2072
  ASIC,2080
  Veterans' Affairs,2144
  "Environment, Water, Heritage and the Arts",3115
  Foreign Affairs & Trade,3160
  Australian Bureau of Statistics,3332
  FaHCSIA,3685
  "Agriculture, Fisheries & Forestry",4892
  Health & Ageing,5232
  Medicare Australia,5695
  Customs,5940
  DEEWR,6054
  Human Services,6607
  Immigration & Citizenship,7014
  Defence,21458
  Australian Taxation Office,24070
  Centrelink,27312
 
  "Agriculture, Fisheries and Forestry",5219
  Australian Fisheries Management Authority,227
  APVMA,165
  Wheat Exports Australia,13
  Attorney-General's,1618
  Administrative Appeals Tribunal,162
  ACLEI,27
  AUSTRAC,332
  Australian Crime Commission,635
  Aust. Customs & Border Prot. Service,5767
  Aust. Human Rights Commission,139
  Commonwealth DPP,514
  CrimTrac Agency,181
  Family Court of Australia,670
  Federal Court of Australia,362
  Federal Magistrates Court,195
  ITSA,344
  National Native Title Tribunal,176
  Office of the Parl. Counsel,55
  DBCDE,700
  ACMA,662
  Climate Change and Energy Efficiency,984
  ORER,44
  Defence,22481
  Defence Housing Australia,614
  DEEWR,5468
  ABCC,150
  Comcare,667
  Fair Work Australia,310
  Office of the Fair Work Ombudsman,955
  Safe Work Australia,121
  FaHCSIA,3500
  Aboriginal Hostels Ltd.,563
  EOWA,34
  Torres Strait Regional Authority,107
  Finance and Deregulation,1964
  Australian Electoral Commission,913
  ComSuper,545
  Future Fund Management Agency,80
  Foreign Affairs and Trade,3354
  ACIAR,50
  AusAID,1245
  AUSTRADE,546
  Health and Ageing,5396
  Aust. Institute of Health and Welfare,393
  AOTDTA,33
  ARPANSA,162
  Cancer Australia,22
  FSANZ,133
  National Blood Authority,50
  NHMRC,248
  Private Health Insurance Ombudsman,14
  Professional Services Review,30
  Human Services,6443
  Centrelink,25225
  Medicare Australia,5842
  Immigration and Citizenship,7903
  MRT RRT,283
  Infrastructure,1001
  Australian Transport Safety Bureau,109
  DIISR,3594
  AIATSIS,136
  Australian Research Council,112
  Prime Minister and Cabinet,1140
  Aust. Institute of Family Studies,81
  ANAO,360
  Aust. National Maritime Museum,127
  Australian Public Service Commission,332
  Australian Sports Anti-Doping Authority,67
  Commonwealth Ombudsman,183
  National Archives of Australia,494
  National Library of Australia,494
  National Museum of Australia,287
  National Film and Sound Archive,235
  Office of the Australian Information Commissioner,86
  OIGIS,13
  Office of National Assessments,148
  Old Parliament House,92
  Screen Australia,35
  Regional Australia,336
  National Capital Authority,69
  "Resources, Energy and Tourism",1262
  NOPSA,60
  "Sustainability, Environment, Water, Population and Communities",2949
  Bureau of Meteorology,1775
  GBRMPA,266
  Murray Darling Basin Authority,314
  National Water Commission,74
  Treasury,1397
  Australian Bureau of Statistics,3540
  ACCC,825
  ASIC,1905
  Australian Taxation Office,24902
  National Competition Council,9
  Productivity Commission,203
  Veterans' Affairs,2146
  Australian War Memorial,302
 
  <?php
 
  include_once("../include/common.inc.php");
 
 
  setlocale(LC_CTYPE, 'C');
 
  $headers = Array("#id", "name", "request_email", "short_name", "notes", "publication_scheme", "home_page", "tag_string");
 
  $db = $server->get_db('disclosr-agencies');
  $headers = Array();
  try {
  $rows = $db->get_view("app", "fieldNames?group=true", null, true)->rows;
 
  $dataValues = Array();
  foreach ($rows as $row) {
  $headers[] = $row->key;
  }
  } catch (SetteeRestClientException $e) {
  setteErrorHandler($e);
  }
 
  $fp = fopen('php://output', 'w');
  if ($fp && $db) {
  header('Content-Type: text/csv; charset=utf-8');
  header('Content-Disposition: attachment; filename="export.' . date("c") . '.csv"');
  header('Pragma: no-cache');
  header('Expires: 0');
  fputcsv($fp, $headers);
  try {
  $agencies = $db->get_view("app", "byCanonicalName", null, true)->rows;
  //print_r($rows);
  foreach ($agencies as $agency) {
  // print_r($agency);
 
  if ( !isset($agency->value->status)) {
  $row = Array();
  $agencyArray = object_to_array($agency->value);
  foreach ($headers as $fieldName) {
  if (isset($agencyArray[$fieldName])) {
  if (is_array($agencyArray[$fieldName])) {
  $row[] = implode(";",$agencyArray[$fieldName]);
  } else {
  $row[] = $agencyArray[$fieldName];
  }
  } else {
  $row[] = "";
  }
  }
 
  fputcsv($fp, array_values($row));
 
 
  }
  }
  } catch (SetteeRestClientException $e) {
  setteErrorHandler($e);
  }
 
  die;
  }
  ?>
 
  <?php
 
  include_once("../include/common.inc.php");
 
 
  setlocale(LC_CTYPE, 'C');
 
  $headers = Array("name");
 
  $db = $server->get_db('disclosr-agencies');
  try {
  $rows = $db->get_view("app", "all", null, true)->rows;
 
  $dataValues = Array();
  foreach ($rows as $row) {
  if (isset($row->value->statistics->employees)) {
 
  $headers = array_unique(array_merge($headers, array_keys(object_to_array($row->value->statistics->employees))));
  }
  }
  } catch (SetteeRestClientException $e) {
  setteErrorHandler($e);
  }
 
  $fp = fopen('php://output', 'w');
  if ($fp && $db) {
  header('Content-Type: text/csv; charset=utf-8');
  header('Content-Disposition: attachment; filename="export.employeestats.' . date("c") . '.csv"');
  header('Pragma: no-cache');
  header('Expires: 0');
  fputcsv($fp, $headers);
  try {
  $agencies = $db->get_view("app", "all", null, true)->rows;
  //print_r($agencies);
  foreach ($agencies as $agency) {
 
  if (isset($agency->value->statistics->employees)) {
  $row = Array();
  $agencyEmployeesArray = object_to_array($agency->value->statistics->employees);
  foreach ($headers as $fieldName) {
  if ($fieldName == "name") {
  $row[] = $agency->value->name;
  } else if (isset($agencyEmployeesArray[$fieldName])) {
  $row[] = $agencyEmployeesArray[$fieldName]["value"];
  } else {
  $row[] = 0;
  }
  }
 
  fputcsv($fp, array_values($row));
  }
  }
  } catch (SetteeRestClientException $e) {
  setteErrorHandler($e);
  }
 
  die;
  }
  ?>
 
file:a/admin/import.php (deleted)
<?php  
 
require_once 'include/common.inc.php';  
try {  
$server->create_db('disclosr-agencies');  
} catch (SetteeRestClientException $e) {  
setteErrorHandler($e);  
}  
$db = $server->get_db('disclosr-agencies');  
createAgencyDesignDoc();  
$conn = new PDO("pgsql:dbname=contractDashboard;user=postgres;password=snmc;host=localhost");  
$namesQ = 'select agency.abn, string_agg("agencyName",\'|\') as names from agency inner join agency_nametoabn on agency.abn::text = agency_nametoabn.abn group by agency.abn;';  
$abntonames = Array();  
foreach ($conn->query($namesQ) as $row) {  
$abntonames[$row['abn']] = explode("|", $row['names']);  
}  
$result = $conn->query("select * from agency");  
while ($agency = $result->fetch(PDO::FETCH_ASSOC)) {  
$agency['_id'] = md5($agency['abn']);  
$agency['otherNames'] = $abntonames[$agency['abn']];  
if (sizeof($abntonames[$agency['abn']]) == 1)  
$agency['name'] = $abntonames[$agency['abn']][0];  
$agency["lastScraped"] = "1/1/1970";  
$agency["scrapeDepth"] = 1;  
try {  
$doc = $db->save($agency);  
//print_r($doc);  
echo $agency['abn'] . " imported \n<br>";  
} catch (SetteeRestClientException $e) {  
setteErrorHandler($e);  
}  
}  
?>  
 
  <?php
 
  require_once '../include/common.inc.php';
 
  $db = $server->get_db('disclosr-agencies');
  $rows = $db->get_view("app", "byName")->rows;
  $nametoid = Array();
  $sums = Array();
  foreach ($rows as $row) {
  $nametoid[trim($row->key)] = $row->value;
  }
  $employeeCSVs = Array("2002-2003" => "0203apsemployees.csv",
  "2003-2004" => "0304apsemployees.csv",
  "2004-2005" => "0405apsemployees.csv",
  "2005-2006" => "0506apsemployees.csv",
  "2006-2007" => "0607apsemployees.csv",
  "2007-2008" => "0708apsemployees.csv",
  "2008-2009" => "0809apsemployees.csv",
  "2009-2010" => "0910apsemployees.csv",
  "2010-2011" => "1011apsemployees.csv"
  );
  foreach ($employeeCSVs as $timePeriod => $employeeCSV) {
  echo $employeeCSV . "<br>" . PHP_EOL;
  $row = 1;
  if (($handle = fopen($employeeCSV, "r")) !== FALSE) {
  while (($data = fgetcsv($handle, 1000, ",")) !== FALSE) {
  //print_r($data);
  $name = trim($data[0]);
  if (isset($nametoid[$name])) {
  $id = $nametoid[$name];
  //echo $id . "<br>" . PHP_EOL;
  @$sums[$id][$timePeriod] += $data[1];
  } else {
  echo "<br>ERROR NAME MISSING FROM ID LIST<br><bR>" . PHP_EOL;
 
  die();
 
  }
  }
  fclose($handle);
  }
  }
  foreach ($sums as $id => $sum) {
  echo $id. "<br>" . PHP_EOL;
  $doc = $db->get($id);
  // print_r($doc);
  if (isset($doc->statistics)) $doc->statistics = Array();
  foreach ($sum as $timePeriod => $value) {
  $doc->statistics["employees"][$timePeriod] = Array("value"=>$value, "source"=>"http://apsc.gov.au/stateoftheservice/");
  }
  $db->save($doc);
  }
  // employees: timeperiod, source = apsc state of service, value
  ?>
 
  <?php
 
  require_once '../include/common.inc.php';
  try {
  $server->create_db('disclosr-agencies');
  } catch (SetteeRestClientException $e) {
  setteErrorHandler($e);
  }
  $db = $server->get_db('disclosr-agencies');
  createAgencyDesignDoc();
  $conn = new PDO("pgsql:dbname=contractDashboard;user=postgres;password=snmc;host=localhost");
  $namesQ = 'select agency.abn, string_agg("agencyName",\'|\') as names from agency inner join agency_nametoabn on agency.abn::text = agency_nametoabn.abn group by agency.abn;';
  $abntonames = Array();
  foreach ($conn->query($namesQ) as $row) {
  $abntonames[$row['abn']] = explode("|", $row['names']);
  }
  $result = $conn->query("select * from agency");
  while ($agency = $result->fetch(PDO::FETCH_ASSOC)) {
  $agency['_id'] = md5($agency['abn']);
  $agency['otherNames'] = $abntonames[$agency['abn']];
  if (sizeof($abntonames[$agency['abn']]) == 1)
  $agency['name'] = $abntonames[$agency['abn']][0];
  $agency["lastScraped"] = "1/1/1970";
  $agency["scrapeDepth"] = 1;
  try {
  $doc = $db->save($agency);
  //print_r($doc);
  echo $agency['abn'] . " imported \n<br>";
  } catch (SetteeRestClientException $e) {
  setteErrorHandler($e);
  }
  }
  ?>
 
  <?php
  // http://gold.gov.au/reports/department/index.xml
  require_once '../include/common.inc.php';
  try {
  $server->create_db('disclosr-agencies');
  } catch (SetteeRestClientException $e) {
  setteErrorHandler($e);
  }
  $db = $server->get_db('disclosr-agencies');
  createAgencyDesignDoc();
 
  ?>
 
  <?php
 
  require_once '../include/common.inc.php';
  try {
  $server->create_db('disclosr-agencies');
  } catch (SetteeRestClientException $e) {
  setteErrorHandler($e);
  }
  $db = $server->get_db('disclosr-agencies');
  createAgencyDesignDoc();
 
  // twitter https://docs.google.com/spreadsheet/fm?id=tsJVd9EYoAjbl014y3qMgWg.03918275400592898296.8568379511161083736&hl=en&fmcmd=5&gid=0
  // RSS https://docs.google.com/spreadsheet/fm?id=tbqjwIDHKHiVYF_glJ93GgA.03918275400592898296.8789688748524615194&authkey=CJDP-uQG&hl=en_GB&fmcmd=5&gid=0
  // facebook https://docs.google.com/spreadsheet/fm?id=tkcqoo9wrgzNWmoANuVhsBw.03918275400592898296.3040387705062056060&authkey=CKzl7r0I&hl=en_GB&fmcmd=5&gid=0
  ?>
 
  <?php
 
  require_once '../include/common.inc.php';
  $db = $server->get_db('disclosr-agencies');
  createAgencyDesignDoc();
  ?>
 
  <?php
 
  include_once('../include/common.inc.php');
  include_header();
  // Include the diff class
  echo '<STYLE TYPE="text/css">
  <!--
  @import url(../lib/php-diff/example/styles.css);
  -->
  </STYLE>
  ';
  require_once dirname(__FILE__) . '/../lib/php-diff/lib/Diff.php';
  // Generate a side by side diff
  require_once dirname(__FILE__) . '/../lib/php-diff/lib/Diff/Renderer/Html/SideBySide.php';
  $renderer = new Diff_Renderer_Html_SideBySide;
 
 
 
  $db = $server->get_db('disclosr-agencies');
  $docs = Array();
  try {
  $rows = $db->get_view("app", "getConflicts")->rows;
  //print_r($rows);
  foreach ($rows as $row) {
  echo '<h2>' . $row->id . '</h2>';
  echo "Comparing " . $row->value[0] . " and " . $row->value[1];
  $docA = explode(",", json_encode($db->get($row->id . "?rev=" . $row->value[0])));
  $docB = explode(",", json_encode($db->get($row->id . "?rev=" . $row->value[1])));
  // Options for generating the diff
  $options = array(
  //'ignoreWhitespace' => true,
  //'ignoreCase' => true,
  );
 
  // Initialize the diff class
  $diff = new Diff($docA, $docB, $options);
  echo $diff->Render($renderer);
  }
  } catch (SetteeRestClientException $e) {
  setteErrorHandler($e);
  }
  include_footer();
  ?>
<?php <?php
   
include_once('include/common.inc.php'); include_once('../include/common.inc.php');
include_header(); include_header();
   
$db = $server->get_db('disclosr-agencies'); $db = $server->get_db('disclosr-agencies');
$docs = Array(); $docs = Array();
try { try {
$rows = $db->get_view("app", "byABN")->rows; $rows = $db->get_view("app", "byABN")->rows;
//print_r($rows); //print_r($rows);
foreach ($rows as $row) { foreach ($rows as $row) {
$docs["a" . $row->key] = $row->value; $docs["a" . $row->key] = $row->value;
} }
} catch (SetteeRestClientException $e) { } catch (SetteeRestClientException $e) {
setteErrorHandler($e); setteErrorHandler($e);
} }
//print_r($docs); //print_r($docs);
$row = 1; $row = 1;
if (($handle = fopen("cacfma.csv", "r")) !== FALSE) { if (($handle = fopen("cacfma.csv", "r")) !== FALSE) {
while (($data = fgetcsv($handle, 1000, ",")) !== FALSE) { while (($data = fgetcsv($handle, 1000, ",")) !== FALSE) {
$row++; $row++;
echo $data[0] . " " . str_replace("?", "", $data[1]) . "<br />\n"; echo $data[0] . " " . str_replace("?", "", $data[1]) . "<br />\n";
$name = $data[0]; $name = $data[0];
$abn = trim(str_replace("?", "", $data[1])); $abn = trim(str_replace("?", "", $data[1]));
$aabn = "a".$abn; $aabn = "a".$abn;
if (isset($docs[$aabn])) { if (isset($docs[$aabn])) {
echo "Existing agency ABN detected<br>"; echo "Existing agency ABN detected<br>";
if (!in_array($name, object_to_array($docs[$aabn]->otherNames)) && $name != $docs[$aabn]->name) { if (!in_array($name, object_to_array($docs[$aabn]->otherNames)) && $name != $docs[$aabn]->name) {
$docs[$aabn]->otherNames[] = $name; $docs[$aabn]->otherNames[] = $name;
try { try {
$docs[$aabn] = $db->save($docs[$aabn]); $docs[$aabn] = $db->save($docs[$aabn]);
//print_r($doc); //print_r($doc);
echo $abn . " additional names imported \n<br>"; echo $abn . " additional names imported \n<br>";
} catch (SetteeRestClientException $e) { } catch (SetteeRestClientException $e) {
setteErrorHandler($e); setteErrorHandler($e);
} }
} }
} else { } else {
echo "New agency ABN detected<br>"; echo "New agency ABN detected<br>";
$agency['_id'] = md5($aabn); $agency['_id'] = md5($aabn);
$agency['name'] = $name; $agency['name'] = $name;
$agency["abn"] = $abn; $agency["abn"] = $abn;
try { try {
$doc = $db->save($agency); $doc = $db->save($agency);
print_r($doc); print_r($doc);
echo $abn . " imported \n<br>"; echo $abn . " imported \n<br>";
} catch (SetteeRestClientException $e) { } catch (SetteeRestClientException $e) {
setteErrorHandler($e); setteErrorHandler($e);
} }
} }
echo "<hr>"; echo "<hr>";
} }
fclose($handle); fclose($handle);
} }
include_footer(); include_footer();
?> ?>
  <?php
 
  include_once("../include/common.inc.php");
 
  function shortName($name) {
  $name = trim($name);
  if (strstr($name,"Minister ") || strstr($name,"Treasurer") || strstr($name,"Parliamentary Secretary")) {
  $badWords = Array ("Assisting the Prime Minister on","Assisting on"," the "," of "," for "," on "," and "," to ",","," ","'","`");
  return str_replace($badWords,"",$name);
  }
 
  else {
  $out = Array();
  preg_match_all('/[A-Z]/', $name, $out);
  return implode("", $out[0]);
  }
  }
 
  setlocale(LC_CTYPE, 'C');
 
  $headers = Array("#id", "name", "request_email", "short_name", "notes", "publication_scheme", "home_page", "tag_string");
 
  $db = $server->get_db('disclosr-agencies');
 
  $tag = Array();
  try {
  $rows = $db->get_view("app", "byDeptStateName", null, true)->rows;
  //print_r($rows);
  foreach ($rows as $row) {
  $tag[$row->id] = phrase_to_tag(dept_to_portfolio($row->key));
  }
  } catch (SetteeRestClientException $e) {
  setteErrorHandler($e);
  die();
  }
 
  $foiEmail = Array();
  try {
  $rows = $db->get_view("app", "foiEmails", null, true)->rows;
  //print_r($rows);
  foreach ($rows as $row) {
  $foiEmail[$row->key] = $row->value;
  }
  } catch (SetteeRestClientException $e) {
  setteErrorHandler($e);
  die();
  }
 
  $fp = fopen('php://output', 'w');
  if ($fp && $db) {
  header('Content-Type: text/csv; charset=utf-8');
  header('Content-Disposition: attachment; filename="export.' . date("c") . '.csv"');
  header('Pragma: no-cache');
  header('Expires: 0');
  fputcsv($fp, $headers);
  try {
  $agencies = $db->get_view("app", "byCanonicalName", null, true)->rows;
  //print_r($rows);
  foreach ($agencies as $agency) {
  // print_r($agency);
 
  if (isset($agency->value->foiEmail) && $agency->value->foiEmail != "null" && !isset($agency->value->status)) {
  $row = Array();
  $row["#id"] = $agency->id;
  $row["name"] = trim($agency->value->name);
  if (isset($agency->value->foiEmail)) {
  $row["request_email"] = $agency->value->foiEmail;
  } else {
  if ($agency->value->orgType == "FMA-DepartmentOfState") {
  $row["request_email"] = "foi@" . GetDomain($agency->value->website);
  } else {
  $row["request_email"] = $foiEmail[$agency->value->parentOrg];
  }
  }
  if (isset($agency->value->shortName)) {
  $row["short_name"] = $agency->value->shortName;
  } else {
  $row["short_name"] = shortName($agency->value->name);
  }
  $row["notes"] = "";
  $row["publication_scheme"] = (isset($agency->value->infoPublicationSchemeURL) ? $agency->value->infoPublicationSchemeURL : "");
  $row["home_page"] = (isset($agency->value->website) ? $agency->value->website : "");
  if ($agency->value->orgType == "FMA-DepartmentOfState") {
  $row["tag_string"] = $tag[$agency->value->_id] . " " . $agency->value->orgType;
  } else {
  $row["tag_string"] = $tag[$agency->value->parentOrg] . " " . $agency->value->orgType;
  }
 
  fputcsv($fp, array_values($row));
 
  if (isset($agency->value->foiBodies)) {
  foreach ($agency->value->foiBodies as $foiBody) {
  $row['name'] = iconv("UTF-8", "ASCII//TRANSLIT",$foiBody);
  $row["short_name"] = shortName($foiBody);
  fputcsv($fp, array_values($row));
  }
  }
  }
  }
  } catch (SetteeRestClientException $e) {
  setteErrorHandler($e);
  }
 
  die;
  }
  ?>
 
  <?php
 
  include_once("../include/common.inc.php");
  setlocale(LC_CTYPE, 'C');
  header('Content-Type: text/csv');
  header('Content-Disposition: attachment; filename="public_body_categories_en.rb"');
  header('Pragma: no-cache');
  header('Expires: 0');
  echo 'PublicBodyCategories.add(:en, [' . PHP_EOL;
  echo ' "Portfolios",' . PHP_EOL;
  $db = $server->get_db('disclosr-agencies');
 
  try {
  $rows = $db->get_view("app", "byDeptStateName", null, true)->rows;
  //print_r($rows);
  foreach ($rows as $row) {
  echo ' [ "' . phrase_to_tag(dept_to_portfolio($row->key)) . '","' . dept_to_portfolio($row->key) . '","part of the ' . dept_to_portfolio($row->key) . ' portfolio" ],' . PHP_EOL;
  }
  } catch (SetteeRestClientException $e) {
  setteErrorHandler($e);
  }
  echo '])';
  ?>
 
file:b/charts.php (new)
  <?php
  include_once('include/common.inc.php');
  include_header();
  $db = $server->get_db('disclosr-agencies');
  ?>
  <div class="foundation-header">
  <h1><a href="about.php">Charts</a></h1>
  <h4 class="subheader">Lorem ipsum.</h4>
  </div>
  <div id="placeholder" style="width:900px;height:500px;"></div>
  <script id="source">
  window.onload = function() {
  $(document).ready(function() {
  var d1 = [];
  var labels = [];
  <?php
  try {
  $rows = $db->get_view("app", "scoreHas?group=true", null, true)->rows;
 
 
  $dataValues = Array();
  foreach ($rows as $row) {
  $dataValues[$row->value] = $row->key;
  }
  $i = 0;
  ksort($dataValues);
  foreach ($dataValues as $value => $key) {
 
  echo " d1.push([$i, $value]);" . PHP_EOL;
  echo " labels.push('$key');" . PHP_EOL;
  $i++;
  }
  } catch (SetteeRestClientException $e) {
  setteErrorHandler($e);
  }
  ?>
  function trackformatter(obj) {
  if (labels[Math.floor(obj.x)]) {
  return (labels[Math.floor(obj.x)])+"="+obj.y;
 
  } else {
  return "";
  }
  }
  function tickformatter(val, axis) {
  if (labels[Math.floor(val)]) {
  return '<p style="margin-top:8em;-webkit-transform:rotate(-90deg);">'+(labels[Math.floor(val)])+"</b>";
 
  } else {
  return "";
  }
  }
  Flotr.draw(document.getElementById("placeholder"), [ {data: d1}], {
  HtmlText: true,
  bars : {
  show : true
  },
  mouse : {
  track : true,
  relative : true,
  trackFormatter: trackformatter
  },yaxis: {
  min:0
  },
  xaxis: {
 
  minorTickFreq: 0.6,
  noTicks : 19,
  tickFormatter: tickformatter
  }
  });
 
 
 
  });
  };
 
  </script>
 
  <?php
  include_footer();
  ?>
  <?php
 
  /**
  * Databaase class.
  */
  class SetteeDatabase {
 
  /**
  * Base URL of the CouchDB REST API
  */
  private $conn_url;
 
  /**
  * HTTP REST Client instance
  */
  protected $rest_client;
 
  /**
  * Name of the database
  */
  private $dbname;
 
  /**
  * Default constructor
  */
  function __construct($conn_url, $dbname) {
  $this->conn_url = $conn_url;
  $this->dbname = $dbname;
  $this->rest_client = SetteeRestClient::get_instance($this->conn_url);
  }
 
 
  /**
  * Get UUID from CouchDB
  *
  * @return
  * CouchDB-generated UUID string
  *
  */
  function gen_uuid() {
  $ret = $this->rest_client->http_get('_uuids');
  return $ret['decoded']->uuids[0]; // should never be empty at this point, so no checking
  }
 
  /**
  * Create or update a document database
  *
  * @param $document
  * PHP object, a PHP associative array, or a JSON String representing the document to be saved. PHP Objects and arrays are JSON-encoded automatically.
  *
  * <p>If $document has a an "_id" property set, it will be used as document's unique id (even for "create" operation).
  * If "_id" is missing, CouchDB will be used to generate a UUID.
  *
  * <p>If $document has a "_rev" property (revision), document will be updated, rather than creating a new document.
  * You have to provide "_rev" if you want to update an existing document, otherwise operation will be assumed to be
  * one of creation and you will get a duplicate document exception from CouchDB. Also, you may not provide "_rev" but
  * not provide "_id" since that is an invalid input.
  *
  * @param $allowRevAutoDetection
  * Default: false. When true and _rev is missing from the document, save() function will auto-detect latest revision
  * for a document and use it. This option is "false" by default because it involves an extra http HEAD request and
  * therefore can make save() operation slightly slower if such auto-detection is not required.
  *
  * @return
  * document object with the database id (uuid) and revision attached;
  *
  * @throws SetteeCreateDatabaseException
  */
  function save($document, $allowRevAutoDetection = false) {
  if (is_string($document)) {
  $document = json_decode($document);
  }
 
  // Allow passing of $document as an array (for syntactic simplicity and also because in JSON world it does not matter)
  if(is_array($document)) {
  $document = (object) $document;
  }
 
  if (empty($document->_id) && empty($document->_rev)) {
  $id = $this->gen_uuid();
  }
  elseif (empty($document->_id) && !empty($document->_rev)) {
  throw new SetteeWrongInputException("Error: You can not save a document with a revision provided, but missing id");
  }
  else {
  $id = $document->_id;
 
  if ($allowRevAutoDetection) {
  try {
  $rev = $this->get_rev($id);
  } catch (SetteeRestClientException $e) {
  // auto-detection may fail legitimately, if a document has never been saved before (new doc), so skipping error
  }
  if (!empty($rev)) {
  $document->_rev = $rev;
  }
  }
  }
 
  $full_uri = $this->dbname . "/" . $this->safe_urlencode($id);
  $document_json = json_encode($document, JSON_NUMERIC_CHECK);
 
  $ret = $this->rest_client->http_put($full_uri, $document_json);
 
  $document->_id = $ret['decoded']->id;
  $document->_rev = $ret['decoded']->rev;
 
  return $document;
  }
 
  /**
  * @param $doc
  * @param $name
  * @param $content
  * Content of the attachment in a string-buffer format. This function will automatically base64-encode content for
  * you, so you don't have to do it.
  * @param $mime_type
  * Optional. Will be auto-detected if not provided
  * @return void
  */
  public function add_attachment($doc, $name, $content, $mime_type = null) {
  if (empty($doc->_attachments) || !is_object($doc->_attachments)) {
  $doc->_attachments = new stdClass();
  }
 
  if (empty($mime_type)) {
  $mime_type = $this->rest_client->content_mime_type($content);
  }
 
  $doc->_attachments->$name = new stdClass();
  $doc->_attachments->$name->content_type = $mime_type;
  $doc->_attachments->$name->data = base64_encode($content);
  }
 
  /**
  * @param $doc
  * @param $name
  * @param $file
  * Full path to a file (e.g. as returned by PHP's realpath function).
  * @param $mime_type
  * Optional. Will be auto-detected if not provided
  * @return void
  */
  public function add_attachment_file($doc, $name, $file, $mime_type = null) {
  $content = file_get_contents($file);
  $this->add_attachment($doc, $name, $content, $mime_type);
  }
 
  /**
  *
  * Retrieve a document from CouchDB
  *
  * @throws SetteeWrongInputException
  *
  * @param $id
  * Unique ID (usually: UUID) of the document to be retrieved.
  * @return
  * database document in PHP object format.
  */
  function get($id) {
  if (empty($id)) {
  throw new SetteeWrongInputException("Error: Can't retrieve a document without a uuid.");
  }
 
  $full_uri = $this->dbname . "/" . $this->safe_urlencode($id);
  $full_uri = str_replace("%3Frev%3D","?rev=",$full_uri);
  $ret = $this->rest_client->http_get($full_uri);
  return $ret['decoded'];
  }
 
  /**
  *
  * Get the latest revision of a document with document id: $id in CouchDB.
  *
  * @throws SetteeWrongInputException
  *
  * @param $id
  * Unique ID (usually: UUID) of the document to be retrieved.
  * @return
  * database document in PHP object format.
  */
  function get_rev($id) {
  if (empty($id)) {
  throw new SetteeWrongInputException("Error: Can't query a document without a uuid.");
  }
 
  $full_uri = $this->dbname . "/" . $this->safe_urlencode($id);
  $headers = $this->rest_client->http_head($full_uri);
  if (empty($headers['Etag'])) {
  throw new SetteeRestClientException("Error: could not retrieve revision. Server unexpectedly returned empty Etag");
  }
  $etag = str_replace('"', '', $headers['Etag']);
  return $etag;
  }
 
  /**
  * Delete a document
  *
  * @param $document
  * a PHP object or JSON representation of the document that has _id and _rev fields.
  *
  * @return void
  */
  function delete($document) {
  if (!is_object($document)) {
  $document = json_decode($document);
  }
 
  $full_uri = $this->dbname . "/" . $this->safe_urlencode($document->_id) . "?rev=" . $document->_rev;
  $this->rest_client->http_delete($full_uri);
  }
 
 
  /*----------------- View-related functions --------------*/
 
  /**
  * Create a new view or update an existing one.
  *
  * @param $design_doc
  * @param $view_name
  * @param $map_src
  * Source code of the map function in Javascript
  * @param $reduce_src
  * Source code of the reduce function in Javascript (optional)
  * @return void
  */
  function save_view($design_doc, $view_name, $map_src, $reduce_src = null) {
  $obj = new stdClass();
  $obj->_id = "_design/" . urlencode($design_doc);
  $view_name = urlencode($view_name);
  $obj->views->$view_name->map = $map_src;
  if (!empty($reduce_src)) {
  $obj->views->$view_name->reduce = $reduce_src;
  }
 
  // allow safe updates (even if slightly slower due to extra: rev-detection check).
  return $this->save($obj, true);
  }
 
  /**
  * Create a new view or update an existing one.
  *
  * @param $design_doc
  * @param $view_name
  * @param $key
  * key parameter to a view. Can be a single value or an array (for a range). If passed an array, function assumes
  * that first element is startkey, second: endkey.
  * @param $descending
  * return results in descending order. Please don't forget that if you are using a startkey/endkey, when you change
  * order you also need to swap startkey and endkey values!
  *
  * @return void
  */
  function get_view($design_doc, $view_name, $key = null, $descending = false) {
  $id = "_design/" . urlencode($design_doc);
  $view_name = urlencode($view_name);
  $id .= "/_view/$view_name";
 
  $data = array();
  if (!empty($key)) {
  if (is_string($key)) {
  $data = "key=" . '"' . $key . '"';
  }
  elseif (is_array($key)) {
  list($startkey, $endkey) = $key;
  $data = "startkey=" . '"' . $startkey . '"&' . "endkey=" . '"' . $endkey . '"';
  }
 
  if ($descending) {
  $data .= "&descending=true";
  }
  }
 
 
 
  if (empty($id)) {
  throw new SetteeWrongInputException("Error: Can't retrieve a document without a uuid.");
  }
 
  $full_uri = $this->dbname . "/" . $this->safe_urlencode($id);
  $full_uri = str_replace("%253Fgroup%253Dtrue","?group=true",$full_uri);
  $ret = $this->rest_client->http_get($full_uri, $data);
  return $ret['decoded'];
 
  }
 
  /**
  * @param $id
  * @return
  * return a properly url-encoded id.
  */
  private function safe_urlencode($id) {
  //-- System views like _design can have "/" in their URLs.
  $id = rawurlencode($id);
  if (substr($id, 0, 1) == '_') {
  $id = str_replace('%2F', '/', $id);
  }
  return $id;
  }
 
  /** Getter for a database name */
  function get_name() {
  return $this->dbname;
  }
 
  }
  @echo off
 
  set CLASSPATH="../conf;"
  Echo Cleared classpath
 
  set JAVA_OPTS="-server -Xmx1g"
  Echo set Java Opts %JAVA_OPTS%
 
  set CLASS=com.github.rnewson.couchdb.lucene.Main
  Echo set couchdb-lucene class %CLASS%
 
  setlocal ENABLEDELAYEDEXPANSION
  if defined CLASSPATH (set CLASSPATH=%CLASSPATH%;.) else (set CLASSPATH=.)
  FOR /R ..\lib %%G IN (*.jar) DO set CLASSPATH=!CLASSPATH!;%%G
  Echo The Classpath definition is %CLASSPATH%
 
  java "%JAVA_OPTS%" -cp %CLASSPATH% %CLASS%
 
directory:a/couchdb/settee -> directory:b/couchdb/settee
   
<?php <?php
   
include_once('include/common.inc.php'); include_once('include/common.inc.php');
include_header(); include_header();
   
function displayValue($key, $value, $mode) { function displayValue($key, $value, $mode) {
global $db; global $db, $schemas;
if ($mode == "view") { if ($mode == "view") {
   
  echo "<tr>";
   
  echo "<td>" . $schemas['agency']["properties"][$key]['x-title'] . "<br><small>" . $schemas['agency']["properties"][$key]['description'] . "</small></td><td>";
if (is_array($value)) { if (is_array($value)) {
echo "<tr><td>$key</td><td><ol>"; echo "<ol>";
foreach ($value as $subkey => $subvalue) { foreach ($value as $subkey => $subvalue) {
echo "<li>$subvalue</li>"; if (isset($schemas['agency']["properties"][$key]['x-property'])) {
  echo '<li property="' . $schemas['agency']["properties"][$key]['x-property'] . '">';
  } else {
  echo "<li>";
  }
  echo "$subvalue</li>";
} }
echo "</ol></td></tr>"; echo "</ol></td></tr>";
} else { } else {
echo "<tr><td>$key</td><td>$value</td></tr>"; if (isset($schemas['agency']["properties"][$key]['x-property'])) {
  echo '<span property="' . $schemas['agency']["properties"][$key]['x-property'] . '">';
  } else {
  echo "<span>";
  }
  if ((strpos($key, "URL") > 0 || $key == 'website') && $value != "") {
  echo "<a href='$value'>view</a></span>";
  } else {
  echo "$value</span>";
  }
} }
  echo "</td></tr>";
} }
if ($mode == "edit") { if ($mode == "edit") {
if (is_array($value)) { if (is_array($value)) {
echo '<div class="row"> echo '<div class="row">
<div class="seven columns"> <div class="seven columns">
<fieldset> <fieldset>
<h5>' . $key . '</h5>'; <h5>' . $key . '</h5>';
foreach ($value as $subkey => $subvalue) { foreach ($value as $subkey => $subvalue) {
echo "<label>$subkey</label><input class='input-text' type='text' id='$key$subkey' name='$key" . '[' . $subkey . "]' value='$subvalue'/></tr>"; echo "<label>$subkey</label><input class='input-text' type='text' id='$key$subkey' name='$key" . '[' . $subkey . "]' value='$subvalue'/></tr>";
} }
echo "</fieldset> echo "</fieldset>
</div> </div>
</div>"; </div>";
} else { } else {
if (strpos($key, "_") === 0) { if (strpos($key, "_") === 0) {
echo"<input type='hidden' id='$key' name='$key' value='$value'/>"; echo"<input type='hidden' id='$key' name='$key' value='$value'/>";
} else if ($key == "parentOrg") { } else if ($key == "parentOrg") {
echo "<label for='$key'>$key</label><select id='$key' name='$key'>"; echo "<label for='$key'>$key</label><select id='$key' name='$key'><option value=''> Select... </option>";
$rows = $db->get_view("app", "byDeptStateName")->rows; $rows = $db->get_view("app", "byDeptStateName")->rows;
//print_r($rows); //print_r($rows);
foreach ($rows as $row) { foreach ($rows as $row) {
echo "<option value='{$row->value}'".(($row->value == $value) ? "SELECTED":"")." >".str_replace("Department of ","",$row->key)."</option>"; echo "<option value='{$row->value}'" . (($row->value == $value) ? "SELECTED" : "") . " >" . str_replace("Department of ", "", $row->key) . "</option>";
} }
echo" </select>"; echo" </select>";
} else if (strpos($key, "has") === 0) { } else {
echo "<label for='$key'><input type='checkbox' id='$key' name='$key' ".(($value=='true')?"checked='$value'":"")."> $key</label>";  
} else {  
echo "<label>$key</label><input class='input-text' type='text' id='$key' name='$key' value='$value'/>"; echo "<label>$key</label><input class='input-text' type='text' id='$key' name='$key' value='$value'/>";
if ((strpos($key,"URL") > 0 || $key == 'website')&& $value != "") { if ((strpos($key, "URL") > 0 || $key == 'website') && $value != "") {
echo "<a href='$value'>view</a>"; echo "<a href='$value'>view</a>";
} }
if ($key == 'abn') { if ($key == 'abn') {
echo "<a href='http://www.abr.business.gov.au/SearchByAbn.aspx?SearchText=$value'>view abn</a>"; echo "<a href='http://www.abr.business.gov.au/SearchByAbn.aspx?SearchText=$value'>view abn</a>";
} }
} }
} }
} }
// //
} }
   
function addDefaultFields($row) { function addDefaultFields($row) {
global $schemas; global $schemas;
$defaultFields = array_keys($schemas['agency']['properties']); $defaultFields = array_keys($schemas['agency']['properties']);
foreach ($defaultFields as $defaultField) { foreach ($defaultFields as $defaultField) {
if (!isset($row[$defaultField])) { if (!isset($row[$defaultField])) {
if ($schemas['agency']['properties'][$defaultField]['type'] == "string") { if ($schemas['agency']['properties'][$defaultField]['type'] == "string") {
if (strpos($defaultField, "has") === 0) {  
$row[$defaultField] = "false"; $row[$defaultField] = "";
   
} else {  
$row[$defaultField] = "";  
}  
} }
if ($schemas['agency']['properties'][$defaultField]['type'] == "array") { if ($schemas['agency']['properties'][$defaultField]['type'] == "array") {
   
$row[$defaultField] = Array(""); $row[$defaultField] = Array("");
} }
} }
} }
return $row; return $row;
} }
   
$db = $server->get_db('disclosr-agencies'); $db = $server->get_db('disclosr-agencies');
   
if (isset($_REQUEST['id'])) { if (isset($_REQUEST['id'])) {
//get an agency record as json/html, search by name/abn/id //get an agency record as json/html, search by name/abn/id
// by name = startkey="Ham"&endkey="Ham\ufff0" // by name = startkey="Ham"&endkey="Ham\ufff0"
// edit? // edit?
   
$row = $db->get($_REQUEST['id']); $row = $db->get($_REQUEST['id']);
//print_r($row); //print_r($row);
if (sizeof($_POST) > 0) { if (sizeof($_POST) > 0) {
//print_r($_POST); //print_r($_POST);
  foreach ($_POST as $postkey => $postvalue) {
  if ($postvalue == "") {
  unset($_POST[$postkey]);
  }
  if (is_array($postvalue) && count($postvalue) == 1 && $postvalue[0] == "") {
  unset($_POST[$postkey]);
  }
  }
if (isset($_POST['_id']) && $db->get_rev($_POST['_id']) == $_POST['_rev']) { if (isset($_POST['_id']) && $db->get_rev($_POST['_id']) == $_POST['_rev']) {
echo "Edited version was latest version, continue saving"; echo "Edited version was latest version, continue saving";
$newdoc = $_POST; $newdoc = $_POST;
$newdoc['metadata']['lastModified'] = time(); $newdoc['metadata']['lastModified'] = time();
$row = $db->save($newdoc); $row = $db->save($newdoc);
} else { } else {
echo "ALERT doc revised by someone else while editing."; echo "ALERT doc revised by someone else while editing. Document not saved.";
} }
} }
   
$mode = "edit"; $mode = "edit";
$row = addDefaultFields(object_to_array($row)); if ($mode == "edit") {
  $row = addDefaultFields(object_to_array($row));
  } else {
  $row = object_to_array($row);
  }
   
if ($mode == "view") { if ($mode == "view") {
echo '<table width="100%">'; echo '<div typeof="schema:GovernmentOrganisation" about="#' . $row['_id'] . '"><table width="100%">';
echo '<tr> <td colspan="2"><h3>' . $row['name'] . "</h3></td></tr>"; echo '<tr> <td colspan="2"><h3>' . $row['name'] . "</h3></td></tr>";
echo "<tr><th>Field Name</th><th>Field Value</th></tr>"; echo "<tr><th>Field Name</th><th>Field Value</th></tr>";
} }
if ($mode == "edit") { if ($mode == "edit") {
?> ?>
<input id="addfield" type="button" value="Add Field"/> <input id="addfield" type="button" value="Add Field"/>
<script> <script>
window.onload = function() { window.onload = function() {
$(document).ready(function() { $(document).ready(function() {
// put all your jQuery goodness in here. // put all your jQuery goodness in here.
// http://charlie.griefer.com/blog/2009/09/17/jquery-dynamically-adding-form-elements/ // http://charlie.griefer.com/blog/2009/09/17/jquery-dynamically-adding-form-elements/
$('#addfield').click(function() { $('#addfield').click(function() {
var field_name=window.prompt("fieldname?",""); var field_name=window.prompt("fieldname?","");
if (field_name !="") { if (field_name !="") {
$('#submitbutton').before($('<span></span>') $('#submitbutton').before($('<span></span>')
.append("<label>"+field_name+"</label>") .append("<label>"+field_name+"</label>")
.append("<input class='input-text' type='text' id='"+field_name+"' name='"+field_name+"'/>") .append("<input class='input-text' type='text' id='"+field_name+"' name='"+field_name+"'/>")
); );
} }
}); });
}); });
}; };
</script> </script>
<form id="editform" class="nice" method="post"> <form id="editform" class="nice" method="post">
<?php <?php
   
  }
  foreach ($row as $key => $value) {
  echo displayValue($key, $value, $mode);
  }
  if ($mode == "view") {
  echo "</table></div>";
  }
  if ($mode == "edit") {
  echo '<input id="submitbutton" type="submit"/></form>';
  }
  } else {
   
  try {
  /* $rows = $db->get_view("app", "showNamesABNs")->rows;
  //print_r($rows);
  foreach ($rows as $row) {
  // print_r($row);
  echo '<li><a href="getAgency.php?id=' . $row->key . '">' .
  (isset($row->value->name) && $row->value->name != "" ? $row->value->name : "NO NAME " . $row->value->abn)
  . '</a></li>';
  } */
  $rows = $db->get_view("app", "byName")->rows;
  //print_r($rows);
  echo '<ul>';
  foreach ($rows as $row) {
  // print_r($row);
  echo '<li typeof="schema:GovernmentOrganisation foaf:Organization" about="getAgency.php?id=' . $row->value . '">
  <a href="getAgency.php?id=' . $row->value . '" rel="schema:url foaf:page" property="schema:name foaf:name">' .
  $row->key
  . '</a></li>';
} }
foreach ($row as $key => $value) { echo "</ul>";
echo displayValue($key, $value, $mode); } catch (SetteeRestClientException $e) {
} setteErrorHandler($e);
if ($mode == "view") { }
echo "</table>"; }
} include_footer();
if ($mode == "edit") { ?>
echo '<input id="submitbutton" type="submit"/></form>';  
}  
} else {  
   
try {  
/*$rows = $db->get_view("app", "showNamesABNs")->rows;  
//print_r($rows);  
foreach ($rows as $row) {  
// print_r($row);  
echo '<li><a href="getAgency.php?id=' . $row->key . '">' .  
(isset($row->value->name) && $row->value->name != "" ? $row->value->name : "NO NAME " . $row->value->abn)  
. '</a></li>';  
}*/  
$rows = $db->get_view("app", "byName")->rows;  
//print_r($rows);  
foreach ($rows as $row) {  
// print_r($row);  
echo '<li><a href="getAgency.php?id=' . $row->value . '">' .  
$row->key  
. '</a></li>';  
}  
} catch (SetteeRestClientException $e) {  
setteErrorHandler($e);  
}  
}  
include_footer();  
?>  
file:a/graph.php -> file:b/graph.php
<?php <?php
include_once('include/common.inc.php'); include_once('include/common.inc.php');
//include_header(); //include_header();
?> $format = "html";
<script src="http://ajax.googleapis.com/ajax/libs/jquery/1.3.2/jquery.min.js"></script> if (isset($_REQUEST['format'])) {
<script src="lib/springy/springy.js"></script> $format = $_REQUEST['format'];
<script src="lib/springy/springyui.js"></script> }
<script>  
var graph = new Graph(); function add_node($id, $label) {
var nodes = []; global $format;
<?php if ($format == "html") {
  echo "nodes[\"$id\"] = graph.newNode({label: \"$label\"});" . PHP_EOL;
  }
  if ($format == "dot" && $label != "") {
  echo "$id [label=\"$label\"];". PHP_EOL;
  }
  }
   
  function add_edge($from, $to, $color) {
  global $format;
  if ($format == "html") {
  echo "graph.newEdge(nodes[\"$from\"], nodes['$to'], {color: '$color'});" . PHP_EOL;
  }
  if ($format == "dot") {
  echo "$from -> $to ".($color != ""? "[color=$color]":"").";". PHP_EOL;
  }
  }
   
  if ($format == "html") {
  ?>
  <script src="http://ajax.googleapis.com/ajax/libs/jquery/1.3.2/jquery.min.js"></script>
  <script src="lib/springy/springy.js"></script>
  <script src="lib/springy/springyui.js"></script>
  <script>
  var graph = new Graph();
  var nodes = [];
  <?php
  }
  if ($format == "dot") {
  echo 'digraph g {'. PHP_EOL;
  }
$db = $server->get_db('disclosr-agencies'); $db = $server->get_db('disclosr-agencies');
echo "nodes[\"fedg\"] = graph.newNode({label: \"Federal Government - Commonwealth of Australia\"});" . PHP_EOL; add_node("fedg","Federal Government - Commonwealth of Australia");
try { try {
$rows = $db->get_view("app", "byCanonicalName", null, true)->rows; $rows = $db->get_view("app", "byCanonicalName", null, true)->rows;
//print_r($rows); //print_r($rows);
foreach ($rows as $row) { foreach ($rows as $row) {
echo "nodes[\"{$row->value}\"] = graph.newNode({label: \"{$row->key}\"});" . PHP_EOL; add_node($row->id, $row->key);
} }
} catch (SetteeRestClientException $e) { } catch (SetteeRestClientException $e) {
setteErrorHandler($e); setteErrorHandler($e);
} }
   
try { try {
$rows = $db->get_view("app", "byDeptStateName", null, true)->rows; $rows = $db->get_view("app", "byDeptStateName", null, true)->rows;
//print_r($rows); //print_r($rows);
foreach ($rows as $row) { foreach ($rows as $row) {
echo "graph.newEdge(nodes[\"fedg\"], nodes['{$row->value}'], {color: '#00A0B0'});" . PHP_EOL; add_edge("fedg", $row->value, 'yellow');
} }
} catch (SetteeRestClientException $e) { } catch (SetteeRestClientException $e) {
setteErrorHandler($e); setteErrorHandler($e);
} }
   
try { try {
$rows = $db->get_view("app", "parentOrgs", null, true)->rows; $rows = $db->get_view("app", "parentOrgs", null, true)->rows;
// print_r($rows); // print_r($rows);
foreach ($rows as $row) { foreach ($rows as $row) {
echo "graph.newEdge(nodes[\"{$row->key}\"], nodes['{$row->value}'], {color: '#FFA0B0'});" . PHP_EOL; add_edge($row->key, $row->value, 'blue');
} }
} catch (SetteeRestClientException $e) { } catch (SetteeRestClientException $e) {
setteErrorHandler($e); setteErrorHandler($e);
} }
?> if ($format == "html") {
window.onload = function() { ?>
$(document).ready(function() { window.onload = function() {
var springy = $('#springydemo').springy({ $(document).ready(function() {
graph: graph var springy = $('#springydemo').springy({
  graph: graph
  });
}); });
}); };
}; </script>
</script>  
   
<canvas id="springydemo" width="1260" height="680" /> <canvas id="springydemo" width="1260" height="680" />
<?php <?php
  }
  if ($format == "dot") {
  echo "}";
  }
//include_footer(); //include_footer();
?> ?>
   
   
<?php <?php
   
  date_default_timezone_set("Australia/Sydney");
   
  $basePath = "";
  if (strstr($_SERVER['PHP_SELF'], "alaveteli/")
  || strstr($_SERVER['PHP_SELF'], "admin/")
  || strstr($_SERVER['PHP_SELF'], "lib/")
  || strstr($_SERVER['PHP_SELF'], "include/"))
  $basePath = "../";
   
include_once ('couchdb.inc.php'); include_once ('couchdb.inc.php');
include_once ('template.inc.php'); include_once ('template.inc.php');
  require_once $basePath.'lib/Requests/library/Requests.php';
   
  Requests::register_autoloader();
# Convert a stdClass to an Array. http://www.php.net/manual/en/language.types.object.php#102735 # Convert a stdClass to an Array. http://www.php.net/manual/en/language.types.object.php#102735
   
function object_to_array(stdClass $Class) { function object_to_array(stdClass $Class) {
# Typecast to (array) automatically converts stdClass -> array. # Typecast to (array) automatically converts stdClass -> array.
$Class = (array) $Class; $Class = (array) $Class;
   
# Iterate through the former properties looking for any stdClass properties. # Iterate through the former properties looking for any stdClass properties.
# Recursively apply (array). # Recursively apply (array).
foreach ($Class as $key => $value) { foreach ($Class as $key => $value) {
if (is_object($value) && get_class($value) === 'stdClass') { if (is_object($value) && get_class($value) === 'stdClass') {
$Class[$key] = object_to_array($value); $Class[$key] = object_to_array($value);
} }
} }
return $Class; return $Class;
} }
   
# Convert an Array to stdClass. http://www.php.net/manual/en/language.types.object.php#102735 # Convert an Array to stdClass. http://www.php.net/manual/en/language.types.object.php#102735
   
function array_to_object(array $array) { function array_to_object(array $array) {
# Iterate through our array looking for array values. # Iterate through our array looking for array values.
# If found recurvisely call itself. # If found recurvisely call itself.
foreach ($array as $key => $value) { foreach ($array as $key => $value) {
if (is_array($value)) { if (is_array($value)) {
$array[$key] = array_to_object($value); $array[$key] = array_to_object($value);
} }
} }
   
# Typecast to (object) will automatically convert array -> stdClass # Typecast to (object) will automatically convert array -> stdClass
return (object) $array; return (object) $array;
} }
?>  
   
  function dept_to_portfolio($deptName) {
  return trim(str_replace("Department of", "", str_replace("Department of the", "Department of", $deptName)));
  }
  function phrase_to_tag ($phrase) {
  return str_replace(" ","_",str_replace("'","",str_replace(",","",strtolower($phrase))));
  }
  function GetDomain($url)
  {
  $nowww = ereg_replace('www\.','',$url);
  $domain = parse_url($nowww);
  if(!empty($domain["host"]))
  {
  return $domain["host"];
  } else
  {
  return $domain["path"];
  }
  }
   
   
<?php <?php
   
include "schemas/schemas.inc.php"; include $basePath . "schemas/schemas.inc.php";
   
  require ($basePath . 'couchdb/settee/src/settee.php');
   
  function createDocumentsDesignDoc() {
  /*"views": {
  "web_server": {
  "map": "function(doc) {\n emit(doc.web_server, 1);\n}",
  "reduce": "function (key, values, rereduce) {\n return sum(values);\n}"
  },
  "byAgency": {
  "map": "function(doc) {\n emit(doc.agencyID, 1);\n}",
  "reduce": "function (key, values, rereduce) {\n return sum(values);\n}"
  },
  "byURL": {
  "map": "function(doc) {\n emit(doc.url, doc);\n}"
  },
  "agency": {
  "map": "function(doc) {\n emit(doc.agencyID, doc);\n}"
  },
  "byWebServer": {
  "map": "function(doc) {\n emit(doc.web_server, doc);\n}"
  }
  }*/
  }
   
function createAgencyDesignDoc() { function createAgencyDesignDoc() {
global $db; global $db;
$obj = new stdClass(); $obj = new stdClass();
$obj->_id = "_design/" . urlencode("app"); $obj->_id = "_design/" . urlencode("app");
$obj->language = "javascript"; $obj->language = "javascript";
$obj->views->all->map = "function(doc) { emit(doc._id, doc); };"; $obj->views->all->map = "function(doc) { emit(doc._id, doc); };";
$obj->views->byABN->map = "function(doc) { emit(doc.abn, doc); };"; $obj->views->byABN->map = "function(doc) { emit(doc.abn, doc); };";
$obj->views->byCanonicalName->map = "function(doc) { $obj->views->byCanonicalName->map = "function(doc) {
if (doc.parentOrg || doc.orgType == 'FMA-DepartmentOfState') { if (doc.parentOrg || doc.orgType == 'FMA-DepartmentOfState') {
emit(doc.name, doc._id); emit(doc.name, doc);
} }
};"; };";
$obj->views->byDeptStateName->map = "function(doc) { $obj->views->byDeptStateName->map = "function(doc) {
if (doc.orgType == 'FMA-DepartmentOfState') { if (doc.orgType == 'FMA-DepartmentOfState') {
emit(doc.name, doc._id); emit(doc.name, doc._id);
} }
};"; };";
$obj->views->parentOrgs->map = "function(doc) { $obj->views->parentOrgs->map = "function(doc) {
if (doc.parentOrg) { if (doc.parentOrg) {
emit(doc._id, doc.parentOrg); emit(doc._id, doc.parentOrg);
} }
};"; };";
$obj->views->byName->map = "function(doc) { $obj->views->byName->map = 'function(doc) {
  if (typeof(doc["status"]) == "undefined" || doc["status"] != "suspended") {
emit(doc.name, doc._id); emit(doc.name, doc._id);
  if (typeof(doc.shortName) != "undefined" && doc.shortName != doc.name) {
  emit(doc.shortName, doc._id);
  }
for (name in doc.otherNames) { for (name in doc.otherNames) {
if (doc.otherNames[name] != '' && doc.otherNames[name] != doc.name) { if (doc.otherNames[name] != "" && doc.otherNames[name] != doc.name) {
emit(doc.otherNames[name], doc._id); emit(doc.otherNames[name], doc._id);
} }
} }
  for (name in doc.foiBodies) {
  if (doc.foiBodies[name] != "" && doc.foiBodies[name] != doc.name) {
  emit(doc.foiBodies[name], doc._id);
  }
  }
  }
  };';
   
  $obj->views->foiEmails->map = "function(doc) {
  emit(doc._id, doc.foiEmail);
};"; };";
   
$obj->views->byLastModified->map = "function(doc) { emit(doc.metadata.lastModified, doc); }"; $obj->views->byLastModified->map = "function(doc) { emit(doc.metadata.lastModified, doc); }";
$obj->views->getActive->map = 'function(doc) { if (doc.status == "active") { emit(doc._id, doc); } };'; $obj->views->getActive->map = 'function(doc) { if (doc.status == "active") { emit(doc._id, doc); } };';
$obj->views->getSuspended->map = 'function(doc) { if (doc.status == "suspended") { emit(doc._id, doc); } };'; $obj->views->getSuspended->map = 'function(doc) { if (doc.status == "suspended") { emit(doc._id, doc); } };';
$obj->views->getScrapeRequired->map = "function(doc) { emit(doc.abn, doc); };"; $obj->views->getScrapeRequired->map = "function(doc) {
   
  var lastScrape = Date.parse(doc.metadata.lastScraped);
   
  var today = new Date();
   
  if (!lastScrape || lastScrape.getTime() + 1000 != today.getTime()) {
  emit(doc._id, doc);
  }
   
  };";
$obj->views->showNamesABNs->map = "function(doc) { emit(doc._id, {name: doc.name, abn: doc.abn}); };"; $obj->views->showNamesABNs->map = "function(doc) { emit(doc._id, {name: doc.name, abn: doc.abn}); };";
$obj->views->getConflicts->map = "function(doc) { $obj->views->getConflicts->map = "function(doc) {
if (doc._conflicts) { if (doc._conflicts) {
emit(null, [doc._rev].concat(doc._conflicts)); emit(null, [doc._rev].concat(doc._conflicts));
} }
}"; }";
// http://stackoverflow.com/questions/646628/javascript-startswith // http://stackoverflow.com/questions/646628/javascript-startswith
$obj->views->score->map = 'if(!String.prototype.startsWith){ $obj->views->scoreHas->map = 'if(!String.prototype.startsWith){
String.prototype.startsWith = function (str) { String.prototype.startsWith = function (str) {
return !this.indexOf(str); return !this.indexOf(str);
} }
} }
  if(!String.prototype.endsWith){
  String.prototype.endsWith = function(suffix) {
      return this.indexOf(suffix, this.length - suffix.length) !== -1;
  };
  }
function(doc) { function(doc) {
count = 0;  
if (typeof(doc["status"]) == "undefined" || doc["status"] != "suspended") { if (typeof(doc["status"]) == "undefined" || doc["status"] != "suspended") {
for(var propName in doc) { for(var propName in doc) {
if(typeof(doc[propName]) != "undefined" && propName.startsWith("l")) { if(typeof(doc[propName]) != "undefined" && (propName.startsWith("has") || propName.endsWith("URL"))) {
count++ emit(propName, 1);
} }
} }
emit(count+doc._id, {id:doc._id, name: doc.name, score:count}); emit("total", 1);
} }
}'; }';
  $obj->views->scoreHas->map = 'if(!String.prototype.startsWith){
  String.prototype.startsWith = function (str) {
  return !this.indexOf(str);
  }
  }
  if(!String.prototype.endsWith){
  String.prototype.endsWith = function(suffix) {
      return this.indexOf(suffix, this.length - suffix.length) !== -1;
  };
  }
  function(doc) {
  if (typeof(doc["status"]) == "undefined" || doc["status"] != "suspended") {
  for(var propName in doc) {
  if(typeof(doc[propName]) != "undefined" && (propName.startsWith("has") || propName.endsWith("URL"))) {
  emit(propName, 1);
  }
  }
  emit("total", 1);
  }
  }';
  $obj->views->scoreHas->reduce = 'function (key, values, rereduce) {
  return sum(values);
  }';
  $obj->views->fieldNames->map = '
  function(doc) {
  for(var propName in doc) {
  emit(propName, doc._id);
  }
   
  }';
  $obj->views->fieldNames->reduce = 'function (key, values, rereduce) {
  return values.length;
  }';
// allow safe updates (even if slightly slower due to extra: rev-detection check). // allow safe updates (even if slightly slower due to extra: rev-detection check).
return $db->save($obj, true); return $db->save($obj, true);
} }
   
require ('couchdb/settee/src/settee.php'); if (php_uname('n') == "vanille") {
  $serverAddr = 'http://192.168.178.21:5984/';
   
  } else
  if (php_uname('n') == "KYUUBEY") {
   
if( php_uname('n') == "vanille") { $serverAddr = 'http://192.168.1.148:5984/';
   
$server = new SetteeServer('http://192.168.178.21:5984');  
} else  
if( php_uname('n') == "kyuubey") {  
   
$server = new SetteeServer('http://192.168.1.8:5984');  
} else { } else {
$server = new SetteeServer('http://127.0.0.1:5984'); $serverAddr = 'http://127.0.0.1:5984/';
} }
  $server = new SetteeServer($serverAddr);
function setteErrorHandler($e) { function setteErrorHandler($e) {
echo $e->getMessage() . "<br>" . PHP_EOL; echo $e->getMessage() . "<br>" . PHP_EOL;
} }
   
?>  
   
<?php <?php
   
function include_header() { function include_header() {
  global $basePath;
?> ?>
<!DOCTYPE html> <!DOCTYPE html>
   
<!-- paulirish.com/2008/conditional-stylesheets-vs-css-hacks-answer-neither/ --> <!-- paulirish.com/2008/conditional-stylesheets-vs-css-hacks-answer-neither/ -->
<!--[if lt IE 7]> <html class="no-js lt-ie9 lt-ie8 lt-ie7" lang="en"> <![endif]--> <!--[if lt IE 7]> <html class="no-js lt-ie9 lt-ie8 lt-ie7" lang="en"> <![endif]-->
<!--[if IE 7]> <html class="no-js lt-ie9 lt-ie8" lang="en"> <![endif]--> <!--[if IE 7]> <html class="no-js lt-ie9 lt-ie8" lang="en"> <![endif]-->
<!--[if IE 8]> <html class="no-js lt-ie9" lang="en"> <![endif]--> <!--[if IE 8]> <html class="no-js lt-ie9" lang="en"> <![endif]-->
<!--[if gt IE 8]><!--> <html lang="en"> <!--<![endif]--> <!--[if gt IE 8]><!--> <html lang="en"> <!--<![endif]-->
<head> <head>
<meta charset="utf-8" /> <meta charset="utf-8" />
   
<!-- Set the viewport width to device width for mobile --> <!-- Set the viewport width to device width for mobile -->
<meta name="viewport" content="width=device-width" /> <meta name="viewport" content="width=device-width" />
   
<title>Disclosr</title> <title>Disclosr</title>
   
<!-- Included CSS Files --> <!-- Included CSS Files -->
<link rel="stylesheet" href="stylesheets/foundation.css"> <link rel="stylesheet" href="<?php echo $basePath ?>stylesheets/foundation.css">
<link rel="stylesheet" href="stylesheets/app.css"> <link rel="stylesheet" href="<?php echo $basePath ?>stylesheets/app.css">
   
<!--[if lt IE 9]> <!--[if lt IE 9]>
<link rel="stylesheet" href="stylesheets/ie.css"> <link rel="stylesheet" href="<?php echo $basePath ?>stylesheets/ie.css">
<![endif]--> <![endif]-->
   
   
<!-- IE Fix for HTML5 Tags --> <!-- IE Fix for HTML5 Tags -->
<!--[if lt IE 9]> <!--[if lt IE 9]>
<script src="http://html5shiv.googlecode.com/svn/trunk/html5.js"></script> <script src="http://html5shiv.googlecode.com/svn/trunk/html5.js"></script>
<![endif]--> <![endif]-->
   
</head> </head>
<body> <body xmlns:schema="http://schema.org/" xmlns:foaf="http://xmlns.com/foaf/0.1/">
   
<!-- navBar --> <!-- navBar -->
<div id="navbar" class="container"> <div id="navbar" class="container">
<div class="row"> <div class="row">
<div class="four columns"> <div class="four columns">
<h1><a href="/">Disclosr</a></h1> <h1><a href="/">Disclosr</a></h1>
</div> </div>
<div class="eight columns hide-on-phones"> <div class="eight columns hide-on-phones">
<strong class="right"> <strong class="right">
<a href="getAgency.php">Agencies</a> <a href="getAgency.php">Agencies</a>
<a href="about.php">About/FAQ</a> <a href="about.php">About/FAQ</a>
</strong> </strong>
</div> </div>
</div> </div>
</div> </div>
<!-- /navBar --> <!-- /navBar -->
   
<!-- container --> <!-- container -->
<div class="container"> <div class="container">
<?php } <?php }
   
function include_footer() { ?> function include_footer() {
  global $basePath;
  ?>
   
</div> </div>
<!-- container --> <!-- container -->
   
   
   
   
<!-- Included JS Files --> <!-- Included JS Files -->
<script src="javascripts/foundation.js"></script> <script src="<?php echo $basePath; ?>javascripts/foundation.js"></script>
<script src="javascripts/app.js"></script> <script src="<?php echo $basePath; ?>javascripts/app.js"></script>
<script src="http://code.jquery.com/jquery-1.7.1.min.js"></script> <script src="http://code.jquery.com/jquery-1.7.1.min.js"></script>
   
  <script type="text/javascript" src="javascripts/flotr2/flotr2.js"></script>
   
</body> </body>
</html> </html>
   
<?php } <?php }
   
?>  
   
/* Foundation v2.1.4 http://foundation.zurb.com */ /* Foundation v2.1.4 http://foundation.zurb.com */
$(document).ready(function () { $(document).ready(function () {
   
/* Use this js doc for all application specific JS */ /* Use this js doc for all application specific JS */
   
/* TABS --------------------------------- */ /* TABS --------------------------------- */
/* Remove if you don't need :) */ /* Remove if you don't need :) */
   
function activateTab($tab) { function activateTab($tab) {
var $activeTab = $tab.closest('dl').find('a.active'), var $activeTab = $tab.closest('dl').find('a.active'),
contentLocation = $tab.attr("href") + 'Tab'; contentLocation = $tab.attr("href") + 'Tab';
   
//Make Tab Active //Make Tab Active
$activeTab.removeClass('active'); $activeTab.removeClass('active');
$tab.addClass('active'); $tab.addClass('active');
   
//Show Tab Content //Show Tab Content
$(contentLocation).closest('.tabs-content').children('li').hide(); $(contentLocation).closest('.tabs-content').children('li').hide();
$(contentLocation).show(); $(contentLocation).show();
} }
   
$('dl.tabs').each(function () { $('dl.tabs').each(function () {
//Get all tabs //Get all tabs
var tabs = $(this).children('dd').children('a'); var tabs = $(this).children('dd').children('a');
tabs.click(function (e) { tabs.click(function (e) {
activateTab($(this)); activateTab($(this));
}); });
}); });
   
if (window.location.hash) { if (window.location.hash) {
activateTab($('a[href="' + window.location.hash + '"]')); activateTab($('a[href="' + window.location.hash + '"]'));
} }
   
/* ALERT BOXES ------------ */ /* ALERT BOXES ------------ */
$(".alert-box").delegate("a.close", "click", function(event) { $(".alert-box").delegate("a.close", "click", function(event) {
event.preventDefault(); event.preventDefault();
$(this).closest(".alert-box").fadeOut(function(event){ $(this).closest(".alert-box").fadeOut(function(event){
$(this).remove(); $(this).remove();
}); });
}); });
   
   
/* PLACEHOLDER FOR FORMS ------------- */ /* PLACEHOLDER FOR FORMS ------------- */
/* Remove this and jquery.placeholder.min.js if you don't need :) */ /* Remove this and jquery.placeholder.min.js if you don't need :) */
   
$('input, textarea').placeholder(); //$('input, textarea').placeholder();
   
   
   
/* UNCOMMENT THE LINE YOU WANT BELOW IF YOU WANT IE6/7/8 SUPPORT AND ARE USING .block-grids */ /* UNCOMMENT THE LINE YOU WANT BELOW IF YOU WANT IE6/7/8 SUPPORT AND ARE USING .block-grids */
// $('.block-grid.two-up>li:nth-child(2n+1)').css({clear: 'left'}); // $('.block-grid.two-up>li:nth-child(2n+1)').css({clear: 'left'});
// $('.block-grid.three-up>li:nth-child(3n+1)').css({clear: 'left'}); // $('.block-grid.three-up>li:nth-child(3n+1)').css({clear: 'left'});
// $('.block-grid.four-up>li:nth-child(4n+1)').css({clear: 'left'}); // $('.block-grid.four-up>li:nth-child(4n+1)').css({clear: 'left'});
// $('.block-grid.five-up>li:nth-child(5n+1)').css({clear: 'left'}); // $('.block-grid.five-up>li:nth-child(5n+1)').css({clear: 'left'});
   
   
   
/* DROPDOWN NAV ------------- */ /* DROPDOWN NAV ------------- */
   
var currentFoundationDropdown = null; var currentFoundationDropdown = null;
$('.nav-bar li a, .nav-bar li a:after').each(function() { $('.nav-bar li a, .nav-bar li a:after').each(function() {
$(this).data('clicks', 0); $(this).data('clicks', 0);
}); });
$('.nav-bar li a, .nav-bar li a:after').live('click', function(e) { $('.nav-bar li a, .nav-bar li a:after').live('click', function(e) {
e.preventDefault(); e.preventDefault();
if (currentFoundationDropdown !== $(this).index() || currentFoundationDropdown === null) { if (currentFoundationDropdown !== $(this).index() || currentFoundationDropdown === null) {
$(this).data('clicks', 0); $(this).data('clicks', 0);
currentFoundationDropdown = $(this).index(); currentFoundationDropdown = $(this).index();
} }
$(this).data('clicks', ($(this).data('clicks') + 1)); $(this).data('clicks', ($(this).data('clicks') + 1));
var f = $(this).siblings('.flyout'); var f = $(this).siblings('.flyout');
if (!f.is(':visible') && $(this).parent('.has-flyout').length > 1) { if (!f.is(':visible') && $(this).parent('.has-flyout').length > 1) {
$('.nav-bar li .flyout').hide(); $('.nav-bar li .flyout').hide();
f.show(); f.show();
} else if (($(this).data('clicks') > 1) || ($(this).parent('.has-flyout').length < 1)) { } else if (($(this).data('clicks') > 1) || ($(this).parent('.has-flyout').length < 1)) {
window.location = $(this).attr('href'); window.location = $(this).attr('href');
} }
}); });
$('.nav-bar').live('click', function(e) { $('.nav-bar').live('click', function(e) {
e.stopPropagation(); e.stopPropagation();
if ($(e.target).parents().is('.flyout') || $(e.target).is('.flyout')) { if ($(e.target).parents().is('.flyout') || $(e.target).is('.flyout')) {
e.preventDefault(); e.preventDefault();
} }
}); });
// $('body').bind('touchend', function(e) { // $('body').bind('touchend', function(e) {
// if (!$(e.target).parents().is('.nav-bar') || !$(e.target).is('.nav-bar')) { // if (!$(e.target).parents().is('.nav-bar') || !$(e.target).is('.nav-bar')) {
// $('.nav-bar li .flyout').is(':visible').hide(); // $('.nav-bar li .flyout').is(':visible').hide();
// } // }
// }); // });
   
/* DISABLED BUTTONS ------------- */ /* DISABLED BUTTONS ------------- */
/* Gives elements with a class of 'disabled' a return: false; */ /* Gives elements with a class of 'disabled' a return: false; */
   
}); });
   
directory:b/javascripts/flotr2 (new)
 
directory:b/lib/Requests (new)
 
directory:b/lib/php-diff (new)
 
<?php <?php
   
$schemas['agency'] = Array( $schemas['agency'] = Array(
"description" => "Representation of government agency and online transparency measures", "description" => "Representation of government agency and online transparency measures",
"type" => "object", "type" => "object",
"properties" => Array( "properties" => Array(
"name" => Array("type" => "string", "required" => true, "x-title"=> "Agency Name", "description" => "Agency Name, most recent and broadest"), "name" => Array("type" => "string", "required" => true, "x-property" => "schema:name foaf:name", "x-title" => "Name", "description" => "Name, most recent and broadest"),
"otherNames" => Array("type" => "array", "required" => true, "x-title" => "Agency Past/Other Names", "description" => "Agency Names", "shortName" => Array("type" => "string", "required" => false, "x-title" => "Short Name", "description" => "Name shortened, usually to an acronym"),
  "foiEmail" => Array("type" => "string", "required" => false, "x-title" => "FOI Contact Email", "description" => "FOI contact email if not foi@"),
  "sameAs" => Array("type" => "array", "required" => false, "x-property"=>"owl:sameAs","x-title" => "Same As", "description" => "Same as other URLs/URIs for this entity",
"items" => Array("type" => "string")), "items" => Array("type" => "string")),
"orgType"=> Array("type" => "string", "required" => true, "x-title"=> "Agency Name", "description" => "Agency Name, most recent and broadest"), "otherNames" => Array("type" => "array", "required" => true, "x-title" => "Past/Other Names", "description" => "Other names for organisation",
"parentOrg"=> Array("type" => "string", "required" => true, "x-title"=> "Agency Name", "description" => "Agency Name, most recent and broadest"), "items" => Array("type" => "string")),
"website"=> Array("type" => "string", "required" => true, "x-title"=> "Agency Name", "description" => "Agency Name, most recent and broadest"), "foiBodies" => Array("type" => "array", "required" => true, "x-title" => "FOI Bodies","x-property"=>"schema:members foaf:knows", "description" => "Organisational units within this agency that are subject to FOI Act but are not autonomous",
"abn" => Array("type" => "string", "required" => true, "x-title"=> "Agency Name", "description" => "Agency Name, most recent and broadest"), "items" => Array("type" => "string")),
"contractListURL" => Array("type" => "string", "required" => true, "x-title"=> "Agency Name", "description" => "Agency Name, most recent and broadest"), "orgType" => Array("type" => "string", "required" => true, "x-title" => "Organisation Type", "description" => "Org type based on legal formation via FMA/CAC legislation etc."),
"grantsReportingURL" => Array("type" => "string", "required" => true, "x-title"=> "Agency Name", "description" => "Agency Name, most recent and broadest"), "parentOrg" => Array("type" => "string", "required" => true, "x-title" => "Parent Organisation", "description" => "Parent organisation, usually a department of state"),
"annualReportURL" => Array("type" => "string", "required" => true, "x-title"=> "Agency Name", "description" => "Agency Name, most recent and broadest"), "website" => Array("type" => "string", "required" => true, "x-title" => "Website", "x-property" => "schema:url foaf:homepage", "description" => "Website URL"),
"consultanciesURL" => Array("type" => "string", "required" => true, "x-title"=> "Agency Name", "description" => "Agency Name, most recent and broadest"), "abn" => Array("type" => "string", "required" => true, "x-title" => "Australian Business Number", "description" => "ABN from business register"),
"legalExpenditureURL"=> Array("type" => "string", "required" => true, "x-title"=> "Agency Name", "description" => "Agency Name, most recent and broadest"), "contractListURL" => Array("type" => "string", "required" => true, "x-title" => "Contract Listing", "description" => "Departmental and agency contracts, <a href='http://www.aph.gov.au/senate/pubs/standing_orders/d05.htm'>mandated by the Senate</a>"),
"recordsListURL"=> Array("type" => "string", "required" => true, "x-title"=> "Agency Name", "description" => "Agency Name, most recent and broadest"), "grantsReportingURL" => Array("type" => "string", "required" => true, "x-title" => "Grants Awarded",
"FOIDocumentsURL"=> Array("type" => "string", "required" => true, "x-title"=> "Agency Name", "description" => "Agency Name, most recent and broadest"), "description" => "Departmental and agency grants <a href='http://www.aph.gov.au/senate/pubs/standing_orders/d05.htm'>mandated by the Senate</a> and <a href='http://www.finance.gov.au/publications/fmg-series/23-commonwealth-grant-guidelines.html'>Commonwealth grants guidelines</a> "),
"infoPublicationSchemeURL"=> Array("type" => "string", "required" => true, "x-title"=> "Agency Name", "description" => "Agency Name, most recent and broadest"), "annualReportURL" => Array("type" => "string", "required" => true, "x-title" => "Annual Report(s)", "description" => ""),
"appointmentsURL"=> Array("type" => "string", "required" => true, "x-title"=> "Agency Name", "description" => "Agency Name, most recent and broadest"), "consultanciesURL" => Array("type" => "string", "required" => true, "x-title" => "Consultants Hired", "description" => ""),
"hasRSS"=> Array("type" => "string", "required" => true, "x-title"=> "Agency Name", "description" => "Agency Name, most recent and broadest"), "legalExpenditureURL" => Array("type" => "string", "required" => true, "x-title" => "Legal Services Expenditure", "description" => "Legal Services Expenditure mandated by Legal Services Directions 2005"),
"hasMailingList"=> Array("type" => "string", "required" => true, "x-title"=> "Agency Name", "description" => "Agency Name, most recent and broadest"), "recordsListURL" => Array("type" => "string", "required" => true, "x-title" => "Files/Records Held", "description" => "Indexed lists of departmental and agency files, <a href='http://www.aph.gov.au/senate/pubs/standing_orders/d05.htm'>mandated by the Senate</a>"),
"hasTwitter"=> Array("type" => "string", "required" => true, "x-title"=> "Agency Name", "description" => "Agency Name, most recent and broadest"), "FOIDocumentsURL" => Array("type" => "string", "required" => true, "x-title" => "FOI Documents Released", "description" => "FOI Disclosure Log URL"),
"hasFacebook"=> Array("type" => "string", "required" => true, "x-title"=> "Agency Name", "description" => "Agency Name, most recent and broadest"), "FOIDocumentsRSSURL" => Array("type" => "string", "required" => false, "x-title" => "RSS Feed of FOI Documents Released", "description" => "FOI Disclosure Log in RSS format"),
"hasYouTube"=> Array("type" => "string", "required" => true, "x-title"=> "Agency Name", "description" => "Agency Name, most recent and broadest"), "hasFOIPDF" => Array("type" => "array", "required" => false, "x-title" => "Has FOI Documents Released in PDF", "description" => "FOI Disclosure Log contains any PDFs",
  "items" => Array("type" => "string")),
  "infoPublicationSchemeURL" => Array("type" => "string", "required" => true, "x-title" => "Information Publication Scheme", "description" => ""),
  "appointmentsURL" => Array("type" => "string", "required" => true, "x-title" => "Agency Appointments/Boards", "description" => "Departmental and agency appointments and vacancies , <a href='http://www.aph.gov.au/senate/pubs/standing_orders/d05.htm'>mandated by the Senate</a>"),
  "advertisingURL" => Array("type" => "string", "required" => true, "x-title" => "Approved Advertising Campaigns", "description" => " Agency advertising and public information projects, <a href='http://www.aph.gov.au/senate/pubs/standing_orders/d05.htm'>mandated by the Senate</a> "),
  "hasRSS" => Array("type" => "array", "required" => true, "x-title" => "Has RSS", "description" => ""),
  "hasMailingList" => Array("type" => "array", "required" => true, "x-title" => "Has Mailing List", "description" => "",
  "items" => Array("type" => "string")),
  "hasTwitter" => Array("type" => "array", "required" => true, "x-title" => "Has Twitter", "description" => "",
  "items" => Array("type" => "string")),
  "hasFacebook" => Array("type" => "array", "required" => true, "x-title" => "Has Facebook", "description" => "",
  "items" => Array("type" => "string")),
  "hasYouTube" => Array("type" => "array", "required" => true, "x-title" => "Has YouTube", "description" => "",
  "items" => Array("type" => "string")),
  "hasFlickr" => Array("type" => "array", "required" => true, "x-title" => "Has Flickr", "description" => "",
  "items" => Array("type" => "string")),
  "hasCCBY" => Array("type" => "array", "required" => true, "x-title" => "Has CC-BY", "description" => "Has any page licenced Creative Commons - Attribution",
  "items" => Array("type" => "string")),
  "hasRestrictiveLicence" => Array("type" => "array","required" => true, "x-title" => "Has Restrictive Licence", "description" => "Has any page licenced under terms more restrictive than Crown Copyright",
  "items" => Array("type" => "string")),
  "hasCrownCopyright" => Array("type" => "array", "required" => true, "x-title" => "Has Standard Crown Copyright licence", "description" => "Has any page still licenced under the former Commonwealth Copyright Administration",
  "items" => Array("type" => "string")),
), ),
/*"org":{"type":"object", /* "org":{"type":"object",
"properties":{ "properties":{
"organizationName":{"type":"string"}, "organizationName":{"type":"string"},
"organizationUnit":{"type":"string"}}, "organizationUnit":{"type":"string"}},
} }
}*/ } */
); );
?> ?>
   
file:b/scrape.py (new)
  #http://packages.python.org/CouchDB/client.html
  import couchdb
  import urllib2
  from BeautifulSoup import BeautifulSoup
  import re
  import hashlib
  from urlparse import urljoin
  import time
  import os
  import mimetypes
  import re
  import urllib
  import urlparse
 
  def canonurl(url):
  r"""Return the canonical, ASCII-encoded form of a UTF-8 encoded URL, or ''
  if the URL looks invalid.
  >>> canonurl('\xe2\x9e\xa1.ws') # tinyarro.ws
  'http://xn--hgi.ws/'
  """
  # strip spaces at the ends and ensure it's prefixed with 'scheme://'
  url = url.strip()
  if not url:
  return ''
  if not urlparse.urlsplit(url).scheme:
  url = 'http://' + url
 
  # turn it into Unicode
  #try:
  # url = unicode(url, 'utf-8')
  #except UnicodeDecodeError:
  # return '' # bad UTF-8 chars in URL
 
  # parse the URL into its components
  parsed = urlparse.urlsplit(url)
  scheme, netloc, path, query, fragment = parsed
 
  # ensure scheme is a letter followed by letters, digits, and '+-.' chars
  if not re.match(r'[a-z][-+.a-z0-9]*$', scheme, flags=re.I):
  return ''
  scheme = str(scheme)
 
  # ensure domain and port are valid, eg: sub.domain.<1-to-6-TLD-chars>[:port]
  match = re.match(r'(.+\.[a-z0-9]{1,6})(:\d{1,5})?$', netloc, flags=re.I)
  if not match:
  return ''
  domain, port = match.groups()
  netloc = domain + (port if port else '')
  netloc = netloc.encode('idna')
 
  # ensure path is valid and convert Unicode chars to %-encoded
  if not path:
  path = '/' # eg: 'http://google.com' -> 'http://google.com/'
  path = urllib.quote(urllib.unquote(path.encode('utf-8')), safe='/;')
 
  # ensure query is valid
  query = urllib.quote(urllib.unquote(query.encode('utf-8')), safe='=&?/')
 
  # ensure fragment is valid
  fragment = urllib.quote(urllib.unquote(fragment.encode('utf-8')))
 
  # piece it all back together, truncating it to a maximum of 4KB
  url = urlparse.urlunsplit((scheme, netloc, path, query, fragment))
  return url[:4096]
 
  #http://diveintopython.org/http_web_services/etags.html
  class NotModifiedHandler(urllib2.BaseHandler):
  def http_error_304(self, req, fp, code, message, headers):
  addinfourl = urllib2.addinfourl(fp, headers, req.get_full_url())
  addinfourl.code = code
  return addinfourl
 
  def fetchURL(docsdb, url, fieldName, agencyID, scrape_again=True):
  url = canonurl(url)
  hash = hashlib.md5(url).hexdigest().encode("utf-8")
  req = urllib2.Request(url)
  print "Fetching %s" % url
  if url.startswith("mailto") or url.startswith("javascript") or url.startswith("#") or url == None or url == "":
  print "Not a valid HTTP url"
  return (None,None)
  doc = docsdb.get(hash)
  if doc == None:
  doc = {'_id': hash, 'agencyID': agencyID, 'url': url, 'fieldName':fieldName}
  else:
  if (('page_scraped' in doc) and (time.time() - doc['page_scraped']) < 999999):
  print "Uh oh, trying to scrape URL again too soon!"
  last_attachment_fname = doc["_attachments"].keys()[-1]
  last_attachment = docsdb.get_attachment(doc,last_attachment_fname)
  return (doc['mime_type'],last_attachment)
  if scrape_again == False:
  print "Not scraping this URL again as requested"
  return (None,None)
 
  time.sleep(3) # wait 3 seconds to give webserver time to recover
 
  #if there is a previous version stored in couchdb, load caching helper tags
  if doc.has_key('etag'):
  req.add_header("If-None-Match", doc['etag'])
  if doc.has_key('last_modified'):
  req.add_header("If-Modified-Since", doc['last_modified'])
 
  opener = urllib2.build_opener(NotModifiedHandler())
  try:
  url_handle = opener.open(req)
  headers = url_handle.info() # the addinfourls have the .info() too
  doc['etag'] = headers.getheader("ETag")
  doc['last_modified'] = headers.getheader("Last-Modified")
  doc['date'] = headers.getheader("Date")
  doc['page_scraped'] = time.time()
  doc['web_server'] = headers.getheader("Server")
  doc['powered_by'] = headers.getheader("X-Powered-By")
  doc['file_size'] = headers.getheader("Content-Length")
  content_type = headers.getheader("Content-Type")
  if content_type != None:
  doc['mime_type'] = content_type.split(";")[0]
  else:
  (type,encoding) = mimetypes.guess_type(url)
  doc['mime_type'] = type
  if hasattr(url_handle, 'code'):
  if url_handle.code == 304:
  print "the web page has not been modified"
  return (None,None)
  else:
  content = url_handle.read()
  docsdb.save(doc)
  doc = docsdb.get(hash) # need to get a _rev
  docsdb.put_attachment(doc, content, str(time.time())+"-"+os.path.basename(url), doc['mime_type'])
  return (doc['mime_type'], content)
  #store as attachment epoch-filename
  except urllib2.URLError as e:
  error = ""
  if hasattr(e, 'reason'):
  error = "error %s in downloading %s" % (str(e.reason), url)
  elif hasattr(e, 'code'):
  error = "error %s in downloading %s" % (e.code, url)
  print error
  doc['error'] = error
  docsdb.save(doc)
  return (None,None)
 
 
 
  def scrapeAndStore(docsdb, url, depth, fieldName, agencyID):
  (mime_type,content) = fetchURL(docsdb, url, fieldName, agencyID)
  if content != None and depth > 0:
  if mime_type == "text/html" or mime_type == "application/xhtml+xml" or mime_type =="application/xml":
  # http://www.crummy.com/software/BeautifulSoup/documentation.html
  soup = BeautifulSoup(content)
  navIDs = soup.findAll(id=re.compile('nav|Nav|menu|bar'))
  for nav in navIDs:
  print "Removing element", nav['id']
  nav.extract()
  navClasses = soup.findAll(attrs={'class' : re.compile('nav|menu|bar')})
  for nav in navClasses:
  print "Removing element", nav['class']
  nav.extract()
  links = soup.findAll('a') # soup.findAll('a', id=re.compile("^p-"))
  linkurls = set([])
  for link in links:
  if link.has_key("href"):
  if link['href'].startswith("http"):
  # lets not do external links for now
  # linkurls.add(link['href'])
  None
  if link['href'].startswith("mailto"):
  # not http
  None
  if link['href'].startswith("javascript"):
  # not http
  None
  else:
  linkurls.add(urljoin(url,link['href'].replace(" ","%20")))
  for linkurl in linkurls:
  #print linkurl
  scrapeAndStore(docsdb, linkurl, depth-1, fieldName, agencyID)
 
  couch = couchdb.Server('http://127.0.0.1:5984/')
 
  # select database
  agencydb = couch['disclosr-agencies']
  docsdb = couch['disclosr-documents']
 
  for row in agencydb.view('app/getScrapeRequired'): #not recently scraped agencies view?
  agency = agencydb.get(row.id)
  print agency['name']
  for key in agency.keys():
  if key == 'website':
  scrapeAndStore(docsdb, agency[key],0,key,agency['_id'])
  if key.endswith('URL'):
  print key
  depth = 1
  if 'scrapeDepth' in agency.keys():
  depth = agency['scrapeDepth']
  scrapeAndStore(docsdb, agency[key],depth,key,agency['_id'])
 
  agency['metadata']['lastScraped'] = time.time()
  agencydb.save(agency)
 
file:b/search.php (new)
  <?php
  include_once('include/common.inc.php');
  include_header();
  ?>
  <div class="foundation-header">
  <h1><a href="search.php">Search</a></h1>
  </div>
  <form>
  <input type="text" name="q" value="<?php if (isset($_REQUEST['q']))echo $_REQUEST['q'];?>"/>
  <input type="submit"/>
  </form>
 
  <?php
  if (isset($_REQUEST['q'])) {
  $request = Requests::get($serverAddr."disclosr-documents/_fti/_design/lucene/by_all?include_docs=true&q=".$_REQUEST['q']);
  $results = json_decode($request->body);
  $db = $server->get_db('disclosr-documents');
  foreach ($results->rows as $result) {
  //print_r($result);
  //$row = $db->get($result->id);
  echo $result->doc->_id." ".$result->doc->url."<br>".PHP_EOL;
  }
  }
  include_footer();
  ?>
<?php  
 
include_once("./lib/common.inc.php");  
setlocale(LC_CTYPE, 'C');  
// source: http://stackoverflow.com/questions/81934/easy-way-to-export-a-sql-table-without-access-to-the-server-or-phpmyadmin#81951  
 
$unspsc = Array();  
$unspscresult = $conn->prepare('select * from "UNSPSCcategories" where "UNSPSC"::text like \'%00000\';');  
$unspscresult->execute();  
foreach ($unspscresult->fetchAll() as $row) {  
$unspsc[$row['UNSPSC']] = $row['Title'];  
}  
 
$query = $conn->prepare('  
SELECT "CNID",contractnotice."agencyName",agency_nametoabn.abn as "agencyABN",  
EXTRACT(EPOCH FROM "publishDate") as "publishDate",  
EXTRACT(EPOCH FROM "contractStart") as "contractStart",  
EXTRACT(EPOCH FROM "contractEnd") as "contractEnd",  
value,description,category,  
"supplierName",(case when "supplierABN" != 0 THEN "supplierABN"::text ELSE "supplierName" END) as supplierID,  
(\'https://www.tenders.gov.au/?event=public.advancedsearch.keyword&keyword=CN\'::text || "CNID"::text) as sourceURL  
FROM contractnotice join agency_nametoabn on contractnotice."agencyName"=agency_nametoabn."agencyName"  
where "childCN" is null'  
, array(PDO::ATTR_CURSOR => PDO::FETCH_ORI_NEXT));  
$query->execute();  
$errors = $conn->errorInfo();  
if ($errors[2] != "") {  
die("Export terminated, db error" . print_r($errors, true));  
}  
 
$num_fields = $query->columnCount();  
$headers = Array();  
for ($i = 0; $i < $num_fields; $i++) { // for each column in query, make a CSV header  
$meta = $query->getColumnMeta($i);  
$headers[] = $meta['name'];  
}  
$fp = fopen('php://output', 'w');  
if ($fp && $query) {  
header('Content-Type: text/csv');  
header('Content-Disposition: attachment; filename="export.' . date("c") . '.csv"');  
header('Pragma: no-cache');  
header('Expires: 0');  
fputcsv($fp, $headers);  
while ($row = $query->fetch(PDO::FETCH_NUM, PDO::FETCH_ORI_NEXT)) {  
foreach ($row as $key => &$colvalue) {  
 
$colvalue = preg_replace('/[^[:print:]]/', '', utf8_encode($colvalue));  
if ($headers[$key] == "publishDate" || $headers[$key] == "contractStart"  
|| $headers[$key] == "contractEnd") {  
$colvalue = date("Y-m-d", $colvalue);  
}  
/* if ($headers[$key] == "CNID") {  
$colvalue = str_replace("A","", $colvalue);  
}*/  
if ($headers[$key] == "cat1" || $headers[$key] == "cat2"  
|| $headers[$key] == "cat3") {  
$colvalue = $unspsc[$colvalue];  
}  
}  
fputcsv($fp, array_values($row));  
}  
die;  
}  
?>  
 
file:a/unimplemented/scrape.py (deleted)
#http://packages.python.org/CouchDB/client.html  
import couchdb  
import urllib2  
from BeautifulSoup import BeautifulSoup  
import re  
 
couch = couchdb.Server() # Assuming localhost:5984  
# If your CouchDB server is running elsewhere, set it up like this:  
# couch = couchdb.Server('http://example.com:5984/')  
 
# select database  
agencydb = couch['disclosr-agencies']  
 
for row in agencydb.view('app/getScrapeRequired'): #not recently scraped agencies view?  
agency = agencydb.get(row.id)  
print agency['agencyName']  
 
#http://diveintopython.org/http_web_services/etags.html  
class NotModifiedHandler(urllib2.BaseHandler):  
def http_error_304(self, req, fp, code, message, headers):  
addinfourl = urllib2.addinfourl(fp, headers, req.get_full_url())  
addinfourl.code = code  
return addinfourl  
 
def scrapeAndStore(URL, depth, agency):  
URL = "http://www.hole.fi/jajvirta/weblog/"  
req = urllib2.Request(URL)  
 
#if there is a previous version sotred in couchdb, load caching helper tags  
if etag:  
req.add_header("If-None-Match", etag)  
if last_modified:  
req.add_header("If-Modified-Since", last_modified)  
 
opener = urllib2.build_opener(NotModifiedHandler())  
url_handle = opener.open(req)  
headers = url_handle.info() # the addinfourls have the .info() too  
etag = headers.getheader("ETag")  
last_modified = headers.getheader("Last-Modified")  
web_server = headers.getheader("Server")  
file_size = headers.getheader("Content-Length")  
mime_type = headers.getheader("Content-Type")  
 
if hasattr(url_handle, 'code') and url_handle.code == 304:  
print "the web page has not been modified"  
else:  
print "error %s in downloading %s", url_handle.code, URL  
#record/alert error to error database  
 
#do scraping  
html = ?  
# http://www.crummy.com/software/BeautifulSoup/documentation.html  
soup = BeautifulSoup(html)  
links = soup.findAll('a') # soup.findAll('a', id=re.compile("^p-"))  
for link in links:  
print link['href']  
#for each unique link  
#if html mimetype  
# go down X levels,  
# diff with last stored attachment, store in document  
#if not  
# remember to save parentURL and title (link text that lead to document)  
 
#store as attachment epoch-filename