| Welcome to the GovHack toolkit. This page provides all the information you need to prepare hackfest entries. | |
| These tools can be used to make entries like: mobile apps, web apps, data visualisations/infographics | |
| - govhack toolkit!!!! | |
| - key datasets, directory.gov.au gazetter/AEC electorates/suburbs/postcodes/LGAs | |
| - definitions, open licence reuse permissive hacker hack data journalism data bis UCX etc. | |
| - what do govhack entries look like? | |
| - roles; coder, designer UX/graphics | |
| geo | geo |
| - http://converter.mygeodata.eu/vector kml exporter for shp | |
| - http://bost.ocks.org/mike/map/ | |
| R | R |
| - http://blog.yhathq.com/posts/10-R-packages-I-wish-I-knew-about-earlier.html | |
| - excel -> R/rattle/ deducer? http://www.r-bloggers.com/updates-to-the-deducer-family-of-packages/ | |
| - tools.disclo.gs - how to use data | - tools.disclo.gs - how to use data |
| - developer tools inc. linked data | - developer tools inc. linked data |
| - gephi -> neo4j | - gephi -> neo4j |
| - neo4j lets you build on, do massive queries of who is friends with who | - neo4j lets you build on, do massive queries of who is friends with who |
| - don't use network viz if what you actually have is a tree/hierarchy with no interconnections http://www.randelshofer.ch/treeviz/ http://thejit.org/demos/ http://mbostock.github.com/protovis/ex/treemap.html http://blog.pixelingene.com/2011/07/building-a-tree-diagram-in-d3-js/ | - don't use network viz if what you actually have is a tree/hierarchy with no interconnections http://www.randelshofer.ch/treeviz/ http://thejit.org/demos/ http://mbostock.github.com/protovis/ex/treemap.html http://blog.pixelingene.com/2011/07/building-a-tree-diagram-in-d3-js/ |
| - http://mbostock.github.com/d3/ex/pack.html http://mbostock.github.com/d3/ex/tree.html | - http://mbostock.github.com/d3/ex/pack.html http://mbostock.github.com/d3/ex/tree.html |
| - postgis/quantum gis | - postgis/quantum gis |
| - (google earth is alright but many limitations) NASA World Wind? | - (google earth is alright but many limitations) NASA World Wind? |
| - ABS statistical areas | - ABS statistical areas |
| - http://irs.gis-lab.info/ wms or http://www.gdal.org/frmt_wms_openstreetmap_tms.xml | - base layers like agri, http://irs.gis-lab.info/ wms or http://www.gdal.org/frmt_wms_openstreetmap_tms.xml |
| - can do nearest/isin/union queries - personal geocoder | - can do nearest/isin/union queries - personal geocoder |
| - cloudmade geocoder, google maps my maps | - cloudmade geocoder, google maps my maps |
| - base layers like agri or nearmap | |
| - scraperwiki with new pytemplate libraries | - scraperwiki with new pytemplate libraries |
| - makes an API for your data to get in sqlite/json/csv | - makes an API for your data to get in sqlite/json/csv |
| - three.js with cables vis? Overviewer from AP? | |
| - govhack library | - govhack library |
| - http://hpneo.github.com/gmaps/examples.html | |
| - https://graphics.stanford.edu/wikis/cs448b-12-fall/ data viz theory | - https://graphics.stanford.edu/wikis/cs448b-12-fall/ data viz theory |
| - https://mixpanel.com/education/understanding-the-first-five-minutes | |
| - http://drawingbynumbers.org/toolsandresources | - http://drawingbynumbers.org/toolsandresources |
| - http://wmbriggs.com/blog/?p=6465 | - http://wmbriggs.com/blog/?p=6465 |
| - http://ofps.oreilly.com/titles/9781449339739/k_00000002.html list of d3 alternatives | - http://ofps.oreilly.com/titles/9781449339739/k_00000002.html list of d3 alternatives |
| - http://craigkerstiens.com/2012/10/01/understanding-postgres-performance/ | - http://craigkerstiens.com/2012/10/01/understanding-postgres-performance/ |
| - https://github.com/clips/pattern for easy NLP/network analysis/data mining | - https://github.com/clips/pattern for easy NLP/network analysis/data mining |
| - https://github.com/theodi/open-data-tech-review/wiki othr cleanup/linked data toola | - https://github.com/theodi/open-data-tech-review/wiki othr cleanup/linked data toola |
| - http://selection.datavisualization.ch/ data viz tools catalog | - http://selection.datavisualization.ch/ data viz tools catalog |
| - manipulating data - grep/find replace/sed/regex | - manipulating data - grep/find replace/sed/regex |
| - d3 tools and tutorial http://enjalot.com/ http://news.ycombinator.com/item?id=4608440 | - d3 tools and tutorial http://enjalot.com/ http://news.ycombinator.com/item?id=4608440 |
| - Why d3 is the way it is and how to make charts http://bost.ocks.org/mike/chart/ | - Why d3 is the way it is and how to make charts http://bost.ocks.org/mike/chart/ |
| - how to make an xkcd chart http://bl.ocks.org/3914862 | - how to make an xkcd chart http://bl.ocks.org/3914862 |
| - data viz | - data viz |
| - http://k2company.com/blog/2012/09/06/toolbox-for-learning-machine-learning-and-data-science/ | - http://k2company.com/blog/2012/09/06/toolbox-for-learning-machine-learning-and-data-science/ |
| - http://williamparry.blogspot.com.au/2011/04/putting-data-into-google-fusion-tables.html google fusion tutorial | - http://williamparry.blogspot.com.au/2011/04/putting-data-into-google-fusion-tables.html google fusion tutorial |
| - andrewharvey4.wordpress.com postgis/asgs tutorial | - andrewharvey4.wordpress.com postgis/asgs tutorial |
| - http://www.slideshare.net/maxdemarzi/etl-into-neo4j | - http://www.slideshare.net/maxdemarzi/etl-into-neo4j |
| - http://www.twotorials.com/ for R | - http://www.twotorials.com/ for R |
| - http://www.r-bloggers.com/gradient-word-clouds/ http://www.rstudio.com/shiny/ http://blog.ouseful.info/2012/11/28/quick-shiny-demo-exploring-nhs-winter-sit-rep-data/ https://github.com/timelyportfolio/shiny-d3-plot https://github.com/trestletech/shiny-sandbox/tree/master/grn | - http://www.r-bloggers.com/gradient-word-clouds/ http://www.rstudio.com/shiny/ http://blog.ouseful.info/2012/11/28/quick-shiny-demo-exploring-nhs-winter-sit-rep-data/ https://github.com/timelyportfolio/shiny-d3-plot https://github.com/trestletech/shiny-sandbox/tree/master/grn |
| - http://is-r.tumblr.com/post/38240018815/making-prettier-network-graphs-with-sna-and-igraph | - http://is-r.tumblr.com/post/38240018815/making-prettier-network-graphs-with-sna-and-igraph |
| - http://www.r-bloggers.com/video-simpler-tricks-and-tools-help-debugging-git-latex-and-workflow-with-r-by-prof-rob-hyndman/ | - http://www.r-bloggers.com/video-simpler-tricks-and-tools-help-debugging-git-latex-and-workflow-with-r-by-prof-rob-hyndman/ |
| - http://yihui.name/knitr/ makes reports including google widgets/charts/maps via http://www.r-bloggers.com/googlevis-0-3-2-is-released-better-integration-with-knitr/ | - http://yihui.name/knitr/ makes reports including google widgets/charts/maps via http://www.r-bloggers.com/googlevis-0-3-2-is-released-better-integration-with-knitr/ |
| - http://chartsnthings.tumblr.com/post/36978271916/r-tutorial-simple-charts http://flowingdata.com/2012/12/17/getting-started-with-charts-in-r/ | - http://chartsnthings.tumblr.com/post/36978271916/r-tutorial-simple-charts http://flowingdata.com/2012/12/17/getting-started-with-charts-in-r/ |
| - http://dydra.com/ | - http://dydra.com/ |
| - http://selection.datavisualization.ch/ data viz tools list | - http://selection.datavisualization.ch/ data viz tools list |
| - http://nodexl.codeplex.com/ network graphs for excel | - http://nodexl.codeplex.com/ network graphs for excel |
| - http://sunfoundation.tumblr.com/ | - http://sunfoundation.tumblr.com/ |
| - analysing - linked data tools | - analysing - linked data tools |
| - http://govcampau.wikispaces.com/useful+tools | - http://govcampau.wikispaces.com/useful+tools |
| - http://linkeddata.org/home | - http://linkeddata.org/home |
| # Hosted Developer Tools {#hosted-developer-tools} | Welcome to the GovHack toolkit. This page provides all the information you need to prepare hackfest entries. |
| These tools can be used to make entries like: mobile apps, web apps, data visualisations/infographics | |
| ### Sourceforge | |
| Subversion, Git, Mercurial, Bazaar, CVS, issue tracker, wiki, release file downloads. Unlimited free use for open source projects. | # General Data Hacking and Programming References {#general-data-hacking-and-programming-references} |
| ## The basics of being a data scientist | |
| You can create your own Sourceforge project at [http://sourceforge.net/](http://sourceforge.net/) | |
| ### Google Code Project Hosting | |
| Git, Mercurial, and Subversion code. Issue tracker, wiki, release file downloads. Unlimited free use for open source projects. | |
| You can host your Google Code project and get access to developer tools, APIs and documentation at [http://code.google.com/](http://code.google.com/) | |
| # The basics of being a data scientist | |
| * Have a hypothesis � even if you’re making a tool/api that helps people with their questions too, remember what the objective of that is. | * Have a hypothesis � even if you’re making a tool/api that helps people with their questions too, remember what the objective of that is. |
| * Find the people and tools you need to prove/show/find. This rest of this page will help with the latter. | * Find the people and tools you need to prove/show/find. This rest of this page will help with the latter. |
| * Analyse and present results � were they what you expected? Do they help explain to others what you have found out? Can present as a interactive data visualisation or a web/mobile application or just a infographic/motion graphics video that tells a story. | * Analyse and present results � were they what you expected? Do they help explain to others what you have found out? Can present as a interactive data visualisation or a web/mobile application or just a infographic/motion graphics video that tells a story. |
| Please note, there are a combination of Analysis and Visualisation tools in each of the major categories below. | Please note, there are a combination of Analysis and Visualisation tools in each of the major categories below. |
| [](http://www.govhack.org/wp-content/uploads/How-to-participate-in-GovHack_html_m6a65720f.gif)</dt> | |
| Illustration from Data Journalism Handbook, CC BY-SA 3.0</dd> | |
| The best high level reference is the �Understanding Data� and �Delivering Data� chapters of the Data Journalism Handbook which is available online for free at | |
| [datajournalismhandbook.org](http://datajournalismhandbook.org/) | |
| You can learn the technical skills from scratch in Visualize This: The FlowingData Guide to Design, Visualization, and Statistics by Nathan Yau or for more advanced | |
| practical advice check out Data Analysis with Open Source Tools by Philipp K. Janert | |
| For further reading in this space | |
| [http://flowingdata.com/2012/04/27/data-and-visualization-blogs-worth-following/](http://flowingdata.com/2012/04/27/data-and-visualization-blogs-worth-following/) | |
| **Statistics** | |
| [http://greenteapress.com/thinkstats/html/index.html](http://greenteapress.com/thinkstats/html/index.html) | |
| **Programming** | |
| Programming is valuable skill for manipulating and displaying data. | |
| Basic tutorials for a variety of languages are available for free online or you can learn | |
| interactively with websites like [http://www.codecademy.com/](http://www.codecademy.com/#!/exercises/0\. for JavaScript or [http://www.learnpython.org/ ](http://www.learnpython.org/)or [http://tryruby.org](http://tryruby.org/) | |
| [https://developer.mozilla.org/en/JavaScript](https://developer.mozilla.org/en/JavaScript) –\. especially for web applications and visualisations, you’ll need a basic understanding of JS. Common libraries like prototype or jQuery can help | |
| **Accessibility/User Experience** | |
| WCAG guidelines not only make a web app accessible but make it a better experience for all users! Even if not making an app, good to consider these things to do and not do: [http://www.w3.org/TR/WCAG/](http://www.w3.org/TR/WCAG/) | |
| ## Who can be a hack day participant | |
| - roles; coder, designer UX/graphics | |
| ## Definitions | |
| - definitions, open licence reuse permissive hacker hack data journalism data bis UCX etc. | |
| ## key datasets | |
| - key datasets, directory.gov.au gazetter/AEC electorates/suburbs/postcodes/LGAs | |
| ## examples | |
| ## The Open Budget | |
| ## WeatherTree weather API | |
| ## PlanningAlerts | |
| [](http://www.govhack.org/wp-content/uploads/How-to-participate-in-GovHack_html_2f0199ff1.png)Description: Planning Alerts takes data from local government development applications and sends alerts to users based on what applications are lodged in their area. | |
| Programming Language: Ruby | |
| Source Control: [Git](https://github.com/openaustralia/planningalerts-app) | |
| Issue Tracking: [Atlassian JIRA](http://tickets.openaustraliafoundation.org.au/browse/PA/) | |
| ## LobbyLens | |
| [](http://www.govhack.org/wp-content/uploads/129-Screenshot-LobbyClue_-_Chromium.png) | |
| Description: Displays connections between government contracts, business details, politician responsibilities, lobbyists, clients of lobbyists, political donors and the location of these entities. | |
| Programing Language: PHP | |
| Source Control: SVN (Subversion) | |
| Issue Tracking: A whiteboard | |
| ## Bus.lambdacomplex.org | |
| [](http://www.govhack.org/wp-content/uploads/How-to-participate-in-GovHack_html_3789acae.jpg) | |
| Description: Online Canberra Bus Timetables and Trip Planner. | |
| Programing Language: PHP/Ruby | |
| Source Control: Git | |
| Issue Tracking: Github | |
| ## Developer Tools For Your Computer {#developer-tools-for-your-computer} | |
| server admin / technical tools | |
| many projects will require some kind of internet presence, webpage etc. | |
| - css framework like bootstrap or zurb foundation | |
| video tools, youtube video editor/slideshow, FOSS video editing tools | |
| - http://ubuntu-tutorials.com/2008/11/11/relaying-postfix-smtp-via-smtpgmailcom/ | |
| - amon | |
| ### Source Control –\. Git / Subversion | |
| [](http://progit.org/book/) | |
| [http://progit.org/book/](http://progit.org/book/) | |
| [http://svnbook.red-bean.com/](http://svnbook.red-bean.com/) | |
| [http://tortoisesvn.net/](http://tortoisesvn.net/) | |
| [http://code.google.com/p/tortoisegit/](http://code.google.com/p/tortoisegit/) | |
| ### Task Tracking - | |
| Issue/task trackers allow you to outline the tasks required for your project and assign them to people to do. | |
| [Trello](https://trello.com/) and [Workflowy](https://workflowy.com/) are free, lightweight project management tools suitable for a rapid project! | |
| # Hosted Developer Tools {#hosted-developer-tools} | |
| ### Sourceforge | |
| Subversion, Git, Mercurial, Bazaar, CVS, issue tracker, wiki, release file downloads. Unlimited free use for open source projects. | |
| You can create your own Sourceforge project at [http://sourceforge.net/](http://sourceforge.net/) | |
| ### Google Code Project Hosting | |
| Git, Mercurial, and Subversion code. Issue tracker, wiki, release file downloads. Unlimited free use for open source projects. | |
| You can host your Google Code project and get access to developer tools, APIs and documentation at [http://code.google.com/](http://code.google.com/) | |
| # API Development {#api-development} | # API Development {#api-development} |
| So an API isn’t just an XML file  | So an API isn’t just an XML file  |
| A good web based data API: | A good web based data API: |
| * Is logically organised | * Is logically organised |
| * Can filter returned data | * Can filter returned data |
| * Can return results in different open formats (CSV/JSON etc.) | * Can return results in different open formats (CSV/JSON etc.) |
| * Is efficient and responsive by using caching and databases appropriately | * Is efficient and responsive by using caching and databases appropriately |
| * Handles errors gracefully | * Handles errors gracefully |
| * Monitors and controls access (to show benefit realised of API and prevent abuse) | * Monitors and controls access (to show benefit realised of API and prevent abuse) |
| * Provides appropriate documentation with examples | * Provides appropriate documentation with examples |
| Some people like sensis [http://](http://developers.sensis.com.au/)[developers.sensis.com.<wbr>au</wbr>](http://developers.sensis.com.au/)[/](http://developers.sensis.com.au/) use a provider like[http://](http://mashery.com/)[mashery.com](http://mashery.com/)[/](http://mashery.com/) or [https](https://apigee.com/)[://](https://apigee.com/)[apigee.com](https://apigee.com/) or [http://](http://apiaxle.com/)[apiaxle.com](http://apiaxle.com/)[/](http://apiaxle.com/) or [http://www.3scale.net/](http://www.3scale.net/) which handles making a good API for them. | Some people like sensis [http://](http://developers.sensis.com.au/)[developers.sensis.com.<wbr>au</wbr>](http://developers.sensis.com.au/)[/](http://developers.sensis.com.au/) use a provider like[http://](http://mashery.com/)[mashery.com](http://mashery.com/)[/](http://mashery.com/) or [https](https://apigee.com/)[://](https://apigee.com/)[apigee.com](https://apigee.com/) or [http://](http://apiaxle.com/)[apiaxle.com](http://apiaxle.com/)[/](http://apiaxle.com/) or [http://www.3scale.net/](http://www.3scale.net/) which handles making a good API for them. |
| Atlassian have a great page on what makes a good API [https](https://developer.atlassian.com/display/REST/Atlassian+REST+API+Design+Guidelines+version+1)[://](https://developer.atlassian.com/display/REST/Atlassian+REST+API+Design+Guidelines+version+1)[developer.atlassian.<wbr>com</wbr>](https://developer.atlassian.com/display/REST/Atlassian+REST+API+Design+Guidelines+version+1)[/display/REST/](https://developer.atlassian.com/display/REST/Atlassian+REST+API+Design+Guidelines+version+1)[Atlassian](https://developer.atlassian.com/display/REST/Atlassian+REST+API+Design+Guidelines+version+1)[+<wbr>REST+API+Design+Guidelines+<wbr>version+1</wbr></wbr>](https://developer.atlassian.com/display/REST/Atlassian+REST+API+Design+Guidelines+version+1) | Atlassian have a great page on what makes a good API [https](https://developer.atlassian.com/display/REST/Atlassian+REST+API+Design+Guidelines+version+1)[://](https://developer.atlassian.com/display/REST/Atlassian+REST+API+Design+Guidelines+version+1)[developer.atlassian.<wbr>com</wbr>](https://developer.atlassian.com/display/REST/Atlassian+REST+API+Design+Guidelines+version+1)[/display/REST/](https://developer.atlassian.com/display/REST/Atlassian+REST+API+Design+Guidelines+version+1)[Atlassian](https://developer.atlassian.com/display/REST/Atlassian+REST+API+Design+Guidelines+version+1)[+<wbr>REST+API+Design+Guidelines+<wbr>version+1</wbr></wbr>](https://developer.atlassian.com/display/REST/Atlassian+REST+API+Design+Guidelines+version+1) |
| API | API |
| - howto.gov api tutorial | - howto.gov api tutorial |
| - http://shancarter.com/data_converter/ | - http://shancarter.com/data_converter/ |
| - http://training.sunlightfoundation.com/module/data-visualizations-google-docs/ | - http://training.sunlightfoundation.com/module/data-visualizations-google-docs/ |
| - api documentation | - api documentation |
| - WSDL or http://swagger.wordnik.com/ or https://github.com/mashery/iodocs | - WSDL or http://swagger.wordnik.com/ or https://github.com/mashery/iodocs |
| - http://symfony.com/ https://github.com/FriendsOfSymfony/FOSRestBundle http://williamdurand.fr/2012/08/02/rest-apis-with-symfony2-the-right-way/ https://github.com/nelmio/NelmioApiDocBundle better apis https://github.com/liip/LiipHelloBundle | - http://symfony.com/ https://github.com/FriendsOfSymfony/FOSRestBundle http://williamdurand.fr/2012/08/02/rest-apis-with-symfony2-the-right-way/ https://github.com/nelmio/NelmioApiDocBundle better apis https://github.com/liip/LiipHelloBundle |
| - https://github.com/elc/rapi_doc https://github.com/Pajk/apipie-rails | - https://github.com/elc/rapi_doc https://github.com/Pajk/apipie-rails |
| - | - |
| http://amberonrails.com/building-stripes-api/ | |
| # Infographics and Data Visualisation {#data-visualisation} | # Infographics and Data Visualisation {#data-visualisation} |
| Infographics try to contextualise charts and graphs to tell a story. Data vis builds on this to find new ways to design insight. | Infographics try to contextualise charts and graphs to tell a story. Data vis builds on this to find new ways to design insight. |
| Most of the categories to follow have visualisation tools specific to their purpose. | Most of the categories to follow have visualisation tools specific to their purpose. |
| You can find some data visualisation “essential”\. tools below: | You can find some data visualisation “essential”\. tools below: |
| [http://www.visualisingdata.com/index.php/2011/07/part-6-the-essential-collection-of-visualisation-resources/](http://www.visualisingdata.com/index.php/2011/07/part-6-the-essential-collection-of-visualisation-resources/) | [http://www.visualisingdata.com/index.php/2011/07/part-6-the-essential-collection-of-visualisation-resources/](http://www.visualisingdata.com/index.php/2011/07/part-6-the-essential-collection-of-visualisation-resources/) |
| Also check out [http://thejit.org](http://thejit.org/) & [http://www.senchalabs.org/<wbr>philogl/</wbr>](http://www.senchalabs.org/philogl/) (contributed by Matt Adcock) | Also check out [http://thejit.org](http://thejit.org/) & [http://www.senchalabs.org/<wbr>philogl/</wbr>](http://www.senchalabs.org/philogl/) (contributed by Matt Adcock) |
| Have to use visual art concepts, good color schemes http://www.r-bloggers.com/the-paul-tol-21-color-salute/ | |
| # Mobile | # Mobile |
| html5 jquery mobile like directory.gov.au | html5 jquery mobile like directory.gov.au |
| - android datviz | - android datviz |
| - http://code.google.com/p/afreechart/ http://code.google.com/p/snowdon/ http://code.google.com/p/chartdroid/ http://androidplot.com/ http://code.google.com/p/achartengine/ | - http://code.google.com/p/afreechart/ http://code.google.com/p/snowdon/ http://code.google.com/p/chartdroid/ http://androidplot.com/ http://code.google.com/p/achartengine/ |
| # Geographical Data Tools {#geographical-data-tools} | # Geographical Data Tools {#geographical-data-tools} |
| Check out the[ GeoRabble Boundary Mapper’s Cookbook](http://georabble.org/2012/05/31/the-boundary-mappers-cookbook/) to see how you can tie all these things together! | Check out the[ GeoRabble Boundary Mapper’s Cookbook](http://georabble.org/2012/05/31/the-boundary-mappers-cookbook/) to see how you can tie all these things together! |
| ## Wrangling | |
| ## Converting | |
| There are many spatial data formats and often the one your tool requires is not the one the dataset is provided in | |
| Online | |
| - http://converter.mygeodata.eu/vector kml exporter for shp | |
| or locally using GDAL | |
| ## geocoding | |
| cloudmade, google (but you must display on a Google Map). | |
| ## Analysis | ## Analysis |
| ### PostGIS | ### PostGIS |
| [](http://www.govhack.org/wp-content/uploads/postgisexample.jpg)PostGIS is an extension for the PostgreSQL database server that allows you to store and manipulate geospatial data on a large scale. For example finding which points are in an area or what points are closest . It is also very useful for storing geospatial data because it can convert between all major formats including ESRI Shape files and Google Earth/Maps KML. | [](http://www.govhack.org/wp-content/uploads/postgisexample.jpg)PostGIS is an extension for the PostgreSQL database server that allows you to store and manipulate geospatial data on a large scale. For example finding which points are in an area or what points are closest . It is also very useful for storing geospatial data because it can convert between all major formats including ESRI Shape files and Google Earth/Maps KML. |
| ### Quantum GIS | ### Quantum GIS |
| [](http://www.govhack.org/wp-content/uploads/How-to-participate-in-GovHack_html_m50afbe88.jpg)QGIS is a graphical desktop application that allows viewing and editing of geospatial data. Some good base maps are available by adding the WMS layer/server [http://irs.gis-lab.info/](http://irs.gis-lab.info/) | [](http://www.govhack.org/wp-content/uploads/How-to-participate-in-GovHack_html_m50afbe88.jpg)QGIS is a graphical desktop application that allows viewing and editing of geospatial data. Some good base |