diff --git a/app/views/_tailwind/documentation/how_to_write_a_scraper.html.erb b/app/views/_tailwind/documentation/how_to_write_a_scraper.html.erb index 84ee6dca6..84b386d3c 100644 --- a/app/views/_tailwind/documentation/how_to_write_a_scraper.html.erb +++ b/app/views/_tailwind/documentation/how_to_write_a_scraper.html.erb @@ -2,7 +2,7 @@ <%# TODO: Format all the content to match the rest of the site %> <%# TODO: Extract this prose block into a component %> -
+

<%= yield :page_title %>

@@ -72,158 +72,164 @@ <%= link_to "create a new scraper", "https://morph.io/scrapers/new" %> that downloads and saves the following information:

-

- <%# TODO: Add highlight to required? %> - The following fields are required. All development applications should have these bits of information. -

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
fieldExample valueDescription
council_referenceTA/00323/2012 -

- The ID that the council has given the planning application. This also must be the unique key for this data set. -

-
address1 Sowerby St, Goulburn, NSW -

- The physical address that this application relates to. This will be geocoded so doesn't need to be a specific - format but obviously the more explicit it is the more likely it will be successfully geo-coded. If the original - address did not include the state (e.g. "QLD") at the end, then add it. -

-
descriptionGround floor alterations to rear and first floor addition -

- A text description of what the planning application seeks to carry out. -

-
info_urlhttp://foo.gov.au/app?key=527230 -

- A URL that provides more information about the planning application. -

-

- This should be a persistent URL that preferably is specific to this particular application. In many cases councils force - users to click through a license to access planning application. In this case be careful about what URL you provide. - Test clicking the link in a browser that hasn't established a session with the council's site to ensure users of Planning Alerts - will be able to click the link and not be presented with an error. -

-
date_scraped2012-08-01 -

- The date that your scraper is collecting this data (i.e. now). Should be in - <%= link_to "ISO 8601", "http://en.wikipedia.org/wiki/ISO_8601" %> - format. -

-

- Use the following Ruby code: - Date.today.to_s -

-
-

- Note that there used to be a field "comment_url" - above that was required. This is no longer used though you might - still see it referenced in older scrapers. -

-

- The following fields are optional because not every planning authority provides them. Please do include them if data is available. -

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
fieldExample valueDescription
date_received2012-06-23 -

- The date this application was received by council. Should be in - <%= link_to "ISO 8601", "http://en.wikipedia.org/wiki/ISO_8601" %> - format. -

-
on_notice_from2012-08-01 -

- The date from when public submissions can be made about this application. Should be in - <%= link_to "ISO 8601", "http://en.wikipedia.org/wiki/ISO_8601" %> - format. -

-
on_notice_to2012-08-14 -

- The date until when public submissions can be made about this application. Should be in - <%= link_to "ISO 8601", "http://en.wikipedia.org/wiki/ISO_8601" %> - format. -

-
comment_emailfoo@bar.com -

- Only set this in - extremely unusual - situations. Allows each application in a single - planning authority to go to a different email address. This should never be set for 99.9% - of authorities as a single email address is used for all comments. Currently this is only - used for SA Planning Portal where comments are ideally sent back to the originating - local council so that the staff in state government don't have to do the redirection by hand. -

-
comment_authorityAcme Council -

- Only set this in - extremely unusual - situations. Give the name associated with the comment_email address. -

-
+
+

Required fields

+

+ <%# TODO: Add highlight to required? %> + The following fields are required. All development applications should have these bits of information. +

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
fieldExample valueDescription
council_referenceTA/00323/2012 +

+ The ID that the council has given the planning application. This also must be the unique key for this data set. +

+
address1 Sowerby St, Goulburn, NSW +

+ The physical address that this application relates to. This will be geocoded so doesn't need to be a specific + format but obviously the more explicit it is the more likely it will be successfully geo-coded. If the original + address did not include the state (e.g. "QLD") at the end, then add it. +

+
descriptionGround floor alterations to rear and first floor addition +

+ A text description of what the planning application seeks to carry out. +

+
info_urlhttp://foo.gov.au/app?key=527230 +

+ A URL that provides more information about the planning application. +

+

+ This should be a persistent URL that preferably is specific to this particular application. In many cases councils force + users to click through a license to access planning application. In this case be careful about what URL you provide. + Test clicking the link in a browser that hasn't established a session with the council's site to ensure users of Planning Alerts + will be able to click the link and not be presented with an error. +

+
date_scraped2012-08-01 +

+ The date that your scraper is collecting this data (i.e. now). Should be in + <%= link_to "ISO 8601", "http://en.wikipedia.org/wiki/ISO_8601" %> + format. +

+

+ Use the following Ruby code: + Date.today.to_s +

+
+

+ Note that there used to be a field "comment_url" + above that was required. This is no longer used though you might + still see it referenced in older scrapers. +

+
+
+

Optional fields

+

+ The following fields are optional because not every planning authority provides them. Please do include them if data is available. +

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
fieldExample valueDescription
date_received2012-06-23 +

+ The date this application was received by council. Should be in + <%= link_to "ISO 8601", "http://en.wikipedia.org/wiki/ISO_8601" %> + format. +

+
on_notice_from2012-08-01 +

+ The date from when public submissions can be made about this application. Should be in + <%= link_to "ISO 8601", "http://en.wikipedia.org/wiki/ISO_8601" %> + format. +

+
on_notice_to2012-08-14 +

+ The date until when public submissions can be made about this application. Should be in + <%= link_to "ISO 8601", "http://en.wikipedia.org/wiki/ISO_8601" %> + format. +

+
comment_emailfoo@bar.com +

+ Only set this in + extremely unusual + situations. Allows each application in a single + planning authority to go to a different email address. This should never be set for 99.9% + of authorities as a single email address is used for all comments. Currently this is only + used for SA Planning Portal where comments are ideally sent back to the originating + local council so that the staff in state government don't have to do the redirection by hand. +

+
comment_authorityAcme Council +

+ Only set this in + extremely unusual + situations. Give the name associated with the comment_email address. +

+
+