N.B. This is a work-in-progress
We're doing this so we can put in a submission to an Australian Senate committee that has the entire history of correspondence on RightToKnow. The committee is considering a bill related to Freedom of Information.
Well we would if we could but unfortunately the api doesn't return more than a small number of the most recent requests and so it's not possible to use the api to get the urls for all the requests. So, instead we are to scraping the pages. I know, it's not pretty.
To download the zipped version of the request we need to be logged in. I think this is the case to limit load on the server due to spiders, etc.. So we need login details so we can do this automatically for every request.
Make sure however that you don't user a superuser account to do this. Otherwise it will be able to see (and download) requests that are hidden. This could be very bad.