diff --git a/.github/README.md b/.github/README.md deleted file mode 100644 index e070e1e..0000000 --- a/.github/README.md +++ /dev/null @@ -1,44 +0,0 @@ -# scraper - -A set of utilities focused on scraping the [VEX Robotics Q&A](https://www.robotevents.com/V5RC/2024-2025/QA/). - -## Usage - -### Retreiving All Questions - -```ts -import { getAllQuestions } from "@qnaplus/scraper"; - -(async () => { - const questions = await getAllQuestions(); -})(); -``` - -### Retreiving and Filtering Questions - -```ts -import { getQuestions } from "@qnaplus/scraper"; - -(async () => { - // gets all questions from the current season - const currentSeasonQuestions = await getQuestions(); - - // get all questions from a particular season - const specificSeasonQuestions = await getQuestions(["2020-2021"]); - - // get all VURC questions from the 2020-2021 season - const filteredQuestions = await getQuestions({ - VURC: ["2020-2021"] - }); -})(); -``` - -### Retrieving Unanswered Questions - -```ts -import { getUnansweredQuestions } from "@qnaplus/scraper"; - -(async () => { - const questions = await getUnansweredQuestions(); -})(); -``` diff --git a/README.md b/README.md new file mode 100644 index 0000000..8ba6c15 --- /dev/null +++ b/README.md @@ -0,0 +1,70 @@ +# scraper + +A set of utilities focused on scraping the [VEX Robotics Q&A](https://www.robotevents.com/V5RC/2024-2025/QA/). + +## Usage + +```ts +import { + getAllQuestions, + getQuestions, + getUnansweredQuestions +} from "@qnaplus/scraper"; + +(async () => { + const questions = await getAllQuestions(); + + // gets all questions from the current season + const currentSeasonQuestions = await getQuestions(); + + // get all questions from a particular season + const specificSeasonQuestions = await getQuestions(["2020-2021"]); + + // get all VURC questions from the 2020-2021 season + const filteredQuestions = await getQuestions({ + VURC: ["2020-2021"] + }); + + // Get all unanswered questions for the current season + const questions = await getUnansweredQuestions(); +})(); +``` + +## Logging + +`@qnaplus/scraper` supports logging through [pino](https://github.com/pinojs/pino). This means that for most functions, you can provide a logger instance to trace the operations within the function, like so: + +```ts +import { getUnansweredQuestions } from "@qnaplus/scraper"; +import { pino } from "pino"; + +(async () => { + const logger = pino(); + const questions = await getUnansweredQuestions({ logger }); +})(); +``` + +## Strategies + +At times, the default strategy (powered by [got-scraping](https://github.com/apify/got-scraping)) used to make requests will fail. `@qnaplus/scraper` allows you to pass custom clients to work around this. Create your own or install [@qnaplus/scraper-strategies](https://github.com/qnaplus/scraper-strategies) for additional clients. + +```ts +import { getUnansweredQuestions } from "@qnaplus/scraper"; +import { CurlImpersonateScrapingClient } from "@qnaplus/scraper-strategies"; +import { pino } from "pino"; + +(async () => { + const logger = pino(); + const questions = await getUnansweredQuestions({ + logger, + client: new CurlImpersonateScrapingClient(logger) + }); +})(); +``` + +## Contributing + +1. Clone the repository. Ensure you have [volta](https://volta.sh/) installed, or you are using the nodejs/yarn versions defined in the `volta` field in the `package.json`. +2. Add your contribution(s) and commit them. This project loosely follows [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/). +3. In a separate commit, run `yarn neat`. +4. In a separate commit, bump the version. diff --git a/package.json b/package.json index 398744c..06d24e2 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "@qnaplus/scraper", - "version": "2.0.1", + "version": "2.0.2", "description": "Utility package to scrape questions from the VEX Robotics Q&A.", "main": "dist/index.js", "repository": {