Skip to content

Latest commit

 

History

History
132 lines (95 loc) · 3.15 KB

README.md

File metadata and controls

132 lines (95 loc) · 3.15 KB
Raw

Gitter Archive

Gitter Chat room message downloader

npm npm

Usage

Install yarn package manager.

Install gitter-archive-cli as global.

$ yarn global gitter-archive-cli

# OR

$ npm install -g gitter-archive-cli

Add a .gitterarchiverc[.json|.js|.yml|.yaml] or gitterarchive.config.js file in the directory where you execute the CLI in the following format.

The configuration is loaded using cosmiconfig module.
The archive and no archive room lists are matched using extglob module.

NOTE: if you're copy pasting the following example, remove the comments.

{
  "rooms": {
    "noArchiveList": [
      // rooms not to archive
      // e.g.

      "FreeCodeCamp/FreeCodeCamp"
    ],
    "archiveList": [
      // list of rooms to archive
      // e.g.

      "FreeCodeCamp/*"
    ]
  },

  "groups": {
    "enabled": [{
      "uri": "FreeCodeCamp",
      "id": "57542cf4c43b8c6019778297"
    }],
    "disabled": [
      // groups not to be scanned
    ]
  }
}

Copy the example.env as .env and fill in the details.
Start the archive process.

$ gitter-archive

Use node v7 to run.

# run build
$ yarn run build

# start gitter archive
$ yarn run start

Development

$ yarn global nodemon
$ nodemon -q -x yarn run start:dev

OR for build watch

$ yarn global nodemon
$ yarn run build:watch

Then link it as global

# for globally making the cli and the module available
$ npm link

CLI

Features

CLI options (TODO)

  • --boost - boost archiving using skip parameter
  • -b - beforeId
  • --gid - group id (required)
  • --guri - group uri (optional)
  • --uri - room uri (optional)
  • --id - room id (required)
  • --ignore, -i - ignore room uri pattern (multiple)

Notes

With a limit of 100 messages/request and 100 requests/minute,
total downloadable messages/minute = 100 x 100 = 10,000 messages per minute

❓ With a skip limit of 5000, how can the scrapping be improved?

Known Bugs

  • After complete archive of rooms, they're marked isArchived as true.
    Hence, the next time the app is run, it removes the archived rooms from the list.
    The expected behavior is that it should simply ignore the archived rooms and only consider the non archived rooms.
    This results in archive pattern download behaving weird i.e. since it cannot verify if a particular room was archived, it redownloads the room and appends it to the end. (resulting in duplication)
  • HTTP Status 429 rate limiter error. Requests are not being rate limited even if using rate-limiter-api

TODO

  • Add cli processor modules
  • Update rooms
  • CLI should auto create .env and .gitterarchiverc.json if not present
  • Fetch group id given a room name
  • Fetch by room name