You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
If you have Docker installed, use `docker-compose up` to start up the multi-container for `Splash` and `Playwright`-integration.
35
36
36
37
As a last step, set up your config variables by copying the `.env.example`-file and modifying it if necessary:
@@ -40,7 +41,7 @@ As a last step, set up your config variables by copying the `.env.example`-file
40
41
# Running crawlers
41
42
42
43
- A crawler can be run with `scrapy crawl <spider-name>`.
43
-
- (It assumes that you have an edu-sharing v6.0+ instance in your `.env` settings configured which can accept the data.)
44
+
- (It assumes that you have an edu-sharing v8.1+ instance in your `.env` settings configured which can accept the data.)
44
45
- If a crawler has [Scrapy Spider Contracts](https://docs.scrapy.org/en/latest/topics/contracts.html#spiders-contracts) implemented, you can test those by running `scrapy check <spider-name>`
45
46
46
47
@@ -60,8 +61,10 @@ docker compose up
60
61
61
62
- We use Scrapy as a framework. Please check out the guides for Scrapy spider (https://docs.scrapy.org/en/latest/intro/tutorial.html)
62
63
- To create a new spider, create a file inside `converter/spiders/<myname>_spider.py`
63
-
- We recommend inheriting the `LomBase` class in order to get out-of-the-box support for our metadata model
64
-
- You may also Inherit a Base Class for crawling data, if your site provides LRMI metadata, the `LrmiBase` is a good start. If your system provides an OAI interface, you may use the `OAIBase`
64
+
- We recommend inheriting the `LomBase` class to get out-of-the-box support for our metadata model
65
+
- You may also inherit a base class (see: `converter/spiders/base_classes/`) for crawling data.
66
+
- If your site provides LRMI metadata, the `LrmiBase` is a good start.
67
+
- If your system provides an OAI interface, you may use the `OAIBase`
65
68
- As a sample/template, please take a look at the `sample_spider.py` or `sample_spider_alternative.py`
66
69
- To learn more about the LOM standard we're using, you'll find useful information at https://en.wikipedia.org/wiki/Learning_object_metadata
0 commit comments