multi page marketing site scraping

[shanbady]

What are the relevant tickets?

Closes https://github.com/mitodl/hq/issues/7081

Description (What does it do?)

This PR allows us to scrape multiple pages for a given marketing site (for purposes of embedding).

How can this be tested?

1. Checkout this branch
2. rebuild your web and celery containers
3. set settings.EMBEDDINGS_EXTERNAL_FETCH_USE_WEBDRIVER = True
4. docker compose down/up celery
5. run the task to fetch marketing page data

from learning_resources.tasks import scrape_marketing_pages
scrape_marketing_pages.run()

6. inspect the content of the marketing pages:

from learning_resources.models import ContentFile
cfs = ContentFile.objects.filter(file_type="marketing_page")
print(cfs.first().content)

7. for micromasters program pages - it should contain content from all the pages (tabs at the top):

from learning_resources.models import ContentFile
ContentFile.objects.filter(file_type="marketing_page", learning_resource__url__icontains='micromasters')

[github-actions]

OpenAPI Changes

Show/hide No detectable change.

[abeglova]

lgtm