Skip to content
This repository has been archived by the owner on May 2, 2024. It is now read-only.

Move node_modules outside of edx-platform in Tutor's openedx image #150

Closed
15 tasks
kdmccormick opened this issue Jan 5, 2023 · 1 comment
Closed
15 tasks

Move node_modules outside of edx-platform in Tutor's openedx image #150

kdmccormick opened this issue Jan 5, 2023 · 1 comment
Assignees
@kdmccormick

Comments

@kdmccormick
Copy link
Collaborator

kdmccormick commented Jan 5, 2023 •
edited
Loading

Background

This is a sub-task of #146

The problem: NPM packages are installed into the openedx image at /openedx/edx-platform/node_modules. So, when you mount your own edx-platform, you override that node_modules folder. Now, you must re-install node_modules yourself (tutor dev run --mount=edx-platform lms npm install), which takes a long time and is completely redundant with the node_modules that you had to download when you downloaded the openedx image. If you forget to do this, then your edx-platform frontend will be broken.

The solution: Move node_modules somewhere else in the openedx image, such as /openedx/node_modules.

Tasks

  • In Tutor's Dockerfile, copy package.json and package-lock.json up to /openedx. Run npm install from within /openedx, causing the packages to be installed at /openedx/node_modules.
  • In Tutor's Dockerfile, update the PATH to add /openedx/node_modules/.bin instead of /openedx/edx-platform/node_modules/.bin
  • The edx-platform asset pipeline unfortunately needs to reach into node_modules and copy stuff out of it, notably in the case of node_modules dependencies that also need to be copied into "vendor" directories for use by non-Webpack frontend code. So: We need to update the assets pipeline to tolerate the fact that node modules have moved to /openedx/node_modules. As necessary, do one or more of these:
  • Test several frontend workflows to ensure that they still work with the changed node_modules location.
  • Triumphantly strike the npm clean-install step from Tutor's dev env setup instructions.
  • Add a section to the troubleshooting section along the lines of:
    • My NPM packages are out of date!
      • Are you bind-mounting edx-platform? If so: When you bind-mount edx-platform and it has a node_modules directory, then those node_modules will shadow any matching node_modules installed into the openedx image. So, if your edx-platform/node_modules is outdated, then the outdated packages will shadow the up-to-date packages on your image. To resolve this, simply remove your edx-platform/node_modules folder.

Notes

NPM checks first for node_modules in the current working directory, then checks the parent directory, then the grandparent directory, and so on. This is why we are able to locate NPM packages at /openedx/node_modules without totally breaking edx-platform. node_modules that exist in a mounted edx-platform
@kdmccormick kdmccormick self-assigned this Jan 5, 2023
@kdmccormick kdmccormick mentioned this issue Jan 5, 2023
Closed
@kdmccormick kdmccormick mentioned this issue Jan 5, 2023
Closed
@kdmccormick kdmccormick moved this from 📥 New to 📋 Refined in Developer Experience Working Group Jan 5, 2023
@kdmccormick kdmccormick moved this from 📋 Refined to 🛠️ In Progress in Developer Experience Working Group Jan 5, 2023
This was referenced Jan 6, 2023
Closed
Closed
Closed
kdmccormick added a commit to kdmccormick/tutor that referenced this issue Jan 10, 2023
@kdmccormick
perf: move node_modules out of edx-platform repo
a5d27a1 
Background: NPM packages were installed into the openedx image at
/openedx/edx-platform/node_modules. So, when one mounted their own copy of
edx-platform, the built-in node_modules folder was overriden. This
forced edx-platform developers to re-run ``npm install``, which is
redundant, slow, resource-intensive, and added complexity to the first-time
developer setup process. If one forgot to run ``npm install``, their
LMS/CMS frontend would be broken. And, when edx-platform's NPM requirements
were updated, developers would not receive these updates from the openedx
image; they would need to think to re-run ``npm install`` themselves.

The solution: Move /openedx/edx-platform/node_modules to
/openedx/node_modules. Note that this new location is outside of the
edx-platform repository. So, when a developer mounts their own copy of
edx-platform, the image's node_modules will remain intact.

Closes openedx-unsupported/wg-developer-experience#150
@kdmccormick kdmccormick moved this from 🛠️ In Progress to 👀 In Review in Developer Experience Working Group Jan 10, 2023
@kdmccormick kdmccormick moved this from 👀 In Review to ⚠️ Blocked in Developer Experience Working Group Jan 11, 2023
@kdmccormick kdmccormick moved this from ⚠️ Blocked to 🛠️ In Progress in Developer Experience Working Group Jan 17, 2023
@kdmccormick kdmccormick mentioned this issue Jan 17, 2023
Closed
kdmccormick added a commit to kdmccormick/tutor that referenced this issue Jan 18, 2023
@kdmccormick
perf: move node_modules out of edx-platform repo
45e9e40 
Background: NPM packages were installed into the openedx image at
/openedx/edx-platform/node_modules. So, when one mounted their own copy of
edx-platform, the built-in node_modules folder was overriden. This
forced edx-platform developers to re-run ``npm install``, which is
redundant, slow, resource-intensive, and added complexity to the first-time
developer setup process. If one forgot to run ``npm install``, their
LMS/CMS frontend would be broken. And, when edx-platform's NPM requirements
were updated, developers would not receive these updates from the openedx
image; they would need to think to re-run ``npm install`` themselves.

The solution: Move /openedx/edx-platform/node_modules to
/openedx/node_modules. Note that this new location is outside of the
edx-platform repository. So, when a developer mounts their own copy of
edx-platform, the image's node_modules will remain intact.

Closes openedx-unsupported/wg-developer-experience#150
This was referenced Jan 18, 2023
Closed
Closed
Closed
kdmccormick added a commit to kdmccormick/tutor that referenced this issue Jan 20, 2023
@kdmccormick
perf: move node_modules out of edx-platform repo
9cff095 
Background: NPM packages were installed into the openedx image at
/openedx/edx-platform/node_modules. So, when one mounted their own copy of
edx-platform, the built-in node_modules folder was overriden. This
forced edx-platform developers to re-run ``npm install``, which is
redundant, slow, resource-intensive, and added complexity to the first-time
developer setup process. If one forgot to run ``npm install``, their
LMS/CMS frontend would be broken. And, when edx-platform's NPM requirements
were updated, developers would not receive these updates from the openedx
image; they would need to think to re-run ``npm install`` themselves.

The solution: Move /openedx/edx-platform/node_modules to
/openedx/node_modules. Note that this new location is outside of the
edx-platform repository. So, when a developer mounts their own copy of
edx-platform, the image's node_modules will remain intact.

Closes openedx-unsupported/wg-developer-experience#150
@kdmccormick
Copy link
Collaborator Author

Blocked by openedx/edx-platform#31798

@kdmccormick kdmccormick moved this from 🛠️ In Progress to ⚠️ Blocked in Developer Experience Working Group Feb 24, 2023
@kdmccormick kdmccormick mentioned this issue Mar 8, 2023
Merged
kdmccormick added a commit to kdmccormick/tutor that referenced this issue Mar 13, 2023
@kdmccormick
feat: full edx-platform setup with tutor dev launch -m ...
505e9c4 
Before this commit, setting up an edx-platform development environment
took multiple steps:

  tutor dev launch
  tutor dev run --mount=/path/to/edx-platform lms bash
  >> pip install -e .
  >> npm clean-install
  >> openedx-assets build --env=dev

This commit moves the steps under ``run`` into an init task, which
is automatically run by ``launch``. Thus, setup is now one command:

  tutor dev launch --mount=edx-platform

These extra init steps are only applicable when bind-mounting
edx-platform (because bind-mounting the repository overrides
some important artifacts that exist on the image, which must be
re-generated). Thus, the new init tasks exists early if it detects
that it is *not* operating on a bind-mounted repository.

Part of: openedx-unsupported/wg-developer-experience#146
Closes: openedx-unsupported/wg-developer-experience#152

This works around (but does not close) these related issues:
* openedx-unsupported/wg-developer-experience#150
* https://github.com/openedx/wg-developer-experience/issues/151
kdmccormick added a commit to kdmccormick/tutor that referenced this issue Mar 13, 2023
@kdmccormick
feat: full edx-platform setup with tutor dev launch -m ...
9be70cc 
Before this commit, setting up an edx-platform development environment
took multiple steps:

  tutor dev launch
  tutor dev run --mount=/path/to/edx-platform lms bash
  >> pip install -e .
  >> npm clean-install
  >> openedx-assets build --env=dev

This commit moves the steps under ``run`` into an init task, which
is automatically run by ``launch``. Thus, setup is now one command:

  tutor dev launch --mount=edx-platform

These extra init steps are only applicable when bind-mounting
edx-platform (because bind-mounting the repository overrides
some important artifacts that exist on the image, which must be
re-generated). Thus, the new init tasks exists early if it detects
that it is *not* operating on a bind-mounted repository.

Finally, we try to simplify the Open edX development docs so that
it is clearer how bind-mounting fits into the development process.

Part of: openedx-unsupported/wg-developer-experience#146
Closes: openedx-unsupported/wg-developer-experience#152

This works around (but does not close) these related issues:
* openedx-unsupported/wg-developer-experience#150
* https://github.com/openedx/wg-developer-experience/issues/151
kdmccormick added a commit to kdmccormick/tutor that referenced this issue Mar 13, 2023
@kdmccormick
feat: full edx-platform setup with tutor dev launch -m ...
10f8233 
Before this commit, setting up an edx-platform development environment
took multiple steps:

  tutor dev launch
  tutor dev run --mount=/path/to/edx-platform lms bash
  >> pip install -e .
  >> npm clean-install
  >> openedx-assets build --env=dev

This commit moves the steps under ``run`` into an init task, which
is automatically run by ``launch``. Thus, setup is now one command:

  tutor dev launch --mount=edx-platform

These extra init steps are only applicable when bind-mounting
edx-platform (because bind-mounting the repository overrides
some important artifacts that exist on the image, which must be
re-generated). Thus, the new init tasks exists early if it detects
that it is *not* operating on a bind-mounted repository.

Finally, we try to simplify the Open edX development docs so that
it is clearer how bind-mounting fits into the development process.

Part of: openedx-unsupported/wg-developer-experience#146
Closes: openedx-unsupported/wg-developer-experience#152

This works around (but does not close) these related issues:
* openedx-unsupported/wg-developer-experience#150
* https://github.com/openedx/wg-developer-experience/issues/151
kdmccormick added a commit to kdmccormick/tutor that referenced this issue Mar 13, 2023
@kdmccormick
feat: full edx-platform setup with tutor dev launch -m ...
5876a05 
Before this commit, setting up an edx-platform development environment
took multiple steps:

   tutor dev launch
   tutor dev run --mount=/path/to/edx-platform lms bash
   >> pip install -e .
   >> npm clean-install
   >> openedx-assets build --env=dev

This commit moves the steps under ``run`` into an init task, which
is automatically run by ``launch``. Thus, setup is now one command:

   tutor dev launch --mount=edx-platform

These extra init steps are only applicable when bind-mounting
edx-platform (because bind-mounting the repository overrides
some important artifacts that exist on the image, which must be
re-generated). Thus, the new init tasks exists early if it detects
that it is *not* operating on a bind-mounted repository.

Finally, we try to simplify the Open edX development docs so that
it is clearer how bind-mounting fits into the development process.

Part of: openedx-unsupported/wg-developer-experience#146
Closes: openedx-unsupported/wg-developer-experience#152

This works around (but does not close) these related issues:
* openedx-unsupported/wg-developer-experience#150
* https://github.com/openedx/wg-developer-experience/issues/151
@kdmccormick kdmccormick mentioned this issue Mar 13, 2023
Closed
kdmccormick added a commit to kdmccormick/tutor that referenced this issue Mar 13, 2023
@kdmccormick
feat: full edx-platform setup with tutor dev launch -m ...
7222ab1 
Before this commit, setting up an edx-platform development environment
took multiple steps:

   tutor dev launch
   tutor dev run --mount=/path/to/edx-platform lms bash
   >> pip install -e .
   >> npm clean-install
   >> openedx-assets build --env=dev

This commit moves the steps under ``run`` into an init task, which
is automatically run by ``launch``. Thus, setup is now one command:

   tutor dev launch --mount=edx-platform

These extra init steps are only applicable when bind-mounting
edx-platform (because bind-mounting the repository overrides
some important artifacts that exist on the image, which must be
re-generated). Thus, the new init tasks exists early if it detects
that it is *not* operating on a bind-mounted repository.

Finally, we try to simplify the Open edX development docs so that
it is clearer how bind-mounting fits into the development process.

Part of: openedx-unsupported/wg-developer-experience#146
Closes: openedx-unsupported/wg-developer-experience#152

This works around (but does not close) these related issues:
* openedx-unsupported/wg-developer-experience#150
* https://github.com/openedx/wg-developer-experience/issues/151
kdmccormick added a commit to kdmccormick/tutor that referenced this issue Mar 13, 2023
@kdmccormick
feat: full edx-platform setup with tutor dev launch -m ...
ebab66a 
Before this commit, setting up an edx-platform development environment
took multiple steps:

   tutor dev launch
   tutor dev run --mount=/path/to/edx-platform lms bash
   >> pip install -e .
   >> npm clean-install
   >> openedx-assets build --env=dev

This commit moves the steps under ``run`` into an init task, which
is automatically run by ``launch``. Thus, setup is now one command:

   tutor dev launch --mount=edx-platform

These extra init steps are only applicable when bind-mounting
edx-platform (because bind-mounting the repository overrides
some important artifacts that exist on the image, which must be
re-generated). Thus, the new init tasks exists early if it detects
that it is *not* operating on a bind-mounted repository.

Finally, we try to simplify the Open edX development docs so that
it is clearer how bind-mounting fits into the development process.

Part of: openedx-unsupported/wg-developer-experience#146
Closes: openedx-unsupported/wg-developer-experience#152

This works around (but does not close) these related issues:
* openedx-unsupported/wg-developer-experience#150
* https://github.com/openedx/wg-developer-experience/issues/151
kdmccormick added a commit to kdmccormick/tutor that referenced this issue Mar 13, 2023
@kdmccormick
feat: full edx-platform setup with tutor dev launch -m ...
cac093b 
Before this commit, setting up an edx-platform development environment
took multiple steps:

   tutor dev launch
   tutor dev run --mount=/path/to/edx-platform lms bash
   >> pip install -e .
   >> npm clean-install
   >> openedx-assets build --env=dev

This commit moves the steps under ``run`` into an init task, which
is automatically run by ``launch``. Thus, setup is now one command:

   tutor dev launch --mount=edx-platform

These extra init steps are only applicable when bind-mounting
edx-platform (because bind-mounting the repository overrides
some important artifacts that exist on the image, which must be
re-generated). Thus, the new init tasks exists early if it detects
that it is *not* operating on a bind-mounted repository.

Finally, we try to simplify the Open edX development docs so that
it is clearer how bind-mounting fits into the development process.

Part of: openedx-unsupported/wg-developer-experience#146
Closes: openedx-unsupported/wg-developer-experience#152

This works around (but does not close) these related issues:
* openedx-unsupported/wg-developer-experience#150
* https://github.com/openedx/wg-developer-experience/issues/151
@kdmccormick kdmccormick mentioned this issue Mar 13, 2023
Merged
kdmccormick added a commit to kdmccormick/tutor that referenced this issue Mar 14, 2023
@kdmccormick
feat: full edx-platform setup with tutor dev launch -m ...
4fbe896 
Before this commit, setting up an edx-platform development environment
took multiple steps:

   tutor dev launch
   tutor dev run --mount=/path/to/edx-platform lms bash
   >> pip install -e .
   >> npm clean-install
   >> openedx-assets build --env=dev

This commit moves the steps under ``run`` into an init task, which
is automatically run by ``launch``. Thus, setup is now one command:

   tutor dev launch --mount=edx-platform

These extra init steps are only applicable when bind-mounting
edx-platform (because bind-mounting the repository overrides
some important artifacts that exist on the image, which must be
re-generated). Thus, the new init tasks exists early if it detects
that it is *not* operating on a bind-mounted repository.

Finally, we try to simplify the Open edX development docs so that
it is clearer how bind-mounting fits into the development process.

Part of: openedx-unsupported/wg-developer-experience#146
Closes: openedx-unsupported/wg-developer-experience#152

This works around (but does not close) these related issues:
* openedx-unsupported/wg-developer-experience#150
* https://github.com/openedx/wg-developer-experience/issues/151
regisb pushed a commit to overhangio/tutor that referenced this issue Mar 15, 2023
@kdmccormick
feat: full edx-platform setup with tutor dev launch -m ...
f984361 
Before this commit, setting up an edx-platform development environment
took multiple steps:

   tutor dev launch
   tutor dev run --mount=/path/to/edx-platform lms bash
   >> pip install -e .
   >> npm clean-install
   >> openedx-assets build --env=dev

This commit moves the steps under ``run`` into an init task, which
is automatically run by ``launch``. Thus, setup is now one command:

   tutor dev launch --mount=edx-platform

These extra init steps are only applicable when bind-mounting
edx-platform (because bind-mounting the repository overrides
some important artifacts that exist on the image, which must be
re-generated). Thus, the new init tasks exists early if it detects
that it is *not* operating on a bind-mounted repository.

Finally, we try to simplify the Open edX development docs so that
it is clearer how bind-mounting fits into the development process.

These bind-mounts:

* ../build/openedx/themes:/openedx/themes
* ../build/openedx/requirements:/openedx/requirements

existed in the dev lms and cms containers, but they did
not exist in the lms-job and cms-job containers.

This means that themes and requirements that were *built into the
image* would exist in the job containers, but live updates to the
themes and requirements would not apply.

To resolve this, we set ``volumes:`` on the lms-job and cms-job
services so that they match the volumes for the normal lms and
cms services.

Part of: openedx-unsupported/wg-developer-experience#146
Closes: openedx-unsupported/wg-developer-experience#152

This works around (but does not close) these related issues:
* openedx-unsupported/wg-developer-experience#150
* https://github.com/openedx/wg-developer-experience/issues/151
@kdmccormick kdmccormick mentioned this issue Mar 15, 2023
Closed
@kdmccormick kdmccormick closed this as not planned Won't fix, can't repro, duplicate, stale Aug 18, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@kdmccormick kdmccormick

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@kdmccormick