Prevent npm from ignoring .gitignore #36

codeofdiego · 2024-03-19T13:46:58Z

Issue

shanson 22 hours ago
@Diego I just published the big “boilerplate” refactor as a beta version to NPM and then tried to create a new app:
npx create-belt-app@beta MyNewApp123
But I’m getting this error:
[Error: ENOENT: no such file or directory, open '/Users/shanson/.npm/_npx/a87384e08d7b5c87/node_modules/create-belt-app/templates/boilerplate/.gitignore'] {
errno: -2,
code: 'ENOENT',
syscall: 'open',
path: '/Users/shanson/.npm/_npx/a87384e08d7b5c87/node_modules/create-belt-app/templates/boilerplate/.gitignore'
}
I’ve found that the templates/boilerplate/.gitignore does not get packed into the published NPM module, as NPM ignores .gitignore files. Relevant issue. Can you take a look at this and propose or work on a solution? Getting a fix in place so the tool works is a top priority ahead of other Belt tasks.

Options

Use .gitignore.eta or other template naming
1. Worked for npm! However, won't work for local builds from the boilerplate folder and adds the risk of introducing undesired files to source control.
Use exclusion on parent !.gitignore
1. Did not work.
Add "templates/boilerplate/.gitignore" explicitly to the package.json's files array.
1. Did not work.
Zip boilerplate and unpack on postinstall script
1. Did not try as it sounded excessive in light of other options.
Add .npmignore to root and/or boilerplate folder
1. Empty
  - Did not work.
2. With exclusion rule for !.gitignore
  - Worked! However, the contents of .gitignore need to be mirrored in .npmignore for NPM to follow the same exclusions and prevent generated project files from being packed.
  - I also tried to add a more generalist rule to the parent .npmignore as !**/.gitignore to make NPM keep any .gitignore file but that didn't work. To keep them the .npmignore file has to be in the same folder (sibling file).

Solution

I went with solution 5.2 as it uses the built-in .npmignore functionality, adds less complexity to the project over other options, and resolves the issue as designed by NPM.

It worked! However, contents of `.gitignore` need to be mirrored in `.npmignore` in order for NPM to follow the same exclusions and prevent generated project files from being packed.

rakeshpetit · 2024-03-22T09:54:04Z

templates/boilerplate/.npmignore

+/.cache
+
+# Prevent NPM from ignoring the .gitignore file
+!.gitignore


We can add a line break in the end to avoid that minus.

rakeshpetit · 2024-03-22T09:55:50Z

templates/boilerplate/.gitignore

@@ -1,5 +1,7 @@
 # Learn more https://docs.github.com/en/get-started/getting-started-with-git/ignoring-files

+## ATTENTION: Files ignored here must be mirrored in the sibling .npmignore file


Is this not a problem with other package managers such as yarn or bun? Is it because we hit this problem only while using npx?

@rakeshpetit this is an issue with the npm package registry when packaging the library and publishing to npm. Yarn and bun don’t have package registries, so are not relevant here.

Since this file gets copied into new apps generated by Belt, will this comment cause confusion?

stevehanson

Thanks for the detailed writeup and for tackling this.

This approach seems to work well, though I have some concerns about having these three files (boilerplate/.gitignore, boilerplate/.npmignore, .gitignore), each with similar but slightly different purposes. I'm worried that this will become too difficult to reason about:

boilerplate/.gitignore: the gitignore for the generated app. This also has a secondary function as a gitignore for the boilerplate directory in this repo
boilerplate/.npmignore: tells npm which files to ignore when packaging, main purpose of this is to tell it to include .gitignore which otherwise would be excluded. Have to include everything that would normally be in .gitignore or else we run the risk of accidentally packaging unintended files, even when git status shows a clean working copy. This seems like a big danger.
.gitignore: this file in the root can specify additional files to ignore from the boilerplate directory in the Belt repo. Right now, we use this to specify files that we'd want to be committed in a generated app but not in the Belt repo (eg. package lock files)

My concerns over managing the .npmignore and the risk of accidentally packaging unintended files even when git shows a clean working copy make me inclined to want to find an alternative solution. Maybe this is still the best way forward, but I am interested in seeing if there might be another option.

Are there any other solutions that come to mind for this? I can do some exploration as well.

stevehanson · 2024-03-22T13:47:13Z

templates/boilerplate/.gitignore

@@ -1,5 +1,7 @@
 # Learn more https://docs.github.com/en/get-started/getting-started-with-git/ignoring-files

+## ATTENTION: Files ignored here must be mirrored in the sibling .npmignore file


Since this file gets copied into new apps generated by Belt, will this comment cause confusion?

stevehanson · 2024-03-22T14:00:31Z

templates/boilerplate/.npmignore

@@ -0,0 +1,45 @@
+# Learn more https://docs.github.com/en/get-started/getting-started-with-git/ignoring-files


Since the .gitignore gets copied to user apps and therefore isn't an ideal place to document the behavior we're fixing, what would you think about instead documenting the purpose of this file and instructions for how to use it as a comment here?

stevehanson · 2024-03-22T15:23:36Z

I wanted to add some additional context and propose a potential solution.

Context

templates/boilerplate/.gitignore file serves three purposes today:

Template that should be copied to new app
Used by git in Belt repo to keep certain files from being committed to Belt
Used by npm to determine which files to package

The .npmignore solution works well for all of these, and it additionally decouples (2) and (3) above, for better or worse.

Downsides of npmignore

The main downsides I see are that it introduces potential for accidentally packaging unintended files because it is difficult to keep in sync with .gitignore.

Proposal

What if we keep the .npmignore solution that you introduced, but instead of managing it independently, we auto-generate it as part of an npm prepack script? The prepack script is called automatically before npm pack or npm publish. Auto-generating this file ensures that it matches the .gitignore exactly (but with any necessary changes needed). I believe this alleviates all of the downsides I listed above. We could probably keep the file out of version control, since it's auto-generated and only needed when packaging.

Does this seem like it would solve the issues?

stevehanson · 2024-03-22T21:25:23Z

Unfortunately, it seems that this approach will not work after all, details below.

I just pushed up the change to dynamically generate the .npmignore from the .gitignore. I then published a beta to NPM with:

# bumps version to 0.3.1
npm version patch

# this runs "yarn build && npm publish"
yarn pub:beta

The output of the npm publish looked good and indicated that the .gitignore was added to the package:

I then ran npx create-belt-app@beta MyBetaApp and got the same error as we were getting before, no such file or directory for templates/boilerplate/.gitignore. Looking at the package in my file system, after confirming that is is indeed the beta version that I just published, I see that the .gitignore has been renamed to .npmignore in the published package:

even though npm shows that the .gitignore file is correctly published with v0.3.1 of the package, and that there is no .npmignore file.

This seems to be documented in the linked issue thread from the PR description as well as in npm/npm#12917 and npm/npm#7252. It seems that even though we're packaging the file correctly, npm install when installing it renames the .gitignore to .npmignore. So, it looks like we're back to the drawing board on this one.

stevehanson · 2024-03-22T22:09:38Z

I started playing around with an alternative approach:

Rename .gitignore to .gitignore.eta
Add a script to generate .gitignore from .gitignore.eta
Add a pre-commit hook that ensures the .gitignore and .gitignore.eta file are kept in sync

This seems to be working but will need some cleanup, code is here: https://github.com/thoughtbot/belt/compare/fix-gitignore...sh/fix-gitignore?expand=1. Feel free to merge it into this branch, or I can do that.

Right now if you try to commit and the .gitignore is not in sync with .gitignore.eta, the commit will fail with a message to run a script to update it. It might be nicer to automatically run that script and git add the updated .gitignore.

stevehanson · 2024-03-29T16:03:38Z

I'll go ahead and assign myself on this and get the fix in place.

stevehanson · 2024-03-29T18:52:07Z

Closing in favor of #39

This PR supercedes #36. See that ticket for full context. At a high level, the issues are: * Our app boilerplate includes a `.gitignore` file that should be copied for all new apps * This `.gitignore` file also needs to be in place so the git repo ignores certain files in `templates/boilerplate` _and_ so NPM doesn't package those same files with the package * NPM does not by default allow including a `.gitignore` file in a package * Even though we eventually found a way to add the `.gitignore` in the raw package, NPM renames this to `.npmignore` when installing our package with `npm install` Solution: * Rename `templates/boilerplate/.gitignore` to `templates/boilerplate/.gitignore.eta` * When copying the boilerplate for new apps, the `.eta` suffix will automatically be removed. This takes advantage of the ETA template parsing that is already implemented in Belt * Also include `templates/boilerplate/.gitignore`, so that git and npm properly ignore the specified files * To ensure that `.gitignore` and `.gitignore.eta` stay in sync, add a `pre-commit` Husky hook that validates the files match. If they do not match, instruct the developer to run `node bin/generate-gitignore.js` to re-generate `.gitignore` from `.gitignore.eta` * I opted to make this manual and not have the pre-commit hook auto-generate the `.gitignore` because I didn't want it to inadvertently overwrite changes someone had accidentally made to `.gitignore`. Here is a [🎥 video walkthrough](https://www.loom.com/share/8d0b82fe01094295ac8fcce5afc7e9fa?sid=eff2f5d0-e397-4663-939b-327b9f0c9636) as well where I explain this all. --------- Co-authored-by: Diego Oliveira <[email protected]>

Prevent npm from ignoring .gitignore

c683f41

It worked! However, contents of `.gitignore` need to be mirrored in `.npmignore` in order for NPM to follow the same exclusions and prevent generated project files from being packed.

codeofdiego requested a review from stevehanson as a code owner March 19, 2024 13:46

codeofdiego self-assigned this Mar 19, 2024

rakeshpetit reviewed Mar 22, 2024

View reviewed changes

stevehanson reviewed Mar 22, 2024

View reviewed changes

Stephen Hanson added 2 commits March 22, 2024 16:00

Generate npmignore when publishing so it stays in sync

a5ee39e

0.3.1

368c1f8

stevehanson self-assigned this Mar 29, 2024

stevehanson mentioned this pull request Mar 29, 2024

Fix issue where .gitignore not part of NPM package #39

Merged

stevehanson closed this Mar 29, 2024

stevehanson deleted the fix-gitignore branch March 29, 2024 18:52

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Prevent npm from ignoring .gitignore #36

Prevent npm from ignoring .gitignore #36

codeofdiego commented Mar 19, 2024

rakeshpetit Mar 22, 2024

rakeshpetit Mar 22, 2024

stevehanson Mar 22, 2024

stevehanson Mar 22, 2024

stevehanson left a comment

stevehanson Mar 22, 2024

stevehanson Mar 22, 2024

stevehanson commented Mar 22, 2024

stevehanson commented Mar 22, 2024

stevehanson commented Mar 22, 2024 •

edited

Loading

stevehanson commented Mar 29, 2024

stevehanson commented Mar 29, 2024

		@@ -1,5 +1,7 @@
		# Learn more https://docs.github.com/en/get-started/getting-started-with-git/ignoring-files

		## ATTENTION: Files ignored here must be mirrored in the sibling .npmignore file

		@@ -0,0 +1,45 @@
		# Learn more https://docs.github.com/en/get-started/getting-started-with-git/ignoring-files

Prevent npm from ignoring .gitignore #36

Prevent npm from ignoring .gitignore #36

Conversation

codeofdiego commented Mar 19, 2024

Issue

Options

Solution

rakeshpetit Mar 22, 2024

Choose a reason for hiding this comment

rakeshpetit Mar 22, 2024

Choose a reason for hiding this comment

stevehanson Mar 22, 2024

Choose a reason for hiding this comment

stevehanson Mar 22, 2024

Choose a reason for hiding this comment

stevehanson left a comment

Choose a reason for hiding this comment

stevehanson Mar 22, 2024

Choose a reason for hiding this comment

stevehanson Mar 22, 2024

Choose a reason for hiding this comment

stevehanson commented Mar 22, 2024

Context

Downsides of npmignore

Proposal

stevehanson commented Mar 22, 2024

stevehanson commented Mar 22, 2024 • edited Loading

stevehanson commented Mar 29, 2024

stevehanson commented Mar 29, 2024

stevehanson commented Mar 22, 2024 •

edited

Loading