New feature idea/request: cross-module, project-contained, functions

[CeresBarros]

At this moment, cross-module functions must either be created in .GlobalEnv or be contained in packages (or else repeated in each module's R folder). Although packages is the way to go, it can represent a major barrier to many users -- package development has an additional learning curve to SpaDES. On the other hand, cluttering the .GlobalEnv with functions is not great.

So I thought that it would be great/awesome if SpaDES could use/create project-specific functions to be used across modules. An idea of how to implement this would be:

• project-wide functions are contained in scripts located in separate folder, with is passed to "a new setPath"(e.g "functionsPath").
• these functions are either be assigned to each module's environment, or just once to the sim environment

[eliotmcintire]

I like this as a stepping stone to "build a package!".

On the first point... I think it would be reasonable to "just" have an R directory at the project level (without a functionPath). That is what all packages and modules and other workflow tools are doing (e.g., rcompendium ). Furthermore, it would take quite a bit of work to add a new path at this level as there are a lot of places that rely on the current structure. So, while not against this, it seems like it is not necessary, for the work involved.

On the second point, we just have to source those functions into a new environment that gets inserted (new.env(parent = ....)) between the module functions and SpaDES.core, which should be fairly straightforward.

[CeresBarros]

Uhm.. wait, I think I may have jumped the gun here.
If modules depend on these functions they will need, somewhow, to be part of the module-specific R folder (or a package specified in the metadata) to ensure module transferability/interoperability. Otherwise, anyone pulling the module separately from the original project will not have access to the functions.

[eliotmcintire]

Mmm. Good point. How about this:

The project could "be" an R package (this is what rcompendium does). So, if the modules use the functions from the "project", the reqdPkgs entry in the metadata for each module would simply list that package, and it (via Require) would ensure that it has that package installed... there may be some details that would have to be dealt with... namely, this would only work if the project is "on" GitHub; also, updates to the source code locally would have to dealt with... probably other things too.

[CeresBarros]

Let me see if I get what you meant... Require would detect that the "package" Project A, is really not an R package but a project with an R folder (presumably with a specific name and location within the project folder tree) and simply source the functions (instead of installing the R package)?

[eliotmcintire]

An package is minimally defined by having a DESCRIPTION file (with a few key elements, like title). An R package doesn't "need" anything else. But there are standards that make a package work... namely, things such as R source files in a folder named R, documentation in a folder called man, a NAMESPACE file etc.

So, Require will just "install" the package that is the project because it has a DESCRIPTION file... with this, it will get all the R source files, and install them as if they were a package.

[achubaty]

I like the idea of 'stepping stone' too but I'm not sure if it's quite that straightforward. It's not just the R/ directory that's looked at when building a package, and any/all other files not part of the 'package' may need (at minimum) to be .Rbuildignored.

However, an R package can be in a subdirectory of e.g., a git repo and still be installed using the usual tools:

remotes::install_github("mfrasca/r-logging/pkg")
devtools::install("thePkg") ## current project's 'thePkg/' directory

[eliotmcintire]

We already have all the infrastructure to convert a module to a package, and rcompendium and others do the same... so the logistics of doing this are known.

I think the issue of whether or not to do it is conceptual.

What is a way to make source code useable by multiple modules without knowing how to build and maintain an R package?

My gut feeling is using sub-directories will cause more confusion than it is worth. There is the complicated case of "when put in a sub-directoyr, the name of the package is commonly different than the repository". I find this one particularly annoying and frankly quite difficult to work with as a user. I mean we can create: PredictiveEcology/BC_FFEC/BC_FFEC/R with the package called BC_FFEC, but that seems kind of silly to put it in a subdirectory.

On the other hand, we don't actually have to make any infrastructure to do or not do this. A user can just "do this" without us. But, they kind of have to maintain a package... which was the original motivation for having something different.

If "we" do something, it would be something like:

1. if a project has an R folder at the top level and it uses SpaDES.core...
2. any module that needs these functions adds this package name to their reqdPkgs with the same name as the repository/project.
3. If the module is used within the project with the source code in it, then it does install it like step 2, but also load_all each time (so that the user can be editing/working on the source code).

[CeresBarros]

Sorry for the late reply.
On @eliotmcintire's point no. 1:
So when a module that is not on the project that has the source code in it (so the project's <repo/name> is "only" in reqdPkgs):

1. the user will be forced to used Require to "install" the project, right?
2. will Require simply "source" functions in the R folder, or will somehow the whole project's folder structure be downloaded by Require? (I guess we need to test and see?)

Another overall comment:
I like both ideas of project-level "package" and module-level "package", and can see use cases for both. For instance, (some) fireSense functions could easily be thought of as simply shared across FireSense modules and contained within one of them (if we didn't have the fireSense package). Whereas there may be other functions that are transversal to modules that have no a priori similarities between them (e.g. across fire and caribou modules) - we've typically put these in packages but they could now live in a project.

One thing that the developer will need to remember is that if the modules are hosted publically and depend on project-level functions, the project will need to be public as well