Separating the IDE into "sub projects"

[arthuro555]

A currently trending repository architecture are monorepos with many small subpackages, constituting the main software when put together, like for example gatsby, pixi or CRA.

While looking into this, I had an idea: what if we split some parts of the IDE in separate packages?

I see GDevelop tools based on web technologies slowly popping up (GDInstaller, GDevelop-Project-Manager), it would be nice if we had the newIDE/app/src/UI folder published as a component library on npm, to allow them to use the same UI elements and be familiar to GDevelop users.

We could even split some more "high level" components like the events editor as a package too, to allow interesting tools to be created (for example, this could be used to make a "GDevelop git client", by rendering the diff with the events editor and allowing to fix conflicts using the events editor, or to make a "FSM maker", where you can have an FSM as flow graph and attach/edit some events for one specific node)...

The things I think we could split into npm packages:

• The GDevelop UI elements with the Theming system
  • This would make imports way nicer: import SemiControlledAutoComplete from '../../../../UI/SemiControlledAutoComplete'; -> import { SemiControlledAutoComplete } from '@gdevelop/ui'
• The GDevelop Core (GDevelop.js) and its types definitions (allowing tools like gdexporter to not rely on the hacky gdcore-tools I made)
  • Note that if we do the system well, this could allow a cleaner usage of the library, by for example const gd: libGDevelop = global.gd; const project = new gd.Project() -> import { Project } from '@gdevelop/core'; const project = new Project();
• The GDevelop events editor
• The GDevelop scene editor

This would also make the IDE code less intimidating, it is currently looking very scary with the 51 subfolders in newIDE/app/src 😱 . Having some separated packages like this allow adding more structure, making the project less intimidating and easier to navigate for new contributors.

If those are published on npm and used by the community, in addition to growing the GDevelop ecosystem, this would also:

• Force us to more carefully design the API of components to be flexible yet comprehensible and more reusable
• Force us to document more and better the components
• Discover and fix new bugs with the components when used in different situations, avoiding confusion and extra work if we want to use the component for a similar situation in GDevelop.

All of this would therefore also allow providing a better development experience to everyone.

I hope to hear your opinion on this, because I think it could be a huge positive change.

[4ian]

I think this is going on the borderline of over-engineering ;)
Basically, the justification should never be "I see a trend in doing this, so we should be doing it too". Let's try to consider what gains this would provide and what are the risks/disadvantages.

The most obvious approach is that this would make it easier, as you explained, for third party tools to reuse things. It would be a good thing.
But this cost with the costs of:

• Making us less agile because every component is now maybe public and can be relied on by external tools and so we have yet another public interface (in addition to the game engine and the project serialization) to maintain (or at least use semver).
• Making it harder to make big refactoring if we need to break something/rename something/rework something.
  • This seems not a huge thing, but I've seen real life engineering projects where renaming a prop in a component is not done because "well we risk breaking other stuff, so let's just keep the bad name...". What a shame! In GDevelop, it's easy: "search for your prop name in the codebase, replace it, done. If you made a mistake, Flow (or TS in the future) will yell at you, so you're 99% safe".
• Make it harder to track bugs because you have to "jump" between codebases or projects.
  • This is especially the case in the "micro services" architectures, where debugging can be a nightmare. Less the case for frontends/apps, but still.
• More complexity of the build. Right now it's just "run npm start, you're done". Everything is handled by CRA. If we add a monorepo approach (which by the way, we already are ;)), we start to have issues like "you have to learn lerna, or yarn workspace, etc...". I can already see myself wasting 5 hours to upgrade Storybook because "it breaks the build in a component".

I think we sometimes go for complex tools because of the sake of "see it will be more flexible" and overlook all the risks/complexity that come with it.

it is currently looking very scary with the 51 subfolders in newIDE/app/src 😱 . Having some separated packages like this allow adding more structure

We can already make folders inside subfolders :) And questions like "When do we need to create a package? What if I think that a component should be moved to one package to another? " are answered very simply with folders: drag'n'drop the files, let VS Code auto update the imports, done. 10 seconds. That's how we keep a very fast development speed.

import { Project } from '@gdevelop/core'; const project = new Project();

Same, if you want this, you can just refactor the library to be imported as a module. It's already possible :)

This being said, I know the current approach is not giving all the answers - it's hard for third party tools to integrate. But maybe we can:

1. improve some parts of GDevelop to make it easier (like publish only some very specific components to npm)
2. Explicitly tell for others that it's not our focus to expose them externally.

For example:

newIDE/app/src/UI folder published as a component library on npm

We could do this, but then we have to maintain it. We have so much features and work to do on GDevelop, are we really ready to maintain a UI library, handle bugs in it, answer to feature requests, ensure we follow semver (and so constraining us?).

Basically, I'm afraid we're lacking some focus here. Our focus is to make the greatest game engine that is something so easy to use, intuitive and well designed that it can be used by everyone. Tooling made around GDevelop is very welcome! But we won't publish a UI library for this. If we are Adobe or Microsoft, we can release a design system. But it is a full time job for at least 2 or 3 persons!
What if we decide to do this anyway? That will be a decision that will make us lose the focus. We'll focus more on making a great UI library for our third party tools. But because of this, we'll lack time and the focus to make the great features our users want.

That was a big rant :D But I wanted to give a clear explanation. I think we're really going in the over-engineering side of things in the current situation.
We can publish on npm some things, but it has to be

• hyper easy to update
• without any backward compatibility guarantee.
• laser focused - one thing that does one thing and that does it well.

[arthuro555]

Basically, the justification should never be "I see a trend in doing this, so we should be doing it too"

Well the justification wasn't "I saw the trend let's do it too" but rather when I saw that trend, it gave me this idea. I don't think the monorepo is a novel or trending idea (as the GDevelop repo is one itself) and that we should jump into it because of that, I just thought it is a great solution to the problems I have been having when looking into making GDevelop related tools.

Making us less agile [...]
[...] renaming a prop in a component is not done because "well we risk breaking other stuff, so let's just keep the bad name [...]

I think we are both seeing this differently, the way I see this is that being restricted like that isn't a bad thing, as it is hard to get familiar with the internal API if it can change at any time for any reason. This also forces us to be careful when designing the API and make use make an API we like to work with. I even wrote that as good points:

• Force us to more carefully design the API of components to be flexible yet comprehensible and more reusable
• Force us to document more and better the components

I think it's "un mal pour un bien", just like typing can sometimes be annoying but help in the long term to build better software.

Though, you are the professional engineer here so if you think it isn't a good point I trust your judgement and experience.

We could do this, but then we have to maintain it. We have so much features and work to do on GDevelop, are we really ready to maintain a UI library, handle bugs in it, answer to feature requests, ensure we follow semver (and so constraining us?).
Basically, I'm afraid we're lacking some focus here. Our focus is to make the greatest game engine that is something so easy to use, intuitive and well designed that it can be used by everyone. Tooling made around GDevelop is very welcome! But we won't publish a UI library for this. If we are Adobe or Microsoft, we can release a design system. But it is a full time job for at least 2 or 3 persons!
What if we decide to do this anyway? That will be a decision that will make us lose the focus. We'll focus more on making a great UI library for our third party tools. But because of this, we'll lack time and the focus to make the great features our users want.

Overall I agree with that point, I am indeed in retrospect greatly underestimating the amount of work we would have to do to maintain and have let myself go completely out of scope indeed. Thank you for the clear and complete explanation, I will try in the future to keep the scope better in mind.

[4ian]

Well the justification wasn't "I saw the trend let's do it too" but rather when I saw that trend, it gave me this idea.

Sure, fair enough! Ideas are always welcome, just pretty sure this one is going too far into engineering something complex compared to the advantages provided and our limited time.

Also as with everything, a "no" is never a "no forever" => depends on the situation, what this brings to users, numbers of contributors.
If we have a very large ecosystem of third party tools in the future, it may make sense to do that (or at least something where we publish the UI folder as a npm module, even with no stability guarantees) - but not for now as this is not our main focus (and maybe a large ecosystem should be leveraged otherwise, for example an API to create IDE extensions? Or another an intermediate solution :)).

[4ian]

@arthuro555 Quick additional note:
I've been looking at https://github.com/arthuro555/gdcore-tools and it's true that this kind of package is useful for third party tools, or something similar would be useful for something like the upcoming example repository that I'm working on. But I'm not entirely sure about the exact interface that should be exposed, and maybe we should expose this as not really a package, but a set of "publicly available JS files" that you are "allowed" to require from Node.js (after cloning the repository and running npm install).

In other words, we can agree on a public interface, even if not published on npm.
We can talk more in details after I publish the examples repository :)