Evaluation of monorepo approaches for migration of multi-repo microservice code base

[Spenhouet]

Hi @DavidVujic, thanks for providing python-polylith!

You seem very passionate about the Monorepo topic. Hence, I'd like to share what we are currently working on i.e. why I'm evaluating polylith and some thoughts around that.

We are maintaining a larger multi-repo microservice code-base with services and packages all versioned via semantic versioning. It gives a lot of flexibility (e.g. different python and dependency versions per service) and control (e.g. individual access control per repo).
BUT it comes with a ton of overhead (per project commits, PRs, reviews, merge orders, updating internal dependency versions, maintaining dev environments per repo, CVE analysis for multiple versions per dependency and python version, maintaining multi version documentation, more complex testing, ...).
Due to larger organization changes, our team can no longer produce meaningful progress with this setup.

One of the main motivators for a monorepo is our planned adoption of coding agents. Having an agent make consistent changes across many microservices and python packages, automatically test everything and update docs is magical. A monorepo makes this easier. All context in one place. Single commits for larger cross-service changes.

While evaluating our options I have a constant fear of the unknown unknowns and running into a "the grass on the other side isn't greener".

I started of by creating a parent "monorepo" using Git subtree. This way we could keep our separate repos (and existing CI/CD pipelines, workflows, etc.) while working with a "monorepo". Having a backup, in case the new setup isn't as great as we hoped.
In addition, this allows for some repos to be maintained separately by another team while one team maintains it as part of the mono repo.
We are making heavy use of Git LFS with custom LFS repos and it turns out that Git subtree isn't really compatible with Git LFS.

As IDE we use VS code. A supporting idea is to use VS code multi-root workspaces: https://code.visualstudio.com/docs/editing/workspaces/multi-root-workspaces
This allows the IDE to properly handle each projects venv, tests, etc. even in a monorepo setup.
This is especially important when working with agents, as they need to be able to properly run code and tests for each project.
As Git subtrees wasn't working out, I tried simply to just utilize multi-root workspaces to aggregate all repos in a single view.
For enabling the agent to work on the whole system, this seems to already work well (I wouldn't say great).
But the dev UX is not great. With x repos, the source control view becomes unusable. We'd also keep all the Git overhead.

My current perspective is to migrate some of our repos into a monorepo and combine that with other repos via VS code multi-root workspaces.
But I saw in your YouTube guides and GitHub issue comments that you are not using VS code, which makes me concerned with the compatibility / integration of polylith with VS code workspaces. Will have to test that.

Some of our core tooling/workflow is still based on 4-5 year old decision (pyenv, pip, GitFlow) but the python tooling evolved and matured.
Over time we already updated our tooling with replacing mypy and co. with ruff and pyright. For a while I wanted us to switch to uv, as it combined multiple tools we are using right now and also replaces multiple custom shell scripts we maintain for installing python versions and managing virtual envs.
That's one of the first aspects which make me a bit hesitant about polylith. The docs seem to center on Poetry and mypy, which in my view are outdated in the python ecosystem (maybe a controversial take).

In addition or as alternative to polylith I'm looking into uv workspaces. I found the uv workspaces + Dagger setup described in the following blog post intriguing: https://gafni.dev/blog/cracking-the-python-monorepo/
Can't tell yet, how or if polylith fit's into that.

Another aspect which concerns me is that not all our projects are python based. Tooling like polylith or uv workspaces are python only (which is okay), but it's unclear to me how I'm supposed to integrate other non-python-projects into that monorepo structure.

What I find appealing about polylith is it's opinionated repo structure promoting reusable code and the affected detection. Atm. I'm hoping polylith just complements uv workspaces and Dagger by providing a workflow and diff detection.

I will continue to share my findings. Hope it's helpful for someone.

[DavidVujic]

I'm happy that you have found Polylith and also the things I've shared about monorepos! ⭐

Polylith support most of the modern Package & Dependency Management tools out there: Poetry, uv, hatch, PDM, Maturin, Pixi ...

Polylith works very well with the uv-specific workspaces. Have a look at the example repo (very simple, very little code) for uv and how the workspaces feature can be applied: https://github.com/DavidVujic/python-polylith-example-uv

About the code editor and IDE: VS Code will work for sure. I haven't any facts, but would guess the majority of Polylith users today are on VS Code, vim or PyCharm. You can also expect that the common Python tooling works well with Polylith (such as pytest, mypy, black, isort, ruff).

[Spenhouet]

I did look deeper into python-polylith and uv workspaces.

I did read the documentation on: https://polylith.gitbook.io/polylith (can recommend)
There are things about polylith that I don't get.

1. I find the term "bases" unintuitive. I have no association with that term. I find "app" a much more fitting name.
2. Why separate the build config from "bases" into "projects". In most cases this is going to be a 1:1 mapping. I don't see the benefit of doing that. Rather, it distributes the code, making it cumbersome to navigate the monorepo.
3. I get the idea behind "development", usually that is what we call a "scratch" folder, but a bit of a confusing term, as all of the repo is "development". Still thinking about it, but we might stick with "scratch".

When setting up python-polylith I felt a bit lost, also in part because I tried evaluating uv workspaces at the same time. Doing both things at the same time didn't really work out.
I switched to just evaluating uv workspaces first.

I found that I can replicate most of the polylith concept via uv workspaces.
I combined bases and projects into a folder called "apps" and kept the "components" folder.

# Create a new component
uv init --package --lib components/my-component

# Create a new app
uv init --app apps/my-app

# Add my-component as dependency to my-app
uv add --package my-app my-component

The resulting monorepo structure:

.
├── apps/
│   ├── my-app-1/
│   │   ├── tests/
│   │   │   ├── __init__.py
│   │   │   └── test_main.py
│   │   ├── main.py
│   │   ├── pyproject.toml
│   │   └── README.md
│   └── my-app-2/
│       └── ...
├── components/
│   ├── my-component-1/
│   │   ├── src/
│   │   │   └── my_component_1/
│   │   │       ├── __init__.py
│   │   │       ├── py.typed
│   │   │       └── my_module.py
│   │   ├── tests/
│   │   │   └── test_my_module.py
│   │   ├── pyproject.toml
│   │   └── README.md
│   └── my-component-2/
│       └── ...
├── pyproject.toml
├── uv.lock
└── ...

There are a view things missing:

• Affected detection
• Dependency Matrix

But there are multiple issues on the uv repo asking for tooling like this:

• Discussion: uv workspaces in a monorepo - thoughts on change-only testing astral-sh/uv#6356
• uv monorepo tooling support astral-sh/uv#9626
• Document best practices for a monorepo astral-sh/uv#10960
• Add monorepo support to uv with uv init --monorepo and other supporting commands astral-sh/uv#11095

Eventhough we might not end up using python-polylith, I thought it's worth sharing my thoughts.

[DavidVujic]

I'll try to reply to your questions and thoughts in parts 😅

I find the term "bases" unintuitive. I have no association with that term. I find "app" a much more fitting name.

Polylith use the LEGO metaphor heavily. And if you've built something bigger with LEGO you usually have a base plate, the thin layer where you put the individual bricks.

In Polylith, the base plate equivalent is the Python code in the bases folder. A base is a thin layer, an entry point, to the features of a service.

Why separate the build config from "bases" into "projects". In most cases this is going to be a 1:1 mapping. I don't see the benefit of doing that. Rather, it distributes the code, making it cumbersome to navigate the monorepo.

Mostly, a base is a 1:1 relation to a Polylith project. But it doesn't have to be that. You can have multiple projects using one base, or (in rare cases) multiple bases used in one project. That's why the bases are separate from components and from projects.

A component is just like the LEGO bricks you build things from. They are reusable and ideally small parts (rarely an entire feature). A component can be a "util" or tech thing, or something more feature-like. One component can be made of other, smaller components. The main thing is that all components are ready to be reused and composed into something useful.

A component is usually much smaller than a "library".

I get the idea behind "development", usually that is what we call a "scratch" folder, but a bit of a confusing term, as all of the repo is "development". Still thinking about it, but we might stick with "scratch".

You can do that, yes! The development folder is for scratch-like things. But it also can be for code (such as scripts) that don't fit as reusable components. In one organization, we ran one-off and monthly scripts manually, and put them in a subfolder of development. For me, development is more explicit and clear than scratch - but if that's what you usually use (and if you are using PyCharm/JetBrains tools) it might make more sense for you.

I found that I can replicate most of the polylith concept via uv workspaces.

That's great! Then you might not need Polylith. The purpose of this Architecture is to be able to share code without the need to "package" the smaller parts, and also to visualize the Monorepo with the tooling. But if uv workspaces is what you need, go for it 😄

[Spenhouet]

I really appreciate your thoughts and inside.

with LEGO you usually have a base plate, the thin layer where you put the individual bricks

I like that metaphor. Makes sense. Just having to explaining that to every (new) developer on the project already feels like initiating them into a belief system. 😅

But it doesn't have to be that. You can have multiple projects using one base, or (in rare cases) multiple bases used in one project.

Sounds a lot like "we will cross that bridge when we are there". We always lean on YAGNI.

The main thing is that all components are ready to be reused and composed into something useful.

I really like that aspect of Polylith. It's the main concept I'm gonna try to adopt. Make it easy for developers to share code across projects. Not only the aspect of making it easy to reuse, but also the aspect of making it simple to introduce new components. Currently doing so involves creating a new repo, integrating that repo into our build system (webhooks etc), integrating into CVE scans, creating a new project on SonarCloud, tracking that package on other tooling like Jira, ... it's easily a 1day task. Hence -> no one does it. This is a huge issue. In addition, the whole dependency management of all these packages becomes a nightmare. We wrote extra build pipelines to automatically push updates to requirement.txt files of all repos using that package.

we ran one-off and monthly scripts manually, and put them in a subfolder

We usually put these into a bin folder.

without the need to "package" the smaller parts

That's also what we are aiming for. As far as I can tell, that is also how uv workspaces does it.

I can share how that went for us when we actively worked with it and multiple devs got their hands on it. I'm expecting a lot more things to not work as expected when I start migrating our actual code.

[DavidVujic]

A thing that can be different between a setup using Polylith + uv workspaces and only uv workspaces is the way the "parts" are managed. Commonly, a uv workspace member would be more library-like with its own set of dependencies in a pyproject.toml file.

A Polylith component is usually much more light weight, and doesn't handle any third-party dependencies. This is the responsibliity of the project. The code that commonly would live in a uv workspace member would probably be several Polylith components. This minimizes the risk of having code hidden in a "library" that could be shared/reused. Even though the artifacts may live in different business domains, there is very likely that code can be shared (and avoiding duplications).

Both approaches has their benefits and tradeoffs. I prefer the light weight style of a component without the need to duplicate third-party dependencies across the repo - and most importantly, the ability to reuse the small parts without duplicating code.

The way Polylith interacts with uv workspaces is by setting the projects as workspace members. You'll get the benefits of the workspaces feature, with the LEGO style of the Polylith bricks. 😄

[Spenhouet]

I feel it might be easier to manage third-party libraries when they are bundled with the component requiring that dependency. This way it's clear where the dependency is used and given a component is removed, one does not forget to remove no longer needed third-party libraries. Also when I start using an existing component in another base (app), then I don't need to remember to add it to the project (or else the build will fail).
With uv workspaces, when I build a project (app), then it will only install all third-party libraries based on the dependency tree. Let's say, every LEGO brick comes all-inclusive.

But I see that downside, that e.g. in case we want to update a lib to a new version, we have to do this in x places (luckily IDEs have a replace all across the workspace).

[DavidVujic]

Have a look at the Polylith repo itself (this repo). Even though it might be a bit meta with a polylith repo for polylith itself, containing components named "components" and such 🤯, it might be interesting to have a look at a real setup. Most existing Polylith users out there are not open source projects.

This repo began with one project (then only for Poetry), later on expanding to a standalone CLI and tool-specific hooks (such as the Hatch build hook). The CLI and Poetry project share most code, but have different bases.

[vEpiphyte]

I've been struggling with similar issues; only migrating multiple independent repos which were setup with a single implicit namespace into the same repository. One of the biggest issue I've had was actually carving out the specific modules for packaging applications together, which polylith does really well with the hatchling plugin.

In some more testing, I'm currently looking at the following structure ( investigating retaining a toplevel directory of the project namespace ):

├── projects
│   ├── fooapp
│   │   └── pyproject.toml
│   └── bazapp
│       ├── docker
│       │   ├── release
│       │   │   └── Dockerfile
│       │   └── scripts
│       │       ├── bootstrap.sh
│       │       └── entrypoint.sh
│       └── pyproject.toml
├── pyproject.toml
├── pytest.ini
├── README.rst
├── mynamespace
│   ├── common
│   │   ├── cell.py
│   │   ├── __init__.py
│   │   ├── __pycache__
│   │   └── version.py
│   ├── fooapp
│   │   ├── __init__.py
│   │   ├── service.py
│   │   └── version.py
│   ├── fixtures
│   │   ├── common.py
│   │   ├── __init__.py
│   │   ├── bazfixt.py
│   │   └── README.rst
│   ├── bazapp
│   │   ├── assets
│   │   │   ├── docs
│   │   │   │   ├── doc1.rst
│   │   │   │   ├── doc1.rst
│   │   │   ├── __init__.py
│   │   │   ├── meta.yaml
│   │   ├── __init__.py
│   │   ├── service.py
│   │   └── version.py
├── test
│   ├── common
│   │   ├── __init__.py
│   │   └── test_cell.py
│   ├── fooapp
│   │   ├── __init__.py
│   │   └── test_service.py
│   ├── fixtures
│   │   ├── assets
│   │   │   └── test.json
│   │   ├── __init__.py
│   │   └── test_fixture.py
│   ├── __init__.py
│   ├── bazapp
│   │   ├── __init__.py
│   │   └── test_service.py
└── workspace.toml

The thing with this is that I can continue to use polylith to build wheel packages with the following hooks in the pyproject.toml:

[tool.hatch.build.hooks.polylith-bricks]

[tool.polylith.bricks]
"../../mynamespace/bazapp" = "mynamespace/bazapp"
"../../mynamespace/common" = "mynamespace/common"

( I'm seemingly unable to recreate this behavior solely with hatching build rules! )

But I lose the ability to track sync new dependencies easily - poly sync no longer functions to identify bricks that need to be added to the individual project pyproject.toml files; even if each module in the namespace is listed in the root pyproject.toml file.

I'm not entirely certain how helpful this observation is, but it feels similar to the problems Spenhouet has brought up. It could be rather powerful capability to practically "skip" the implicit (?) requirement to separate the bricks into bases and components and allow for the resolution of bricks solely based on their presence in the root pyproject.toml file.

[DavidVujic]

Oh, you mean that the sync command stopped working when moving the code away from the bases and components folders, as in the repo?

[DavidVujic]

I'm sorry, I misunderstood what you meant and misread your initial example of setup. The Polylith tool expect you to have code in the components and bases folders, to recognize them as Polylith bricks. The tooling is strict about the architecture, and that's why the sync command won't work. I can imagine the poly info won't show you any bricks either, right?

It looks like you are using a Polylith-inspired setup, which looks good! But the tooling won't be very much useful for that. I would recommend to have the code in the components.

[vEpiphyte]

I can imagine the poly info won't show you any bricks either, right?

Correct; it only shows the the 2 projects.

It looks like you are using a Polylith-inspired setup, which looks good! But the tooling won't be very much useful for that. I would recommend to have the code in the components.

Yup - a bit of inspiration and melding together multiple discrete repos which used the same namespace package together under the same repo; with the projects being broken out into their own space to represent the deliverables for discrete applications.

Like I mentioned previously - the hatch build tooling does work fine to build artifacts in this setup ( joining the discrete modules from the namespace package together nicely ); its just the sync / info is not working. Knowing that is a hard requirement for the tooling is good to know and I may be able to get by with a single top level folder ( just bases ); instead of using bases and components as two separate items.

[vEpiphyte]

In thinking about this more - would it be possible to configure the bases and components directories?

I could see an org becoming more accepting of tooling like this if they could re-point the root bases concept to something like a app directory. Maybe by setting that in the workspace.toml like this:

[tool.polylith.structure]
theme = "loose"
bases = "app"
components = "lib"

That would cause the behavior of the tooling to consider the app and lib directories as the location of the bases and components directories for the purposes of resolving where the various bricks are located.

[DavidVujic]

I have thought of something like this before, such as adding another theme with configurable options. I think it would be confusing though, when reading about the Polylith Architecture and the building blocks but having the core structure renamed.

I think it's a misunderstanding that components are like libs. If you have a microservice setup, you might not call it app either (it can be handlers, consumers, REST apis, CLIs or installable packages). Can you elaborate more about why bases and components wouldn't fit?