Planning for breaking changes in 0.12.x / 0.13.x ?

[lihaoyi]

It's been a bit over a year since 0.11.0 was released (June 2023), and I'd like to start thinking about what breaking changes we'd like to do so we can bundle them into a 0.12.0 release. Exact timeline isn't set, but I'm thinking end-2024 or so. We can continue maintaining a 0.11.x branch if we wish, but at some point we'll need to break compat and it would be good to consolidate the breakage so it happens as few times as possible

Some off the top of my head:

• Migrate build.sc to Scala 3 #3152 will be a major change in the Scala version used by the build.sc files. Any OSS libraries you use in your build.sc files may need to be updated, any internal libraries you use may need to be migrated (e.g. if they use macros). Scala 3.x is not source compatible with 2.13.x, so build.sc files may need to be adjusted to satisfy the new syntax. Scala 3 also introduces new capabilities that could change how we want to implement some things, e.g. some modules that are currently modelled as nested traits could be better modelled as traits taking parameters instead.

• Make per-folder build.sc files a first-class feature to support large projects #2637 will likely require changes in the build.sc file desugaring. Not sure exact details, but e.g. we would want to move RootModule from target-resolution-time to codegen-time so that it would apply not just to CLI invocations but also to in-code programmatic references between build.sc files as well (these do not exist today because only one build.sc is supported)

• Should we get rid of Agg in user-facing APIs, or just make it a type alias for Seq? It felt like a good idea at the time, and it's definitely a handy abstraction, but at this point I'm not sure it's worth the complexity and novelty cost v.s. just using Seq everywhere

• Would Make CompilationResult more generic #2781 be a breaking change?

• Redesign tasks that get an evaluator instance #502 does not have any plan or design ready to go yet, but if we do come up with something it would probably be breaking

• Should we make -j0 the default so parallelism becomes opt-out rather than opt-in? That would match the behavior of other tools like SBT or Bazel, and I think at this point parallelism is battle-tested enough we can turn it on by default

• Rework handling of mill-launcher in MillBuildModule #2703 would be a breaking change if we decided to expose the minimal classpath

• First class support for Java and Kotlin: bring mill-kotoin plugin into main repo, and split out a mill.javalib package for java projects to use

• If we don't expect to continue making major change in Mill 0.11.0, we could resolve Complete versioned documentation generation process for dynamic pages since 0.11.0-M8 #2455 by just generating it manually as manually saving the output as a sub-folder in the gh-pages branch that we link to from the main docs

We might not get to all of these, but it's definitely worth thinking about. Any other things that people may want to change that would involve breakage? @lefou @lolgab

[lolgab]

I wanted to change the way Mill environment variables to not propagate the Environment anymore.
It's a breaking change so I would be more confident to do in a major version change.
#2882

[lolgab]

✅ Fixed in #3442 and #3437
Now the client's environment will be used in the server and forkEnv is by default an Map which contains the extra environment to be added to the system ones.

[lefou]

All these points sound good to me. Although in this summer I will probably not have so much time to implement anything of this.

Some of these changes are probably not necessarily breaking compatibility, e.g. making --jobs 0 the default.

More on -j0: I'd like to make our output easier readable, e.g. drop the self-erasing progress in favor of some nicer progress which shows the state of not only the last started job but also of some currently running jobs. Maybe, group and delay output of non-interactive tasks. But I have no clear idea yet. Also, I'd like to change how we handle commands, e.g. make them non-concurrent to other jobs and interactive by default.

I'd also like to see some native Mill launcher, so that repeated mill executions get instant and don't need to fire up the JVM each time. Unfortunately, I don't have much experience yet with either Scala Native or Graal native images. I think the latter might be our go-to solution, since we use junix-socket for Mill client-server communications and it looks like it comes with ready to use native image configs.

[lihaoyi]

More on -j0: I'd like to make our output easier readable, e.g. drop the self-erasing progress in favor of some nicer progress which shows the state of not only the last started job but also of some currently running jobs. Maybe, group and delay output of non-interactive tasks. But I have no clear idea yet.

SBT and Bazel both have similar TUIs for displaying the concurrent tasks going on; we could build something similar

Also, I'd like to change how we handle commands, e.g. make them non-concurrent to other jobs and interactive by default.

This is what Bazel does. Sounds reasonable to me

[sequencer]

My first wish is removing absolute Path dependency(maybe change to RelPath to workspace) to allow moving cache around different CI machines and even allow remote build support.

[lihaoyi]

All these points sound good to me. Although in this summer I will probably not have so much time to implement anything of this.

That's fine, I have some bandwidth, I can maybe put up some bounties, and we might not achieve everything in the list. As long as we have general agreement on the general direction we can move forward at whatever speed is feasible

Some of these changes are probably not necessarily breaking compatibility, e.g. making --jobs 0 the default.

It doesn't break binary compatibility, but it does break semantic compatibility. There historically hasn't been a requirement that build.sc logic is multithreading-safe, even if it happens mostly "for free" due to functional programming. Introducing such a requirement could require a bunch of third-party code to break in subtle ways and is worth a big version bump

[lihaoyi]

@lefou what do you think of targeting non-Scala deveopers with Mill? e.g. mainlining Kotlin support, breaking out a dedicated mill.javalib package, etc. We used to have some half-baked docs suggesting Mill could be used for Java, but I mean doing it properly. e.g. Properly meaning full length doc pages for getting started with Mill for Java, Java Module configs and Java Build Examples, and the same for Kotlin

That would be following the Gradle playbook: nobody uses Groovy or Grails these days, but they use Gradle anyway since it's better than Maven sometimes. Maybe Mill could find traction in Java/Kotlin land too; it's not like people there are in love with Maven/Gradle/Bazel.

I think there's a space for a build tool somewhere between those three: more scalable and flexible than Maven, more simple than Gradle or Bazel, even if a bit less scalable. e.g. Maybe targeting codebases up to 1-1000 developers with 1-1000 modules, but not for monorepos with >1000 developers which probably need Bazel anyway. Currently a lot of projects get started using Maven or Gradle and switch to Bazel later, and if they get started using Mill they could have a better experience up front and also delay the inevitable Bazel migration since Mill works fine with hundreds of modules (e.g. com-lihaoyi/mill has around 500 modules and still works great)

This would open up Mill to some avenues for growth beyond the small Scala community, while also giving folks in the broader JVM community a taste of Scala if Mill manages to become popular

Again, ignoring the work hours necessary. That's orthogonal and maybe can be sorted out later

[lefou]

You're preaching to the converted. I'm all in, since I think Mill is the better alternative for many Gradle users. About the mill.javalib package, we have some prior discussions (#1826 (comment) to just link one of them); in summary, it will be hard to de-entangle Java and Scala concerns and maybe not worth it. That's mainly, because we use Zinc and its workers also for pure-Java modules. There is also some more work needed to get the stackable-trait concept a better match to stackable compiler tool chains. I've seen lots of those stacked tool chains already, and also the issues that can occur. But these technically details aside (which can be solved) I'm all for it! Especially, since it's mostly the Scala-free documentation it needs to speak to the Java audience.

[lihaoyi]

Hah i missed that thread, thanks for the link

I think using Zinc isn't a blocker. We can have JavaModule use javac (in memory or via subprocess) and ScalaModule override it to use Zinc.

I'm thinking that even just having a mill.javalib package in the same scalalib module would be fine. The user will see "java", and won't see "scala" at all even if its used under the hood

[lefou]

About mainlining Kotlin, which is currently usable in Mill via https://github.com/lefou/mill-kotlin. That would be great! Unfortunately, non-Jetbrains tools are still second class citizens to the Kotlin API. There is currently no way to use the fast incremental compilation from Mill. Gradle and IntelliJ use some internal API. I opened issues and discussions about it, but there is still nothing we can use to date. Without that, it might get hard to come up with a stable API around Kotlin settings.

[lefou]

We could also provide a GroovyModule. Not that it's needed nowadays, but maybe it provides the needed support for bespoken legacy tool chains. I don't have insights in any larger closed-source Gradle project, so this is wild guessing.

[lihaoyi]

I think Java/Scala/Kotlin are the three main ones to target. Other languages, which do exist, are generally much smaller. Groovy, Clojure, and so on

I think having limitations around kotlin is reasonable. I don't know how fast kotlin changes, but I assume once tue dust has settled around kotlin 2.0 things shouldnt be moving that quickly. Even for internal APIs

But good to know you're in agreement with the direction.

[lefou]

• Support an alternative to $-imports #3062

I think this is a good opportunity to migrate to using directives.

Pros:

• Easier to parse
• IDEs, code formatter and hightlighters already support using directives
• No need to mangle the import statements before compilation
• No false error markers in IDEs
• Opportunity to add and unify more configuration options

[lefou]

• Refactor CompileModule out of JavaModule #3237

[lihaoyi]

To add to this list, bumping uPickle to 4.x can be part of Mill 0.12.x as well

[lolgab]

Remove default value from debugEnabled: Boolean in Logger.scala

[lolgab]

Remove all methods deprecated before Mill 0.11.0

[lihaoyi]

@lefou just to bring the 0.12.x branching/timeline back here, what do you think of the following dates to aim for:

1. 31 July: Fork off 0.11.x from main
2. 31 Oct: Cut 0.12.0-RC1
3. 31 Nov: Cut 0.12.0-final

Forking off 0.11.x will allow us to treat the two branches differently, e.g. optimizing for binary compatibility at the expense of velocity in 0.11.x, and doing the opposite for 0.12.0.

After 31 July, PRs targeting both main and 0.11.x will need to be done twice, which is tedious. But not every PR will need to be backported to 0.11.x, so hopefully the burden will not be too great.

We might not be able to exactly meet those dates, but it would be good to have dates to work towards just so we're all on the same page

@bishabosha will likely be helping us with the Scala 3 upgrade from 1 August to 31 Oct. This will definitely be a big breaking change (syntax, bincompat, upstream dependency versions, etc.) so it definitely needs to go into 0.12.0

[lefou]

Honestly, to my perception the move to a Scala 3 API isn't even on the horizon. And I fear, if we commit to include it in a final 0.12, we end up with a longer milestone phase than anticipated.

How about switching to a binary but not behavior compatible 0.12 release from main to get some almost ready new features out and defunc / disable some deprecated features in preparation for the next breaking window. The fact is, keeping the plugin ecosystem up-to-date in a milestone phase as well as shortly after a major release is much more work then doing that some extra stuff in Mill's code base to keep bin-compat.

With some productivity features already out, we can focus on Scala 3 and breaking stuff in 0.13. I think, users don't mind to switch from 0.11 to 0.12, as long as they don't need to do much migration work. Most Mill projects don't rely on package managers anyway, since Mill manages version itself, so there is no large distribution tail to worry about. Only those projects relying on features we are going to remove/replace (foreign modules), will have some migration work. For them, it's a great service that they don't need to deal with any other breaking changes like a Scala 3 migration. Especially, when they maintain some private Mill extensions (which is often the case for projects that heavily use imports or foreign builds).

So here is the adapted timeframe:

• next days: cut a 0.11.9 to get pending PRs and the Java-focused documentation out the door
• End of July: Cut a 0.12.0-RC1 with new features and changed behavior
• End of Sept: Fork of 0.12.x from main
• End of Oct: Cut 0.13.0-RC1
• End of Nov: Cut 0.13.0-final

If a 0.12 line is binary compatible with 0.11, we hopefully can avoid a 0.11.x maintenance branch and deliver fixes in 0.12.x releases.

[lihaoyi]

@lefou Your proposed timeframe and branching strategy works for me. Let's do it!

[lefou]

• Should we include more dependencies to `TestModule.<Framework>Module` #3281

[lefou]

I think, before we can target a new major milestone with main, we should fix our versioned documentation story, so we can host a documentation for (at least the latest) 0.11.

I think what essentially needs to be done is to make our pre-processing (generating examples and contrib pages) persistent in a way we can commit to git. That way, we can later re-use it with antora by pointing to the right tags/branches. So, all that's needed is possibly to have either

• have a dedicated directory with generated documentation, or
• have a branch with all all documentation files (hand-written and generated) and update that branch whenever we need too.

[lihaoyi]

I think given how infrequently major versions are released, manually generating a "final" 0.11.x docsite and putting it in a folder in gh-pages may be sufficient. We expect to release 2 new versions in the forseeable future, so the cost of trying to automate it elegantly would dwarf the cost of copy-pasting some html files and tweaking some links.

In fact, doing this automatically shouldnt be hard either, as long as we do it at the html level: have the github action checkout the relevant commits, generate docs, and copy the files to the right place. No need long term storage of materialized documentation fragments anywhere

[lefou]

I thought we agreed to re-enabled the ability to easily render documentation for older versions "later", when we broke that feature in the milestone phase of 0.11? (#2448)

It was one of the major features leading to choose Antora as the documentation tool. TBH, I'd have liked to have versioned documentation for every 0.11.x release. I think these kind of details that might look like overhead to you, are the things that draw the line between a technically capable product and one that is also trusted by users. We broke some things in 0.11.x but never wrote the migration guide. We didn't provide the versioned documentation which could be used to refer to some now deprecated or removed features, so the migration process could become more bearable. Yes, we can wait for users to ask in the chat room or open issues, but we also loose some that way and we never get those who still evaluate. And it's more community work, too. It can be really frustrating to answer the same questions again and again. We promise an easy tool but lack to support those, who can't afford to update their tool chain every week. I'm not arguing whether we really should have docs for every release like 0.11.9 and 0.11.10, but if we need to do something manually, which also required low level fiddling with files, I know it will us slow down later. I have no joy in doing that. I already invest a lot of time in maintaining those other little things many don't see (like assigning milestones to merged PRs and closed issues, so users can easily find when something was added or removed). These are manual task I'm willing to do, as I believe in their usefulness to others. (And although I documented it in our contribution section in the Readme, nobody else does it.) But I don't look forward to remember which html file needs to be regex-ed to adapt to the next release. We should stick to the AsciiDoc files, so that rendering only happens when we generate the page. Mill is a great tool to automate things. We should eat our own dog food, and use Mill to automate the preparation of a version documentation.

Any manual ghpages fiddling will be dropped sooner or later, when the next documentation site comes up. If we target Java users, there will be projects that will use Mill 0.11.x for a long time. We can't simply stop publishing documentation for it next year. Antora already covers our use case, so we don't need to start from scratch. All we need to do is to persist the directory we currently prepare at generation time, so it can be user later. Either by pointing Antora to some branch, tag or directory.

[lihaoyi]

Yes I don't disagree about fixing the older documentation, just a matter of implementation details.

I do think that automated gh-pages fiddling is an avenue worth exploring. I'll put up a PR and we can discuss it a bit more concretely.

[lolgab]

I wish that Mill had a warn log level that would print yellow logs for things that are not errors since you can live with it but are not info as well.
Maybe we can use the breaking version to add it.

Moreover, it would be nice if had proper log levels instead of a debugEnabled boolean, so people could choose to see only errors, or errors + warnings or errors + warnings + infos, etc.

[lefou]

Sounds good.

[lefou]

A inlined lazy message construction would be nice, too, so we will only render a message that is also used somewhere.

[lihaoyi]

@lefou @lolgab The build and publishing is green again, so I'm planning to start with merging the 0.12.0 pull requests into master over the next few days so we can prepare a 0.12.0-RC release

[lefou]

We should mark the last non-breaking commit with a 0.12.0-M0 tag, so the version number clearly communicates the kind of commits to expect. According to @eed3si9n, a 0.12.0-M.0 would be even better from a tooling perspective (semver and ordering in particular), but this would require us to adapt our scripts and download logic.

[lefou]

I tagged 0.12.0-M0, so we can't confuse snapshot versions by their number.

[lefou]

I opened a discussion reflecting the merged changes of Mill 0.12, new features and breaking changes. If anything needs to be done by users, we can add that to that list.

• New features and breaking changes in 0.12.x #3349

[lihaoyi]

Ok!

Since main is targeting 0.12.0 now, we should try to get in whatever breaking changes we want to do as long as they are binary compatible. Then we can test/RC/release those as 0.12.0 and save the bincompat breakage for 0.13.

I have some ideas around rudimentary filesystem isolation i want to experiment with, that could conceivably go into 0.12. Probably other interesting ideas will come up over the next month or two

[lefou]

How about changing TaskModule.defaultCommandName to not only support commands but any kind of target?

Then, we could make the testCached target the default for TestModule.

[lefou]

That must be a new-ish thing.

[lefou]

• Convert defaultCommand to defaultTask, allow usage for non-command tasks #3373

[lefou]

I tested it, this works since Mill 0.11.0.

[lihaoyi]

Makes sense, that was the version where I overhauled the task resolution logic

[lihaoyi]

I don't think we can naively change the default test command to testCached though. Because people are used to typing __.test, that would result in both the .test module and the .test.testCached target being resolved, which is not what people want. To do this, we would likely need to change the .test command to .testUncached or similar, which would be a binary incompatible change that would need to go into 0.13.0

[lihaoyi]

It's now mid August, so a bit behind the discussed timeline of cutting 0.12.0-RC1 by end-July, but I'd like to aim for cutting a 0.12.0-RC1 release by End-August. The remaining timeline (End-Sep for 0.12.0 final, End-Oct for 0.13.0-RC1, End-Nov for 0.13.0 final) can remain unchanged

[lefou]

I think before tagging a 0.12.0-RC1, we should complete the documentation for the new features. This greatly helps fleshing out potential glitches and experimenting with it. Esp. the module.sc support needs some documentation.

[lihaoyi]

Yeah i plan to spend most of next week on docs, should be enough to get them into shape

[lefou]

Has anybody tried out GitHub projects? Is it suitable to managing/planning our next releases?

[lihaoyi]

I haven't; happy to try, but given the small number of people working on stuff (<5) ad-hoc coordination in this discussion thread will probably be ok

[lihaoyi]

Just an update here as end of August approaches. The last two things I want to get done before cutting 0.12.0-RC1 are:

• Land Overhaul build file management with new build.mill/package.mill format #3426. This is a major overhaul of the per-folder build config system, and given that that's one of the big features of this release I think it's worth getting this overhaul in before people start trying it out. Doesn't make sense to cut an -RC1 for a feature, get people to test it, and then land a rewrite immediately after rendering the testing useless

• Rebootstrap Mill on latest main. Currently waiting for a new version of Mill-Scalafix to be cut with Make Scalafix look for config in T.workspace rather than os.pwd joan38/mill-scalafix#199, then after that we should be good to go. Again, for such a big change, it doesn't make sense to cut a -RC1 for other folks to use when we haven't even tried out the new features ourselves in Mill, since they'll just hit the same bugs we would hit when we rebootstrap.

Might take more than the next two days for these two things to happen, but hopefully should be able to complete all this in the next week or so. We can continue making changes even after -RC1 has been cut, but at least the big ticket items should be in beforehand for RC1 to be useful

[lihaoyi]

Both tasks above are complete, and Mill is re-bootstrapped on latest master with the build.sc file converted to build.mill and broken up into per-folder package.mill files where possible. IDE support works as expected, with some known limitations e.g. build.foo.bar module references to the root module do not highlight correctly across different files even though build.foo.bar.myTask references do. Auto-parallelism works. Per-test sandbox folders and Mill process sandbox folders work. The new RootModule codegen-time unwrapping works

I'm planning on cutting 0.12.0-RC1 this coming Sunday 8th Sep, with 0.12.0 final still targeting ~end Sep

[lihaoyi]

Some updates:

• 0.12.0-RC1 was cut on 8 Sep for people to try out. I have PRs out to every repo in com-lihaoyi to update them to 0.12.0-RC1, and so far nothing unexpected has turned up

• Going forward, I'm planning on cutting 0.12.0-RC2 on 15 Sep, and 0.12.0-RC3 on 22 Sep.

• 0.12.0 final is still due end (30th?) Sep

There's still work going on in main even though most of the major breaking changes landed in 0.12.0-RC1, so the subsequent RCs will give a chance for people to try out the ongoing more minor changes before the final 0.12.0 release.

[lihaoyi]

Tagged 0.12.0-RC2 https://github.com/com-lihaoyi/mill#0-12-0-RC2

[lihaoyi]

Just an update here, the last major change I'd like to get into 0.12.0-RC3 is an overhaul of the terminal UI #3577. This is a pretty major addition to making Mill 0.12.x's parallel-evaluation-by-default usable, so I'd like to get it into 0.12.0-RC3 so it has a chance to bake and dogfood a bit before 0.12.0 final.

I estimate it will take another week or so to land that PR, so that pushes back our timeline slightly:

• 0.12.0-RC3 landing 29 Sep
• 0.12.0 final landing 6 Oct