feat: Extract serde_core out of serde crate #2608

osiewicz · 2023-09-04T13:26:06Z

This PR implements approach suggested in #2584 by @epage and @soqb. Also #2181
There were two changes initially that were since reverted during the lifetime of this PR:

I've changed serde_derive to use serde_core instead of serde. This has since been reverted as it would be a breaking change.
Code examples in serde_core aliased serde_core as serde so that the docs reexported in serde refer to serde and not serde_core. This however was not necessary following commit 2258bb4

Before (11.4s for a clean release build with derive feature, 8 cores):

After (7.8s for a clean release build with derive feature, 8 cores):

osiewicz · 2023-09-04T14:00:01Z

Ugh, seems like there's a bunch of CI failures on set up job stage.

epage · 2023-09-04T14:22:55Z

Besides the gains here, this allows serde_json and friends to depend on serde_core, preventing packages using serde w/ derive from serializing things.

NobodyXu · 2023-10-30T12:15:16Z

There're merge conflicts on this branch again which needs to be resolved.

osiewicz · 2023-10-30T12:30:44Z

Hey, the merge conflicts on this PR are usually trivial to resolve (the PR itself doesn't change anything about the code in serde, just it's structure) and they'll pop up with each new serde version released; I am not sure if it makes sense to massage it all the time given that this PR has went almost unnoticed for 2 months now. I'd say it should be fine to resolve them once this PR gets some attention. =)

NobodyXu · 2023-10-30T12:33:59Z

Hmmm I thought this PR is blocked because merge conflicts and would get some attentions once it is resolved, but it turns out that it just doesn't get much attention regardless of merge conflicts.

This PR would really help improve compilation time of serde with derive enabled and I was looking forward to it, it's sad that it has been blocking for so long was no review from maintainer.

osiewicz · 2023-10-30T12:36:06Z

Yeah, I think so too. On the other hand, it was not asked for so no hard feelings there.
I would also love to see it merged.

But yeah, the conflicts will always be there whenever a new version of serde is released, as this PR moves half of the files into other directory (which git seems to be able to resolve conflicts for on it's own but GH can't see that) + the version number changes. Either way, if there's a consensus to merge it, we can just rebase the changes onto the current master and that should do it.

oli-obk · 2023-10-30T12:49:14Z

Two things:

Rebases are preferred over merges. A rebase that only conflicts due to file moves is usually resolved trivially without user input
nightly is broken, you may need to run with nightly, too and adjust some imports
I'm not going to push for this in the next two months as I'm going on leave soon
I can't count 😆

osiewicz · 2023-10-30T12:53:41Z

Hey, should we come back to this PR whenever you're available then? By that I also mean whether I should fix it up in the meantime or is it fine to just come back to it in 2 months and rebase it then?

oli-obk · 2023-10-30T12:59:01Z

I think it would be fine to just not rebase it. Either @dtolnay will pick it up at whatever state it is in while I'm gone and ask you to rebase, or I will when I'm back.

epage · 2023-11-09T15:37:47Z

As a real-world example of how this can be beneficial, I was looking at the timings chart for ripgrep on the parallel frontend blog post. I did some analysis on it. I'm guessing this change on its own would take ripgrep build times on a 24-core machine from 9s to 6-7s.

dtolnay

Thanks for working on this!

Some things I would want to see further consideration of before deciding how to move forward with this design are:

the effect on compiler diagnostics seen by users, or other potential friction from this change, particularly when it comes to users who may be new to Rust
implications for serde 2.0 when it comes to the private helper types that serde_derive relies on
implications for serde 2.0 when it comes to wishlist for 2.0

(Maybe @oli-obk can work with you to flesh these out.)

Compiler diagnostics

After this change, what are the situations in which diagnostics that used to look like "the trait bound `T: serde::ser::Serialize` is not satisfied" will instead say "the trait bound `T: serde_core::ser::Serialize` is not satisfied"? — these occur often when you are using a data format library like serde_json together with a data structure library whose Serde trait impls are behind a feature.

I am not familiar with the heuristics rustc uses to choose whether to display the trait path in these errors as serde::ser::Serialize or serde_core::ser::Serialize or serde_core::Serialize or just Serialize.

For errors that refer to serde, users have been able to unblock themselves well enough by finding that the crate's documentation mentions "serde support" behind a feature, or noticing in cargo add it printed that there is a serde feature, or noticing that the relevant impl serde::Serialize for ... in the crate's source code has a conditional compilation attribute.

A crate is unlikely to document that it has "serde_core support". To some users it will be obvious that a serde trait impl is interchangeable with a serde_core trait impl, but this isn't the case in general. For example thiserror and thiserror_core are unrelated in this way, serde_json and serde_json_core are unrelated, etc. Maybe consider whether serde_traits, or something else, would be better naming.

I think this is probably inevitable given that moving the traits to a different crate that is a dependency of serde is most likely required for ever publishing serde_derive 2.0. (Although semver trick is another option here, which log used successfully in the past.) So this one is a nonblocking concern. Nevertheless it would be good to understand if there is a need for any kind of followup in rustc. Serde is commonly one of the very first crates that someone new to Rust would first use, so there is a lot of leverage to scare users off of Rust with hours of frustration, likely greater than the cost of 3 seconds in a first-time build.

What mitigation options exist, and how do they compare? Is setting [lib] name = "serde" on serde_core better or worse?

Private helper types

Currently this PR has moved all of serde::__private into serde_core.

That may not fly. It means that someone is able to write:

serde = { package = "serde_core", version = "1" }
serde_derive = "1"

which then means all of serde_derive 1.0's helper types need to be present in serde_core in perpetuity, even all the ones serde_derive 2.0 does not use, even if serde 1.0 is not in the build graph. I have not measured recently, but these helper types account for a nontrivial part of the compile time of serde.

What alternative options exist, and how do they compare? Keep helper types in serde for now? Put helper types in a third crate somehow? Some feature voodoo? Progress on rust-lang/rust#54363 would be valuable here too.

Items from 2.0 wishlist

Relocating traits into a different crate is a one-time deal. When we do serde_derive 2.0 (in which I'd like to remove support for many attributes that are too niche, and some changes like default = $expr should take an expression instead of a string literal) and serde 2.0 to go with it, it needs to be written against serde_core 1.0 because the cost–benefit from improving the macro is a lot better than the cost–benefit of breaking the ecosystem with incompatible traits.

That means some of what might be on the table for serde 2.0 needs to be accounted for now. In particular, this would be the time to figure out what(whether) to do about #1070. Do the Serialize and Deserialize impl for Result need to be behind a feature that is enabled only by serde 1.0 and not 2.0? Missing that now would be a blunder because it cannot be fixed later.

Hopefully that one is the only 2.0 item needing consideration, but I am not sure.

serde/Cargo.toml

serde_core/Cargo.toml

serde_core/LICENSE-APACHE

serde_core/src/lib.rs

epage · 2024-02-26T18:22:56Z

After this change, what are the situations in which diagnostics that used to look like "the trait bound T: serde::ser::Serialize is not satisfied" will instead say "the trait bound T: serde_core::ser::Serialize is not satisfied"? — these occur often when you are using a data format library like serde_json together with a data structure library whose Serde trait impls are behind a feature.

Not tested with serde_core (since more changes are needed) but forced some serde compiler errors to see what I'd get.

Using a type in a #[derive(Deserialize)] struct's field that doesn't implement Deserialize. Notice a lack of serde::. I even removed use serde::Deserialize to double check that wasn't the reason why.

error[E0277]: the trait bound `StringA: Deserialize<'_>` is not satisfied
    --> crates/cargo-util-schemas/src/manifest.rs:30:25
     |
30   |     pub cargo_features: Option<Vec<StringA>>,
     |                         ^^^^^^^^^^^^^^^^^^^^ the trait `Deserialize<'_>` is not implemented for `StringA`
     |
     = help: the following other types implement trait `Deserialize<'de>`:
               bool
               char
               isize
               i8
               i16
               i32
               i64
               i128
             and 192 others
     = note: required for `Vec<StringA>` to implement `Deserialize<'_>`
     = note: 1 redundant requirement hidden
     = note: required for `std::option::Option<Vec<StringA>>` to implement `Deserialize<'_>`
note: required by a bound in `next_element`
    --> /home/epage/.cargo/registry/src/index.crates.io-6f17d22bba15001f/serde-1.0.196/src/de/mod.rs:1726:12
     |
1724 |     fn next_element<T>(&mut self) -> Result<Option<T>, Self::Error>
     |        ------------ required by a bound in this associated function
1725 |     where
1726 |         T: Deserialize<'de>,
     |            ^^^^^^^^^^^^^^^^ required by this bound in `SeqAccess::next_element`

Trait not in scope. Note the suggestion of use crate::manifest::_::_serde::Deserialize rather than serde::Deserialize. I suspect the compiler is picking up on an import generated by serde_derive.

error[E0599]: no function or associated item named `deserialize` found for struct `TomlInheritedField` in the current
scope
   --> crates/cargo-util-schemas/src/manifest.rs:292:37
    |
292 |                 TomlInheritedField::deserialize(mvd).map(InheritableField::Inherit)
    |                                     ^^^^^^^^^^^ function or associated item not found in `TomlInheritedField`
...
444 | pub struct TomlInheritedField {
    | ----------------------------- function or associated item `deserialize` not found for this struct
    |
note: if you're trying to build a new `TomlInheritedField`, consider using `TomlInheritedField::new` which returns `To
mlInheritedField`
   --> crates/cargo-util-schemas/src/manifest.rs:449:5
    |
449 |     pub fn new() -> Self {
    |     ^^^^^^^^^^^^^^^^^^^^
    = help: items from traits can only be used if the trait is in scope
help: the following trait is implemented but not in scope; perhaps add a `use` for it:
    |
8   + use crate::manifest::_::_serde::Deserialize;

epage · 2024-02-26T18:26:01Z

What alternative options exist, and how do they compare? Keep helper types in serde for now? Put helper types in a third crate somehow? Some feature voodoo? Progress on rust-lang/rust#54363 would be valuable here too.

My assumption for this is the helpers would be in serde and we keep existing derive users working the same way we do today.

No breaking change for now
Isolate the semver compatibility guarentees

Maybe with serde@2, we could switch to serde_derive + serde_derive_macro where serde_derive has the helpers but doing that is dependent on a lot of different factors that don't seem worth discussing atm.

serde_derive/Cargo.toml

osiewicz · 2024-02-29T00:23:46Z

Just to add to second example provided by @epage; it looks like rustc may sometimes pick up a different path to (De)serialize trait, based on whether user has serde in non-macro scope.
Baseline code:

use serde;

fn main() {
    let _ = u32::deserialize("");
}

Stderr with `use serde`

error[E0599]: no function or associated item named `deserialize` found for type `u32` in the current scope
 --> src/main.rs:7:23
  |
7 |     let _: u32 = u32::deserialize(());
  |                       ^^^^^^^^^^^ function or associated item not found in `u32`
  |
  = help: items from traits can only be used if the trait is in scope
help: the following trait is implemented but not in scope; perhaps add a `use` for it:
  |
2 + use serde::Deserialize;
  |

For more information about this error, try `rustc --explain E0599`.
warning: `throwaway_serde` (bin "throwaway_serde") generated 1 warning
error: could not compile `throwaway_serde` (bin "throwaway_serde") due to previous error; 1 warning emitted

Stderr with `use serde_core`

error[E0599]: no function or associated item named `deserialize` found for type `u32` in the current scope
 --> src/main.rs:7:23
  |
7 |     let _: u32 = u32::deserialize(());
  |                       ^^^^^^^^^^^ function or associated item not found in `u32`
  |
  = help: items from traits can only be used if the trait is in scope
help: the following trait is implemented but not in scope; perhaps add a `use` for it:
  |
2 + use serde_core::Deserialize;
  |

For more information about this error, try `rustc --explain E0599`.
warning: `throwaway_serde` (bin "throwaway_serde") generated 1 warning
error: could not compile `throwaway_serde` (bin "throwaway_serde") due to previous error; 1 warning emitted

When both serde and serde_core are used, the one that's imported first is the one that's suggested in diagnostics.

As for the first example, could it be that rustc always picks "local" path to a (De)serialize trait, since the failure here is related to monomorphization/typeck?

serde_core/src/lib.rs

oli-obk · 2024-05-22T09:07:31Z

When both serde and serde_core are used, the one that's imported first is the one that's suggested in diagnostics.

That's fine. When people are actually directly depending on serde_core, it's fine to suggest importing from it. Almost no one will be directly depending on serde_core. The fact that the import suggestions and other diagnostics always refer to serde is a good sign.

Using a type in a #[derive(Deserialize)] struct's field that doesn't implement Deserialize. Notice a lack of serde::. I even removed use serde::Deserialize to double check that wasn't the reason why.

There are possibly other diagnostics that use the full path instead of just the reduced path, especially on older compiler versions. We've since moved diagnostics to refer to imported paths, but there may be some cases left where we report the full path (and that may then refer to serde_core). That's rare and in almost all cases a compiler bug, so we can ignore that.

oli-obk · 2024-05-22T09:11:26Z

So... the only remaining step is to consider putting various "fishy" Deserialize/Serialize impls (e.g. the one for Result) behind cargo features of serde_core, so that serde 2.0 or other future consumers can just not enable those features unless they deem them valid use cases after all.

This step requires going through every impl in serde_core and making a decision for whether to cargo-feature it or not. Most of them should be obviously fine (integers, atomics, string, vec, ...)

oli-obk · 2024-05-22T09:11:45Z

also next time you merge, maybe rebase and squash instead?

osiewicz · 2024-07-02T23:15:37Z

Outside of Result, I find impls for Mutex pretty suspicious; at the very least, I've never really found myself needing that + it's pretty subtle that the lock is essentially hidden away from you in a library code. That's just my 2 cents though.

osiewicz · 2024-07-06T19:07:34Z

Besides the gains here, this allows serde_json and friends to depend on serde_core, preventing packages using serde w/ derive from serializing things.

I wonder, is there a clear migration path for crates such as serde_json to start using serde_core in place of serde? Say the user depends on serde explicitly (using 1.0 as version constraint) - unless they decide to cargo update their lockfile, will current resolver prefer to have just one copy of serde that is serde_core-compliant, or is it possible that an user will end up with both jumbo serde (as a direct dependency of their code) and serde_core as a dependency of serde_json et al?

epage · 2024-07-07T01:48:17Z

I wonder, is there a clear migration path for crates such as serde_json to start using serde_core in place of serde? Say the user depends on serde explicitly (using 1.0 as version constraint) - unless they decide to cargo update their lockfile, will current resolver prefer to have just one copy of serde that is serde_core-compliant, or is it possible that an user will end up with both jumbo serde (as a direct dependency of their code) and serde_core as a dependency of serde_json et al?

They could have a "never matching" cfg dependency on a new-enough serde that uses serde_core.

Mingun · 2024-08-04T17:29:51Z

So what status of this? It is finished or not? If I understand correctly, the only thing remains is to add bunch of features, one for every manual Serialize / Deserialize implementation that will be in serde_core?

osiewicz · 2024-08-04T17:32:24Z

I don't have anything to add from my side (sans #2608 (comment), though I did not go ahead with this for the lack of feedback), so I'd expect it to be in a merge-able state - except for conflicts, that is.

Mingun · 2024-08-04T18:23:42Z

serde/src/private/de.rs

@@ -298,7 +302,7 @@ mod content {
            // Untagged and internally tagged enums are only supported in
            // self-describing formats.
            let visitor = ContentVisitor { value: PhantomData };
-            deserializer.__deserialize_content(actually_private::T, visitor)
+            deserializer.deserialize_any(visitor)


Another way to deal with that is to remove actually_private::T from the method signature which is needed only to prevent calling this method from the other crates. The method with the similar intention, Visitor::__private_visit_untagged_option does not have such seal and survived separation without problems.

This sounds like a change we could do on master and consider in isolation from this PR

This should help with untangling serde's serialized build. Addresses

Mingun · 2024-08-09T15:20:38Z

@dtolnay, @oli-obk, it seems that all review comments was addressed here or at least @osiewicz thinks so. Please give your opinion.

oli-obk

Please test that everything still works with msrv and a few key versions

oli-obk · 2024-10-21T19:46:23Z

serde/Cargo.toml

@@ -29,7 +30,7 @@ features = ["derive", "rc"]
 [package.metadata.docs.rs]
 features = ["derive", "rc", "unstable"]
 targets = ["x86_64-unknown-linux-gnu"]
-rustdoc-args = ["--generate-link-to-definition"]
+rustdoc-args = ["--cfg", "doc_cfg", "--generate-link-to-definition"]


Is this a rebase artifact? I think we switched to docsrs, so we don't need to explicitly set this anymore (same for serde_core)

oli-obk · 2024-10-22T08:11:32Z

serde/src/lib.rs

@@ -263,28 +233,14 @@ mod lib {
    };
    #[cfg(all(feature = "std", no_target_has_atomic, not(no_std_atomic64)))]


Why are these imports still here, but not the ones with some of the cfgs disabled?

oli-obk · 2024-10-22T08:13:30Z

serde/src/lib.rs

 }

 // None of this crate's error handling needs the `From::from` error conversion
 // performed implicitly by the `?` operator or the standard library's `try!`
 // macro. This simplified macro gives a 5.5% improvement in compile time
 // compared to standard `try!`, and 9% improvement compared to `?`.
+#[doc(hidden)]
+#[macro_export]


Why is this exported?

oli-obk · 2024-10-22T08:14:27Z

serde/src/private/de.rs

@@ -298,7 +302,7 @@ mod content {
            // Untagged and internally tagged enums are only supported in
            // self-describing formats.
            let visitor = ContentVisitor { value: PhantomData };
-            deserializer.__deserialize_content(actually_private::T, visitor)
+            deserializer.deserialize_any(visitor)


This sounds like a change we could do on master and consider in isolation from this PR

oli-obk · 2024-10-22T08:15:09Z

serde/src/private/de.rs

@@ -1048,7 +1052,7 @@ mod content {
        E: de::Error,
    {
        #[cold]
-        fn invalid_type(self, exp: &Expected) -> E {
+        fn invalid_type(self, exp: &dyn Expected) -> E {


Does this cause msrv issues?

oli-obk self-assigned this Feb 9, 2024

dtolnay reviewed Feb 25, 2024

View reviewed changes

dtolnay requested changes Feb 25, 2024

View reviewed changes

osiewicz commented Feb 27, 2024

View reviewed changes

serde_derive/Cargo.toml Outdated Show resolved Hide resolved

oli-obk reviewed Feb 29, 2024

View reviewed changes

serde_core/src/lib.rs Outdated Show resolved Hide resolved

osiewicz force-pushed the extract_serde_core branch 4 times, most recently from 12b7260 to ffe914d Compare May 22, 2024 23:01

osiewicz force-pushed the extract_serde_core branch 2 times, most recently from c71ecab to 7d74d54 Compare July 2, 2024 23:00

Mingun mentioned this pull request Aug 4, 2024

Fix roundtrip of flatten externally tagged enum unit variant #2786

Open

Mingun reviewed Aug 4, 2024

View reviewed changes

osiewicz added 3 commits August 5, 2024 21:50

Extract serde-core out of serde.

6d6ca71

This should help with untangling serde's serialized build. Addresses

Add result feature to serde and serde_core

67b0a95

Update CI to test new result feature

31b0d83

osiewicz force-pushed the extract_serde_core branch 2 times, most recently from 66b00bd to 7eb333d Compare August 5, 2024 20:00

Fixup after rebase

ded3527

osiewicz force-pushed the extract_serde_core branch from 7eb333d to ded3527 Compare August 5, 2024 20:01

oli-obk self-requested a review August 14, 2024 10:56

Mingun mentioned this pull request Aug 22, 2024

Internally-tagged enum representation could be more efficient #1495

Open

oli-obk requested changes Oct 22, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Extract serde_core out of serde crate #2608

feat: Extract serde_core out of serde crate #2608

osiewicz commented Sep 4, 2023 •

edited

Loading

osiewicz commented Sep 4, 2023 •

edited

Loading

epage commented Sep 4, 2023

NobodyXu commented Oct 30, 2023

osiewicz commented Oct 30, 2023 •

edited

Loading

NobodyXu commented Oct 30, 2023

osiewicz commented Oct 30, 2023 •

edited

Loading

oli-obk commented Oct 30, 2023

osiewicz commented Oct 30, 2023

oli-obk commented Oct 30, 2023

epage commented Nov 9, 2023

dtolnay left a comment

epage commented Feb 26, 2024

epage commented Feb 26, 2024

osiewicz commented Feb 29, 2024 •

edited

Loading

oli-obk commented May 22, 2024

oli-obk commented May 22, 2024

oli-obk commented May 22, 2024

osiewicz commented Jul 2, 2024

osiewicz commented Jul 6, 2024

epage commented Jul 7, 2024

Mingun commented Aug 4, 2024

osiewicz commented Aug 4, 2024 •

edited

Loading

Mingun Aug 4, 2024

oli-obk Oct 22, 2024

Mingun commented Aug 9, 2024

oli-obk left a comment

oli-obk Oct 21, 2024

oli-obk Oct 22, 2024

oli-obk Oct 22, 2024

oli-obk Oct 22, 2024

oli-obk Oct 22, 2024

		@@ -263,28 +233,14 @@ mod lib {
		};
		#[cfg(all(feature = "std", no_target_has_atomic, not(no_std_atomic64)))]

feat: Extract serde_core out of serde crate #2608

Are you sure you want to change the base?

feat: Extract serde_core out of serde crate #2608

Conversation

osiewicz commented Sep 4, 2023 • edited Loading

osiewicz commented Sep 4, 2023 • edited Loading

epage commented Sep 4, 2023

NobodyXu commented Oct 30, 2023

osiewicz commented Oct 30, 2023 • edited Loading

NobodyXu commented Oct 30, 2023

osiewicz commented Oct 30, 2023 • edited Loading

oli-obk commented Oct 30, 2023

osiewicz commented Oct 30, 2023

oli-obk commented Oct 30, 2023

epage commented Nov 9, 2023

dtolnay left a comment

Choose a reason for hiding this comment

Compiler diagnostics

Private helper types

Items from 2.0 wishlist

epage commented Feb 26, 2024

epage commented Feb 26, 2024

osiewicz commented Feb 29, 2024 • edited Loading

oli-obk commented May 22, 2024

oli-obk commented May 22, 2024

oli-obk commented May 22, 2024

osiewicz commented Jul 2, 2024

osiewicz commented Jul 6, 2024

epage commented Jul 7, 2024

Mingun commented Aug 4, 2024

osiewicz commented Aug 4, 2024 • edited Loading

Mingun Aug 4, 2024

Choose a reason for hiding this comment

oli-obk Oct 22, 2024

Choose a reason for hiding this comment

Mingun commented Aug 9, 2024

oli-obk left a comment

Choose a reason for hiding this comment

oli-obk Oct 21, 2024

Choose a reason for hiding this comment

oli-obk Oct 22, 2024

Choose a reason for hiding this comment

oli-obk Oct 22, 2024

Choose a reason for hiding this comment

oli-obk Oct 22, 2024

Choose a reason for hiding this comment

oli-obk Oct 22, 2024

Choose a reason for hiding this comment

osiewicz commented Sep 4, 2023 •

edited

Loading

osiewicz commented Sep 4, 2023 •

edited

Loading

osiewicz commented Oct 30, 2023 •

edited

Loading

osiewicz commented Oct 30, 2023 •

edited

Loading

osiewicz commented Feb 29, 2024 •

edited

Loading

osiewicz commented Aug 4, 2024 •

edited

Loading