Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add NerdbankMessagePackFormatter #1100

Draft
wants to merge 25 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
e934175
Update package versions and target frameworks
Dec 18, 2024
6819e6b
Continue implementing NerdbankMessagePackFormatter.
Dec 20, 2024
052592e
Refactor NerdbankMessagePackFormatter for converters
Dec 20, 2024
259558a
Add type converter registration methods to interface
Dec 21, 2024
e87da44
Enhance serialization context and type shape providers
Dec 21, 2024
ca439ea
Enhance NerdbankMessagePackFormatter with new methods
Dec 21, 2024
b12c357
Replaced the `IFormatterContextBuilder` interface with a concrete `Fo…
Dec 21, 2024
345a39c
Downgrade package versions for compatibility
Dec 21, 2024
ad436f7
User RawMessagePack from Nerdbank.MessagePack
Dec 21, 2024
1a795c7
Remove namespace aliases
Dec 21, 2024
0e6b4aa
Add unit tests
Dec 21, 2024
bca2af4
Some tests are now passing.
Dec 22, 2024
d693a1c
Cleaning up formatters. Rename `FormatterContext` to `FormatterProfile`.
Dec 22, 2024
0c06ae5
Adopt MessagePackString and configure test formatters.
Dec 24, 2024
166ae49
Adopt simpler MessagePackString write API
Dec 26, 2024
3d3cb90
Fix bug when searching property names from peek reader
Dec 26, 2024
d8430f4
Begin process of making converters stateless
Dec 27, 2024
6444f55
Remove state from converters.
Dec 27, 2024
bb22ee8
Refactor and add comments for Profile and Profile.Builder.
Dec 27, 2024
e171cdb
Add additional profile configuration to test suite.
Dec 28, 2024
d36d607
Update Benchmarks
Dec 28, 2024
949b3ab
Upgrade NB.MB version; Update tests.
Dec 29, 2024
f94bf1d
Refactor serialization to use ReadOnlySequence<byte>
Jan 1, 2025
eecd92c
Enhance NerdbankMessagePackFormatter with new types
Jan 1, 2025
edad2f8
Update Nerdbank.MessagePack and refactor converters
Jan 11, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 5 additions & 4 deletions Directory.Packages.props
Original file line number Diff line number Diff line change
Expand Up @@ -15,21 +15,22 @@
<PackageVersion Include="MessagePackAnalyzer" Version="$(MessagePackVersion)" />
<PackageVersion Include="Microsoft.AspNetCore.TestHost" Version="2.2.0" />
<PackageVersion Include="Microsoft.AspNetCore" Version="2.2.0" />
<PackageVersion Include="Microsoft.Bcl.AsyncInterfaces" Version="8.0.0" />
<PackageVersion Include="Microsoft.Bcl.AsyncInterfaces" Version="9.0.0" />
<PackageVersion Include="Microsoft.CodeAnalysis.PublicApiAnalyzers" Version="3.3.4" />
<PackageVersion Include="Microsoft.NET.Test.Sdk" Version="17.11.1" />
<PackageVersion Include="Microsoft.VisualStudio.Internal.MicroBuild.NonShipping" Version="$(MicroBuildVersion)" />
<PackageVersion Include="Microsoft.VisualStudio.Threading.Analyzers" Version="$(VisualStudioThreadingVersion)" />
<PackageVersion Include="Microsoft.VisualStudio.Threading" Version="$(VisualStudioThreadingVersion)" />
<PackageVersion Include="Microsoft.VisualStudio.Validation" Version="17.8.8" />
<PackageVersion Include="Nerdbank.MessagePack" Version="0.3.54-beta" />
<PackageVersion Include="Nerdbank.Streams" Version="2.11.74" />
<PackageVersion Include="Newtonsoft.Json" Version="13.0.1" />
<PackageVersion Include="System.Collections.Immutable" Version="6.0.0" />
<PackageVersion Include="System.Collections.Immutable" Version="9.0.0" />
trippwill marked this conversation as resolved.
Show resolved Hide resolved
<PackageVersion Include="System.Diagnostics.DiagnosticSource" Version="6.0.1" />
<PackageVersion Include="System.IO.Pipelines" Version="8.0.0" />
<PackageVersion Include="System.IO.Pipelines" Version="9.0.0" />
<PackageVersion Include="System.IO.Pipes" Version="4.3.0" />
<PackageVersion Include="System.Net.Http" Version="4.3.4" />
<PackageVersion Include="System.Text.Json" Version="8.0.5" />
<PackageVersion Include="System.Text.Json" Version="9.0.0" />
trippwill marked this conversation as resolved.
Show resolved Hide resolved
<PackageVersion Include="System.Threading.Tasks.Dataflow" Version="6.0.0" />
<PackageVersion Include="System.ValueTuple" Version="4.5.0" />
<PackageVersion Include="xunit.combinatorial" Version="1.6.24" />
Expand Down
15 changes: 15 additions & 0 deletions nuget.config
Original file line number Diff line number Diff line change
Expand Up @@ -5,10 +5,25 @@
</config>
<packageSources>
<clear />
<add key="PublicCI" value="https://pkgs.dev.azure.com/andrewarnott/OSS/_packaging/PublicCI/nuget/v3/index.json" />
<add key="nuget.org" value="https://api.nuget.org/v3/index.json" />
<add key="msft_consumption" value="https://pkgs.dev.azure.com/azure-public/vside/_packaging/msft_consumption/nuget/v3/index.json" />
</packageSources>
<disabledPackageSources>
<!-- Defend against user or machine level disabling of sources that we list in this file. -->
<clear />
</disabledPackageSources>
<packageSourceMapping>
<!-- key value for <packageSource> should match key values from <packageSources> element -->
<packageSource key="msft_consumption">
<package pattern="*" />
</packageSource>
<packageSource key="nuget.org">
<package pattern="System.*" />
<package pattern="Microsoft.Bcl.*" />
</packageSource>
<packageSource key="PublicCI">
<package pattern="Nerdbank.MessagePack" />
</packageSource>
</packageSourceMapping>
</configuration>
1 change: 1 addition & 0 deletions src/StreamJsonRpc/FormatterBase.cs
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
using System.IO.Pipelines;
using System.Reflection;
using System.Runtime.Serialization;
using Nerdbank.MessagePack;
using Nerdbank.Streams;
using StreamJsonRpc.Protocol;
using StreamJsonRpc.Reflection;
Expand Down
2 changes: 1 addition & 1 deletion src/StreamJsonRpc/JsonRpc.cs
Original file line number Diff line number Diff line change
Expand Up @@ -1697,7 +1697,7 @@ protected virtual async ValueTask<JsonRpcMessage> DispatchRequestAsync(JsonRpcRe
}

/// <summary>
/// Sends the JSON-RPC message to <see cref="IJsonRpcMessageHandler"/> intance to be transmitted.
/// Sends the JSON-RPC message to <see cref="IJsonRpcMessageHandler"/> instance to be transmitted.
/// </summary>
/// <param name="message">The message to send.</param>
/// <param name="cancellationToken">A token to cancel the send request.</param>
Expand Down
104 changes: 104 additions & 0 deletions src/StreamJsonRpc/NerdbankMessagePackFormatter.CommonString.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,104 @@
// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT license. See LICENSE file in the project root for full license information.

using System.Diagnostics;
using NBMP = Nerdbank.MessagePack;

namespace StreamJsonRpc;

public partial class NerdbankMessagePackFormatter
{
[DebuggerDisplay("{" + nameof(Value) + "}")]
private struct CommonString
trippwill marked this conversation as resolved.
Show resolved Hide resolved
{
internal CommonString(string value)
{
Requires.Argument(value.Length > 0 && value.Length <= 16, nameof(value), "Length must be >0 and <=16.");
this.Value = value;
ReadOnlyMemory<byte> encodedBytes = MessagePack.Internal.CodeGenHelpers.GetEncodedStringBytes(value);
this.EncodedBytes = encodedBytes;

ReadOnlySpan<byte> span = this.EncodedBytes.Span.Slice(1);
this.Key = MessagePack.Internal.AutomataKeyGen.GetKey(ref span); // header is 1 byte because string length <= 16
this.Key2 = span.Length > 0 ? (ulong?)MessagePack.Internal.AutomataKeyGen.GetKey(ref span) : null;
}

/// <summary>
/// Gets the original string.
/// </summary>
internal string Value { get; }

/// <summary>
/// Gets the 64-bit integer that represents the string without decoding it.
/// </summary>
private ulong Key { get; }

/// <summary>
/// Gets the next 64-bit integer that represents the string without decoding it.
/// </summary>
private ulong? Key2 { get; }

/// <summary>
/// Gets the messagepack header and UTF-8 bytes for this string.
/// </summary>
private ReadOnlyMemory<byte> EncodedBytes { get; }

/// <summary>
/// Writes out the messagepack binary for this common string, if it matches the given value.
/// </summary>
/// <param name="writer">The writer to use.</param>
/// <param name="value">The value to be written, if it matches this <see cref="CommonString"/>.</param>
/// <returns><see langword="true"/> if <paramref name="value"/> matches this <see cref="Value"/> and it was written; <see langword="false"/> otherwise.</returns>
internal bool TryWrite(ref NBMP::MessagePackWriter writer, string value)
{
if (value == this.Value)
{
this.Write(ref writer);
return true;
}

return false;
}

internal readonly void Write(ref NBMP::MessagePackWriter writer) => writer.WriteRaw(this.EncodedBytes.Span);

/// <summary>
/// Checks whether a span of UTF-8 bytes equal this common string.
/// </summary>
/// <param name="utf8String">The UTF-8 string.</param>
/// <returns><see langword="true"/> if the UTF-8 bytes are the encoding of this common string; <see langword="false"/> otherwise.</returns>
internal readonly bool TryRead(ReadOnlySpan<byte> utf8String)
{
if (utf8String.Length != this.EncodedBytes.Length - 1)
{
return false;
}

ulong key1 = MessagePack.Internal.AutomataKeyGen.GetKey(ref utf8String);
if (key1 != this.Key)
{
return false;
}

if (utf8String.Length > 0)
{
if (!this.Key2.HasValue)
{
return false;
}

ulong key2 = MessagePack.Internal.AutomataKeyGen.GetKey(ref utf8String);
if (key2 != this.Key2.Value)
{
return false;
}
}
else if (this.Key2.HasValue)
{
return false;
}

return true;
}
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT license. See LICENSE file in the project root for full license information.

using PolyType;

namespace StreamJsonRpc;

/// <summary>
/// Serializes JSON-RPC messages using MessagePack (a fast, compact binary format).
/// </summary>
/// <remarks>
/// The MessagePack implementation used here comes from https://github.com/AArnott/Nerdbank.MessagePack.
/// </remarks>
public sealed partial class NerdbankMessagePackFormatter
{
/// <summary>
/// Provides a builder interface for adding type shape providers.
/// </summary>
public interface ICompositeTypeShapeProviderBuilder
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hopefully this whole interface will be unnecessary when eiriktsarpalis/PolyType#91 is fulfilled.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I simplified this further by removing this interface and merging the methods in the FormatterContectBuilder. But yes, it would be nice to rely on built-in support from PolyType for composing type shape providers.

{
/// <summary>
/// Adds a single type shape provider to the builder.
/// </summary>
/// <param name="provider">The type shape provider to add.</param>
/// <returns>The current builder instance.</returns>
ICompositeTypeShapeProviderBuilder Add(ITypeShapeProvider provider);

/// <summary>
/// Adds a range of type shape providers to the builder.
/// </summary>
/// <param name="providers">The collection of type shape providers to add.</param>
/// <returns>The current builder instance.</returns>
ICompositeTypeShapeProviderBuilder AddRange(IEnumerable<ITypeShapeProvider> providers);

/// <summary>
/// Adds a reflection-based type shape provider to the builder.
/// </summary>
/// <param name="useReflectionEmit">A value indicating whether to use Reflection.Emit for dynamic type generation.</param>
/// <returns>The current builder instance.</returns>
ICompositeTypeShapeProviderBuilder AddReflectionTypeShapeProvider(bool useReflectionEmit);
trippwill marked this conversation as resolved.
Show resolved Hide resolved
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT license. See LICENSE file in the project root for full license information.

using Nerdbank.MessagePack;

namespace StreamJsonRpc;

/// <summary>
/// Serializes JSON-RPC messages using MessagePack (a fast, compact binary format).
/// </summary>
/// <remarks>
/// The MessagePack implementation used here comes from https://github.com/AArnott/Nerdbank.MessagePack.
/// </remarks>
public sealed partial class NerdbankMessagePackFormatter
{
/// <summary>
/// Provides a builder interface for configuring the serialization context.
/// </summary>
public interface IFormatterContextBuilder
trippwill marked this conversation as resolved.
Show resolved Hide resolved
{
/// <summary>
/// Gets the type shape provider builder.
/// </summary>
ICompositeTypeShapeProviderBuilder TypeShapeProviderBuilder { get; }

/// <summary>
/// Registers a type converter for asynchronous enumerable types.
/// </summary>
/// <typeparam name="TElement">The element type of the asynchronous enumerable.</typeparam>
void RegisterAsyncEnumerableTypeConverter<TElement>();

/// <summary>
/// Registers a custom converter for a specific type.
/// </summary>
/// <typeparam name="T">The type for which the converter is registered.</typeparam>
/// <param name="converter">The converter to register.</param>
void RegisterConverter<T>(MessagePackConverter<T> converter);

/// <summary>
/// Registers known subtypes for a base type.
/// </summary>
/// <typeparam name="TBase">The base type for which the subtypes are registered.</typeparam>
/// <param name="mapping">The mapping of known subtypes.</param>
void RegisterKnownSubTypes<TBase>(KnownSubTypeMapping<TBase> mapping);

/// <summary>
/// Registers a type converter for progress types.
/// </summary>
/// <typeparam name="TProgress">The type of the progress to register.</typeparam>
void RegisterProgressTypeConverter<TProgress>();
}
}
88 changes: 88 additions & 0 deletions src/StreamJsonRpc/NerdbankMessagePackFormatter.RawMessagePack.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT license. See LICENSE file in the project root for full license information.

using System.Buffers;
using Nerdbank.MessagePack;
using PolyType.Abstractions;

namespace StreamJsonRpc;

/// <summary>
/// Serializes JSON-RPC messages using MessagePack (a fast, compact binary format).
/// </summary>
/// <remarks>
/// The MessagePack implementation used here comes from https://github.com/AArnott/Nerdbank.MessagePack.
/// </remarks>
public partial class NerdbankMessagePackFormatter
{
private struct RawMessagePack
trippwill marked this conversation as resolved.
Show resolved Hide resolved
{
private readonly ReadOnlySequence<byte> rawSequence;

private readonly ReadOnlyMemory<byte> rawMemory;

private RawMessagePack(ReadOnlySequence<byte> raw)
{
this.rawSequence = raw;
this.rawMemory = default;
}

private RawMessagePack(ReadOnlyMemory<byte> raw)
{
this.rawSequence = default;
this.rawMemory = raw;
}

internal readonly bool IsDefault => this.rawMemory.IsEmpty && this.rawSequence.IsEmpty;

public override readonly string ToString() => "<raw msgpack>";

/// <summary>
/// Reads one raw messagepack token.
/// </summary>
/// <param name="reader">The reader to use.</param>
/// <param name="copy"><see langword="true"/> if the token must outlive the lifetime of the reader's underlying buffer; <see langword="false"/> otherwise.</param>
/// <param name="context">The serialization context to use.</param>
/// <returns>The raw messagepack slice.</returns>
internal static RawMessagePack ReadRaw(ref MessagePackReader reader, bool copy, Nerdbank.MessagePack.SerializationContext context)
{
SequencePosition initialPosition = reader.Position;
reader.Skip(context);
ReadOnlySequence<byte> slice = reader.Sequence.Slice(initialPosition, reader.Position);
return copy ? new RawMessagePack(slice.ToArray()) : new RawMessagePack(slice);
}

internal readonly void WriteRaw(ref MessagePackWriter writer)
{
if (this.rawSequence.IsEmpty)
{
writer.WriteRaw(this.rawMemory.Span);
}
else
{
writer.WriteRaw(this.rawSequence);
}
}

internal readonly object? Deserialize(Type type, FormatterContext options)
{
MessagePackReader reader = this.rawSequence.IsEmpty
? new MessagePackReader(this.rawMemory)
: new MessagePackReader(this.rawSequence);

return options.Serializer.DeserializeObject(
ref reader,
options.ShapeProvider.Resolve(type));
}

internal readonly T Deserialize<T>(FormatterContext options)
{
MessagePackReader reader = this.rawSequence.IsEmpty
? new MessagePackReader(this.rawMemory)
: new MessagePackReader(this.rawSequence);

return options.Serializer.Deserialize<T>(ref reader, options.ShapeProvider)
?? throw new MessagePackSerializationException(Resources.FailureDeserializingJsonRpc);
}
}
}
Loading