Bring over distributed transactions support for Windows

[roji]

@stephentoub @AaronRobinsonMSFT here's a draft PR bringing over Windows support for distributed transactions from netfx.

• The native side seems simple enough; as discussed we now have a System.Transactions.Native.dll (no more mixed mode C++/CLI), built with cmake like other native libraries. No more extra modules, seems simple and clean.
• This is tested to successfully manage and commit a distributed transaction on Windows (manual testing only for now).
• The managed code still needs some cleanup/work (NRT annotations, clean up tracing/logging, properly remove BinaryFormatter-dependent logic). So no need to give this a proper review yet - am mainly looking for confirmation that the native interop and overall approach look good.

/cc @ajcvickers

[ghost]

Note regarding the new-api-needs-documentation label:

This serves as a reminder for when your PR is modifying a ref *.cs file and adding/modifying public APIs, to please make sure the API implementation in the src *.cs file is documented with triple slash comments, so the PR reviewers can sign off that change.

[stephentoub]

Awesome. Nice work, @roji.

[stephentoub]

I know you said you've yet to clean up the managed code, so just for tracking purposes, we'll want to convert the DllImports to LibraryImports.

[roji]

Is this possible, given that this uses [MarshalAs(UnmanagedType.Interface)]? /cc @AaronRobinsonMSFT

[jkotas]

The native side seems simple enough; as discussed we now have a System.Transactions.Native.dll

What are the reasons to implement this as an unmanaged shim and not just C#?

The unmanaged shim will require special build and packaging logic that is not straightforward. We do not have OOB Windows library with unmanaged shim like this, so you are likely going to need to do some build infrastructure work for this. EDIT: I have not realized that this is inbox library, so the new packaging logic should not be required. My question still remains.

[stephentoub]

I expect we'll want to strip out most or all of the serialization-related support.
cc: @GrabYourPitchforks

[roji]

Yep. I still need to do some research, but from what I understand, System.Transactions had functionality where attempting to send a Transaction across AppDomain boundaries would automatically trigger escalation to a distributed transaction, and then propagate the transaction via OleTx. I think this relied on BinaryFormatter, and I definitely don't think we need to do this.

We can always see if we get user demand in the future for this (I doubt it), and if so, reimplement this in some safer way.

[roji]

@jkotas

What are the reasons to implement this as an unmanaged shim and not just C#?

The native code was brought over from netfx as-is - with almost zero changes - the idea being that we want to avoid any risky rewriting which would make this more complex. I don't foresee us working on this much: the goal here is mostly to unblock people still stuck on netfx because their application requires distributed transactions.

Makes sense?

[jkotas]

Makes sense?

I have doubts that this is saving us work in the long run. C/C++ code is high maintenance. It requires updates to the latest SDL requirements every release (like #66154). It needs special handling during the build, packaging, trimming and single-file. etc.

Unfortunately, this is inbox library so we will be stuck with this nearly forever,

[roji]

Closing in favor of #72051, which ports the C++ DtcProxyShim to .NET.