Skip to content

Commit ae11fbb

Browse files
tannergoodingRuihan-Yin
tannergooding
authored and
Ruihan-Yin
committed
Change the ReciprocalEstimate and ReciprocalSqrtEstimate APIs to be mustExpand on RyuJIT (dotnet#102098)
* Change the ReciprocalEstimate and ReciprocalSqrtEstimate APIs to be mustExpand on RyuJIT * Apply formatting patch * Fix the RV64 and LA64 builds * Mark the ReciprocalEstimate and ReciprocalSqrtEstimate methods as AggressiveOptimization to bypass R2R * Mark other usages of ReciprocalEstimate and ReciprocalSqrtEstimate in Corelib with AggressiveOptimization * Mark several non-deterministic APIs as BypassReadyToRun and skip intrinsic expansion in R2R * Cleanup based on PR recommendations to rely on the runtime rather than attributation of non-deterministic intrinsics * Adding a regression test ensuring direct and indirect invocation of non-deterministic intrinsic APIs returns the same result * Add a note about non-deterministic intrinsic expansion to the botr * Apply formatting patch * Ensure vector tests are correctly validating against the scalar implementation * Fix the JIT/SIMD/VectorConvert test and workaround a 32-bit test issue * Skip a test on Mono due to a known/tracked issue * Ensure that lowering on Arm64 doesn't make an assumption about cast shapes * Ensure the tier0opts local is used * Ensure impEstimateIntrinsic bails out for APIs that need to be implemented as user calls
1 parent 6905408 commit ae11fbb

File tree

22 files changed

+639
-284
lines changed

22 files changed

+639
-284
lines changed

‎docs/design/coreclr/botr/vectors-and-intrinsics.md

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -156,6 +156,14 @@ private void SomeVectorizationHelper()
156156
}
157157
```
158158

159+
#### Non-Deterministic Intrinsics in System.Private.Corelib
160+
161+
Some APIs exposed in System.Private.Corelib are intentionally non-deterministic across hardware and instead only ensure determinism within the scope of a single process. To facilitate the support of such APIs, the JIT defines `Compiler::BlockNonDeterministicIntrinsics(bool mustExpand)` which should be used to help block such APIs from expanding in scenarios such as ReadyToRun. Additionally, such APIs should recursively call themselves so that indirect invocation (such as via a delegate, function pointer, reflection, etc) will compute the same result.
162+
163+
An example of such a non-deterministic API is the `ConvertToIntegerNative` APIs exposed on `System.Single` and `System.Double`. These APIs convert from the source value to the target integer type using the fastest mechanism available for the underlying hardware. They exist due to the IEEE 754 specification leaving conversions undefined when the input cannot fit into the output (for example converting `float.MaxValue` to `int`) and thus different hardware having historically provided differing behaviors on these edge cases. They allow developers who do not need to be concerned with edge case handling but where the performance overhead of normalizing results for the default cast operator is too great.
164+
165+
Another example is the various `*Estimate` APIs, such as `float.ReciprocalSqrtEstimate`. These APIs allow a user to likewise opt into a faster result at the cost of some inaccuracy, where the exact inaccuracy encountered depends on the input and the underlying hardware the instruction is executed against.
166+
159167
# Mechanisms in the JIT to generate correct code to handle varied instruction set support
160168

161169
The JIT receives flags which instruct it on what instruction sets are valid to use, and has access to a new jit interface api `notifyInstructionSetUsage(isa, bool supportBehaviorRequired)`.

‎src/coreclr/jit/compiler.h

Lines changed: 25 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -4530,7 +4530,7 @@ class Compiler
45304530
CORINFO_SIG_INFO* sig,
45314531
CorInfoType callJitType,
45324532
NamedIntrinsic intrinsicName,
4533-
bool tailCall);
4533+
bool mustExpand);
45344534
GenTree* impMathIntrinsic(CORINFO_METHOD_HANDLE method,
45354535
CORINFO_SIG_INFO* sig,
45364536
var_types callType,
@@ -4561,7 +4561,8 @@ class Compiler
45614561
GenTree* impPrimitiveNamedIntrinsic(NamedIntrinsic intrinsic,
45624562
CORINFO_CLASS_HANDLE clsHnd,
45634563
CORINFO_METHOD_HANDLE method,
4564-
CORINFO_SIG_INFO* sig);
4564+
CORINFO_SIG_INFO* sig,
4565+
bool mustExpand);
45654566

45664567
#ifdef FEATURE_HW_INTRINSICS
45674568
GenTree* impHWIntrinsic(NamedIntrinsic intrinsic,
@@ -4573,7 +4574,8 @@ class Compiler
45734574
CORINFO_CLASS_HANDLE clsHnd,
45744575
CORINFO_METHOD_HANDLE method,
45754576
CORINFO_SIG_INFO* sig,
4576-
GenTree* newobjThis);
4577+
GenTree* newobjThis,
4578+
bool mustExpand);
45774579

45784580
protected:
45794581
bool compSupportsHWIntrinsic(CORINFO_InstructionSet isa);
@@ -4584,15 +4586,17 @@ class Compiler
45844586
var_types retType,
45854587
CorInfoType simdBaseJitType,
45864588
unsigned simdSize,
4587-
GenTree* newobjThis);
4589+
GenTree* newobjThis,
4590+
bool mustExpand);
45884591

45894592
GenTree* impSpecialIntrinsic(NamedIntrinsic intrinsic,
45904593
CORINFO_CLASS_HANDLE clsHnd,
45914594
CORINFO_METHOD_HANDLE method,
45924595
CORINFO_SIG_INFO* sig,
45934596
CorInfoType simdBaseJitType,
45944597
var_types retType,
4595-
unsigned simdSize);
4598+
unsigned simdSize,
4599+
bool mustExpand);
45964600

45974601
GenTree* getArgForHWIntrinsic(var_types argType,
45984602
CORINFO_CLASS_HANDLE argClass,
@@ -8272,6 +8276,22 @@ class Compiler
82728276
return eeGetEEInfo()->targetAbi == abi;
82738277
}
82748278

8279+
bool BlockNonDeterministicIntrinsics(bool mustExpand)
8280+
{
8281+
// We explicitly block these APIs from being expanded in R2R
8282+
// since we know they are non-deterministic across hardware
8283+
8284+
if (opts.IsReadyToRun() && !IsTargetAbi(CORINFO_NATIVEAOT_ABI))
8285+
{
8286+
if (mustExpand)
8287+
{
8288+
implLimitation();
8289+
}
8290+
return true;
8291+
}
8292+
return false;
8293+
}
8294+
82758295
#if defined(FEATURE_EH_WINDOWS_X86)
82768296
bool eeIsNativeAotAbi;
82778297
bool UsesFunclets() const

‎src/coreclr/jit/hwintrinsic.cpp

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1577,7 +1577,8 @@ GenTree* Compiler::impHWIntrinsic(NamedIntrinsic intrinsic,
15771577
}
15781578
else
15791579
{
1580-
retNode = impSpecialIntrinsic(intrinsic, clsHnd, method, sig, simdBaseJitType, nodeRetType, simdSize);
1580+
retNode =
1581+
impSpecialIntrinsic(intrinsic, clsHnd, method, sig, simdBaseJitType, nodeRetType, simdSize, mustExpand);
15811582
}
15821583

15831584
#if defined(TARGET_ARM64)

‎src/coreclr/jit/hwintrinsicarm64.cpp

Lines changed: 43 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -473,6 +473,7 @@ GenTree* Compiler::impNonConstFallback(NamedIntrinsic intrinsic, var_types simdT
473473
// sig -- signature of the intrinsic call.
474474
// simdBaseJitType -- generic argument of the intrinsic.
475475
// retType -- return type of the intrinsic.
476+
// mustExpand -- true if the intrinsic must return a GenTree*; otherwise, false
476477
//
477478
// Return Value:
478479
// The GT_HWINTRINSIC node, or nullptr if not a supported intrinsic
@@ -483,7 +484,8 @@ GenTree* Compiler::impSpecialIntrinsic(NamedIntrinsic intrinsic,
483484
CORINFO_SIG_INFO* sig,
484485
CorInfoType simdBaseJitType,
485486
var_types retType,
486-
unsigned simdSize)
487+
unsigned simdSize,
488+
bool mustExpand)
487489
{
488490
const HWIntrinsicCategory category = HWIntrinsicInfo::lookupCategory(intrinsic);
489491
const int numArgs = sig->numArgs;
@@ -762,10 +764,18 @@ GenTree* Compiler::impSpecialIntrinsic(NamedIntrinsic intrinsic,
762764
break;
763765
}
764766

765-
case NI_Vector64_ConvertToInt32:
766767
case NI_Vector64_ConvertToInt32Native:
767-
case NI_Vector128_ConvertToInt32:
768768
case NI_Vector128_ConvertToInt32Native:
769+
{
770+
if (BlockNonDeterministicIntrinsics(mustExpand))
771+
{
772+
break;
773+
}
774+
FALLTHROUGH;
775+
}
776+
777+
case NI_Vector64_ConvertToInt32:
778+
case NI_Vector128_ConvertToInt32:
769779
{
770780
assert(sig->numArgs == 1);
771781
assert(simdBaseType == TYP_FLOAT);
@@ -775,10 +785,18 @@ GenTree* Compiler::impSpecialIntrinsic(NamedIntrinsic intrinsic,
775785
break;
776786
}
777787

778-
case NI_Vector64_ConvertToInt64:
779788
case NI_Vector64_ConvertToInt64Native:
780-
case NI_Vector128_ConvertToInt64:
781789
case NI_Vector128_ConvertToInt64Native:
790+
{
791+
if (BlockNonDeterministicIntrinsics(mustExpand))
792+
{
793+
break;
794+
}
795+
FALLTHROUGH;
796+
}
797+
798+
case NI_Vector64_ConvertToInt64:
799+
case NI_Vector128_ConvertToInt64:
782800
{
783801
assert(sig->numArgs == 1);
784802
assert(simdBaseType == TYP_DOUBLE);
@@ -799,10 +817,18 @@ GenTree* Compiler::impSpecialIntrinsic(NamedIntrinsic intrinsic,
799817
break;
800818
}
801819

802-
case NI_Vector64_ConvertToUInt32:
803820
case NI_Vector64_ConvertToUInt32Native:
804-
case NI_Vector128_ConvertToUInt32:
805821
case NI_Vector128_ConvertToUInt32Native:
822+
{
823+
if (BlockNonDeterministicIntrinsics(mustExpand))
824+
{
825+
break;
826+
}
827+
FALLTHROUGH;
828+
}
829+
830+
case NI_Vector64_ConvertToUInt32:
831+
case NI_Vector128_ConvertToUInt32:
806832
{
807833
assert(sig->numArgs == 1);
808834
assert(simdBaseType == TYP_FLOAT);
@@ -812,10 +838,18 @@ GenTree* Compiler::impSpecialIntrinsic(NamedIntrinsic intrinsic,
812838
break;
813839
}
814840

815-
case NI_Vector64_ConvertToUInt64:
816841
case NI_Vector64_ConvertToUInt64Native:
817-
case NI_Vector128_ConvertToUInt64:
818842
case NI_Vector128_ConvertToUInt64Native:
843+
{
844+
if (BlockNonDeterministicIntrinsics(mustExpand))
845+
{
846+
break;
847+
}
848+
FALLTHROUGH;
849+
}
850+
851+
case NI_Vector64_ConvertToUInt64:
852+
case NI_Vector128_ConvertToUInt64:
819853
{
820854
assert(sig->numArgs == 1);
821855
assert(simdBaseType == TYP_DOUBLE);

‎src/coreclr/jit/hwintrinsicxarch.cpp

Lines changed: 33 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -957,6 +957,8 @@ GenTree* Compiler::impNonConstFallback(NamedIntrinsic intrinsic, var_types simdT
957957
// sig -- signature of the intrinsic call.
958958
// simdBaseJitType -- generic argument of the intrinsic.
959959
// retType -- return type of the intrinsic.
960+
// mustExpand -- true if the intrinsic must return a GenTree*; otherwise, false
961+
//
960962
// Return Value:
961963
// the expanded intrinsic.
962964
//
@@ -966,7 +968,8 @@ GenTree* Compiler::impSpecialIntrinsic(NamedIntrinsic intrinsic,
966968
CORINFO_SIG_INFO* sig,
967969
CorInfoType simdBaseJitType,
968970
var_types retType,
969-
unsigned simdSize)
971+
unsigned simdSize,
972+
bool mustExpand)
970973
{
971974
GenTree* retNode = nullptr;
972975
GenTree* op1 = nullptr;
@@ -1098,7 +1101,7 @@ GenTree* Compiler::impSpecialIntrinsic(NamedIntrinsic intrinsic,
10981101
{
10991102
// Vector<T> is TYP_SIMD32, so we should treat this as a call to Vector128.ToVector256
11001103
return impSpecialIntrinsic(NI_Vector128_ToVector256, clsHnd, method, sig, simdBaseJitType, retType,
1101-
simdSize);
1104+
simdSize, mustExpand);
11021105
}
11031106
else if (vectorTByteLength == XMM_REGSIZE_BYTES)
11041107
{
@@ -1208,7 +1211,7 @@ GenTree* Compiler::impSpecialIntrinsic(NamedIntrinsic intrinsic,
12081211
{
12091212
// Vector<T> is TYP_SIMD32, so we should treat this as a call to Vector256.GetLower
12101213
return impSpecialIntrinsic(NI_Vector256_GetLower, clsHnd, method, sig, simdBaseJitType, retType,
1211-
simdSize);
1214+
simdSize, mustExpand);
12121215
}
12131216

12141217
default:
@@ -1248,13 +1251,13 @@ GenTree* Compiler::impSpecialIntrinsic(NamedIntrinsic intrinsic,
12481251
if (intrinsic == NI_Vector256_AsVector)
12491252
{
12501253
return impSpecialIntrinsic(NI_Vector256_GetLower, clsHnd, method, sig, simdBaseJitType, retType,
1251-
simdSize);
1254+
simdSize, mustExpand);
12521255
}
12531256
else
12541257
{
12551258
assert(intrinsic == NI_Vector256_AsVector256);
12561259
return impSpecialIntrinsic(NI_Vector128_ToVector256, clsHnd, method, sig, simdBaseJitType,
1257-
retType, 16);
1260+
retType, 16, mustExpand);
12581261
}
12591262
}
12601263
}
@@ -1281,13 +1284,13 @@ GenTree* Compiler::impSpecialIntrinsic(NamedIntrinsic intrinsic,
12811284
if (intrinsic == NI_Vector512_AsVector)
12821285
{
12831286
return impSpecialIntrinsic(NI_Vector512_GetLower, clsHnd, method, sig, simdBaseJitType, retType,
1284-
simdSize);
1287+
simdSize, mustExpand);
12851288
}
12861289
else
12871290
{
12881291
assert(intrinsic == NI_Vector512_AsVector512);
12891292
return impSpecialIntrinsic(NI_Vector256_ToVector512, clsHnd, method, sig, simdBaseJitType, retType,
1290-
32);
1293+
32, mustExpand);
12911294
}
12921295
break;
12931296
}
@@ -1301,13 +1304,13 @@ GenTree* Compiler::impSpecialIntrinsic(NamedIntrinsic intrinsic,
13011304
if (intrinsic == NI_Vector512_AsVector)
13021305
{
13031306
return impSpecialIntrinsic(NI_Vector512_GetLower128, clsHnd, method, sig, simdBaseJitType,
1304-
retType, simdSize);
1307+
retType, simdSize, mustExpand);
13051308
}
13061309
else
13071310
{
13081311
assert(intrinsic == NI_Vector512_AsVector512);
13091312
return impSpecialIntrinsic(NI_Vector128_ToVector512, clsHnd, method, sig, simdBaseJitType,
1310-
retType, 16);
1313+
retType, 16, mustExpand);
13111314
}
13121315
}
13131316
}
@@ -1436,6 +1439,11 @@ GenTree* Compiler::impSpecialIntrinsic(NamedIntrinsic intrinsic,
14361439
assert(sig->numArgs == 1);
14371440
assert(simdBaseType == TYP_FLOAT);
14381441

1442+
if (BlockNonDeterministicIntrinsics(mustExpand))
1443+
{
1444+
break;
1445+
}
1446+
14391447
op1 = impSIMDPopStack();
14401448
retNode = gtNewSimdCvtNativeNode(retType, op1, CORINFO_TYPE_INT, simdBaseJitType, simdSize);
14411449
break;
@@ -1463,6 +1471,11 @@ GenTree* Compiler::impSpecialIntrinsic(NamedIntrinsic intrinsic,
14631471
assert(sig->numArgs == 1);
14641472
assert(simdBaseType == TYP_DOUBLE);
14651473

1474+
if (BlockNonDeterministicIntrinsics(mustExpand))
1475+
{
1476+
break;
1477+
}
1478+
14661479
if (IsBaselineVector512IsaSupportedOpportunistically())
14671480
{
14681481
op1 = impSIMDPopStack();
@@ -1542,6 +1555,11 @@ GenTree* Compiler::impSpecialIntrinsic(NamedIntrinsic intrinsic,
15421555
assert(sig->numArgs == 1);
15431556
assert(simdBaseType == TYP_FLOAT);
15441557

1558+
if (BlockNonDeterministicIntrinsics(mustExpand))
1559+
{
1560+
break;
1561+
}
1562+
15451563
if (IsBaselineVector512IsaSupportedOpportunistically())
15461564
{
15471565
op1 = impSIMDPopStack();
@@ -1556,6 +1574,7 @@ GenTree* Compiler::impSpecialIntrinsic(NamedIntrinsic intrinsic,
15561574
{
15571575
assert(sig->numArgs == 1);
15581576
assert(simdBaseType == TYP_DOUBLE);
1577+
15591578
if (IsBaselineVector512IsaSupportedOpportunistically())
15601579
{
15611580
op1 = impSIMDPopStack();
@@ -1571,6 +1590,11 @@ GenTree* Compiler::impSpecialIntrinsic(NamedIntrinsic intrinsic,
15711590
assert(sig->numArgs == 1);
15721591
assert(simdBaseType == TYP_DOUBLE);
15731592

1593+
if (BlockNonDeterministicIntrinsics(mustExpand))
1594+
{
1595+
break;
1596+
}
1597+
15741598
if (IsBaselineVector512IsaSupportedOpportunistically())
15751599
{
15761600
op1 = impSIMDPopStack();

0 commit comments

Comments
 (0)