[CDAP-21075] Support DP 2.2(spark 3.5.1) by checking if the new method is present or not.

[sahusanket]

We need to support both Dataproc 2.2 ( spark 3.5.1) and 2.1 (spark 3.3.2).

There were few refactoring and breaking API changes in SPARK-44531 .

---

Changed the spark version in CDAP to 3.5.1 just to check compilation failures and it resulted in only 1 Error in out code base i.e. OpaqueDatasetCollection#toDataframeCollection

In which we were using RowEncoder.apply(StructType) which is no longer present.

Also testing with DP 2.2 with a suitable pipeline (Dedup --> Join) gives java.lang.NoSuchMethodError for that code path, rest of the testing went fine.

---

Based on above points, we can say this is the only code path that requires change.

---

Fix in this pr:
Using reflection we are first determining if new method is present, if so , it is Spark 3.5.1 and we use the following code via reflection to achieve the Encoder :

AgnosticEncoder ae = RowEncoder.encoderFor(schema);
//Convert to expression
return ExpressionEncoder.apply(ae);

[sonarqubecloud]

Quality Gate failed

Failed conditions
0.0% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube Cloud

[tivv]

Could you please cache the Method objects retrieved? Instrospection is a pretty heavy operation, we can do it once and then simply do "invoke".