Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replacing Jackson Factory with Gson Factory and further code updation #32158

Merged
merged 5 commits into from
Aug 19, 2024
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .test-infra/dockerized-jenkins/plugins.txt
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ github-branch-source
gradle
handlebars
handy-uri-templates-2-api
jackson2-api
gson-api
jacoco
javadoc
jaxb
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -731,7 +731,7 @@ class BeamModulePlugin implements Plugin<Project> {
google_ads : "com.google.api-ads:google-ads:$google_ads_version",
google_ads_stubs_v14 : "com.google.api-ads:google-ads-stubs-v14:$google_ads_version",
google_api_client : "com.google.api-client:google-api-client:$google_clients_version", // for the libraries using $google_clients_version below.
google_api_client_jackson2 : "com.google.api-client:google-api-client-jackson2:$google_clients_version",
google_api_client_gson : "com.google.api-client:google-api-client-gson:$google_clients_version",
google_api_client_java6 : "com.google.api-client:google-api-client-java6:$google_clients_version",
google_api_common : "com.google.api:api-common", // google_cloud_platform_libraries_bom sets version
google_api_services_bigquery : "com.google.apis:google-api-services-bigquery:v2-rev20240323-2.0.0", // [bomupgrader] sets version
Expand Down Expand Up @@ -768,7 +768,7 @@ class BeamModulePlugin implements Plugin<Project> {
google_http_client_apache_v2 : "com.google.http-client:google-http-client-apache-v2", // google_cloud_platform_libraries_bom sets version
google_http_client_gson : "com.google.http-client:google-http-client-gson", // google_cloud_platform_libraries_bom sets version
google_http_client_jackson : "com.google.http-client:google-http-client-jackson:1.29.2",
google_http_client_jackson2 : "com.google.http-client:google-http-client-jackson2", // google_cloud_platform_libraries_bom sets version
google_http_client_gson : "com.google.http-client:google-http-client-gson", // google_cloud_platform_libraries_bom sets version
google_http_client_protobuf : "com.google.http-client:google-http-client-protobuf", // google_cloud_platform_libraries_bom sets version
google_oauth_client : "com.google.oauth-client:google-oauth-client:$google_oauth_clients_version",
google_oauth_client_java6 : "com.google.oauth-client:google-oauth-client-java6:$google_oauth_clients_version",
Expand Down
1 change: 1 addition & 0 deletions gradle/wrapper/gradle-wrapper.properties
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
#Sat Aug 10 17:06:24 IST 2024
damccorm marked this conversation as resolved.
Show resolved Hide resolved
distributionBase=GRADLE_USER_HOME
distributionPath=wrapper/dists
distributionUrl=https\://services.gradle.org/distributions/gradle-8.4-bin.zip
Expand Down
4 changes: 2 additions & 2 deletions runners/google-cloud-dataflow-java/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -108,8 +108,8 @@ dependencies {
implementation library.java.google_auth_library_credentials
implementation library.java.google_auth_library_oauth2_http
implementation library.java.google_http_client
implementation library.java.google_http_client_jackson2
permitUnusedDeclared library.java.google_http_client_jackson2 // BEAM-11761
implementation library.java.google_http_client_gson
permitUnusedDeclared library.java.google_http_client_gson // BEAM-11761
implementation library.java.hamcrest
implementation library.java.jackson_annotations
implementation library.java.jackson_core
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@
import com.google.api.client.json.GenericJson;
import com.google.api.client.json.Json;
import com.google.api.client.json.JsonFactory;
import com.google.api.client.json.jackson2.JacksonFactory;
import com.google.api.client.json.gson.GsonFactory;
import com.google.api.client.testing.http.HttpTesting;
import com.google.api.client.testing.http.MockHttpTransport;
import com.google.api.client.testing.http.MockLowLevelHttpRequest;
Expand Down Expand Up @@ -580,7 +580,7 @@ public void testPackageUploadIsSkippedWithNonExistentResource() throws Exception
/** Builds a fake GoogleJsonResponseException for testing API error handling. */
private static GoogleJsonResponseException googleJsonResponseException(
final int status, final String reason, final String message) throws IOException {
final JsonFactory jsonFactory = new JacksonFactory();
final JsonFactory jsonFactory = new GsonFactory();
HttpTransport transport =
new MockHttpTransport() {
@Override
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@
import static org.junit.Assert.assertNotSame;

import com.google.api.client.json.GenericJson;
import com.google.api.client.json.jackson2.JacksonFactory;
import com.google.api.client.json.gson.GsonFactory;
import com.google.api.services.dataflow.model.InstructionOutput;
import com.google.api.services.dataflow.model.ParDoInstruction;
import com.google.api.services.dataflow.model.ParallelInstruction;
Expand Down Expand Up @@ -89,7 +89,7 @@ public class LengthPrefixUnknownCodersTest {
public void setup() {
MockitoAnnotations.initMocks(this);
instruction = new ParallelInstruction();
instruction.setFactory(new JacksonFactory());
instruction.setFactory(new GsonFactory());
}

/** Test wrapping unknown coders with {@code LengthPrefixCoder} */
Expand Down Expand Up @@ -142,7 +142,7 @@ public void testLengthPrefixAndReplaceUnknownCoder() throws Exception {
public void testLengthPrefixInstructionOutputCoder() throws Exception {
InstructionOutput output = new InstructionOutput();
output.setCodec(CloudObjects.asCloudObject(windowedValueCoder, /*sdkComponents=*/ null));
output.setFactory(new JacksonFactory());
output.setFactory(new GsonFactory());

InstructionOutput prefixedOutput = forInstructionOutput(output, false);
assertEqualsAsJson(
Expand Down Expand Up @@ -267,7 +267,7 @@ private static ParallelInstructionNode createReadNode(
.setCodec(CloudObjects.asCloudObject(coder, /*sdkComponents=*/ null))
.setSpec(CloudObject.forClassName(readClassName))));

parallelInstruction.setFactory(new JacksonFactory());
parallelInstruction.setFactory(new GsonFactory());
return ParallelInstructionNode.create(parallelInstruction, Nodes.ExecutionLocation.UNKNOWN);
}

Expand All @@ -276,7 +276,7 @@ private static InstructionOutputNode createInstructionOutputNode(String name, Co
new InstructionOutput()
.setName(name)
.setCodec(CloudObjects.asCloudObject(coder, /*sdkComponents=*/ null));
instructionOutput.setFactory(new JacksonFactory());
instructionOutput.setFactory(new GsonFactory());
return InstructionOutputNode.create(instructionOutput, "fakeId");
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -18,15 +18,15 @@
package org.apache.beam.runners.dataflow.worker.testing;

import com.google.api.client.json.GenericJson;
import com.google.api.client.json.jackson2.JacksonFactory;
import com.google.api.client.json.gson.GsonFactory;
import java.io.IOException;
import org.json.JSONException;
import org.skyscreamer.jsonassert.JSONAssert;

/** Assertions on {@link GenericJson} class. */
public class GenericJsonAssert {

private static final JacksonFactory jacksonFactory = JacksonFactory.getDefaultInstance();
private static final GsonFactory gsonFactory = GsonFactory.getDefaultInstance();

/**
* Asserts that {@code actual} has the same JSON representation as {@code expected}.
Expand All @@ -39,8 +39,8 @@ public static void assertEqualsAsJson(Object expected, Object actual) {

try {
String expectedJsonText =
expected instanceof String ? (String) expected : jacksonFactory.toString(expected);
String actualJsonText = jacksonFactory.toString(actual);
expected instanceof String ? (String) expected : gsonFactory.toString(expected);
String actualJsonText = gsonFactory.toString(actual);
JSONAssert.assertEquals(expectedJsonText, actualJsonText, true);
} catch (JSONException ex) {
throw new IllegalArgumentException("Could not parse JSON", ex);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
package org.apache.beam.runners.dataflow.worker.testing;

import com.google.api.client.json.GenericJson;
import com.google.api.client.json.jackson2.JacksonFactory;
import com.google.api.client.json.gson.GsonFactory;
import java.io.IOException;
import org.hamcrest.Description;
import org.hamcrest.TypeSafeMatcher;
Expand All @@ -35,11 +35,11 @@ public final class GenericJsonMatcher extends TypeSafeMatcher<GenericJson> {

private String expectedJsonText;

private static final JacksonFactory jacksonFactory = JacksonFactory.getDefaultInstance();
private static final GsonFactory gsonFactory = GsonFactory.getDefaultInstance();

private GenericJsonMatcher(GenericJson expected) {
try {
expectedJsonText = jacksonFactory.toString(expected);
expectedJsonText = gsonFactory.toString(expected);
} catch (IOException ex) {
throw new IllegalArgumentException("Could not parse JSON", ex);
}
Expand All @@ -52,7 +52,7 @@ public static GenericJsonMatcher jsonOf(GenericJson genericJson) {
@Override
protected boolean matchesSafely(GenericJson actual) {
try {
String actualJsonText = jacksonFactory.toString(actual);
String actualJsonText = gsonFactory.toString(actual);
JSONCompareResult result =
JSONCompare.compareJSON(expectedJsonText, actualJsonText, JSONCompareMode.STRICT);
return result.passed();
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ dependencies {
implementation library.java.vendored_guava_32_1_2_jre
implementation project(path: ":sdks:java:core", configuration: "shadow")
implementation project(path: ":runners:core-java")
implementation library.java.google_http_client_jackson2
implementation library.java.google_http_client_gson
implementation library.java.google_auth_library_oauth2_http
implementation library.java.google_api_client
implementation library.java.bigdataoss_gcsio
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@
import com.google.api.client.http.HttpRequestInitializer;
import com.google.api.client.http.HttpTransport;
import com.google.api.client.json.JsonFactory;
import com.google.api.client.json.jackson2.JacksonFactory;
import com.google.api.client.json.gson.GsonFactory;
import com.google.api.services.storage.Storage;
import com.google.auth.Credentials;
import com.google.auth.http.HttpCredentialsAdapter;
Expand Down Expand Up @@ -53,7 +53,7 @@ private static class SingletonHelper {

static {
try {
JSON_FACTORY = JacksonFactory.getDefaultInstance();
JSON_FACTORY = GsonFactory.getDefaultInstance();
HTTP_TRANSPORT = GoogleNetHttpTransport.newTrustedTransport();
} catch (GeneralSecurityException | IOException e) {
throw new RuntimeException(e);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@
import com.google.api.client.json.GenericJson;
import com.google.api.client.json.Json;
import com.google.api.client.json.JsonFactory;
import com.google.api.client.json.jackson2.JacksonFactory;
import com.google.api.client.json.gson.GsonFactory;
import com.google.api.client.testing.http.HttpTesting;
import com.google.api.client.testing.http.MockHttpTransport;
import com.google.api.client.testing.http.MockLowLevelHttpRequest;
Expand Down Expand Up @@ -474,7 +474,7 @@ public void testRetryFileSizeNonBatch() throws IOException {

@Test
public void testGetSizeBytesWhenFileNotFoundBatch() throws Exception {
JsonFactory jsonFactory = new JacksonFactory();
JsonFactory jsonFactory = new GsonFactory();

String contentBoundary = "batch_foobarbaz";
String contentBoundaryLine = "--" + contentBoundary;
Expand Down Expand Up @@ -545,7 +545,7 @@ public void testGetSizeBytesWhenFileNotFoundNoBatch() throws Exception {

@Test
public void testGetSizeBytesWhenFileNotFoundBatchRetry() throws Exception {
JsonFactory jsonFactory = new JacksonFactory();
JsonFactory jsonFactory = new GsonFactory();

String contentBoundary = "batch_foobarbaz";
String contentBoundaryLine = "--" + contentBoundary;
Expand Down Expand Up @@ -657,7 +657,7 @@ public LowLevelHttpResponse execute() throws IOException {

@Test
public void testRemoveWhenFileNotFound() throws Exception {
JsonFactory jsonFactory = new JacksonFactory();
JsonFactory jsonFactory = new GsonFactory();

String contentBoundary = "batch_foobarbaz";
String contentBoundaryLine = "--" + contentBoundary;
Expand Down Expand Up @@ -1045,7 +1045,7 @@ private void verifyMetricWasSet(
/** Builds a fake GoogleJsonResponseException for testing API error handling. */
private static GoogleJsonResponseException googleJsonResponseException(
final int status, final String reason, final String message) throws IOException {
final JsonFactory jsonFactory = new JacksonFactory();
final JsonFactory jsonFactory = new GsonFactory();
HttpTransport transport =
new MockHttpTransport() {
@Override
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@
import com.google.api.client.http.LowLevelHttpRequest;
import com.google.api.client.http.LowLevelHttpResponse;
import com.google.api.client.json.JsonFactory;
import com.google.api.client.json.jackson2.JacksonFactory;
import com.google.api.client.json.gson.GsonFactory;
import com.google.api.services.storage.Storage;
import java.io.IOException;
import org.apache.beam.sdk.metrics.Histogram;
Expand All @@ -61,7 +61,7 @@ public class LatencyRecordingHttpRequestInitializerTest {
@Mock private LowLevelHttpResponse mockLowLevelResponse;
@Mock private Histogram mockHistogram;

private final JsonFactory jsonFactory = JacksonFactory.getDefaultInstance();
private final JsonFactory jsonFactory = GsonFactory.getDefaultInstance();
private Storage storage;

@Before
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@
import com.google.api.client.http.LowLevelHttpRequest;
import com.google.api.client.http.LowLevelHttpResponse;
import com.google.api.client.json.JsonFactory;
import com.google.api.client.json.jackson2.JacksonFactory;
import com.google.api.client.json.gson.GsonFactory;
import com.google.api.client.testing.http.MockHttpTransport;
import com.google.api.client.testing.http.MockLowLevelHttpRequest;
import com.google.api.client.testing.http.MockLowLevelHttpResponse;
Expand Down Expand Up @@ -79,7 +79,7 @@ public class RetryHttpRequestInitializerTest {
@Mock private LowLevelHttpResponse mockLowLevelResponse;
@Mock private HttpResponseInterceptor mockHttpResponseInterceptor;

private final JsonFactory jsonFactory = JacksonFactory.getDefaultInstance();
private final JsonFactory jsonFactory = GsonFactory.getDefaultInstance();
private Storage storage;

// Used to test retrying a request more than the default 10 times.
Expand Down
2 changes: 1 addition & 1 deletion sdks/java/io/google-cloud-platform/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,7 @@ dependencies {
implementation library.java.google_cloud_spanner
implementation library.java.google_code_gson
implementation library.java.google_http_client
implementation library.java.google_http_client_jackson2
implementation library.java.google_http_client_gson
// bigdataoss_util declares old google_oauth_client version that does not have
// IdTokenVerifier.verifyPayload method. Let's declare the newer one.
implementation library.java.google_oauth_client
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,6 @@
import com.google.api.client.http.javanet.NetHttpTransport;
import com.google.api.client.json.JsonFactory;
import com.google.api.client.json.gson.GsonFactory;
import com.google.api.client.json.jackson2.JacksonFactory;
import com.google.api.services.healthcare.v1.CloudHealthcare;
import com.google.api.services.healthcare.v1.CloudHealthcare.Projects.Locations.Datasets.FhirStores.Fhir.PatientEverything;
import com.google.api.services.healthcare.v1.CloudHealthcare.Projects.Locations.Datasets.Hl7V2Stores.Messages;
Expand Down Expand Up @@ -736,7 +735,7 @@ private void initClient() throws IOException {

client =
new CloudHealthcare.Builder(
new NetHttpTransport(), new JacksonFactory(), requestInitializer)
new NetHttpTransport(), new GsonFactory(), requestInitializer)
.setApplicationName("apache-beam-hl7v2-io")
.build();
httpClient =
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@
import com.google.api.client.http.LowLevelHttpResponse;
import com.google.api.client.json.GenericJson;
import com.google.api.client.json.Json;
import com.google.api.client.json.jackson2.JacksonFactory;
import com.google.api.client.json.gson.GsonFactory;
import com.google.api.client.testing.http.MockHttpTransport;
import com.google.api.client.testing.http.MockLowLevelHttpRequest;
import com.google.api.client.testing.util.MockSleeper;
Expand Down Expand Up @@ -1597,12 +1597,12 @@ public void testSkipInvalidRowsIgnoreUnknownIgnoreInsertIdsValuesStreaming()
/** A helper to convert a string response back to a {@link GenericJson} subclass. */
private static <T extends GenericJson> T fromString(String content, Class<T> clazz)
throws IOException {
return JacksonFactory.getDefaultInstance().fromString(content, clazz);
return GsonFactory.getDefaultInstance().fromString(content, clazz);
}

/** A helper to wrap a {@link GenericJson} object in a content stream. */
private static InputStream toStream(GenericJson content) throws IOException {
return new ByteArrayInputStream(JacksonFactory.getDefaultInstance().toByteArray(content));
return new ByteArrayInputStream(GsonFactory.getDefaultInstance().toByteArray(content));
}

/** A helper that generates the error JSON payload that Google APIs produce. */
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
import com.google.api.client.http.HttpHeaders;
import com.google.api.client.http.HttpRequestInitializer;
import com.google.api.client.http.javanet.NetHttpTransport;
import com.google.api.client.json.jackson2.JacksonFactory;
import com.google.api.client.json.gson.GsonFactory;
import com.google.api.services.healthcare.v1.model.HttpBody;
import com.google.api.services.storage.Storage;
import com.google.api.services.storage.model.StorageObject;
Expand Down Expand Up @@ -136,7 +136,7 @@ public static void tearDownTempBucket() throws IOException {
request.setReadTimeout(60000); // 1 minute read timeout
};
Storage storage =
new Storage.Builder(new NetHttpTransport(), new JacksonFactory(), requestInitializer)
new Storage.Builder(new NetHttpTransport(), new GsonFactory(), requestInitializer)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like this is now failing code formatting checks, could you please run: ./gradlew :sdks:java:io:google-cloud-platform:spotlessApply from the root of the project to fix?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I ran this on the cmd line but the build failed

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Screenshot from 2024-08-15 21-24-41

May be this is happening because of I am missing some project requirements

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have you tried going through https://github.com/apache/beam/blob/master/CONTRIBUTING.md#setup-your-environment-and-learn-about-language-specific-setup? In particular, the ./local-env-setup.sh piece might help

If you don't want to deal with the dependencies for this, you can also just use the suggestion from the build itself - https://github.com/apache/beam/actions/runs/10406736496/job/28820457976?pr=32158

In this case, the relevant piece is:

Execution failed for task ':sdks:java:io:google-cloud-platform:spotlessJavaCheck'.
> The following files had format violations:
      sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/healthcare/HttpHealthcareApiClient.java
          @@ -734,8 +734,7 @@
           ????????????????CloudHealthcareScopes.CLOUD_PLATFORM,?StorageScopes.CLOUD_PLATFORM_READ_ONLY));
           
           ????client?=
          -????????new?CloudHealthcare.Builder(
          -????????????????new?NetHttpTransport(),?new?GsonFactory(),?requestInitializer)
          +????????new?CloudHealthcare.Builder(new?NetHttpTransport(),?new?GsonFactory(),?requestInitializer)
           ????????????.setApplicationName("apache-beam-hl7v2-io")
           ????????????.build();
           ????httpClient?=
      sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/healthcare/FhirIOTestUtil.java
          @@ -136,8 +136,7 @@
           ??????????request.setReadTimeout(60000);?//?1?minute?read?timeout
           ????????};
           ????Storage?storage?=
          -????????new?Storage.Builder(new?NetHttpTransport(),?new?GsonFactory(),?requestInitializer)
          -????????????.build();
          +????????new?Storage.Builder(new?NetHttpTransport(),?new?GsonFactory(),?requestInitializer).build();
           ????List<StorageObject>?blobs?=?storage.objects().list(DEFAULT_TEMP_BUCKET).execute().getItems();
           ????if?(blobs?!=?null)?{
           ??????for?(StorageObject?blob?:?blobs)?{
  Run './gradlew :sdks:java:io:google-cloud-platform:spotlessApply' to fix these violations.

So to fix, you'd update the lines in FhirIOTestUtil.java and HttpHealthcareApiClient.java to follow the specified format

Copy link
Contributor Author

@cutiepie-10 cutiepie-10 Aug 15, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @damccorm,
I perfoormed the setup manually and checked it again but the only error I am getting is building the project.
The build fails while building compileGroovy.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, could you please just manually edit the lines then? If you share the error you're getting I may be able to help as well

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The error I am getting is mentioned in the comment above.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Screenshot from 2024-08-17 10-44-05

This is the error I am getting while running ./local-env-setup.sh and in the comments above I have shown the error I am getting while building the project.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like a bunch of permission errors - could you try running sudo ./local-env-setup.sh and running it as root? Or if you don't want to give elevated permissions to the whole script, you could modify it to just run certain commands as root.

.build();
List<StorageObject> blobs = storage.objects().list(DEFAULT_TEMP_BUCKET).execute().getItems();
if (blobs != null) {
Expand Down
Loading