GCS Sink additional scenarios. #2

bijay27bit · 2024-12-13T04:24:55Z

No description provided.

bijay27bit · 2024-12-13T04:26:42Z

@itsmekumari Please review internal PR for GCS Sink additional scenarios. Thanks

itsmekumari · 2024-12-13T10:53:28Z

src/e2e-test/features/gcs/sink/GCSSink.feature

@@ -265,3 +265,175 @@ Feature: GCS sink - Verification of GCS Sink plugin
    Then Open and capture logs
    Then Verify the pipeline status is "Succeeded"
    Then Verify data is transferred to target GCS bucket
+
+#Added new scenarios for GCS Sink - Bijay


Remove the commented line here.

itsmekumari · 2024-12-13T10:55:01Z

src/e2e-test/features/gcs/sink/GCSSink.feature

+
+#Added new scenarios for GCS Sink - Bijay
+  @BQ_SOURCE_TEST @GCS_SINK_TEST
+  Scenario:Validate successful records transfer from BigQuery to GCS with macro enabled at sink


Add the macro scenario in a separate feature file with name macro, refer other plugins feature file for naming convention.

itsmekumari · 2024-12-13T11:04:08Z

src/e2e-test/features/gcs/sink/GCSSink.feature

+    Then Verify the pipeline status is "Succeeded"
+    Then Verify data is transferred to target GCS bucket
+
+  @GCS_SINK_TEST @BQ_SOURCE_TEST @GCS_Sink_Required


This scenario should be from GCS source to GCS sink right? Re-check and change accordingly. And why are we making it a macro scenarios, it is already covered in macro enabled scenario anyways.

@itsmekumari
This was the scenario: Verify and validate an end to end test case from BigQuery source to GCS sink using the Advanced File System Properties field
Please suggest.

itsmekumari · 2024-12-13T11:07:15Z

src/e2e-test/features/gcs/sink/GCSSink.feature

+  @GCS_SINK_TEST @BQ_SOURCE_TEST @GCS_Sink_Required
+  Scenario Outline: To verify successful data transfer from BigQuery to GCS for different formats with write header true
+    Given Open Datafusion Project to configure pipeline
+    When Source is BigQuery


Use the latest existing steps from framework.

itsmekumari · 2024-12-13T11:07:48Z

src/e2e-test/features/gcs/sink/GCSSink.feature

+    Then Wait till pipeline is in running state
+    Then Open and capture logs
+    Then Verify the pipeline status is "Succeeded"
+    Then Verify data is transferred to target GCS bucket


Add the validation steps in all the scenarios.

itsmekumari · 2024-12-13T11:09:33Z

src/e2e-test/features/gcs/sink/GCSSink.feature

+    Then Close the GCS properties
+    Then Save the pipeline
+    Then Preview and run the pipeline
+    Then Enter runtime argument value "gcsFileSysProperty" for key "FileSystemPr"


I don't see any value added in parameter file for file system property

File: PluginParameters.properties
Line # 112
gcsFileSysProperty={"textinputformat.record.delimiter": "@"}

itsmekumari · 2024-12-13T11:12:15Z

src/e2e-test/features/gcs/sink/GCSSink.feature

+      | tsv        | text/plain    |
+
+  @BQ_SOURCE_TEST @GCS_SINK_TEST
+  Scenario: To verify data is getting transferred successfully from BigQuery to GCS using advanced file system properties field


Why are we adding macro here again, It is already covered in macro enabled scenario. It should be for without macro enabled

itsmekumari · 2024-12-13T11:13:03Z

src/e2e-test/features/gcs/sink/GCSSink.feature

+  @BQ_SOURCE_TEST @GCS_SINK_TEST
+  Scenario: To verify data is getting transferred successfully from BigQuery to GCS using advanced file system properties field
+    Given Open Datafusion Project to configure pipeline
+    When Source is BigQuery


Use the latest existing steps. This is a common review comment across all scenarios.

itsmekumari · 2024-12-13T11:15:17Z

src/e2e-test/features/gcs/sink/GCSSinkError.feature

@@ -65,3 +65,37 @@ Feature: GCS sink - Verify GCS Sink plugin error scenarios
    Then Select GCS property format "csv"
    Then Click on the Validate button
    Then Verify that the Plugin Property: "format" is displaying an in-line error message: "errorMessageInvalidFormat"
+
+  @GCS_SINK_TEST @BQ_SOURCE_TEST


Change the tag order, for ease of understanding.

itsmekumari · 2024-12-13T12:33:19Z

src/e2e-test/features/gcs/sink/GCSSink.feature

+    Then Verify data is transferred to target GCS bucket
+
+  @GCS_SINK_TEST @BQ_SOURCE_TEST
+  Scenario Outline: To verify data is getting transferred successfully from BigQuery to GCS with contenttype selection


Add the validation for file format as well

itsmekumari

Please check the review comments.

AnkitCLI · 2024-12-13T09:43:57Z

src/e2e-test/features/gcs/sink/GCSSinkError.feature

+    Then Run the Pipeline in Runtime
+    Then Enter runtime argument value "gcsInvalidBucketNameSink" for key "gcsSinkPath"
+    Then Run the Pipeline in Runtime with runtime arguments
+    Then Wait till pipeline is in running state


add Open and capture and close logs steps

AnkitCLI · 2024-12-13T09:44:42Z

src/e2e-test/resources/errorMessage.properties

@@ -33,4 +33,5 @@ errorMessageMultipleFileWithoutClearDefaultSchema=Found a row with 4 fields when
 errorMessageInvalidSourcePath=Invalid bucket name in path 'abc@'. Bucket name should
 errorMessageInvalidDestPath=Invalid bucket name in path 'abc@'. Bucket name should
 errorMessageInvalidEncryptionKey=CryptoKeyName.parse: formattedString not in valid format: Parameter "abc@" must be
+errorMessageInvalidBucketNameSink=Spark program 'phase-1' failed with error: Errors were encountered during validation. Error code: 400, Unable to read or access GCS bucket. Bucket names must be at least 3 characters in length, got 2: 'gg'. Please check the system logs for more details.


No need to add too much message. Just add relevant error message

AnkitCLI · 2024-12-13T09:45:44Z

src/e2e-test/resources/errorMessage.properties

@@ -33,4 +33,5 @@ errorMessageMultipleFileWithoutClearDefaultSchema=Found a row with 4 fields when
 errorMessageInvalidSourcePath=Invalid bucket name in path 'abc@'. Bucket name should
 errorMessageInvalidDestPath=Invalid bucket name in path 'abc@'. Bucket name should
 errorMessageInvalidEncryptionKey=CryptoKeyName.parse: formattedString not in valid format: Parameter "abc@" must be
+errorMessageInvalidBucketNameSink=Spark program 'phase-1' failed with error: Errors were encountered during validation. Error code: 400, Unable to read or access GCS bucket. Bucket names must be at least 3 characters in length, got 2: 'gg'. Please check the system logs for more details.


In this error message it is showing bucket name should be atleast 3 characters. Change the bucket name to more than 3 characters.

AnkitCLI · 2024-12-13T09:46:07Z

src/e2e-test/resources/pluginParameters.properties

@@ -159,6 +159,11 @@ gcsParquetFileSchema=[{"key":"workforce","value":"string"},{"key":"report_year",
  {"key":"race_black","value":"long"},{"key":"race_hispanic_latinx","value":"long"},\
  {"key":"race_native_american","value":"long"},{"key":"race_white","value":"long"},\
  {"key":"tablename","value":"string"}]
+gcsInvalidBucketNameSink=gg


same comment here to change it to more 3 characters.

bijay27bit · 2025-01-07T09:23:57Z

@itsmekumari All review comments have been addressed. Please help to merge with master branch. Thanks

…EgcsNewChangesSink_BT # Conflicts: # src/e2e-test/features/gcs/sink/GCSSink.feature

itsmekumari reviewed Dec 13, 2024

View reviewed changes

itsmekumari requested changes Dec 13, 2024

View reviewed changes

AnkitCLI reviewed Dec 16, 2024

View reviewed changes

bijay27bit force-pushed the E2EgcsNewChangesSink_BT branch 6 times, most recently from caf0a54 to 9c57d24 Compare January 14, 2025 07:52

bijay27bit force-pushed the E2EgcsNewChangesSink_BT branch from c4e93b9 to e8d2829 Compare January 17, 2025 06:37

GCS Scenarios for the review.

7d5f5fa

bijay27bit force-pushed the E2EgcsNewChangesSink_BT branch from 0bbe285 to 7d5f5fa Compare January 17, 2025 06:54

zx-zx added 3 commits January 17, 2025 07:03

GCS Scenarios for the review.

e8c1fa7

Updated the scenario name after mergea.

1a4f1ad

Merge remote-tracking branch 'origin/E2EgcsNewChangesSink_BT' into E2…

35808d7

…EgcsNewChangesSink_BT # Conflicts: # src/e2e-test/features/gcs/sink/GCSSink.feature

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GCS Sink additional scenarios. #2

GCS Sink additional scenarios. #2

bijay27bit commented Dec 13, 2024

bijay27bit commented Dec 13, 2024

itsmekumari Dec 13, 2024

itsmekumari Dec 13, 2024

itsmekumari Dec 13, 2024 •

edited

Loading

bijay27bit Dec 13, 2024

itsmekumari Dec 13, 2024

itsmekumari Dec 13, 2024

itsmekumari Dec 13, 2024

bijay27bit Dec 13, 2024

itsmekumari Dec 13, 2024

itsmekumari Dec 13, 2024

itsmekumari Dec 13, 2024

itsmekumari Dec 13, 2024

itsmekumari left a comment

AnkitCLI Dec 13, 2024

AnkitCLI Dec 13, 2024

AnkitCLI Dec 13, 2024

AnkitCLI Dec 13, 2024

bijay27bit commented Jan 7, 2025

GCS Sink additional scenarios. #2

Are you sure you want to change the base?

GCS Sink additional scenarios. #2

Conversation

bijay27bit commented Dec 13, 2024

bijay27bit commented Dec 13, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

itsmekumari Dec 13, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

itsmekumari left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bijay27bit commented Jan 7, 2025

itsmekumari Dec 13, 2024 •

edited

Loading