Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DatastreamToSQL - log schema map details #1747

Merged
merged 23 commits into from
Aug 3, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
24012f2
adding logging to test schemmap
dhercher Jul 25, 2024
2d44982
open schemmap prtection
dhercher Jul 26, 2024
7a63f52
Handling for String Primary Keys in Bulk Reader. (#1743)
VardhanThigle Jul 26, 2024
b2510a6
Add support for ALO in SpannerChangeStreamsToBigQuery template (#1750)
Dippatel98 Jul 26, 2024
bb9cfff
Fixing exception in String .isSplittable (#1755)
VardhanThigle Jul 29, 2024
bf81829
Remove PIP_NO_DEPS from YamlTemplate Dockerfile (#1748)
Polber Jul 29, 2024
cec8533
Terraform template updates for custom transformation (#1746)
shreyakhajanchi Jul 30, 2024
89cee54
Add string mapper case for IT (#1757)
Deep1998 Jul 30, 2024
17a100e
Fixing Index Discovery for 5.7 and removing innodb_parallel_read_thre…
VardhanThigle Jul 30, 2024
eea7827
adding schema map to dml handler
dhercher Aug 1, 2024
311f85f
removing logs for testing
dhercher Aug 1, 2024
045b3a9
ut for reverse replication shadow tables (#1759)
aksharauke Jul 31, 2024
71dc00a
unit test for source writer (#1749)
aksharauke Aug 1, 2024
9205455
Adding autoReconnect parameters to URL (#1760)
VardhanThigle Aug 1, 2024
02d1464
map should not be static
dhercher Aug 1, 2024
17881c5
merging
dhercher Aug 1, 2024
1af61f8
adding logging to test schemmap
dhercher Jul 25, 2024
33c8c98
open schemmap prtection
dhercher Jul 26, 2024
46cc743
adding schema map to dml handler
dhercher Aug 1, 2024
7c6bde9
removing logs for testing
dhercher Aug 1, 2024
c3e611f
map should not be static
dhercher Aug 1, 2024
6519ab8
Merge branch 'main' into ds2sql-add-logging
dhercher Aug 1, 2024
07f387a
linebreaks
dhercher Aug 2, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -356,6 +356,7 @@
CdcJdbcIO.DataSourceConfiguration dataSourceConfiguration = getDataSourceConfiguration(options);
validateOptions(options, dataSourceConfiguration);
Map<String, String> schemaMap = parseSchemaMap(options.getSchemaMap());
LOG.info("Parsed schema map: {}", schemaMap);

Check warning on line 359 in v2/datastream-to-sql/src/main/java/com/google/cloud/teleport/v2/templates/DataStreamToSQL.java

View check run for this annotation

Codecov / codecov/patch

v2/datastream-to-sql/src/main/java/com/google/cloud/teleport/v2/templates/DataStreamToSQL.java#L359

Added line #L359 was not covered by tests

/*
* Stage 1: Ingest and Normalize Data to FailsafeElement with JSON Strings
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -63,10 +63,12 @@
String driverName = this.dataSourceConfiguration.getDriverClassName().get();
switch (driverName) {
case "org.postgresql.Driver":
datastreamToDML = DatastreamToPostgresDML.of(dataSourceConfiguration);
datastreamToDML =
DatastreamToPostgresDML.of(dataSourceConfiguration).withSchemaMap(this.schemaMap);

Check warning on line 67 in v2/datastream-to-sql/src/main/java/com/google/cloud/teleport/v2/transforms/CreateDml.java

View check run for this annotation

Codecov / codecov/patch

v2/datastream-to-sql/src/main/java/com/google/cloud/teleport/v2/transforms/CreateDml.java#L66-L67

Added lines #L66 - L67 were not covered by tests
break;
case "com.mysql.cj.jdbc.Driver":
datastreamToDML = DatastreamToMySQLDML.of(dataSourceConfiguration);
datastreamToDML =
DatastreamToMySQLDML.of(dataSourceConfiguration).withSchemaMap(this.schemaMap);

Check warning on line 71 in v2/datastream-to-sql/src/main/java/com/google/cloud/teleport/v2/transforms/CreateDml.java

View check run for this annotation

Codecov / codecov/patch

v2/datastream-to-sql/src/main/java/com/google/cloud/teleport/v2/transforms/CreateDml.java#L70-L71

Added lines #L70 - L71 were not covered by tests
break;
default:
throw new IllegalArgumentException(
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ public abstract class DatastreamToDML
private CdcJdbcIO.DataSourceConfiguration dataSourceConfiguration;
private DataSource dataSource;
public String quoteCharacter;
private static Map<String, String> schemaMap = new HashMap<String, String>();
protected Map<String, String> schemaMap = new HashMap<String, String>();

public abstract String getDefaultQuoteCharacter();

Expand Down
Loading