Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: Just read docs and adjust small type #254

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docker/playground/filldb/filldb.sh
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.

cd $TMP_DIR/postgresDBSamples/adventureworks
cd $TMP_DIR/postgresDBSamples/adventureworks || exit

if ! psql -lqt -p 5432 -h playground-db -U postgres | cut -d \| -f 1 | grep -qw $ORIGINAL_DB_NAME; then
psql -p 5432 -h playground-db -U postgres -c "CREATE DATABASE $ORIGINAL_DB_NAME;"
Expand Down
7 changes: 6 additions & 1 deletion docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,12 @@

## Introduction

It is evident that the most appropriate approach for executing logical backup dumping and restoration is by leveraging the core PostgreSQL utilities, specifically `pg_dump` and `pg_restore`. Greenmask has been purposefully designed to align with PostgreSQL's native utilities, ensuring compatibility. Greenmask primarily handles data dumping operations independently and delegates the responsibilities of schema dumping and restoration to `pg_dump` and `pg_restore` respectively, maintaining seamless integration with PostgreSQL's standard tools.
It is evident that the most appropriate approach for executing logical backup dumping
and restoration is by leveraging the core PostgreSQL utilities, specifically `pg_dump` and `pg_restore`.
Greenmask has been purposefully designed to align with PostgreSQL's native utilities, ensuring compatibility.
Greenmask primarily handles data dumping operations independently and delegates
the responsibilities of schema dumping and restoration to `pg_dump` and `pg_restore` respectively,
maintaining seamless integration with PostgreSQL's standard tools.

## Backup process

Expand Down
2 changes: 1 addition & 1 deletion docs/built_in_transformers/dynamic_parameters.md
Original file line number Diff line number Diff line change
Expand Up @@ -117,7 +117,7 @@ constraints, you can use dynamic parameters in the `RandomDate` transformer:
template: '{{ .GetValue | tsModify "18 years" | .EncodeValue }}' # (7)
```

1. Firstly we generate the `RadnomDate` for birthdate column. The result of the transformation will used as the minimum
1. Firstly we generate the `RadnomDate` for birthdate column. The result of the transformation will use as the minimum
value for the next transformation for `hiredate` column.
2. Apply the template for static parameter. It calculates the now date and subtracts `30` years from it. The result
is `1994`. The function tsModify return not a raw data, but time.Time object. For getting the raw value suitable for
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ The `RandomAmountWithCurrency` transformer is specifically designed to populate

## Description

This transformer automatically generates random financial amounts along with corresponding global currency codes (e. g., `250.00 USD`, `300.00 EUR`), injecting them into the designated database column. It provides a straightforward solution for populating financial records with varied and realistic data, suitable for testing payment systems, data anonymization, and simulation of economic models.
This transformer automatically generates random financial amounts along with corresponding global currency codes (e.g., `250.00 USD`, `300.00 EUR`), injecting them into the designated database column. It provides a straightforward solution for populating financial records with varied and realistic data, suitable for testing payment systems, data anonymization, and simulation of economic models.

## Example: Populate the `payments` table with random amounts and currencies

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ The `RandomDayOfWeek` transformer is specifically designed to fill specified dat

## Description

Utilizing the `faker` library, the `RandomDayOfWeek` transformer generates names of days (e. g., Monday, Tuesday) at random. This transformer can be applied to any text or varchar column in a database, introducing variability and realism into data sets that need to represent days of the week in a non-specific manner.
Utilizing the `faker` library, the `RandomDayOfWeek` transformer generates names of days (e.g., Monday, Tuesday) at random. This transformer can be applied to any text or varchar column in a database, introducing variability and realism into data sets that need to represent days of the week in a non-specific manner.

## Example: Populate random days of the week for the `work_schedule` table

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ The `RandomTimezone` transformer is designed to populate specified database colu

## Description

Utilizing a comprehensive library or algorithm for generating timezone data, the `RandomTimezone` transformer provides random timezone strings (e. g., "America/New_York", "Europe/London") for database columns. This feature enables the creation of diverse and realistic datasets by simulating timezone information for user profiles, event timings, or any other data requiring timezone context.
Utilizing a comprehensive library or algorithm for generating timezone data, the `RandomTimezone` transformer provides random timezone strings (e.g., "America/New_York", "Europe/London") for database columns. This feature enables the creation of diverse and realistic datasets by simulating timezone information for user profiles, event timings, or any other data requiring timezone context.

## Example: Populate random timezone strings for the `user_accounts` table

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ The `RandomURL` transformer is designed to populate specified database columns w

## Description

Utilizing advanced algorithms or libraries for generating URL strings, the `RandomURL` transformer injects random, plausible URLs into the designated database column. Each generated URL is structured to include the protocol (e. g., "http://", "https://"), domain name, and path, offering a realistic range of web addresses for various applications.
Utilizing advanced algorithms or libraries for generating URL strings, the `RandomURL` transformer injects random, plausible URLs into the designated database column. Each generated URL is structured to include the protocol (e.g., "http://", "https://"), domain name, and path, offering a realistic range of web addresses for various applications.

## Example: Populate random URLs for the `webpages` table

Expand Down
2 changes: 1 addition & 1 deletion docs/commands/validate.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ All of those cases may be used for CI/CD pipelines to stop the process when some
useful when `--schema` flag is used - this allows to avoid data leakage when schema changed.

You can use the `--table` flag multiple times to specify the tables you want to check. Tables can be written with
or without schema names (e. g., `public.table_name` or `table_name`). If you specify multiple tables from different
or without schema names (e.g., `public.table_name` or `table_name`). If you specify multiple tables from different
schemas, an error will be thrown.

To start validation, use the following command:
Expand Down
2 changes: 1 addition & 1 deletion docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -223,7 +223,7 @@ validate:
```
{ .annotate }

1. A list of tables to validate. If this list is not empty, the validation operation will only be performed for the specified tables. Tables can be written with or without the schema name (e. g., `"public.cart"` or `"orders"`).
1. A list of tables to validate. If this list is not empty, the validation operation will only be performed for the specified tables. Tables can be written with or without the schema name (e.g., `"public.cart"` or `"orders"`).
2. Specifies whether to perform data transformation for a limited set of rows. If set to `true`, data transformation will be performed, and the number of rows transformed will be limited to the value specified in the `rows_limit` parameter (default is `10`).
3. Specifies whether to perform diff operations for the transformed data. If set to `true`, the validation process will **find the differences between the original and transformed data**. See more details in the [validate command documentation](commands/validate.md).
4. Limits the number of rows to be transformed during validation. The default limit is `10` rows, but you can change it by modifying this parameter.
Expand Down
2 changes: 1 addition & 1 deletion docs/database_subset.md
Original file line number Diff line number Diff line change
Expand Up @@ -117,7 +117,7 @@ dump:

Greenmask supports polymorphic references. You can define a virtual reference for a table with polymorphic references
using `polymorphic_exprs` attribute. The `polymorphic_exprs` attribute is a list of expressions that are used to make
a polymorphic reference. For instance we might have a table `comments` that has polymorphic reference to `posts` and
a polymorphic reference. For instance, we might have a table `comments` that has polymorphic reference to `posts` and
`videos`. The table comments might have `commentable_id` and `commentable_type` columns. The `commentable_type` column
contains the type of the table that is referenced by the `commentable_id` column. The example of the config:

Expand Down
2 changes: 1 addition & 1 deletion docs/installation.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ docker run -it greenmask/greenmask:latest

2. GitHub container registry

To run the greenmask container from Github registry, use the following command:
To run the greenmask container from GitHub registry, use the following command:
```shell
docker run -it ghcr.io/greenmaskio/greenmask:latest
```
Expand Down
2 changes: 1 addition & 1 deletion docs/release_notes/greenmask_0_1_5.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ Added a new Greenmask CLI command—[show-transformer](../commands/show-transfor

## Improvements

- The [Hash transformer](../built_in_transformers/standard_transformers/hash.md) has been completely remastered and now has the `function` parameter to choose from several hash algorithm options and the `max_length` parameter to truncate the hash tail.
- The [Hash transformer](../built_in_transformers/standard_transformers/hash.md) has been completely remastered and now has the `function` parameter to choose from some hash algorithm options and the `max_length` parameter to truncate the hash tail.
- Split information about transformers between the `list-transformers` and new `show-transformer` CLI commands, which allows for more comprehensible and useful outputs for both commands
- Added error severity for the `Cmd` parameter validator
- Improved UX for the Greenmask release binaries
Expand Down
2 changes: 1 addition & 1 deletion internal/db/postgres/dumpers/transformation_window.go
Original file line number Diff line number Diff line change
Expand Up @@ -111,7 +111,7 @@ func (tw *transformationWindow) close() {
close(tw.done)
}

// Transform - runs the transformation for the record in the window. This function checks the when
// Transform - runs the transformation for the record in the window. This function checks when
// condition of the transformer and if true sends a signal to the transformer goroutine to run the transformation
func (tw *transformationWindow) Transform(ctx context.Context, r *toolkit.Record) (*toolkit.Record, error) {
tw.r = r
Expand Down
4 changes: 2 additions & 2 deletions internal/db/postgres/transformers/custom/custom_cmd.go
Original file line number Diff line number Diff line change
Expand Up @@ -307,8 +307,8 @@ func (ct *CmdTransformer) stderrForwarder(ctx context.Context) error {
lineNum := 0
// This is required for convenient verbosity of output.
// Write "stderr forwarding" log message each 500ms otherwise just print received stderr data
// If it does not use this logic each line would be covered with "stderr forwarding" message and it will be
// complicated to recognize the traceback or multiline message
// If it does not use this logic each line would be covered with "stderr forwarding" message,
// and it will be complicated to recognize the traceback or multiline message
for {
line, _, err := ct.StderrReader.ReadLine()
if err != nil {
Expand Down
10 changes: 5 additions & 5 deletions pkg/toolkit/expr.go
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
// An expression handler for the toolkit package. It is used to evaluate the when condition of the record.
// An expression handler for the toolkit package. It is used to evaluate when condition of the record.
// Might be used in transformation conditions and other places where the record is used.

package toolkit
Expand Down Expand Up @@ -26,9 +26,9 @@ type WhenCond struct {
env map[string]any
}

// NewWhenCond - creates a new WhenCond object. It compiles the when condition and returns the compiled program
// NewWhenCond - creates a new WhenCond object. It compiles when condition and returns the compiled program
// and the record context with the functions for the columns. The functions represent the column names and return the
// column values. If the when condition is empty, the WhenCond object will always return true.
// column values. If when condition is empty, the WhenCond object will always return true.
func NewWhenCond(when string, driver *Driver, meta map[string]any) (*WhenCond, ValidationWarnings) {
var (
rc *RecordContext
Expand All @@ -51,7 +51,7 @@ func NewWhenCond(when string, driver *Driver, meta map[string]any) (*WhenCond, V
}, nil
}

// Evaluate - evaluates the when condition. If the when condition is empty, it will always return true.
// Evaluate - evaluates when condition. If when condition is empty, it will always return true.
func (wc *WhenCond) Evaluate(r *Record) (bool, error) {
if wc.whenCond == nil {
return true, nil
Expand All @@ -71,7 +71,7 @@ func (wc *WhenCond) Evaluate(r *Record) (bool, error) {
return false, fmt.Errorf("when condition should return boolean, got (%T) and value %+v", cond, cond)
}

// compileCond compiles the when condition and returns the compiled program and the record context
// compileCond compiles when condition and returns the compiled program and the record context
// with the functions for the columns. The functions represent the column names and return the column values.
// meta - additional meta information for debugging the compilation process
func compileCond(whenCond string, driver *Driver, meta map[string]any) (
Expand Down
7 changes: 4 additions & 3 deletions pkg/toolkit/template_functions.go
Original file line number Diff line number Diff line change
Expand Up @@ -459,7 +459,8 @@ func randomDate(randGen *rand.Rand, min, max time.Time) (time.Time, error) {
return *(RandomDate(randGen, &min, &max)), nil
}

// randomFloat - generate float randomly in the interval [min, max] with precision. By default precision is 4 digits
// randomFloat - generate float randomly in the interval [min, max] with precision.
// By default, precision is 4 digits
func randomFloat(randGen *rand.Rand, min, max any, decimal ...any) (float64, error) {
var err error
var p = 4
Expand Down Expand Up @@ -503,8 +504,8 @@ func randomInt(randGen *rand.Rand, min, max any) (int64, error) {
return RandomInt(randGen, minInt, maxInt), nil
}

// randomString - generate random string in the provided min and max length using provided symbols. By default symbols
// are "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890"
// randomString - generate random string in the provided min and max length using provided symbols.
// By default, symbols are "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890"
func randomString(randGen *rand.Rand, minLength, maxLength any, symbols ...string) (string, error) {
s := []rune("abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890")
if len(symbols) > 0 {
Expand Down