Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

#85 Override executed script #119

Merged
merged 5 commits into from
Feb 23, 2021
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -114,13 +114,14 @@ Then, you may add the dependencies to ```datamaintain-core``` and the driver mod

| Key | Description | Default value | Mandatory? | Values examples |
|---|---|---|---|---|
| default.script.action | The default script action | ```RUN``` | no | ```RUN``` or ```MARK_AS_EXECUTED``` |
| scan.path | Path to the folder containing all your scripts | ```./scripts/``` | yes | |
| scan.identifier.regex | Regex that will be used to determine an identifier for each file. It has to contain a capturing group. Identifiers are then used to sort the scripts before running them. | ```(.*)``` (with this regex, the script's whole name will be its identifier) | no | With the regex ```(.*?)_.*```, a script named ```1.23_my-script.js``` will have ```1.23``` as its identifier |
| scan.tags.createFromFolder | If true, scripts will have their parent folders names as tags. Relative path to ```scan.path``` is used. | ```false``` | no | ```false``` or ```true``` |
| tag.*your_tag* | Glob paths to your scripts that you want to apply the tag "your_tag" on. To declare multiple tags, you will have to add multiple properties in your settings. A tag ```my_tag``` will have as as property name ```tag.my_tag``` **WARNING:** ALWAYS declare your tags using absolute paths. Relative paths and even using a tilde (~) won't do the trick. | | no | ```[data/*, script1.js, old/old_script1.js]``` |
| filter.tags.whitelisted | Scripts that have these tags will be considered | None | no | ```DATA,tag``` |
| filter.tags.blacklisted | Scripts that have these tags will be ignored. A script having a whitelisted tag and a blacklisted tag will be ignored | None | no | ```DATA,tag``` |
| execution.mode | Execution mode. Possible values:<br />- ```NORMAL```: Regular execution: your scripts will be run on your database.<br />- ```DRY```: Scripts will not be executed. A full report of what would happen is you ran Datamaintain normally will be logged.<br />- ```FORCE_AS_EXECUTED```: Scripts will not be executed but their execution will be remembered by Datamaintain for later executions. | ```NORMAL``` | no | ```NORMAL```, ```DRY``` or ```FORCE_MARK_AS_EXECUTED``` |
| execution.mode | Execution mode. Possible values:<br />- ```NORMAL```: Regular execution: the action for each script will be done.<br />- ```DRY```: No action will be done on script. A full report of what would happen is you ran Datamaintain normally will be logged.<br />- ```FORCE_AS_EXECUTED```: **Deprecated (will be removed in 2.0, use action) !** Scripts will not be executed but their execution will be remembered by Datamaintain for later executions. | ```NORMAL``` | no | ```NORMAL```, ```DRY``` ~~or FORCE_MARK_AS_EXECUTED~~ |
| verbose | If true, more logs will be printed | ```false``` | no | ```true``` or ```false``` |
| prune.tags.to.run.again | Scripts that have these tags will be run, even they were already executed | None | no | ```tag,again``` |
### Mongo driver configuration
Expand Down
4 changes: 4 additions & 0 deletions modules/cli/src/main/kotlin/datamaintain/cli/App.kt
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ import com.github.ajalt.clikt.parameters.types.choice
import datamaintain.core.Datamaintain
import datamaintain.core.config.CoreConfigKey
import datamaintain.core.config.DatamaintainConfig
import datamaintain.core.script.ScriptAction
import datamaintain.core.step.check.allCheckRuleNames
import datamaintain.core.step.executor.ExecutionMode
import datamaintain.db.driver.mongo.MongoConfigKey
Expand Down Expand Up @@ -71,6 +72,8 @@ class UpdateDb(val runner: (DatamaintainConfig) -> Unit = ::defaultUpdateDbRunne

private val executionMode by option(help = "execution mode").choice(ExecutionMode.values().map { it.name }.map { it to it }.toMap())

private val action by option(help = "script action").choice(ScriptAction.values().map { it.name }.map { it to it }.toMap())

Lysoun marked this conversation as resolved.
Show resolved Hide resolved
private val verbose: Boolean? by option(help = "verbose").flag()

private val mongoSaveOutput: Boolean? by option(help = "save mongo output").flag()
Expand Down Expand Up @@ -121,6 +124,7 @@ class UpdateDb(val runner: (DatamaintainConfig) -> Unit = ::defaultUpdateDbRunne
mongoSaveOutput?.let { props.put(MongoConfigKey.DB_MONGO_SAVE_OUTPUT.key, it.toString()) }
mongoPrintOutput?.let { props.put(MongoConfigKey.DB_MONGO_PRINT_OUTPUT.key, it.toString()) }
executionMode?.let { props.put(CoreConfigKey.EXECUTION_MODE.key, it) }
action?.let { props.put(CoreConfigKey.DEFAULT_SCRIPT_ACTION.key, it) }
tagsMatchers?.forEach {
props.put("${CoreConfigKey.TAG.key}.${it.first}", it.second)
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ package datamaintain.core.config
import datamaintain.core.config.ConfigKey.Companion.overrideBySystemProperties
import datamaintain.core.config.CoreConfigKey.*
import datamaintain.core.db.driver.DatamaintainDriverConfig
import datamaintain.core.script.ScriptAction
import datamaintain.core.script.Tag
import datamaintain.core.script.TagMatcher
import datamaintain.core.step.executor.ExecutionMode
Expand All @@ -23,11 +24,13 @@ data class DatamaintainConfig @JvmOverloads constructor(val path: Path = Paths.g
val tagsMatchers: Set<TagMatcher> = setOf(),
val checkRules: Sequence<String> = emptySequence(),
val executionMode: ExecutionMode = defaultExecutionMode,
val defaultScriptAction: ScriptAction = defaultAction,
val driverConfig: DatamaintainDriverConfig,
val verbose: Boolean = VERBOSE.default!!.toBoolean()) {

companion object {
private val defaultExecutionMode = ExecutionMode.NORMAL
private val defaultAction = ScriptAction.RUN

@JvmStatic
fun buildConfig(configInputStream: InputStream, driverConfig: DatamaintainDriverConfig): DatamaintainConfig {
Expand All @@ -40,6 +43,17 @@ data class DatamaintainConfig @JvmOverloads constructor(val path: Path = Paths.g
@JvmOverloads
fun buildConfig(driverConfig: DatamaintainDriverConfig, props: Properties = Properties()): DatamaintainConfig {
overrideBySystemProperties(props, values().asList())

val executionMode = ExecutionMode.fromNullable(props.getNullableProperty(EXECUTION_MODE), defaultExecutionMode)

val scriptAction = if (ExecutionMode.FORCE_MARK_AS_EXECUTED == executionMode) {
// To be compliant with previous version (and avoir breaking changes)
driccio marked this conversation as resolved.
Show resolved Hide resolved
// we set script action from ExecutionMode.FORCE_MARK_AS_EXECUTED
ScriptAction.MARK_AS_EXECUTED
} else {
ScriptAction.fromNullable(props.getNullableProperty(DEFAULT_SCRIPT_ACTION), defaultAction)
}

return DatamaintainConfig(
Paths.get(props.getProperty(SCAN_PATH)),
Regex(props.getProperty(SCAN_IDENTIFIER_REGEX)),
Expand All @@ -51,7 +65,8 @@ data class DatamaintainConfig @JvmOverloads constructor(val path: Path = Paths.g
.map { TagMatcher.parse(it.first.replace("${TAG.key}.", ""), it.second) }
.toSet(),
extractCheckRules(props.getNullableProperty(CHECK_RULES)),
ExecutionMode.fromNullable(props.getNullableProperty(EXECUTION_MODE), defaultExecutionMode),
executionMode,
scriptAction,
driverConfig,
props.getProperty(VERBOSE).toBoolean()
)
Expand Down Expand Up @@ -107,6 +122,7 @@ enum class CoreConfigKey(override val key: String,
override val default: String? = null) : ConfigKey {
// GLOBAL
VERBOSE("verbose", "false"),
DEFAULT_SCRIPT_ACTION("default.script.action", "RUN"),

// SCAN
SCAN_PATH("scan.path", "./scripts/"),
Expand Down
Original file line number Diff line number Diff line change
@@ -1,9 +1,8 @@
package datamaintain.core.script

enum class ExecutionStatus {
OK, KO, FORCE_MARKED_AS_EXECUTED, SHOULD_BE_EXECUTED;

fun correctlyExecuted() = this == OK ||
this == FORCE_MARKED_AS_EXECUTED ||
this == SHOULD_BE_EXECUTED
OK,
KO,
@Deprecated("Will be removed in 2.0. Useless now we store action on script")
FORCE_MARKED_AS_EXECUTED;
}
Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
package datamaintain.core.script

import java.lang.IllegalStateException
import java.math.BigInteger
import java.nio.file.Path
import java.security.MessageDigest

class FileScript @JvmOverloads constructor(
val path: Path,
identifierRegex: Regex,
override val tags: Set<Tag> = setOf()
override val tags: Set<Tag> = setOf(),
override var action: ScriptAction = ScriptAction.RUN
driccio marked this conversation as resolved.
Show resolved Hide resolved
) : ScriptWithContent {

override val name: String
Expand Down
16 changes: 9 additions & 7 deletions modules/core/src/main/kotlin/datamaintain/core/script/Script.kt
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,6 @@ package datamaintain.core.script

import datamaintain.core.step.executor.Execution

import java.time.Duration
import java.time.LocalDateTime

interface Script {
val name: String
val checksum: String
Expand All @@ -16,24 +13,27 @@ data class ExecutedScript @JvmOverloads constructor(
override val checksum: String,
override val identifier: String,
val executionStatus: ExecutionStatus,
var action: ScriptAction,
val executionDurationInMillis: Long? = null,
val executionOutput: String? = null
) : Script {
companion object {
fun forceMarkAsExecuted(script: ScriptWithContent) =
fun simulateExecuted(script: ScriptWithContent, executionStatus: ExecutionStatus) =
ExecutedScript(
script.name,
script.checksum,
script.identifier,
ExecutionStatus.FORCE_MARKED_AS_EXECUTED
executionStatus,
script.action
)

fun shouldBeExecuted(script: ScriptWithContent) =
fun build(script: ScriptWithContent, execution: Execution) =
ExecutedScript(
script.name,
script.checksum,
script.identifier,
ExecutionStatus.SHOULD_BE_EXECUTED
execution.executionStatus,
script.action
driccio marked this conversation as resolved.
Show resolved Hide resolved
)

fun build(script: ScriptWithContent, execution: Execution, executionDurationInMillis: Long) =
Expand All @@ -42,6 +42,7 @@ data class ExecutedScript @JvmOverloads constructor(
script.checksum,
script.identifier,
execution.executionStatus,
script.action,
executionDurationInMillis,
execution.executionOutput
)
driccio marked this conversation as resolved.
Show resolved Hide resolved
Expand All @@ -51,6 +52,7 @@ data class ExecutedScript @JvmOverloads constructor(
interface ScriptWithContent : Script {
val content: String
val tags: Set<Tag>
var action: ScriptAction
}


Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
package datamaintain.core.script

enum class ScriptAction {

/**
* Indicates the script need to be executed and then marked as executed in DB.
*/
RUN,

/**
* Indicates the script only need to be marked as executed in DB. No execution.
*/
MARK_AS_EXECUTED;

companion object {
fun fromNullable(name: String?, defaultMode: ScriptAction): ScriptAction {
return if (name != null) {
valueOf(name)
} else {
defaultMode
}
}
}

}
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,12 @@ class Scanner(private val context: Context) {
logger.info { "Scan ${rootFolder.absolutePath}..." }
val scannedFiles = rootFolder.walk()
.filter { it.isFile }
.map { FileScript(it.toPath(), context.config.identifierRegex,
buildTags(context.config, rootFolder, it).toSet()) }
.map { FileScript(
it.toPath(),
context.config.identifierRegex,
buildTags(context.config, rootFolder, it).toSet(),
context.config.defaultScriptAction
) }
.sortedBy { it.name }
.onEach { context.reportBuilder.addScannedScript(it) }
.onEach {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ package datamaintain.core.step.executor
enum class ExecutionMode {
NORMAL,
DRY,
@Deprecated(message = "Will be removed in 2.0. Use ScriptAction.MARK_AS_EXECUTED to manage FORCE_MARK_AS_EXECUTED")
FORCE_MARK_AS_EXECUTED;

companion object {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,9 @@ import datamaintain.core.Context
import datamaintain.core.report.Report
import datamaintain.core.script.ExecutedScript
import datamaintain.core.script.ExecutionStatus
import datamaintain.core.script.ScriptAction
import datamaintain.core.script.ScriptWithContent
import mu.KotlinLogging
import java.time.Clock
import java.time.LocalDateTime
import kotlin.system.measureTimeMillis

private val logger = KotlinLogging.logger {}
Expand All @@ -16,43 +15,70 @@ class Executor(private val context: Context) {

fun execute(scripts: List<ScriptWithContent>): Report {
logger.info { "Executes scripts.." }

for (script in scripts) {
val executedScript = when (context.config.executionMode) {
ExecutionMode.NORMAL -> {
var execution = Execution(ExecutionStatus.SHOULD_BE_EXECUTED)
val executionDurationInMillis = measureTimeMillis {
execution = context.dbDriver.executeScript(script)
}
ExecutedScript.build(script, execution, executionDurationInMillis)
}
ExecutionMode.FORCE_MARK_AS_EXECUTED -> ExecutedScript.forceMarkAsExecuted(script)
ExecutionMode.DRY -> ExecutedScript.shouldBeExecuted(script)
ExecutionMode.NORMAL -> doAction(script)
else -> simulateAction(script)
driccio marked this conversation as resolved.
Show resolved Hide resolved
}

context.reportBuilder.addExecutedScript(executedScript)

when (executedScript.executionStatus) {
ExecutionStatus.OK -> {
markAsExecuted(executedScript)
logger.info { "${executedScript.name} executed" }
}
ExecutionStatus.FORCE_MARKED_AS_EXECUTED -> {
markAsExecuted(executedScript)
logger.info { "${executedScript.name} only marked (not really executed)" }
}
ExecutionStatus.KO -> {
context.reportBuilder.inError(executedScript)
logger.info { "${executedScript.name} has not been correctly executed" }
// TODO handle interactive shell
return context.reportBuilder.toReport()
}
else -> logger.info { "${executedScript.name} should be executed (dry run)" }
else -> {}
driccio marked this conversation as resolved.
Show resolved Hide resolved
}
}

logger.info { "" }
return context.reportBuilder.toReport()
}

private fun doAction(script: ScriptWithContent): ExecutedScript {
return when (script.action) {
ScriptAction.RUN -> {
var execution = Execution(ExecutionStatus.KO)

val executionDurationInMillis = measureTimeMillis {
execution = context.dbDriver.executeScript(script)
}

val executedScript = ExecutedScript.build(script, execution, executionDurationInMillis)

if (executedScript.executionStatus == ExecutionStatus.OK) {
markAsExecuted(executedScript)
logger.info { "${executedScript.name} executed" }
}

executedScript
}
ScriptAction.MARK_AS_EXECUTED -> {
val executedScript = ExecutedScript.build(script, Execution(ExecutionStatus.OK))

markAsExecuted(executedScript)
logger.info { "${executedScript.name} only marked as executed (so not executed)" }

executedScript
}
}
}

private fun simulateAction(script: ScriptWithContent): ExecutedScript {
when (script.action) {
ScriptAction.RUN ->
logger.info { "${script.name} would have been executed" }
ScriptAction.MARK_AS_EXECUTED ->
logger.info { "${script.name} would have been only marked as executed (so not executed)" }
}

return ExecutedScript.simulateExecuted(script, ExecutionStatus.OK)
}

private fun markAsExecuted(it: ExecutedScript) {
try {
context.dbDriver.markAsExecuted(it)
Expand Down
4 changes: 4 additions & 0 deletions modules/core/src/test/kotlin/datamaintain/FilterTest.kt
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ import datamaintain.core.config.DatamaintainConfig
import datamaintain.core.db.driver.DatamaintainDriver
import datamaintain.core.db.driver.FakeDriverConfig
import datamaintain.core.script.FileScript
import datamaintain.core.script.ScriptAction
import datamaintain.core.script.Tag
import datamaintain.core.step.Filter
import datamaintain.core.step.executor.ExecutionMode
Expand Down Expand Up @@ -35,6 +36,7 @@ internal class FilterTest {
emptySet(),
emptySequence(),
ExecutionMode.NORMAL,
ScriptAction.RUN,
FakeDriverConfig()),
dbDriver = dbDriver)

Expand Down Expand Up @@ -84,6 +86,7 @@ internal class FilterTest {
emptySet(),
emptySequence(),
ExecutionMode.NORMAL,
ScriptAction.RUN,
FakeDriverConfig()),
dbDriver = dbDriver)

Expand Down Expand Up @@ -139,6 +142,7 @@ internal class FilterTest {
emptySet(),
emptySequence(),
ExecutionMode.NORMAL,
ScriptAction.RUN,
FakeDriverConfig()),
dbDriver = dbDriver)

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,8 @@ data class InMemoryScript(
override val name: String,
override val content: String,
override val identifier: String,
override val tags: Set<Tag> = setOf()) : ScriptWithContent {
override val tags: Set<Tag> = setOf(),
override var action: ScriptAction = ScriptAction.RUN) : ScriptWithContent {

override val checksum: String by lazy {
content.hash()
Expand Down
Loading