diff --git a/dev/.documenter-siteinfo.json b/dev/.documenter-siteinfo.json index 26b138b..b82d09a 100644 --- a/dev/.documenter-siteinfo.json +++ b/dev/.documenter-siteinfo.json @@ -1 +1 @@ -{"documenter":{"julia_version":"1.10.5","generation_timestamp":"2024-09-03T13:53:29","documenter_version":"1.6.0"}} \ No newline at end of file +{"documenter":{"julia_version":"1.10.5","generation_timestamp":"2024-09-03T22:09:39","documenter_version":"1.6.0"}} \ No newline at end of file diff --git a/dev/api.MD b/dev/api.MD index f8125d4..838c2f4 100644 --- a/dev/api.MD +++ b/dev/api.MD @@ -26,7 +26,7 @@ Order = [:type] ``` -## Functions -- additional to standard `AbstractArray` functionality +## Functions #### Index ```@index Order = [:function, :macro] diff --git a/dev/api/index.html b/dev/api/index.html index a1c255a..f02c292 100644 --- a/dev/api/index.html +++ b/dev/api/index.html @@ -1,12 +1,12 @@ -API reference · PerfTest.jl

API reference

This is the official API reference of PerfTest. Note that it can also be queried interactively from the Julia REPL using the help mode:

julia> using PerfTest
+API reference · PerfTest.jl
API reference
This is the official API reference of PerfTest. Note that it can also be queried interactively from the Julia REPL using the help mode:
julia> using PerfTest
 julia>?
-help?> PerfTest
Types
Index
Documentation
PerfTest.ASTRule — Type
Used by the AST walker to check for expressions that match condition, if they do then modifier will be applied to the expression.
This is the basic building block of the code transformer, a set of these rules compounds to all the needed manipulations to create the testing suite.
source
PerfTest.ASTWalkDepthRecord — Type
This structure is used to record a test set hierarchy during a AST walk. In any specific point of the walk the array will TODO
source
PerfTest.Context — Type
In order to perform with the test suite generation, the AST walk needs to keep a context register to integrate features that rely on the scope hierarchy.
source
PerfTest.CustomMetric — Type
Saves flags needed during the execution of the AST walk. It holds if:
  • The walk is on an expression that is a test target
  • The walk is on an expression that is inside a config macro
  • Several flags that affect the roofline methodology
source
PerfTest.DepthRecord — Type
This structure is used to record a test set frame during a AST walk. See ASTWalkDepthRecord for more info.
source
PerfTest.EnvironmentFlags — Type
Saves flags needed during the execution of the AST walk. It holds if:
  • The walk is on an expression that is a test target
  • The walk is on an expression that is inside a config macro
  • Several flags that affect the roofline methodology
source
PerfTest.Methodology_Result — Type
This struct is used in the test suite to save a methodology result, which in turn is constituted of a group of metric results and their references. Additionally, custom elements that are not subject to test are also saved, e.g. informational metrics, printing functions.
source
PerfTest.Metric_Constraint — Type
This struct is used in the test suite to save a metric test result and its associated data, it saves the reference used and the toreance intervals in absolute and percentual values, also it shows if the test succeded and some additional variables for data printing
source
PerfTest.Metric_Reference — Type
This struct is used in the test suite to save a metric reference, a reference is meant to be later compared with a result, its combination gives the Metric_Constraint struct. It holds:
  • A reference value.
  • low_is_bad registers if in this metric lower values are less desired than higher ones, or the opposite (e.g. time vs  FLOP/s).
source
PerfTest.Metric_Result — Type
This struct is used in the test suite to save a metric measurement, therefore its saves the metric name, its units space and its value.
source
PerfTest.Perftest_Result — Type
This struct saves a complete test suite result for one execution. It also saves the raw measurements obtained from the targets.
source
PerfTest.Struct_Eff_Mem_Throughput — Type
This struct holds the configuration of the basic effective memory throughput methodology.
  • enabled is used to enable or disable the methodology
  • tolerance defines the interval of ratios (eff.mem.through. / max. bandwidth) that make the test succeed.
source
PerfTest.Struct_Metric_Config — Type
This struct can hold the configuration of any metric.
  • enabled is used to enable or disable the methodology
  • regression_threshold, when comparing the measure with a reference, defines how far can the measurement be from the reference
source
PerfTest.Struct_Regression — Type
This struct holds the configuration of the basic metric regression methodology.
enabled is used to enable or disable the methodology save_failed will record historical measurements of failed tests if true general_regression_threshold sets the torelance interval for the test comparison
regression_calculation can be:
  • :latest    The reference will be the latest saved result
  • :average The reference will be the average of all saved results
source
PerfTest.Struct_Roofline_Config — Type
This struct holds the configuration of the basic roofline methodology.
  • enabled is used to enable or disable the methodology
  • tolerance defines the interval of ratios (eff.mem.through. / max. bandwidth) that make the test succeed.
source
Functions – additional to standard AbstractArray functionality
Index
Documentation
PerfTest._treeRun — Method
This method gets a input julia expression, and a context register and executes a transformation of the input that converts a recipe script (input) into a fully-fledged testing suite (return value).
Arguments
  • input_expr the recipe/source expression. (internally, a.k.a source code space)
  • context a register that will store information useful for the transformation over its run over the AST of the input
source
PerfTest.autoflopExpressionParser — Method
This is one of the parser functions that expand any formula block for metric definition. This function will parse the :autoflop symbol and substitute it with the flop count of the test target
source
PerfTest.by_index — Method
This method expects a hierarchy tree (dict) in the form of nested dictionaries and a vector of dictionary keys idx. The function will recursively apply the keys to get to a final element.
It is usually put to work with the DepthRecord struct.
source
PerfTest.checkAuxiliaryCustomMetrics — Method
This method is used to generate the code that computes the value of every auxiliary custom metric enabled in th current context, where the code is generated.
source
PerfTest.checkAuxiliaryMetric — Method
This method is used to generate the code that computes the value of a given auxiliary custom metric in the context the code is generated.
source
PerfTest.checkCustomMetric — Method
This function is used to generate the code that evaluates if a custom metric result f a target is within a specified reference.
WARNING
Predefined symbols needed before this code is added to the generated space:
  • reference_value
  • metric_results
source
PerfTest.checkCustomMetrics — Method
This function is used to generate the code that evaluates if a custom metric result f a target is within a specified reference.
WARNING
Predefined symbols needed before this code is added to the generated space:
  • reference_value
  • metric_results
source
PerfTest.checkMedianTime — Method
This function is used to generate the code that evaluates if the median time of execution of a target is within a specified reference.
WARNING
Predefined symbols needed before this code is added to the generated space:
  • reference_value
  • metric_results
source
PerfTest.checkMinTime — Method
This function is used to generate the code that evaluates if the minimum time of execution of a target is within a specified reference.
WARNING
Predefined symbols needed before this code is added to the generated space:
  • reference_value
  • metric_results
source
PerfTest.customMetricExpressionParser — Method
This is one of the parser functions that expand any formula block for metric definition. This function will parse all primitive metric symbols with the structure where the corresponding value of the metric is.
source
PerfTest.customMetricReferenceExpressionParser — Method
This is one of the parser functions that expand any formula block for metric definition. This function will parse all primitive metric symbols with the structure where the corresponding reference value for the metric is.
source
PerfTest.extractMethodologyResultArray — Method
This method will return a flattened array of all of the results for all the methodologies exercised in the provided dictionary.
Example:
"Test Set 1"      -> "Test 1"          -> Methodology A result          -> Methodology B result  "Test Set 2"      -> "Test 1"          -> Methodology A result Returns:  M. A result (Test 1)  M. B result (Test 1)  M. A result (Test 2)
source
PerfTest.extractNamesResultArray — Method
This method will return a flattened array of the whole test result hierarchy.
Example
Example:
"Test Set 1"      -> "Test 1"          -> Methodology A result          -> Methodology B result  "Test Set 2"      -> "Test 1"          -> Methodology A result Returns:  "Test Set 1 -> Test 1 -> Methodology A"  "Test Set 1 -> Test 1 -> Methodology B"  "Test Set 2 -> Test 1 -> Methodology A"
source
PerfTest.flattenedInterpolation — Method
This method interpolates the inside_expr into outside_expr anywhere it finds the token substitution_token, which is a symbol. The outside_expr has to be a block or a quote block. It has the particularity that it will remove block heads from the inside_expr and add the nested elements onto the location where the token it.
Example:
outside_expr = :(:A; 4)
inside_expr = :(begin 2;3 end)
substitution_token = :A
returns = :(2;3;4)
source
PerfTest.fullParsingSuite — Method
This function combines a collection of rules to turn a formula block into a functioning expression to calculate any metric defined by said formula
source
PerfTest.genTestName! — Method
Function that generates a test name if needed, it is used to name   test targets to distinguish them if several go in the same testset.
source
PerfTest.getMetricValue — Method
Given a series of methodology results, the the raw values of all the metrics contained in the methodology results.
source
PerfTest.grepOutput — Method
From a string, it will divide it by lines and retrieve the ones that match the regular expression provided.
source
PerfTest.iteratorExpressionParser — Method
This is one of the parser functions that expand any formula block for metric definition. This function will parse the :iterator symbol and substitute it with the current value of the innermost test set loop of the current test target execution
source
PerfTest.metaGet — Method
Runs over an array of expressions trying to match the desired one.   If not found returns "Nothing".
"sym" should follow the MacroTools nomenclature for the @capture macro
source
PerfTest.onCustomMetricDefinition — Method
This function is called to register a custom metric, it will parse the arguments of the definition macro and add the metric to the context to be later used in test targets on the same scope.
source
PerfTest.onMemoryThroughputDefinition — Method
This function is used to register a special custom metric, which is the effective memory throughput calculation, and is registered in the same way as any other but with a special flag that the EMT metholodogy will use to get and use the metric.
source
PerfTest.openDataFile — Method
This method is used to get historical data of a performance test suite from a save file located in path.
source
PerfTest.popQuoteBlocks — Method
Useful to correct operations limited by the tree walking  Will remove quote blocks inside the main block without recursion and push  their expressions into the main block
source
PerfTest.printDepth! — Method
This method is used to print the test names, with consideration on the hierarchy and adding indentation whenever necessary
source
PerfTest.printIntervalLanding — Function
This method is used to print a graphical representation on a test result and the admisible intervals it can take. The result will and the two bounds will be printed in order.
source
PerfTest.printMethodology — Method
This function is used to print the information relative to a methodology, relative to a a specific test execution result. This will usually print a series of metrics and might also print plots.
source
PerfTest.printMetric — Method
This method is used to dump into the output the information about a metric and the value obtained in a specific test.
source
PerfTest.printedOutputExpressionParser — Method
This is one of the parser functions that expand any formula block for metric definition. This function will parse the :printed_output symbol and substitute it with the standard output of the test target execution
source
PerfTest.printfail — Method
This method dumps into the output a test result in case of failure. The output will be formatted to make it easy to read.
source
PerfTest.removeBlock — Method
Pops expr which has a head that is :block or :quote and returns array of nested expressions which are the arguments of such head.
source
PerfTest.retvalExpressionParser — Method
This is one of the parser functions that expand any formula block for metric definition. This function will parse the appropiate symbol and substitute it by the return value of the test target execution.
source
PerfTest.ruleSet — Method
This method builds what is known as a rule set. Which is a function that will evaluate if an expression triggers a rule in a set and if that is the case apply the rule modifier. See the ASTRule documentation for more information.
WARNING: the rule set will apply the FIRST rule that matches with the expression, therefore other matches will be ignored
Arguments
  • context the context structure of the tree run, it will be ocassinally used by some rules on the set.
  • rules the collection of rules that will belong to the resulting set.
source
PerfTest.saveDataFile — Method
This method is used to save historical data of a performance test suite to a save file located in path.
source
PerfTest.saveExprAsFile — Function
Utility to save an expression (expr) to a Julia file stored at path
Requires a :toplevel symbol to be the head of the expression.
source
PerfTest.testsetUpdate! — Method
Function used to register a new test set in the hierarchy record of the context, where name is the name of the test set.
source
PerfTest.treeRun — Method
This method implements the transformation that converts a recipe script into a fully-fledged testing suite. The function will return a Julia expression with the resulting performance testing suite. This can be then executed or saved in a file for later usage.
Arguments
  • path the path of the script to be transformed.
source
PerfTest.@define_eff_memory_throughput — Macro
This macro is used to define the memory bandwidth of a target in order to execute the effective memory thorughput methodology.
Arguments
  • formula block : an expression that returns a single value, which would be the metric value. The formula can have any julia expression inside and additionally some special symbols are supported. The formula may be evaluated several times, so its applied to every target in every test set or just once, if the formula is defined inside a test set, which makes it only applicable to it.
Special symbols:
  • :median_time : will be substituted by the median time the target took to execute in the benchmark.
  • :minimum_time: will be substituted by the minimum time the target took to execute in the benchmark.
  • :ret_value : will be substituted by the return value of the target.
  • :autoflop: will be substituted by the FLOP count the target.
  • :printed_output : will be substituted by the standard output stream of the target.
  • :iterator : will be substituted by the current iterator value in a loop test set.
Example:
The following definition assumes that each execution of the target expression involves transacting 1000 bytes. Therefore the bandwith is 1000 / execution time.
@define_eff_memory_throughput begin
+help?> PerfTest
Types
Index
Documentation
PerfTest.ASTRule — Type
Used by the AST walker to check for expressions that match condition, if they do then modifier will be applied to the expression.
This is the basic building block of the code transformer, a set of these rules compounds to all the needed manipulations to create the testing suite.
source
PerfTest.ASTWalkDepthRecord — Type
This structure is used to record a test set hierarchy during a AST walk. In any specific point of the walk the array will TODO
source
PerfTest.Context — Type
In order to perform with the test suite generation, the AST walk needs to keep a context register to integrate features that rely on the scope hierarchy.
source
PerfTest.CustomMetric — Type
Saves flags needed during the execution of the AST walk. It holds if:
  • The walk is on an expression that is a test target
  • The walk is on an expression that is inside a config macro
  • Several flags that affect the roofline methodology
source
PerfTest.DepthRecord — Type
This structure is used to record a test set frame during a AST walk. See ASTWalkDepthRecord for more info.
source
PerfTest.EnvironmentFlags — Type
Saves flags needed during the execution of the AST walk. It holds if:
  • The walk is on an expression that is a test target
  • The walk is on an expression that is inside a config macro
  • Several flags that affect the roofline methodology
source
PerfTest.Methodology_Result — Type
This struct is used in the test suite to save a methodology result, which in turn is constituted of a group of metric results and their references. Additionally, custom elements that are not subject to test are also saved, e.g. informational metrics, printing functions.
source
PerfTest.Metric_Constraint — Type
This struct is used in the test suite to save a metric test result and its associated data, it saves the reference used and the toreance intervals in absolute and percentual values, also it shows if the test succeded and some additional variables for data printing
source
PerfTest.Metric_Reference — Type
This struct is used in the test suite to save a metric reference, a reference is meant to be later compared with a result, its combination gives the Metric_Constraint struct. It holds:
  • A reference value.
  • low_is_bad registers if in this metric lower values are less desired than higher ones, or the opposite (e.g. time vs  FLOP/s).
source
PerfTest.Metric_Result — Type
This struct is used in the test suite to save a metric measurement, therefore its saves the metric name, its units space and its value.
source
PerfTest.Perftest_Result — Type
This struct saves a complete test suite result for one execution. It also saves the raw measurements obtained from the targets.
source
PerfTest.Struct_Eff_Mem_Throughput — Type
This struct holds the configuration of the basic effective memory throughput methodology.
  • enabled is used to enable or disable the methodology
  • tolerance defines the interval of ratios (eff.mem.through. / max. bandwidth) that make the test succeed.
source
PerfTest.Struct_Metric_Config — Type
This struct can hold the configuration of any metric.
  • enabled is used to enable or disable the methodology
  • regression_threshold, when comparing the measure with a reference, defines how far can the measurement be from the reference
source
PerfTest.Struct_Regression — Type
This struct holds the configuration of the basic metric regression methodology.
enabled is used to enable or disable the methodology save_failed will record historical measurements of failed tests if true general_regression_threshold sets the torelance interval for the test comparison
regression_calculation can be:
  • :latest    The reference will be the latest saved result
  • :average The reference will be the average of all saved results
source
PerfTest.Struct_Roofline_Config — Type
This struct holds the configuration of the basic roofline methodology.
  • enabled is used to enable or disable the methodology
  • tolerance defines the interval of ratios (eff.mem.through. / max. bandwidth) that make the test succeed.
source
Functions
Index
Documentation
PerfTest._treeRun — Method
This method gets a input julia expression, and a context register and executes a transformation of the input that converts a recipe script (input) into a fully-fledged testing suite (return value).
Arguments
  • input_expr the recipe/source expression. (internally, a.k.a source code space)
  • context a register that will store information useful for the transformation over its run over the AST of the input
source
PerfTest.autoflopExpressionParser — Method
This is one of the parser functions that expand any formula block for metric definition. This function will parse the :autoflop symbol and substitute it with the flop count of the test target
source
PerfTest.by_index — Method
This method expects a hierarchy tree (dict) in the form of nested dictionaries and a vector of dictionary keys idx. The function will recursively apply the keys to get to a final element.
It is usually put to work with the DepthRecord struct.
source
PerfTest.checkAuxiliaryCustomMetrics — Method
This method is used to generate the code that computes the value of every auxiliary custom metric enabled in th current context, where the code is generated.
source
PerfTest.checkAuxiliaryMetric — Method
This method is used to generate the code that computes the value of a given auxiliary custom metric in the context the code is generated.
source
PerfTest.checkCustomMetric — Method
This function is used to generate the code that evaluates if a custom metric result f a target is within a specified reference.
WARNING
Predefined symbols needed before this code is added to the generated space:
  • reference_value
  • metric_results
source
PerfTest.checkCustomMetrics — Method
This function is used to generate the code that evaluates if a custom metric result f a target is within a specified reference.
WARNING
Predefined symbols needed before this code is added to the generated space:
  • reference_value
  • metric_results
source
PerfTest.checkMedianTime — Method
This function is used to generate the code that evaluates if the median time of execution of a target is within a specified reference.
WARNING
Predefined symbols needed before this code is added to the generated space:
  • reference_value
  • metric_results
source
PerfTest.checkMinTime — Method
This function is used to generate the code that evaluates if the minimum time of execution of a target is within a specified reference.
WARNING
Predefined symbols needed before this code is added to the generated space:
  • reference_value
  • metric_results
source
PerfTest.customMetricExpressionParser — Method
This is one of the parser functions that expand any formula block for metric definition. This function will parse all primitive metric symbols with the structure where the corresponding value of the metric is.
source
PerfTest.customMetricReferenceExpressionParser — Method
This is one of the parser functions that expand any formula block for metric definition. This function will parse all primitive metric symbols with the structure where the corresponding reference value for the metric is.
source
PerfTest.extractMethodologyResultArray — Method
This method will return a flattened array of all of the results for all the methodologies exercised in the provided dictionary.
Example:
"Test Set 1"      -> "Test 1"          -> Methodology A result          -> Methodology B result  "Test Set 2"      -> "Test 1"          -> Methodology A result Returns:  M. A result (Test 1)  M. B result (Test 1)  M. A result (Test 2)
source
PerfTest.extractNamesResultArray — Method
This method will return a flattened array of the whole test result hierarchy.
Example
Example:
"Test Set 1"      -> "Test 1"          -> Methodology A result          -> Methodology B result  "Test Set 2"      -> "Test 1"          -> Methodology A result Returns:  "Test Set 1 -> Test 1 -> Methodology A"  "Test Set 1 -> Test 1 -> Methodology B"  "Test Set 2 -> Test 1 -> Methodology A"
source
PerfTest.flattenedInterpolation — Method
This method interpolates the inside_expr into outside_expr anywhere it finds the token substitution_token, which is a symbol. The outside_expr has to be a block or a quote block. It has the particularity that it will remove block heads from the inside_expr and add the nested elements onto the location where the token it.
Example:
outside_expr = :(:A; 4)
inside_expr = :(begin 2;3 end)
substitution_token = :A
returns = :(2;3;4)
source
PerfTest.fullParsingSuite — Method
This function combines a collection of rules to turn a formula block into a functioning expression to calculate any metric defined by said formula
source
PerfTest.genTestName! — Method
Function that generates a test name if needed, it is used to name   test targets to distinguish them if several go in the same testset.
source
PerfTest.getMetricValue — Method
Given a series of methodology results, the the raw values of all the metrics contained in the methodology results.
source
PerfTest.grepOutput — Method
From a string, it will divide it by lines and retrieve the ones that match the regular expression provided.
source
PerfTest.iteratorExpressionParser — Method
This is one of the parser functions that expand any formula block for metric definition. This function will parse the :iterator symbol and substitute it with the current value of the innermost test set loop of the current test target execution
source
PerfTest.metaGet — Method
Runs over an array of expressions trying to match the desired one.   If not found returns "Nothing".
"sym" should follow the MacroTools nomenclature for the @capture macro
source
PerfTest.onCustomMetricDefinition — Method
This function is called to register a custom metric, it will parse the arguments of the definition macro and add the metric to the context to be later used in test targets on the same scope.
source
PerfTest.onMemoryThroughputDefinition — Method
This function is used to register a special custom metric, which is the effective memory throughput calculation, and is registered in the same way as any other but with a special flag that the EMT metholodogy will use to get and use the metric.
source
PerfTest.openDataFile — Method
This method is used to get historical data of a performance test suite from a save file located in path.
source
PerfTest.popQuoteBlocks — Method
Useful to correct operations limited by the tree walking  Will remove quote blocks inside the main block without recursion and push  their expressions into the main block
source
PerfTest.printDepth! — Method
This method is used to print the test names, with consideration on the hierarchy and adding indentation whenever necessary
source
PerfTest.printIntervalLanding — Function
This method is used to print a graphical representation on a test result and the admisible intervals it can take. The result will and the two bounds will be printed in order.
source
PerfTest.printMethodology — Method
This function is used to print the information relative to a methodology, relative to a a specific test execution result. This will usually print a series of metrics and might also print plots.
source
PerfTest.printMetric — Method
This method is used to dump into the output the information about a metric and the value obtained in a specific test.
source
PerfTest.printedOutputExpressionParser — Method
This is one of the parser functions that expand any formula block for metric definition. This function will parse the :printed_output symbol and substitute it with the standard output of the test target execution
source
PerfTest.printfail — Method
This method dumps into the output a test result in case of failure. The output will be formatted to make it easy to read.
source
PerfTest.removeBlock — Method
Pops expr which has a head that is :block or :quote and returns array of nested expressions which are the arguments of such head.
source
PerfTest.retvalExpressionParser — Method
This is one of the parser functions that expand any formula block for metric definition. This function will parse the appropiate symbol and substitute it by the return value of the test target execution.
source
PerfTest.ruleSet — Method
This method builds what is known as a rule set. Which is a function that will evaluate if an expression triggers a rule in a set and if that is the case apply the rule modifier. See the ASTRule documentation for more information.
WARNING: the rule set will apply the FIRST rule that matches with the expression, therefore other matches will be ignored
Arguments
  • context the context structure of the tree run, it will be ocassinally used by some rules on the set.
  • rules the collection of rules that will belong to the resulting set.
source
PerfTest.saveDataFile — Method
This method is used to save historical data of a performance test suite to a save file located in path.
source
PerfTest.saveExprAsFile — Function
Utility to save an expression (expr) to a Julia file stored at path
Requires a :toplevel symbol to be the head of the expression.
source
PerfTest.testsetUpdate! — Method
Function used to register a new test set in the hierarchy record of the context, where name is the name of the test set.
source
PerfTest.treeRun — Method
This method implements the transformation that converts a recipe script into a fully-fledged testing suite. The function will return a Julia expression with the resulting performance testing suite. This can be then executed or saved in a file for later usage.
Arguments
  • path the path of the script to be transformed.
source
PerfTest.@define_eff_memory_throughput — Macro
This macro is used to define the memory bandwidth of a target in order to execute the effective memory thorughput methodology.
Arguments
  • formula block : an expression that returns a single value, which would be the metric value. The formula can have any julia expression inside and additionally some special symbols are supported. The formula may be evaluated several times, so its applied to every target in every test set or just once, if the formula is defined inside a test set, which makes it only applicable to it.
Special symbols:
  • :median_time : will be substituted by the median time the target took to execute in the benchmark.
  • :minimum_time: will be substituted by the minimum time the target took to execute in the benchmark.
  • :ret_value : will be substituted by the return value of the target.
  • :autoflop: will be substituted by the FLOP count the target.
  • :printed_output : will be substituted by the standard output stream of the target.
  • :iterator : will be substituted by the current iterator value in a loop test set.
Example:
The following definition assumes that each execution of the target expression involves transacting 1000 bytes. Therefore the bandwith is 1000 / execution time.
@define_eff_memory_throughput begin
       1000 / :median_time
-end
source
PerfTest.@define_metric — Macro
This macro is used to define a new custom metric.
Arguments
  • name : the name of the metric for identification purposes.
  • units : the unit space that the metric values will be in.
  • formula block : an expression that returns a single value, which would be the metric value. The formula can have any julia expression inside and additionally some special symbols are supported. The formula may be evaluated several times, so its applied to every target in every test set or just once, if the formula is defined inside a test set, which makes it only applicable to it.
Special symbols:
  • :median_time : will be substituted by the median time the target took to execute in the benchmark.
  • :minimum_time: will be substituted by the minimum time the target took to execute in the benchmark.
  • :ret_value : will be substituted by the return value of the target.
  • :autoflop: will be substituted by the FLOP count the target.
  • :printed_output : will be substituted by the standard output stream of the target.
  • :iterator : will be substituted by the current iterator value in a loop test set.
source
PerfTest.@on_perftest_exec — Macro
The expression given to this macro will only be executed in the generated suite, and will be deleted if the source code is executed as is.
source
PerfTest.@on_perftest_ignore — Macro
The expression given to this macro will only be executed in the source code, and will be deleted in the generated performance test suite.
source
PerfTest.@perftest — Macro
This macro is used to signal that the wrapped expression is a performance test target, and therefore its performance will be sampled and then evaluated following the current suite configuration.
If the macro is evaluated it does not modify the target at all. The effects of the macro only show when the script is transformed into a performance testing suite.
This macro is sensitive to context since other adjacent macros can change how the target will be evaluated.
Arguments
  • The target expression
Example
@perftest 2 + 3
source
PerfTest.@perftest_config — Macro
Perftest_config macro, used to set customised configuration on the suite generated by the source script
Configuration inside this macro must follow the syntax below:
@perftest_config
+end
source
PerfTest.@define_metric — Macro
This macro is used to define a new custom metric.
Arguments
  • name : the name of the metric for identification purposes.
  • units : the unit space that the metric values will be in.
  • formula block : an expression that returns a single value, which would be the metric value. The formula can have any julia expression inside and additionally some special symbols are supported. The formula may be evaluated several times, so its applied to every target in every test set or just once, if the formula is defined inside a test set, which makes it only applicable to it.
Special symbols:
  • :median_time : will be substituted by the median time the target took to execute in the benchmark.
  • :minimum_time: will be substituted by the minimum time the target took to execute in the benchmark.
  • :ret_value : will be substituted by the return value of the target.
  • :autoflop: will be substituted by the FLOP count the target.
  • :printed_output : will be substituted by the standard output stream of the target.
  • :iterator : will be substituted by the current iterator value in a loop test set.
source
PerfTest.@on_perftest_exec — Macro
The expression given to this macro will only be executed in the generated suite, and will be deleted if the source code is executed as is.
source
PerfTest.@on_perftest_ignore — Macro
The expression given to this macro will only be executed in the source code, and will be deleted in the generated performance test suite.
source
PerfTest.@perftest — Macro
This macro is used to signal that the wrapped expression is a performance test target, and therefore its performance will be sampled and then evaluated following the current suite configuration.
If the macro is evaluated it does not modify the target at all. The effects of the macro only show when the script is transformed into a performance testing suite.
This macro is sensitive to context since other adjacent macros can change how the target will be evaluated.
Arguments
  • The target expression
Example
@perftest 2 + 3
source
PerfTest.@perftest_config — Macro
Perftest_config macro, used to set customised configuration on the suite generated by the source script
Configuration inside this macro must follow the syntax below:
@perftest_config
     key = value
     key.subkey = value
-end
Where key can be any configuration parameter, in some cases parameters will consist on a set of subparameters denoted by the "." to refer to them.
source
PerfTest.@roofline — Macro
This macro enables roofline modelling, if put just before a target declaration (@perftest) it will proceed to evaluate it using a roofline model.
Mandatory arguments
  • formula block: the macro has to wrap a block that holds a formula to obtain the operational intensity of target algorithms.
Optional arguments
  • cpu_peak : a manual input value for the maximum attainable FLOPS, this will override the empirical runtime benchmark
  • membw_peak : a manual input value for the maximum memory bandwith, this will override the empirical runtime benchmark
  • target_opint : a desired operational intensity for the target, this will turn operational intensity into a test metric
  • actual_flops: another formula that defines the actual performance of the test.
  • target_ratio : the acceptable ratio between the actual performance and the projected performance from the roofline, this will turn actual performance into a test metric.
Special symbols:
  • :median_time : will be substituted by the median time the target took to execute in the benchmark.
  • :minimum_time: will be substituted by the minimum time the target took to execute in the benchmark.
  • :ret_value : will be substituted by the return value of the target.
  • :autoflop: will be substituted by the FLOP count the target.
  • :printed_output : will be substituted by the standard output stream of the target.
  • :iterator : will be substituted by the current iterator value in a loop test set.
Any formula block specified in this macro supports these symbols.
Example
@roofline actual_flops=:autoflop target_ratio=0.05 begin
+end
Where key can be any configuration parameter, in some cases parameters will consist on a set of subparameters denoted by the "." to refer to them.
source
PerfTest.@roofline — Macro
This macro enables roofline modelling, if put just before a target declaration (@perftest) it will proceed to evaluate it using a roofline model.
Mandatory arguments
  • formula block: the macro has to wrap a block that holds a formula to obtain the operational intensity of target algorithms.
Optional arguments
  • cpu_peak : a manual input value for the maximum attainable FLOPS, this will override the empirical runtime benchmark
  • membw_peak : a manual input value for the maximum memory bandwith, this will override the empirical runtime benchmark
  • target_opint : a desired operational intensity for the target, this will turn operational intensity into a test metric
  • actual_flops: another formula that defines the actual performance of the test.
  • target_ratio : the acceptable ratio between the actual performance and the projected performance from the roofline, this will turn actual performance into a test metric.
Special symbols:
  • :median_time : will be substituted by the median time the target took to execute in the benchmark.
  • :minimum_time: will be substituted by the minimum time the target took to execute in the benchmark.
  • :ret_value : will be substituted by the return value of the target.
  • :autoflop: will be substituted by the FLOP count the target.
  • :printed_output : will be substituted by the standard output stream of the target.
  • :iterator : will be substituted by the current iterator value in a loop test set.
Any formula block specified in this macro supports these symbols.
Example
@roofline actual_flops=:autoflop target_ratio=0.05 begin
     mem = ((:iterator + 1) * :iterator)
     :autoflop / mem
-end
The code block defines operational intensity, whilst the other arguments define how to measure and compare the actual performance with the roofline performance. If the actual to projected performance ratio goes below the target, the test fails.
source

+end

The code block defines operational intensity, whilst the other arguments define how to measure and compare the actual performance with the roofline performance. If the actual to projected performance ratio goes below the target, the test fails.

source
diff --git a/dev/examples.MD b/dev/examples.MD index 376eec5..1a1e2b9 100644 --- a/dev/examples.MD +++ b/dev/examples.MD @@ -1,9 +1,9 @@ # Examples -```@contents -Pages = ["examples/memcopyCellArray3D.md"] -``` + + + -```@contents -Pages = ["examples/memcopyCellArray3D_ParallelStencil.md"] -``` + + + diff --git a/dev/examples/index.html b/dev/examples/index.html index 4c8fbc6..23b2b52 100644 --- a/dev/examples/index.html +++ b/dev/examples/index.html @@ -1,2 +1,2 @@ -... · PerfTest.jl
+... · PerfTest.jl

Examples

<!– @contents --> <!-- Pages = ["examples/memcopyCellArray3D.md"] --> <!-- –>

<!– @contents --> <!-- Pages = ["examples/memcopyCellArray3D_ParallelStencil.md"] --> <!-- –>

diff --git a/dev/examples/memcopyCellArray3D/index.html b/dev/examples/memcopyCellArray3D/index.html index 04f241d..450e794 100644 --- a/dev/examples/memcopyCellArray3D/index.html +++ b/dev/examples/memcopyCellArray3D/index.html @@ -50,4 +50,4 @@ end memcopy3D() -

The corresponding file can be found ...

+

The corresponding file can be found ...

diff --git a/dev/examples/memcopyCellArray3D_ParallelStencil/index.html b/dev/examples/memcopyCellArray3D_ParallelStencil/index.html index 96f37a7..142d501 100644 --- a/dev/examples/memcopyCellArray3D_ParallelStencil/index.html +++ b/dev/examples/memcopyCellArray3D_ParallelStencil/index.html @@ -53,4 +53,4 @@ end memcopy3D() -

The corresponding file can be found ...

+

The corresponding file can be found ...

diff --git a/dev/index.html b/dev/index.html index 58ac5f7..0ab48a8 100644 --- a/dev/index.html +++ b/dev/index.html @@ -1,2 +1,2 @@ -Introduction · PerfTest.jl

PerfTest.jl Star on GitHub

The package PerfTest provides the user with a performance regression unit testing framework. This framework consists of a collection of macros used to declaratively define a performance test suite. Scripts with said macros can then be transformed into performance suites using the transform method. This package is focused on providing an easy and fast way to develop performance suites, with additional features to customise them following the demands of the use case.

Dependencies

PerfTest relies on:

  • [MacroTools]
  • [JLD2]
  • [MPI]
  • [STREAMBenchmark]
  • [GFlops]
  • [UnicodePlots]
  • [Test]
  • [Suppressor]

Contributors

This project has been developed as a Master's thesis by Daniel Sergio Vega Rodriguez. Thus, the contributors to this project have been so far:

  • Daniel Sergio Vega Rodriguez (@Dvegrod), Università della Svizzera italiana (USI): developer
  • Dr. Samuel Omlin (@omlins), Swiss National Supercomputing Centre (CSCS), ETH Zurich: original idea and Master's thesis supervisor
  • Prof. Olaf Schenk, Università della Svizzera italiana (USI): Master's thesis supervisor
  • Dr. Pasadakis Dimosthenis, Università della Svizzera italiana (USI): additional advisor
+Introduction · PerfTest.jl

PerfTest.jl Star on GitHub

The package PerfTest provides the user with a performance regression unit testing framework. This framework consists of a collection of macros used to declaratively define a performance test suite. Scripts with said macros can then be transformed into performance suites using the transform method. This package is focused on providing an easy and fast way to develop performance suites, with additional features to customise them following the demands of the use case.

Dependencies

PerfTest relies on:

  • [MacroTools]
  • [JLD2]
  • [MPI]
  • [STREAMBenchmark]
  • [GFlops]
  • [UnicodePlots]
  • [Test]
  • [Suppressor]

Contributors

This project has been developed as a Master's thesis by Daniel Sergio Vega Rodriguez. Thus, the contributors to this project have been so far:

  • Daniel Sergio Vega Rodriguez (@Dvegrod), Università della Svizzera italiana (USI): developer
  • Dr. Samuel Omlin (@omlins), Swiss National Supercomputing Centre (CSCS), ETH Zurich: original idea and Master's thesis supervisor
  • Prof. Olaf Schenk, Università della Svizzera italiana (USI): Master's thesis supervisor
  • Dr. Pasadakis Dimosthenis, Università della Svizzera italiana (USI): additional advisor
diff --git a/dev/objects.inv b/dev/objects.inv index 9edad09..a603074 100644 Binary files a/dev/objects.inv and b/dev/objects.inv differ diff --git a/dev/search_index.js b/dev/search_index.js index f42df75..9709c83 100644 --- a/dev/search_index.js +++ b/dev/search_index.js @@ -1,3 +1,3 @@ var documenterSearchIndex = {"docs": -[{"location":"api/","page":"API reference","title":"API reference","text":"CurrentModule = PerfTest","category":"page"},{"location":"api/#API-reference","page":"API reference","title":"API reference","text":"","category":"section"},{"location":"api/","page":"API reference","title":"API reference","text":"This is the official API reference of PerfTest. Note that it can also be queried interactively from the Julia REPL using the help mode:","category":"page"},{"location":"api/","page":"API reference","title":"API reference","text":"julia> using PerfTest\njulia>?\nhelp?> PerfTest","category":"page"},{"location":"api/#Types","page":"API reference","title":"Types","text":"","category":"section"},{"location":"api/#Index","page":"API reference","title":"Index","text":"","category":"section"},{"location":"api/","page":"API reference","title":"API reference","text":"Order = [:type]","category":"page"},{"location":"api/#Documentation","page":"API reference","title":"Documentation","text":"","category":"section"},{"location":"api/","page":"API reference","title":"API reference","text":"Modules = [PerfTest]\nOrder = [:type]","category":"page"},{"location":"api/#PerfTest.ASTRule","page":"API reference","title":"PerfTest.ASTRule","text":"Used by the AST walker to check for expressions that match condition, if they do then modifier will be applied to the expression.\n\nThis is the basic building block of the code transformer, a set of these rules compounds to all the needed manipulations to create the testing suite.\n\n\n\n\n\n","category":"type"},{"location":"api/#PerfTest.ASTWalkDepthRecord","page":"API reference","title":"PerfTest.ASTWalkDepthRecord","text":"This structure is used to record a test set hierarchy during a AST walk. In any specific point of the walk the array will TODO\n\n\n\n\n\n","category":"type"},{"location":"api/#PerfTest.Context","page":"API reference","title":"PerfTest.Context","text":"In order to perform with the test suite generation, the AST walk needs to keep a context register to integrate features that rely on the scope hierarchy.\n\n\n\n\n\n","category":"type"},{"location":"api/#PerfTest.CustomMetric","page":"API reference","title":"PerfTest.CustomMetric","text":"Saves flags needed during the execution of the AST walk. It holds if:\n\nThe walk is on an expression that is a test target\nThe walk is on an expression that is inside a config macro\nSeveral flags that affect the roofline methodology\n\n\n\n\n\n","category":"type"},{"location":"api/#PerfTest.DepthRecord","page":"API reference","title":"PerfTest.DepthRecord","text":"This structure is used to record a test set frame during a AST walk. See ASTWalkDepthRecord for more info.\n\n\n\n\n\n","category":"type"},{"location":"api/#PerfTest.EnvironmentFlags","page":"API reference","title":"PerfTest.EnvironmentFlags","text":"Saves flags needed during the execution of the AST walk. It holds if:\n\nThe walk is on an expression that is a test target\nThe walk is on an expression that is inside a config macro\nSeveral flags that affect the roofline methodology\n\n\n\n\n\n","category":"type"},{"location":"api/#PerfTest.Methodology_Result","page":"API reference","title":"PerfTest.Methodology_Result","text":"This struct is used in the test suite to save a methodology result, which in turn is constituted of a group of metric results and their references. Additionally, custom elements that are not subject to test are also saved, e.g. informational metrics, printing functions.\n\n\n\n\n\n","category":"type"},{"location":"api/#PerfTest.Metric_Constraint","page":"API reference","title":"PerfTest.Metric_Constraint","text":"This struct is used in the test suite to save a metric test result and its associated data, it saves the reference used and the toreance intervals in absolute and percentual values, also it shows if the test succeded and some additional variables for data printing\n\n\n\n\n\n","category":"type"},{"location":"api/#PerfTest.Metric_Reference","page":"API reference","title":"PerfTest.Metric_Reference","text":"This struct is used in the test suite to save a metric reference, a reference is meant to be later compared with a result, its combination gives the Metric_Constraint struct. It holds:\n\nA reference value.\nlow_is_bad registers if in this metric lower values are less desired than higher ones, or the opposite (e.g. time vs FLOP/s).\n\n\n\n\n\n","category":"type"},{"location":"api/#PerfTest.Metric_Result","page":"API reference","title":"PerfTest.Metric_Result","text":"This struct is used in the test suite to save a metric measurement, therefore its saves the metric name, its units space and its value.\n\n\n\n\n\n","category":"type"},{"location":"api/#PerfTest.Perftest_Datafile_Root","page":"API reference","title":"PerfTest.Perftest_Datafile_Root","text":"This struct is the root of the data recording file, it can save several performance test suite execution results.\n\n\n\n\n\n","category":"type"},{"location":"api/#PerfTest.Perftest_Result","page":"API reference","title":"PerfTest.Perftest_Result","text":"This struct saves a complete test suite result for one execution. It also saves the raw measurements obtained from the targets.\n\n\n\n\n\n","category":"type"},{"location":"api/#PerfTest.Struct_Eff_Mem_Throughput","page":"API reference","title":"PerfTest.Struct_Eff_Mem_Throughput","text":"This struct holds the configuration of the basic effective memory throughput methodology.\n\nenabled is used to enable or disable the methodology\ntolerance defines the interval of ratios (eff.mem.through. / max. bandwidth) that make the test succeed.\n\n\n\n\n\n","category":"type"},{"location":"api/#PerfTest.Struct_Metric_Config","page":"API reference","title":"PerfTest.Struct_Metric_Config","text":"This struct can hold the configuration of any metric.\n\nenabled is used to enable or disable the methodology\nregression_threshold, when comparing the measure with a reference, defines how far can the measurement be from the reference\n\n\n\n\n\n","category":"type"},{"location":"api/#PerfTest.Struct_Regression","page":"API reference","title":"PerfTest.Struct_Regression","text":"This struct holds the configuration of the basic metric regression methodology.\n\nenabled is used to enable or disable the methodology save_failed will record historical measurements of failed tests if true general_regression_threshold sets the torelance interval for the test comparison\n\nregression_calculation can be:\n\n:latest The reference will be the latest saved result\n:average The reference will be the average of all saved results\n\n\n\n\n\n","category":"type"},{"location":"api/#PerfTest.Struct_Roofline_Config","page":"API reference","title":"PerfTest.Struct_Roofline_Config","text":"This struct holds the configuration of the basic roofline methodology.\n\nenabled is used to enable or disable the methodology\ntolerance defines the interval of ratios (eff.mem.through. / max. bandwidth) that make the test succeed.\n\n\n\n\n\n","category":"type"},{"location":"api/#PerfTest.Struct_Tolerance","page":"API reference","title":"PerfTest.Struct_Tolerance","text":"Tolerance interval structure. Used to save intervals around a threshold during test comparisons.\n\n\n\n\n\n","category":"type"},{"location":"api/#Functions-–-additional-to-standard-AbstractArray-functionality","page":"API reference","title":"Functions – additional to standard AbstractArray functionality","text":"","category":"section"},{"location":"api/#Index-2","page":"API reference","title":"Index","text":"","category":"section"},{"location":"api/","page":"API reference","title":"API reference","text":"Order = [:function, :macro]","category":"page"},{"location":"api/#Documentation-2","page":"API reference","title":"Documentation","text":"","category":"section"},{"location":"api/","page":"API reference","title":"API reference","text":"Modules = [PerfTest]\nOrder = [:function, :macro]","category":"page"},{"location":"api/#PerfTest._treeRun-Tuple{Expr, PerfTest.Context, Vararg{Any}}","page":"API reference","title":"PerfTest._treeRun","text":"This method gets a input julia expression, and a context register and executes a transformation of the input that converts a recipe script (input) into a fully-fledged testing suite (return value).\n\nArguments\n\ninput_expr the recipe/source expression. (internally, a.k.a source code space)\ncontext a register that will store information useful for the transformation over its run over the AST of the input\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.autoflopExpressionParser-Tuple{Expr}","page":"API reference","title":"PerfTest.autoflopExpressionParser","text":"This is one of the parser functions that expand any formula block for metric definition. This function will parse the :autoflop symbol and substitute it with the flop count of the test target\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.auxiliarMetricPrint-Tuple{PerfTest.Metric_Result, Int64}","page":"API reference","title":"PerfTest.auxiliarMetricPrint","text":"This function is used to dump metric information regading auxiliar metrics, which are not used in testing.\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.buildPrimitiveMetrics-Tuple{}","page":"API reference","title":"PerfTest.buildPrimitiveMetrics","text":"This function generates the code that make primitive metrics values available to all metodologies and custom metrics.\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.by_index-Tuple{Union{BenchmarkTools.BenchmarkGroup, Dict}, Vector{PerfTest.DepthRecord}}","page":"API reference","title":"PerfTest.by_index","text":"This method expects a hierarchy tree (dict) in the form of nested dictionaries and a vector of dictionary keys idx. The function will recursively apply the keys to get to a final element.\n\nIt is usually put to work with the DepthRecord struct.\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.checkAuxiliaryCustomMetrics-Tuple{PerfTest.Context}","page":"API reference","title":"PerfTest.checkAuxiliaryCustomMetrics","text":"This method is used to generate the code that computes the value of every auxiliary custom metric enabled in th current context, where the code is generated.\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.checkAuxiliaryMetric-Tuple{PerfTest.CustomMetric}","page":"API reference","title":"PerfTest.checkAuxiliaryMetric","text":"This method is used to generate the code that computes the value of a given auxiliary custom metric in the context the code is generated.\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.checkCustomMetric-Tuple{PerfTest.CustomMetric}","page":"API reference","title":"PerfTest.checkCustomMetric","text":"This function is used to generate the code that evaluates if a custom metric result f a target is within a specified reference.\n\nWARNING\n\nPredefined symbols needed before this code is added to the generated space:\n\nreference_value\nmetric_results\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.checkCustomMetrics-Tuple{PerfTest.Context}","page":"API reference","title":"PerfTest.checkCustomMetrics","text":"This function is used to generate the code that evaluates if a custom metric result f a target is within a specified reference.\n\nWARNING\n\nPredefined symbols needed before this code is added to the generated space:\n\nreference_value\nmetric_results\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.checkMedianTime-Tuple{PerfTest.Struct_Tolerance}","page":"API reference","title":"PerfTest.checkMedianTime","text":"This function is used to generate the code that evaluates if the median time of execution of a target is within a specified reference.\n\nWARNING\n\nPredefined symbols needed before this code is added to the generated space:\n\nreference_value\nmetric_results\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.checkMinTime-Tuple{PerfTest.Struct_Tolerance}","page":"API reference","title":"PerfTest.checkMinTime","text":"This function is used to generate the code that evaluates if the minimum time of execution of a target is within a specified reference.\n\nWARNING\n\nPredefined symbols needed before this code is added to the generated space:\n\nreference_value\nmetric_results\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.configFallBack-Tuple{Any, Symbol}","page":"API reference","title":"PerfTest.configFallBack","text":"A little automatism to jump to defaults if the configuration provided is absent Kind can be: :regression\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.customMetricExpressionParser-Tuple{Expr}","page":"API reference","title":"PerfTest.customMetricExpressionParser","text":"This is one of the parser functions that expand any formula block for metric definition. This function will parse all primitive metric symbols with the structure where the corresponding value of the metric is.\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.customMetricReferenceExpressionParser-Tuple{Expr}","page":"API reference","title":"PerfTest.customMetricReferenceExpressionParser","text":"This is one of the parser functions that expand any formula block for metric definition. This function will parse all primitive metric symbols with the structure where the corresponding reference value for the metric is.\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.extractMethodologyResultArray-Tuple{Dict, Symbol}","page":"API reference","title":"PerfTest.extractMethodologyResultArray","text":"This method will return a flattened array of all of the results for all the methodologies exercised in the provided dictionary.\n\nExample:\n\n\"Test Set 1\" -> \"Test 1\" -> Methodology A result -> Methodology B result \"Test Set 2\" -> \"Test 1\" -> Methodology A result Returns: M. A result (Test 1) M. B result (Test 1) M. A result (Test 2)\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.extractNamesResultArray-Tuple{Dict, Symbol}","page":"API reference","title":"PerfTest.extractNamesResultArray","text":"This method will return a flattened array of the whole test result hierarchy.\n\nExample\n\nExample:\n\n\"Test Set 1\" -> \"Test 1\" -> Methodology A result -> Methodology B result \"Test Set 2\" -> \"Test 1\" -> Methodology A result Returns: \"Test Set 1 -> Test 1 -> Methodology A\" \"Test Set 1 -> Test 1 -> Methodology B\" \"Test Set 2 -> Test 1 -> Methodology A\"\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.flattenedInterpolation-Tuple{Expr, Expr, Symbol}","page":"API reference","title":"PerfTest.flattenedInterpolation","text":"This method interpolates the inside_expr into outside_expr anywhere it finds the token substitution_token, which is a symbol. The outside_expr has to be a block or a quote block. It has the particularity that it will remove block heads from the inside_expr and add the nested elements onto the location where the token it.\n\nExample:\n\noutside_expr = :(:A; 4)\n\ninside_expr = :(begin 2;3 end)\n\nsubstitution_token = :A\n\nreturns = :(2;3;4)\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.fullParsingSuite-Tuple{Expr}","page":"API reference","title":"PerfTest.fullParsingSuite","text":"This function combines a collection of rules to turn a formula block into a functioning expression to calculate any metric defined by said formula\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.genTestName!-Tuple{PerfTest.Context}","page":"API reference","title":"PerfTest.genTestName!","text":"Function that generates a test name if needed, it is used to name test targets to distinguish them if several go in the same testset.\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.getMetricValue-Tuple{Vector{PerfTest.Methodology_Result}, String}","page":"API reference","title":"PerfTest.getMetricValue","text":"Given a series of methodology results, the the raw values of all the metrics contained in the methodology results.\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.getNumber-Tuple{String}","page":"API reference","title":"PerfTest.getNumber","text":"From a string (field), it will parse the first number it finds as a Float\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.grepOutput-Tuple{String, Union{Regex, String}}","page":"API reference","title":"PerfTest.grepOutput","text":"From a string, it will divide it by lines and retrieve the ones that match the regular expression provided.\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.grepOutputXGetNumber-Tuple{String, String}","page":"API reference","title":"PerfTest.grepOutputXGetNumber","text":"Given a string output, it will retrieve the first number in the first line that contains the string string.\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.iteratorExpressionParser-Tuple{Expr}","page":"API reference","title":"PerfTest.iteratorExpressionParser","text":"This is one of the parser functions that expand any formula block for metric definition. This function will parse the :iterator symbol and substitute it with the current value of the innermost test set loop of the current test target execution\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.loadFileAsExpr-Tuple{AbstractString}","page":"API reference","title":"PerfTest.loadFileAsExpr","text":"Utility to get an expression from a Julia file stored at path\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.metaGet-Tuple{AbstractVector, Symbol}","page":"API reference","title":"PerfTest.metaGet","text":"Runs over an array of expressions trying to match the desired one. If not found returns \"Nothing\".\n\n\"sym\" should follow the MacroTools nomenclature for the @capture macro\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.metaGetString-Tuple{AbstractVector}","page":"API reference","title":"PerfTest.metaGetString","text":"\n\n\n\n","category":"method"},{"location":"api/#PerfTest.onCustomMetricDefinition-Tuple{Expr, PerfTest.Context, Set{Symbol}}","page":"API reference","title":"PerfTest.onCustomMetricDefinition","text":"This function is called to register a custom metric, it will parse the arguments of the definition macro and add the metric to the context to be later used in test targets on the same scope.\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.onMemoryThroughputDefinition-Tuple{Expr, PerfTest.Context}","page":"API reference","title":"PerfTest.onMemoryThroughputDefinition","text":"This function is used to register a special custom metric, which is the effective memory throughput calculation, and is registered in the same way as any other but with a special flag that the EMT metholodogy will use to get and use the metric.\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.openDataFile-Tuple{AbstractString}","page":"API reference","title":"PerfTest.openDataFile","text":"This method is used to get historical data of a performance test suite from a save file located in path.\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.p_blue-Tuple{Any}","page":"API reference","title":"PerfTest.p_blue","text":"Prints the element in color blue\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.p_green-Tuple{Any}","page":"API reference","title":"PerfTest.p_green","text":"Prints the element in color green\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.p_red-Tuple{Any}","page":"API reference","title":"PerfTest.p_red","text":"Prints the element in color red\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.p_yellow-Tuple{Any}","page":"API reference","title":"PerfTest.p_yellow","text":"Prints the element in color yellow\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.perftestConfigEnter-Tuple{Expr, PerfTest.Context}","page":"API reference","title":"PerfTest.perftestConfigEnter","text":"Function to trigger the configuration mode on the context register\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.perftestConfigExit-Tuple{Expr, PerfTest.Context}","page":"API reference","title":"PerfTest.perftestConfigExit","text":"Function to deactivate the configuration mode on the context register\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.popQuoteBlocks-Tuple{Expr}","page":"API reference","title":"PerfTest.popQuoteBlocks","text":"Useful to correct operations limited by the tree walking Will remove quote blocks inside the main block without recursion and push their expressions into the main block\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.printDepth!-Tuple{AbstractArray}","page":"API reference","title":"PerfTest.printDepth!","text":"This method is used to print the test names, with consideration on the hierarchy and adding indentation whenever necessary\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.printIntervalLanding","page":"API reference","title":"PerfTest.printIntervalLanding","text":"This method is used to print a graphical representation on a test result and the admisible intervals it can take. The result will and the two bounds will be printed in order.\n\n\n\n\n\n","category":"function"},{"location":"api/#PerfTest.printMethodology-Tuple{PerfTest.Methodology_Result, Int64}","page":"API reference","title":"PerfTest.printMethodology","text":"This function is used to print the information relative to a methodology, relative to a a specific test execution result. This will usually print a series of metrics and might also print plots.\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.printMetric-Tuple{PerfTest.Metric_Result, PerfTest.Metric_Constraint, Int64}","page":"API reference","title":"PerfTest.printMetric","text":"This method is used to dump into the output the information about a metric and the value obtained in a specific test.\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.printedOutputExpressionParser-Tuple{Expr}","page":"API reference","title":"PerfTest.printedOutputExpressionParser","text":"This is one of the parser functions that expand any formula block for metric definition. This function will parse the :printed_output symbol and substitute it with the standard output of the test target execution\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.printfail-Tuple{BenchmarkTools.TrialJudgement, BenchmarkTools.Trial, BenchmarkTools.Trial, PerfTest.FloatRange, Int64}","page":"API reference","title":"PerfTest.printfail","text":"This method dumps into the output a test result in case of failure. The output will be formatted to make it easy to read.\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.removeBlock-Tuple{Expr}","page":"API reference","title":"PerfTest.removeBlock","text":"Pops expr which has a head that is :block or :quote and returns array of nested expressions which are the arguments of such head.\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.retvalExpressionParser-Tuple{Expr}","page":"API reference","title":"PerfTest.retvalExpressionParser","text":"This is one of the parser functions that expand any formula block for metric definition. This function will parse the appropiate symbol and substitute it by the return value of the test target execution.\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.rooflineMacroParse-Tuple{Expr, PerfTest.Context}","page":"API reference","title":"PerfTest.rooflineMacroParse","text":"Parses roofline user request and sets up data for roofline computation.\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.ruleSet-Tuple{PerfTest.Context, Vector{PerfTest.ASTRule}}","page":"API reference","title":"PerfTest.ruleSet","text":"This method builds what is known as a rule set. Which is a function that will evaluate if an expression triggers a rule in a set and if that is the case apply the rule modifier. See the ASTRule documentation for more information.\n\nWARNING: the rule set will apply the FIRST rule that matches with the expression, therefore other matches will be ignored\n\nArguments\n\ncontext the context structure of the tree run, it will be ocassinally used by some rules on the set.\nrules the collection of rules that will belong to the resulting set.\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.saveDataFile-Tuple{AbstractString, PerfTest.Perftest_Datafile_Root}","page":"API reference","title":"PerfTest.saveDataFile","text":"This method is used to save historical data of a performance test suite to a save file located in path.\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.saveExprAsFile","page":"API reference","title":"PerfTest.saveExprAsFile","text":"Utility to save an expression (expr) to a Julia file stored at path\n\nRequires a :toplevel symbol to be the head of the expression.\n\n\n\n\n\n","category":"function"},{"location":"api/#PerfTest.setupCPUPeakFlopBenchmark-Tuple{}","page":"API reference","title":"PerfTest.setupCPUPeakFlopBenchmark","text":"This method is used to generate the code responsible for sampling the maximum CPU FLOPS based on the avaiable threads in every resulting suite.\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.setupMemoryBandwidthBenchmark-Tuple{}","page":"API reference","title":"PerfTest.setupMemoryBandwidthBenchmark","text":"This method is used to generate the code responsible for sampling the maximum memory bandwith in every resulting suite.\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.testsetUpdate!-Tuple{PerfTest.Context, String}","page":"API reference","title":"PerfTest.testsetUpdate!","text":"Function used to register a new test set in the hierarchy record of the context, where name is the name of the test set.\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.treeRun-Tuple{AbstractString}","page":"API reference","title":"PerfTest.treeRun","text":"This method implements the transformation that converts a recipe script into a fully-fledged testing suite. The function will return a Julia expression with the resulting performance testing suite. This can be then executed or saved in a file for later usage.\n\nArguments\n\npath the path of the script to be transformed.\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.unblockAndConcat-Tuple{Vector{Expr}}","page":"API reference","title":"PerfTest.unblockAndConcat","text":"This function is useful to move expressions to the toplevel when they are enclosed inside a block\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.@auxiliary_metric-Tuple{Any, Any, Any}","page":"API reference","title":"PerfTest.@auxiliary_metric","text":"Defines a custom metric for informational purposes that will not be used for testing but will be printed as output.\n\n\n\n\n\n","category":"macro"},{"location":"api/#PerfTest.@define_eff_memory_throughput-Tuple{Any}","page":"API reference","title":"PerfTest.@define_eff_memory_throughput","text":"This macro is used to define the memory bandwidth of a target in order to execute the effective memory thorughput methodology.\n\nArguments\n\nformula block : an expression that returns a single value, which would be the metric value. The formula can have any julia expression inside and additionally some special symbols are supported. The formula may be evaluated several times, so its applied to every target in every test set or just once, if the formula is defined inside a test set, which makes it only applicable to it.\n\nSpecial symbols:\n\n:median_time : will be substituted by the median time the target took to execute in the benchmark.\n:minimum_time: will be substituted by the minimum time the target took to execute in the benchmark.\n:ret_value : will be substituted by the return value of the target.\n:autoflop: will be substituted by the FLOP count the target.\n:printed_output : will be substituted by the standard output stream of the target.\n:iterator : will be substituted by the current iterator value in a loop test set.\n\nExample:\n\nThe following definition assumes that each execution of the target expression involves transacting 1000 bytes. Therefore the bandwith is 1000 / execution time.\n\n@define_eff_memory_throughput begin\n 1000 / :median_time\nend\n\n\n\n\n\n","category":"macro"},{"location":"api/#PerfTest.@define_metric-Tuple{Any}","page":"API reference","title":"PerfTest.@define_metric","text":"This macro is used to define a new custom metric.\n\nArguments\n\nname : the name of the metric for identification purposes.\nunits : the unit space that the metric values will be in.\nformula block : an expression that returns a single value, which would be the metric value. The formula can have any julia expression inside and additionally some special symbols are supported. The formula may be evaluated several times, so its applied to every target in every test set or just once, if the formula is defined inside a test set, which makes it only applicable to it.\n\nSpecial symbols:\n\n:median_time : will be substituted by the median time the target took to execute in the benchmark.\n:minimum_time: will be substituted by the minimum time the target took to execute in the benchmark.\n:ret_value : will be substituted by the return value of the target.\n:autoflop: will be substituted by the FLOP count the target.\n:printed_output : will be substituted by the standard output stream of the target.\n:iterator : will be substituted by the current iterator value in a loop test set.\n\n\n\n\n\n","category":"macro"},{"location":"api/#PerfTest.@lpad-Tuple{Any}","page":"API reference","title":"PerfTest.@lpad","text":"Macro that adds a space at the beggining of a string\n\n\n\n\n\n","category":"macro"},{"location":"api/#PerfTest.@on_perftest_exec-Tuple{Any}","page":"API reference","title":"PerfTest.@on_perftest_exec","text":"The expression given to this macro will only be executed in the generated suite, and will be deleted if the source code is executed as is.\n\n\n\n\n\n","category":"macro"},{"location":"api/#PerfTest.@on_perftest_ignore-Tuple{Any}","page":"API reference","title":"PerfTest.@on_perftest_ignore","text":"The expression given to this macro will only be executed in the source code, and will be deleted in the generated performance test suite.\n\n\n\n\n\n","category":"macro"},{"location":"api/#PerfTest.@perftest-Tuple{Any}","page":"API reference","title":"PerfTest.@perftest","text":"This macro is used to signal that the wrapped expression is a performance test target, and therefore its performance will be sampled and then evaluated following the current suite configuration.\n\nIf the macro is evaluated it does not modify the target at all. The effects of the macro only show when the script is transformed into a performance testing suite.\n\nThis macro is sensitive to context since other adjacent macros can change how the target will be evaluated.\n\nArguments\n\nThe target expression\n\nExample\n\n@perftest 2 + 3\n\n\n\n\n\n","category":"macro"},{"location":"api/#PerfTest.@perftest_config-Tuple{Any}","page":"API reference","title":"PerfTest.@perftest_config","text":"Perftest_config macro, used to set customised configuration on the suite generated by the source script\n\nConfiguration inside this macro must follow the syntax below:\n\n@perftest_config\n key = value\n key.subkey = value\nend\n\nWhere key can be any configuration parameter, in some cases parameters will consist on a set of subparameters denoted by the \".\" to refer to them.\n\n\n\n\n\n","category":"macro"},{"location":"api/#PerfTest.@roofline","page":"API reference","title":"PerfTest.@roofline","text":"This macro enables roofline modelling, if put just before a target declaration (@perftest) it will proceed to evaluate it using a roofline model.\n\nMandatory arguments\n\nformula block: the macro has to wrap a block that holds a formula to obtain the operational intensity of target algorithms.\n\nOptional arguments\n\ncpu_peak : a manual input value for the maximum attainable FLOPS, this will override the empirical runtime benchmark\nmembw_peak : a manual input value for the maximum memory bandwith, this will override the empirical runtime benchmark\ntarget_opint : a desired operational intensity for the target, this will turn operational intensity into a test metric\nactual_flops: another formula that defines the actual performance of the test.\ntarget_ratio : the acceptable ratio between the actual performance and the projected performance from the roofline, this will turn actual performance into a test metric.\n\nSpecial symbols:\n\n:median_time : will be substituted by the median time the target took to execute in the benchmark.\n:minimum_time: will be substituted by the minimum time the target took to execute in the benchmark.\n:ret_value : will be substituted by the return value of the target.\n:autoflop: will be substituted by the FLOP count the target.\n:printed_output : will be substituted by the standard output stream of the target.\n:iterator : will be substituted by the current iterator value in a loop test set.\n\nAny formula block specified in this macro supports these symbols.\n\nExample\n\n@roofline actual_flops=:autoflop target_ratio=0.05 begin\n mem = ((:iterator + 1) * :iterator)\n :autoflop / mem\nend\n\nThe code block defines operational intensity, whilst the other arguments define how to measure and compare the actual performance with the roofline performance. If the actual to projected performance ratio goes below the target, the test fails.\n\n\n\n\n\n","category":"macro"},{"location":"usage/#Usage","page":"Usage","title":"Usage","text":"","category":"section"},{"location":"usage/","page":"Usage","title":"Usage","text":"Have a look at the Examples and see the API reference for details on the usage of PerfTest.","category":"page"},{"location":"usage/","page":"Usage","title":"Usage","text":"tip: Hint\nParallelStencil.jl enables straightforward working with CellArrays. It automatically allocates CellArray when the keyword arguments celldims or celltype are given to the architecture-agnostic allocation macros @zeros, @ones, @rand, @falses, @trues and @fill (refer to the documentation of ParallelStencil.jl for more details).","category":"page"},{"location":"usage/#Installation","page":"Usage","title":"Installation","text":"","category":"section"},{"location":"usage/","page":"Usage","title":"Usage","text":"PerfTest can be installed directly with the Julia package manager from the Julia REPL:","category":"page"},{"location":"usage/","page":"Usage","title":"Usage","text":"julia>]\n pkg> add PerfTest","category":"page"},{"location":"examples/#Examples","page":"...","title":"Examples","text":"","category":"section"},{"location":"examples/","page":"...","title":"...","text":"Pages = [\"examples/memcopyCellArray3D.md\"]","category":"page"},{"location":"examples/","page":"...","title":"...","text":"Pages = [\"examples/memcopyCellArray3D_ParallelStencil.md\"]","category":"page"},{"location":"examples/memcopyCellArray3D_ParallelStencil/#Memory-copy-of-with-4x4-cells-using-[ParallelStencil.jl](https://github.com/omlins/ParallelStencil.jl)","page":"Memory copy of with 4x4 cells using ParallelStencil.jl","title":"Memory copy of with 4x4 cells using ParallelStencil.jl","text":"","category":"section"},{"location":"examples/memcopyCellArray3D_ParallelStencil/","page":"Memory copy of with 4x4 cells using ParallelStencil.jl","title":"Memory copy of with 4x4 cells using ParallelStencil.jl","text":"Main.mdinclude(joinpath(Main.EXAMPLEROOT, \"memcopyCellArray3D_ParallelStencil.jl\"))","category":"page"},{"location":"examples/memcopyCellArray3D_ParallelStencil/","page":"Memory copy of with 4x4 cells using ParallelStencil.jl","title":"Memory copy of with 4x4 cells using ParallelStencil.jl","text":"The corresponding file can be found ...","category":"page"},{"location":"examples/memcopyCellArray3D/#Memory-copy-of-with-4x4-SMatrix-cells","page":"Memory copy of with 4x4 SMatrix cells","title":"Memory copy of with 4x4 SMatrix cells","text":"","category":"section"},{"location":"examples/memcopyCellArray3D/","page":"Memory copy of with 4x4 SMatrix cells","title":"Memory copy of with 4x4 SMatrix cells","text":"Main.mdinclude(joinpath(Main.EXAMPLEROOT, \"memcopyCellArray3D.jl\"))","category":"page"},{"location":"examples/memcopyCellArray3D/","page":"Memory copy of with 4x4 SMatrix cells","title":"Memory copy of with 4x4 SMatrix cells","text":"The corresponding file can be found ...","category":"page"},{"location":"#[PerfTest.jl](https://github.com/JuliaPerf/PerfTest.jl)-[![Star-on-GitHub](https://img.shields.io/github/stars/JuliaPerf/PerfTest.jl.svg)](https://github.com/JuliaPerf/PerfTest.jl/stargazers)","page":"Introduction","title":"PerfTest.jl (Image: Star on GitHub)","text":"","category":"section"},{"location":"","page":"Introduction","title":"Introduction","text":"The package PerfTest provides the user with a performance regression unit testing framework. This framework consists of a collection of macros used to declaratively define a performance test suite. Scripts with said macros can then be transformed into performance suites using the transform method. This package is focused on providing an easy and fast way to develop performance suites, with additional features to customise them following the demands of the use case.","category":"page"},{"location":"#Dependencies","page":"Introduction","title":"Dependencies","text":"","category":"section"},{"location":"","page":"Introduction","title":"Introduction","text":"PerfTest relies on:","category":"page"},{"location":"","page":"Introduction","title":"Introduction","text":"[MacroTools]\n[JLD2]\n[MPI]\n[STREAMBenchmark]\n[GFlops]\n[UnicodePlots]\n[Test]\n[Suppressor]","category":"page"},{"location":"#Contributors","page":"Introduction","title":"Contributors","text":"","category":"section"},{"location":"","page":"Introduction","title":"Introduction","text":"This project has been developed as a Master's thesis by Daniel Sergio Vega Rodriguez. Thus, the contributors to this project have been so far:","category":"page"},{"location":"","page":"Introduction","title":"Introduction","text":"Daniel Sergio Vega Rodriguez (@Dvegrod), Università della Svizzera italiana (USI): developer\nDr. Samuel Omlin (@omlins), Swiss National Supercomputing Centre (CSCS), ETH Zurich: original idea and Master's thesis supervisor\nProf. Olaf Schenk, Università della Svizzera italiana (USI): Master's thesis supervisor\nDr. Pasadakis Dimosthenis, Università della Svizzera italiana (USI): additional advisor","category":"page"}] +[{"location":"api/","page":"API reference","title":"API reference","text":"CurrentModule = PerfTest","category":"page"},{"location":"api/#API-reference","page":"API reference","title":"API reference","text":"","category":"section"},{"location":"api/","page":"API reference","title":"API reference","text":"This is the official API reference of PerfTest. Note that it can also be queried interactively from the Julia REPL using the help mode:","category":"page"},{"location":"api/","page":"API reference","title":"API reference","text":"julia> using PerfTest\njulia>?\nhelp?> PerfTest","category":"page"},{"location":"api/#Types","page":"API reference","title":"Types","text":"","category":"section"},{"location":"api/#Index","page":"API reference","title":"Index","text":"","category":"section"},{"location":"api/","page":"API reference","title":"API reference","text":"Order = [:type]","category":"page"},{"location":"api/#Documentation","page":"API reference","title":"Documentation","text":"","category":"section"},{"location":"api/","page":"API reference","title":"API reference","text":"Modules = [PerfTest]\nOrder = [:type]","category":"page"},{"location":"api/#PerfTest.ASTRule","page":"API reference","title":"PerfTest.ASTRule","text":"Used by the AST walker to check for expressions that match condition, if they do then modifier will be applied to the expression.\n\nThis is the basic building block of the code transformer, a set of these rules compounds to all the needed manipulations to create the testing suite.\n\n\n\n\n\n","category":"type"},{"location":"api/#PerfTest.ASTWalkDepthRecord","page":"API reference","title":"PerfTest.ASTWalkDepthRecord","text":"This structure is used to record a test set hierarchy during a AST walk. In any specific point of the walk the array will TODO\n\n\n\n\n\n","category":"type"},{"location":"api/#PerfTest.Context","page":"API reference","title":"PerfTest.Context","text":"In order to perform with the test suite generation, the AST walk needs to keep a context register to integrate features that rely on the scope hierarchy.\n\n\n\n\n\n","category":"type"},{"location":"api/#PerfTest.CustomMetric","page":"API reference","title":"PerfTest.CustomMetric","text":"Saves flags needed during the execution of the AST walk. It holds if:\n\nThe walk is on an expression that is a test target\nThe walk is on an expression that is inside a config macro\nSeveral flags that affect the roofline methodology\n\n\n\n\n\n","category":"type"},{"location":"api/#PerfTest.DepthRecord","page":"API reference","title":"PerfTest.DepthRecord","text":"This structure is used to record a test set frame during a AST walk. See ASTWalkDepthRecord for more info.\n\n\n\n\n\n","category":"type"},{"location":"api/#PerfTest.EnvironmentFlags","page":"API reference","title":"PerfTest.EnvironmentFlags","text":"Saves flags needed during the execution of the AST walk. It holds if:\n\nThe walk is on an expression that is a test target\nThe walk is on an expression that is inside a config macro\nSeveral flags that affect the roofline methodology\n\n\n\n\n\n","category":"type"},{"location":"api/#PerfTest.Methodology_Result","page":"API reference","title":"PerfTest.Methodology_Result","text":"This struct is used in the test suite to save a methodology result, which in turn is constituted of a group of metric results and their references. Additionally, custom elements that are not subject to test are also saved, e.g. informational metrics, printing functions.\n\n\n\n\n\n","category":"type"},{"location":"api/#PerfTest.Metric_Constraint","page":"API reference","title":"PerfTest.Metric_Constraint","text":"This struct is used in the test suite to save a metric test result and its associated data, it saves the reference used and the toreance intervals in absolute and percentual values, also it shows if the test succeded and some additional variables for data printing\n\n\n\n\n\n","category":"type"},{"location":"api/#PerfTest.Metric_Reference","page":"API reference","title":"PerfTest.Metric_Reference","text":"This struct is used in the test suite to save a metric reference, a reference is meant to be later compared with a result, its combination gives the Metric_Constraint struct. It holds:\n\nA reference value.\nlow_is_bad registers if in this metric lower values are less desired than higher ones, or the opposite (e.g. time vs FLOP/s).\n\n\n\n\n\n","category":"type"},{"location":"api/#PerfTest.Metric_Result","page":"API reference","title":"PerfTest.Metric_Result","text":"This struct is used in the test suite to save a metric measurement, therefore its saves the metric name, its units space and its value.\n\n\n\n\n\n","category":"type"},{"location":"api/#PerfTest.Perftest_Datafile_Root","page":"API reference","title":"PerfTest.Perftest_Datafile_Root","text":"This struct is the root of the data recording file, it can save several performance test suite execution results.\n\n\n\n\n\n","category":"type"},{"location":"api/#PerfTest.Perftest_Result","page":"API reference","title":"PerfTest.Perftest_Result","text":"This struct saves a complete test suite result for one execution. It also saves the raw measurements obtained from the targets.\n\n\n\n\n\n","category":"type"},{"location":"api/#PerfTest.Struct_Eff_Mem_Throughput","page":"API reference","title":"PerfTest.Struct_Eff_Mem_Throughput","text":"This struct holds the configuration of the basic effective memory throughput methodology.\n\nenabled is used to enable or disable the methodology\ntolerance defines the interval of ratios (eff.mem.through. / max. bandwidth) that make the test succeed.\n\n\n\n\n\n","category":"type"},{"location":"api/#PerfTest.Struct_Metric_Config","page":"API reference","title":"PerfTest.Struct_Metric_Config","text":"This struct can hold the configuration of any metric.\n\nenabled is used to enable or disable the methodology\nregression_threshold, when comparing the measure with a reference, defines how far can the measurement be from the reference\n\n\n\n\n\n","category":"type"},{"location":"api/#PerfTest.Struct_Regression","page":"API reference","title":"PerfTest.Struct_Regression","text":"This struct holds the configuration of the basic metric regression methodology.\n\nenabled is used to enable or disable the methodology save_failed will record historical measurements of failed tests if true general_regression_threshold sets the torelance interval for the test comparison\n\nregression_calculation can be:\n\n:latest The reference will be the latest saved result\n:average The reference will be the average of all saved results\n\n\n\n\n\n","category":"type"},{"location":"api/#PerfTest.Struct_Roofline_Config","page":"API reference","title":"PerfTest.Struct_Roofline_Config","text":"This struct holds the configuration of the basic roofline methodology.\n\nenabled is used to enable or disable the methodology\ntolerance defines the interval of ratios (eff.mem.through. / max. bandwidth) that make the test succeed.\n\n\n\n\n\n","category":"type"},{"location":"api/#PerfTest.Struct_Tolerance","page":"API reference","title":"PerfTest.Struct_Tolerance","text":"Tolerance interval structure. Used to save intervals around a threshold during test comparisons.\n\n\n\n\n\n","category":"type"},{"location":"api/#Functions","page":"API reference","title":"Functions","text":"","category":"section"},{"location":"api/#Index-2","page":"API reference","title":"Index","text":"","category":"section"},{"location":"api/","page":"API reference","title":"API reference","text":"Order = [:function, :macro]","category":"page"},{"location":"api/#Documentation-2","page":"API reference","title":"Documentation","text":"","category":"section"},{"location":"api/","page":"API reference","title":"API reference","text":"Modules = [PerfTest]\nOrder = [:function, :macro]","category":"page"},{"location":"api/#PerfTest._treeRun-Tuple{Expr, PerfTest.Context, Vararg{Any}}","page":"API reference","title":"PerfTest._treeRun","text":"This method gets a input julia expression, and a context register and executes a transformation of the input that converts a recipe script (input) into a fully-fledged testing suite (return value).\n\nArguments\n\ninput_expr the recipe/source expression. (internally, a.k.a source code space)\ncontext a register that will store information useful for the transformation over its run over the AST of the input\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.autoflopExpressionParser-Tuple{Expr}","page":"API reference","title":"PerfTest.autoflopExpressionParser","text":"This is one of the parser functions that expand any formula block for metric definition. This function will parse the :autoflop symbol and substitute it with the flop count of the test target\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.auxiliarMetricPrint-Tuple{PerfTest.Metric_Result, Int64}","page":"API reference","title":"PerfTest.auxiliarMetricPrint","text":"This function is used to dump metric information regading auxiliar metrics, which are not used in testing.\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.buildPrimitiveMetrics-Tuple{}","page":"API reference","title":"PerfTest.buildPrimitiveMetrics","text":"This function generates the code that make primitive metrics values available to all metodologies and custom metrics.\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.by_index-Tuple{Union{BenchmarkTools.BenchmarkGroup, Dict}, Vector{PerfTest.DepthRecord}}","page":"API reference","title":"PerfTest.by_index","text":"This method expects a hierarchy tree (dict) in the form of nested dictionaries and a vector of dictionary keys idx. The function will recursively apply the keys to get to a final element.\n\nIt is usually put to work with the DepthRecord struct.\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.checkAuxiliaryCustomMetrics-Tuple{PerfTest.Context}","page":"API reference","title":"PerfTest.checkAuxiliaryCustomMetrics","text":"This method is used to generate the code that computes the value of every auxiliary custom metric enabled in th current context, where the code is generated.\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.checkAuxiliaryMetric-Tuple{PerfTest.CustomMetric}","page":"API reference","title":"PerfTest.checkAuxiliaryMetric","text":"This method is used to generate the code that computes the value of a given auxiliary custom metric in the context the code is generated.\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.checkCustomMetric-Tuple{PerfTest.CustomMetric}","page":"API reference","title":"PerfTest.checkCustomMetric","text":"This function is used to generate the code that evaluates if a custom metric result f a target is within a specified reference.\n\nWARNING\n\nPredefined symbols needed before this code is added to the generated space:\n\nreference_value\nmetric_results\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.checkCustomMetrics-Tuple{PerfTest.Context}","page":"API reference","title":"PerfTest.checkCustomMetrics","text":"This function is used to generate the code that evaluates if a custom metric result f a target is within a specified reference.\n\nWARNING\n\nPredefined symbols needed before this code is added to the generated space:\n\nreference_value\nmetric_results\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.checkMedianTime-Tuple{PerfTest.Struct_Tolerance}","page":"API reference","title":"PerfTest.checkMedianTime","text":"This function is used to generate the code that evaluates if the median time of execution of a target is within a specified reference.\n\nWARNING\n\nPredefined symbols needed before this code is added to the generated space:\n\nreference_value\nmetric_results\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.checkMinTime-Tuple{PerfTest.Struct_Tolerance}","page":"API reference","title":"PerfTest.checkMinTime","text":"This function is used to generate the code that evaluates if the minimum time of execution of a target is within a specified reference.\n\nWARNING\n\nPredefined symbols needed before this code is added to the generated space:\n\nreference_value\nmetric_results\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.configFallBack-Tuple{Any, Symbol}","page":"API reference","title":"PerfTest.configFallBack","text":"A little automatism to jump to defaults if the configuration provided is absent Kind can be: :regression\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.customMetricExpressionParser-Tuple{Expr}","page":"API reference","title":"PerfTest.customMetricExpressionParser","text":"This is one of the parser functions that expand any formula block for metric definition. This function will parse all primitive metric symbols with the structure where the corresponding value of the metric is.\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.customMetricReferenceExpressionParser-Tuple{Expr}","page":"API reference","title":"PerfTest.customMetricReferenceExpressionParser","text":"This is one of the parser functions that expand any formula block for metric definition. This function will parse all primitive metric symbols with the structure where the corresponding reference value for the metric is.\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.extractMethodologyResultArray-Tuple{Dict, Symbol}","page":"API reference","title":"PerfTest.extractMethodologyResultArray","text":"This method will return a flattened array of all of the results for all the methodologies exercised in the provided dictionary.\n\nExample:\n\n\"Test Set 1\" -> \"Test 1\" -> Methodology A result -> Methodology B result \"Test Set 2\" -> \"Test 1\" -> Methodology A result Returns: M. A result (Test 1) M. B result (Test 1) M. A result (Test 2)\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.extractNamesResultArray-Tuple{Dict, Symbol}","page":"API reference","title":"PerfTest.extractNamesResultArray","text":"This method will return a flattened array of the whole test result hierarchy.\n\nExample\n\nExample:\n\n\"Test Set 1\" -> \"Test 1\" -> Methodology A result -> Methodology B result \"Test Set 2\" -> \"Test 1\" -> Methodology A result Returns: \"Test Set 1 -> Test 1 -> Methodology A\" \"Test Set 1 -> Test 1 -> Methodology B\" \"Test Set 2 -> Test 1 -> Methodology A\"\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.flattenedInterpolation-Tuple{Expr, Expr, Symbol}","page":"API reference","title":"PerfTest.flattenedInterpolation","text":"This method interpolates the inside_expr into outside_expr anywhere it finds the token substitution_token, which is a symbol. The outside_expr has to be a block or a quote block. It has the particularity that it will remove block heads from the inside_expr and add the nested elements onto the location where the token it.\n\nExample:\n\noutside_expr = :(:A; 4)\n\ninside_expr = :(begin 2;3 end)\n\nsubstitution_token = :A\n\nreturns = :(2;3;4)\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.fullParsingSuite-Tuple{Expr}","page":"API reference","title":"PerfTest.fullParsingSuite","text":"This function combines a collection of rules to turn a formula block into a functioning expression to calculate any metric defined by said formula\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.genTestName!-Tuple{PerfTest.Context}","page":"API reference","title":"PerfTest.genTestName!","text":"Function that generates a test name if needed, it is used to name test targets to distinguish them if several go in the same testset.\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.getMetricValue-Tuple{Vector{PerfTest.Methodology_Result}, String}","page":"API reference","title":"PerfTest.getMetricValue","text":"Given a series of methodology results, the the raw values of all the metrics contained in the methodology results.\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.getNumber-Tuple{String}","page":"API reference","title":"PerfTest.getNumber","text":"From a string (field), it will parse the first number it finds as a Float\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.grepOutput-Tuple{String, Union{Regex, String}}","page":"API reference","title":"PerfTest.grepOutput","text":"From a string, it will divide it by lines and retrieve the ones that match the regular expression provided.\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.grepOutputXGetNumber-Tuple{String, String}","page":"API reference","title":"PerfTest.grepOutputXGetNumber","text":"Given a string output, it will retrieve the first number in the first line that contains the string string.\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.iteratorExpressionParser-Tuple{Expr}","page":"API reference","title":"PerfTest.iteratorExpressionParser","text":"This is one of the parser functions that expand any formula block for metric definition. This function will parse the :iterator symbol and substitute it with the current value of the innermost test set loop of the current test target execution\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.loadFileAsExpr-Tuple{AbstractString}","page":"API reference","title":"PerfTest.loadFileAsExpr","text":"Utility to get an expression from a Julia file stored at path\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.metaGet-Tuple{AbstractVector, Symbol}","page":"API reference","title":"PerfTest.metaGet","text":"Runs over an array of expressions trying to match the desired one. If not found returns \"Nothing\".\n\n\"sym\" should follow the MacroTools nomenclature for the @capture macro\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.metaGetString-Tuple{AbstractVector}","page":"API reference","title":"PerfTest.metaGetString","text":"\n\n\n\n","category":"method"},{"location":"api/#PerfTest.onCustomMetricDefinition-Tuple{Expr, PerfTest.Context, Set{Symbol}}","page":"API reference","title":"PerfTest.onCustomMetricDefinition","text":"This function is called to register a custom metric, it will parse the arguments of the definition macro and add the metric to the context to be later used in test targets on the same scope.\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.onMemoryThroughputDefinition-Tuple{Expr, PerfTest.Context}","page":"API reference","title":"PerfTest.onMemoryThroughputDefinition","text":"This function is used to register a special custom metric, which is the effective memory throughput calculation, and is registered in the same way as any other but with a special flag that the EMT metholodogy will use to get and use the metric.\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.openDataFile-Tuple{AbstractString}","page":"API reference","title":"PerfTest.openDataFile","text":"This method is used to get historical data of a performance test suite from a save file located in path.\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.p_blue-Tuple{Any}","page":"API reference","title":"PerfTest.p_blue","text":"Prints the element in color blue\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.p_green-Tuple{Any}","page":"API reference","title":"PerfTest.p_green","text":"Prints the element in color green\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.p_red-Tuple{Any}","page":"API reference","title":"PerfTest.p_red","text":"Prints the element in color red\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.p_yellow-Tuple{Any}","page":"API reference","title":"PerfTest.p_yellow","text":"Prints the element in color yellow\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.perftestConfigEnter-Tuple{Expr, PerfTest.Context}","page":"API reference","title":"PerfTest.perftestConfigEnter","text":"Function to trigger the configuration mode on the context register\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.perftestConfigExit-Tuple{Expr, PerfTest.Context}","page":"API reference","title":"PerfTest.perftestConfigExit","text":"Function to deactivate the configuration mode on the context register\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.popQuoteBlocks-Tuple{Expr}","page":"API reference","title":"PerfTest.popQuoteBlocks","text":"Useful to correct operations limited by the tree walking Will remove quote blocks inside the main block without recursion and push their expressions into the main block\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.printDepth!-Tuple{AbstractArray}","page":"API reference","title":"PerfTest.printDepth!","text":"This method is used to print the test names, with consideration on the hierarchy and adding indentation whenever necessary\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.printIntervalLanding","page":"API reference","title":"PerfTest.printIntervalLanding","text":"This method is used to print a graphical representation on a test result and the admisible intervals it can take. The result will and the two bounds will be printed in order.\n\n\n\n\n\n","category":"function"},{"location":"api/#PerfTest.printMethodology-Tuple{PerfTest.Methodology_Result, Int64}","page":"API reference","title":"PerfTest.printMethodology","text":"This function is used to print the information relative to a methodology, relative to a a specific test execution result. This will usually print a series of metrics and might also print plots.\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.printMetric-Tuple{PerfTest.Metric_Result, PerfTest.Metric_Constraint, Int64}","page":"API reference","title":"PerfTest.printMetric","text":"This method is used to dump into the output the information about a metric and the value obtained in a specific test.\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.printedOutputExpressionParser-Tuple{Expr}","page":"API reference","title":"PerfTest.printedOutputExpressionParser","text":"This is one of the parser functions that expand any formula block for metric definition. This function will parse the :printed_output symbol and substitute it with the standard output of the test target execution\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.printfail-Tuple{BenchmarkTools.TrialJudgement, BenchmarkTools.Trial, BenchmarkTools.Trial, PerfTest.FloatRange, Int64}","page":"API reference","title":"PerfTest.printfail","text":"This method dumps into the output a test result in case of failure. The output will be formatted to make it easy to read.\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.removeBlock-Tuple{Expr}","page":"API reference","title":"PerfTest.removeBlock","text":"Pops expr which has a head that is :block or :quote and returns array of nested expressions which are the arguments of such head.\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.retvalExpressionParser-Tuple{Expr}","page":"API reference","title":"PerfTest.retvalExpressionParser","text":"This is one of the parser functions that expand any formula block for metric definition. This function will parse the appropiate symbol and substitute it by the return value of the test target execution.\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.rooflineMacroParse-Tuple{Expr, PerfTest.Context}","page":"API reference","title":"PerfTest.rooflineMacroParse","text":"Parses roofline user request and sets up data for roofline computation.\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.ruleSet-Tuple{PerfTest.Context, Vector{PerfTest.ASTRule}}","page":"API reference","title":"PerfTest.ruleSet","text":"This method builds what is known as a rule set. Which is a function that will evaluate if an expression triggers a rule in a set and if that is the case apply the rule modifier. See the ASTRule documentation for more information.\n\nWARNING: the rule set will apply the FIRST rule that matches with the expression, therefore other matches will be ignored\n\nArguments\n\ncontext the context structure of the tree run, it will be ocassinally used by some rules on the set.\nrules the collection of rules that will belong to the resulting set.\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.saveDataFile-Tuple{AbstractString, PerfTest.Perftest_Datafile_Root}","page":"API reference","title":"PerfTest.saveDataFile","text":"This method is used to save historical data of a performance test suite to a save file located in path.\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.saveExprAsFile","page":"API reference","title":"PerfTest.saveExprAsFile","text":"Utility to save an expression (expr) to a Julia file stored at path\n\nRequires a :toplevel symbol to be the head of the expression.\n\n\n\n\n\n","category":"function"},{"location":"api/#PerfTest.setupCPUPeakFlopBenchmark-Tuple{}","page":"API reference","title":"PerfTest.setupCPUPeakFlopBenchmark","text":"This method is used to generate the code responsible for sampling the maximum CPU FLOPS based on the avaiable threads in every resulting suite.\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.setupMemoryBandwidthBenchmark-Tuple{}","page":"API reference","title":"PerfTest.setupMemoryBandwidthBenchmark","text":"This method is used to generate the code responsible for sampling the maximum memory bandwith in every resulting suite.\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.testsetUpdate!-Tuple{PerfTest.Context, String}","page":"API reference","title":"PerfTest.testsetUpdate!","text":"Function used to register a new test set in the hierarchy record of the context, where name is the name of the test set.\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.treeRun-Tuple{AbstractString}","page":"API reference","title":"PerfTest.treeRun","text":"This method implements the transformation that converts a recipe script into a fully-fledged testing suite. The function will return a Julia expression with the resulting performance testing suite. This can be then executed or saved in a file for later usage.\n\nArguments\n\npath the path of the script to be transformed.\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.unblockAndConcat-Tuple{Vector{Expr}}","page":"API reference","title":"PerfTest.unblockAndConcat","text":"This function is useful to move expressions to the toplevel when they are enclosed inside a block\n\n\n\n\n\n","category":"method"},{"location":"api/#PerfTest.@auxiliary_metric-Tuple{Any, Any, Any}","page":"API reference","title":"PerfTest.@auxiliary_metric","text":"Defines a custom metric for informational purposes that will not be used for testing but will be printed as output.\n\n\n\n\n\n","category":"macro"},{"location":"api/#PerfTest.@define_eff_memory_throughput-Tuple{Any}","page":"API reference","title":"PerfTest.@define_eff_memory_throughput","text":"This macro is used to define the memory bandwidth of a target in order to execute the effective memory thorughput methodology.\n\nArguments\n\nformula block : an expression that returns a single value, which would be the metric value. The formula can have any julia expression inside and additionally some special symbols are supported. The formula may be evaluated several times, so its applied to every target in every test set or just once, if the formula is defined inside a test set, which makes it only applicable to it.\n\nSpecial symbols:\n\n:median_time : will be substituted by the median time the target took to execute in the benchmark.\n:minimum_time: will be substituted by the minimum time the target took to execute in the benchmark.\n:ret_value : will be substituted by the return value of the target.\n:autoflop: will be substituted by the FLOP count the target.\n:printed_output : will be substituted by the standard output stream of the target.\n:iterator : will be substituted by the current iterator value in a loop test set.\n\nExample:\n\nThe following definition assumes that each execution of the target expression involves transacting 1000 bytes. Therefore the bandwith is 1000 / execution time.\n\n@define_eff_memory_throughput begin\n 1000 / :median_time\nend\n\n\n\n\n\n","category":"macro"},{"location":"api/#PerfTest.@define_metric-Tuple{Any}","page":"API reference","title":"PerfTest.@define_metric","text":"This macro is used to define a new custom metric.\n\nArguments\n\nname : the name of the metric for identification purposes.\nunits : the unit space that the metric values will be in.\nformula block : an expression that returns a single value, which would be the metric value. The formula can have any julia expression inside and additionally some special symbols are supported. The formula may be evaluated several times, so its applied to every target in every test set or just once, if the formula is defined inside a test set, which makes it only applicable to it.\n\nSpecial symbols:\n\n:median_time : will be substituted by the median time the target took to execute in the benchmark.\n:minimum_time: will be substituted by the minimum time the target took to execute in the benchmark.\n:ret_value : will be substituted by the return value of the target.\n:autoflop: will be substituted by the FLOP count the target.\n:printed_output : will be substituted by the standard output stream of the target.\n:iterator : will be substituted by the current iterator value in a loop test set.\n\n\n\n\n\n","category":"macro"},{"location":"api/#PerfTest.@lpad-Tuple{Any}","page":"API reference","title":"PerfTest.@lpad","text":"Macro that adds a space at the beggining of a string\n\n\n\n\n\n","category":"macro"},{"location":"api/#PerfTest.@on_perftest_exec-Tuple{Any}","page":"API reference","title":"PerfTest.@on_perftest_exec","text":"The expression given to this macro will only be executed in the generated suite, and will be deleted if the source code is executed as is.\n\n\n\n\n\n","category":"macro"},{"location":"api/#PerfTest.@on_perftest_ignore-Tuple{Any}","page":"API reference","title":"PerfTest.@on_perftest_ignore","text":"The expression given to this macro will only be executed in the source code, and will be deleted in the generated performance test suite.\n\n\n\n\n\n","category":"macro"},{"location":"api/#PerfTest.@perftest-Tuple{Any}","page":"API reference","title":"PerfTest.@perftest","text":"This macro is used to signal that the wrapped expression is a performance test target, and therefore its performance will be sampled and then evaluated following the current suite configuration.\n\nIf the macro is evaluated it does not modify the target at all. The effects of the macro only show when the script is transformed into a performance testing suite.\n\nThis macro is sensitive to context since other adjacent macros can change how the target will be evaluated.\n\nArguments\n\nThe target expression\n\nExample\n\n@perftest 2 + 3\n\n\n\n\n\n","category":"macro"},{"location":"api/#PerfTest.@perftest_config-Tuple{Any}","page":"API reference","title":"PerfTest.@perftest_config","text":"Perftest_config macro, used to set customised configuration on the suite generated by the source script\n\nConfiguration inside this macro must follow the syntax below:\n\n@perftest_config\n key = value\n key.subkey = value\nend\n\nWhere key can be any configuration parameter, in some cases parameters will consist on a set of subparameters denoted by the \".\" to refer to them.\n\n\n\n\n\n","category":"macro"},{"location":"api/#PerfTest.@roofline","page":"API reference","title":"PerfTest.@roofline","text":"This macro enables roofline modelling, if put just before a target declaration (@perftest) it will proceed to evaluate it using a roofline model.\n\nMandatory arguments\n\nformula block: the macro has to wrap a block that holds a formula to obtain the operational intensity of target algorithms.\n\nOptional arguments\n\ncpu_peak : a manual input value for the maximum attainable FLOPS, this will override the empirical runtime benchmark\nmembw_peak : a manual input value for the maximum memory bandwith, this will override the empirical runtime benchmark\ntarget_opint : a desired operational intensity for the target, this will turn operational intensity into a test metric\nactual_flops: another formula that defines the actual performance of the test.\ntarget_ratio : the acceptable ratio between the actual performance and the projected performance from the roofline, this will turn actual performance into a test metric.\n\nSpecial symbols:\n\n:median_time : will be substituted by the median time the target took to execute in the benchmark.\n:minimum_time: will be substituted by the minimum time the target took to execute in the benchmark.\n:ret_value : will be substituted by the return value of the target.\n:autoflop: will be substituted by the FLOP count the target.\n:printed_output : will be substituted by the standard output stream of the target.\n:iterator : will be substituted by the current iterator value in a loop test set.\n\nAny formula block specified in this macro supports these symbols.\n\nExample\n\n@roofline actual_flops=:autoflop target_ratio=0.05 begin\n mem = ((:iterator + 1) * :iterator)\n :autoflop / mem\nend\n\nThe code block defines operational intensity, whilst the other arguments define how to measure and compare the actual performance with the roofline performance. If the actual to projected performance ratio goes below the target, the test fails.\n\n\n\n\n\n","category":"macro"},{"location":"usage/#Usage","page":"Usage","title":"Usage","text":"","category":"section"},{"location":"usage/","page":"Usage","title":"Usage","text":"Have a look at the Examples and see the API reference for details on the usage of PerfTest.","category":"page"},{"location":"usage/","page":"Usage","title":"Usage","text":"tip: Hint\nParallelStencil.jl enables straightforward working with CellArrays. It automatically allocates CellArray when the keyword arguments celldims or celltype are given to the architecture-agnostic allocation macros @zeros, @ones, @rand, @falses, @trues and @fill (refer to the documentation of ParallelStencil.jl for more details).","category":"page"},{"location":"usage/#Installation","page":"Usage","title":"Installation","text":"","category":"section"},{"location":"usage/","page":"Usage","title":"Usage","text":"PerfTest can be installed directly with the Julia package manager from the Julia REPL:","category":"page"},{"location":"usage/","page":"Usage","title":"Usage","text":"julia>]\n pkg> add PerfTest","category":"page"},{"location":"examples/#Examples","page":"...","title":"Examples","text":"","category":"section"},{"location":"examples/","page":"...","title":"...","text":"