Skip to content

Forward Slicing #1700

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 9 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 14 additions & 2 deletions src/core/steps/all/static-slicing/00-slice.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,18 +4,30 @@ import { PipelineStepStage } from '../../pipeline-step';
import type { DeepReadonly } from 'ts-essentials';
import type { DataflowInformation } from '../../../../dataflow/info';
import type { SlicingCriteria } from '../../../../slicing/criterion/parse';
import { staticSlicing } from '../../../../slicing/static/static-slicer';
import { staticBackwardSlice, staticForwardSlice } from '../../../../slicing/static/static-slicer';
import type { NormalizedAst } from '../../../../r-bridge/lang-4.x/ast/model/processing/decorate';

export interface SliceRequiredInput {
/** The slicing criterion is only of interest if you actually want to slice the R code */
readonly criterion: SlicingCriteria,
/** How many re-visits of the same node are ok? */
readonly threshold?: number
/** The direction to slice in. Defaults to backward slicing if unset. */
readonly direction?: SliceDirection
}

export enum SliceDirection {
Backward = 'backward',
Forward = 'forward'
}

function processor(results: { dataflow?: DataflowInformation, normalize?: NormalizedAst }, input: Partial<SliceRequiredInput>) {
return staticSlicing((results.dataflow as DataflowInformation).graph, results.normalize as NormalizedAst, input.criterion as SlicingCriteria, input.threshold);
switch(input.direction ?? SliceDirection.Backward) {
case SliceDirection.Backward:
return staticBackwardSlice((results.dataflow as DataflowInformation).graph, results.normalize as NormalizedAst, input.criterion as SlicingCriteria, input.threshold);
case SliceDirection.Forward:
return staticForwardSlice((results.dataflow as DataflowInformation).graph, results.normalize as NormalizedAst, input.criterion as SlicingCriteria, input.threshold);
}
}

export const STATIC_SLICE = {
Expand Down
14 changes: 14 additions & 0 deletions src/dataflow/graph/invert-dfg.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
import { DataflowGraph } from './graph';

export function invertDfg(graph: DataflowGraph): DataflowGraph {
const invertedGraph = new DataflowGraph(graph.idMap);
for(const [,v] of graph.vertices(true)) {
invertedGraph.addVertex(v);
}
for(const [from, targets] of graph.edges()) {
for(const [to, { types }] of targets) {
invertedGraph.addEdge(to, from, types);
}
}
return invertedGraph;
}
4 changes: 2 additions & 2 deletions src/documentation/print-core-wiki.ts
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ import { processAccess } from '../dataflow/internal/process/functions/call/built
import { processForLoop } from '../dataflow/internal/process/functions/call/built-in/built-in-for-loop';
import { processRepeatLoop } from '../dataflow/internal/process/functions/call/built-in/built-in-repeat-loop';
import { linkCircularRedefinitionsWithinALoop } from '../dataflow/internal/linker';
import { staticSlicing } from '../slicing/static/static-slicer';
import { staticBackwardSlice } from '../slicing/static/static-slicer';
import { filterOutLoopExitPoints, initializeCleanDataflowInformation } from '../dataflow/info';
import { processDataflowFor } from '../dataflow/processor';
import {
Expand Down Expand Up @@ -380,7 +380,7 @@ Of course, all of these endeavors work not just with the ${shortLink(RShell.name
The slicing is available as an extra step as you can see by inspecting he ${shortLink('DEFAULT_SLICING_PIPELINE', info)}.
Besides ${shortLink('STATIC_SLICE', info)} it contains a ${shortLink('NAIVE_RECONSTRUCT', info)} to print the slice as (executable) R code.

Your main point of interesting here is the ${shortLink(staticSlicing.name, info)} function which relies on a modified
Your main point of interesting here is the ${shortLink(staticBackwardSlice.name, info)} function which relies on a modified
breadth-first search to collect all nodes which are part of the slice.
For more information on how the slicing works, please refer to the [tool demonstration (Section 3.2)](https://doi.org/10.1145/3691620.3695359),
or the [original master's thesis (Chapter 4)](https://doi.org/10.18725/OPARU-50107).
Expand Down
Original file line number Diff line number Diff line change
@@ -1,10 +1,11 @@
import type { StaticSliceQuery, StaticSliceQueryResult } from './static-slice-query-format';
import { staticSlicing } from '../../../slicing/static/static-slicer';
import { staticBackwardSlice, staticForwardSlice } from '../../../slicing/static/static-slicer';
import { reconstructToCode } from '../../../reconstruct/reconstruct';
import { doNotAutoSelect } from '../../../reconstruct/auto-select/auto-select-defaults';
import { makeMagicCommentHandler } from '../../../reconstruct/auto-select/magic-comments';
import { log } from '../../../util/log';
import type { BasicQueryData } from '../../base-query-format';
import { SliceDirection } from '../../../core/steps/all/static-slicing/00-slice';

export function fingerPrintOfQuery(query: StaticSliceQuery): string {
return JSON.stringify(query);
Expand All @@ -20,7 +21,7 @@ export function executeStaticSliceQuery({ dataflow: { graph }, ast }: BasicQuery
}
const { criteria, noReconstruction, noMagicComments } = query;
const sliceStart = Date.now();
const slice = staticSlicing(graph, ast, criteria);
const slice = query.direction === SliceDirection.Forward ? staticForwardSlice(graph, ast, criteria) : staticBackwardSlice(graph, ast, criteria);
const sliceEnd = Date.now();
if(noReconstruction) {
results[key] = { slice: { ...slice, '.meta': { timing: sliceEnd - sliceStart } } };
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ import Joi from 'joi';
import { executeStaticSliceQuery } from './static-slice-query-executor';

import { summarizeIdsIfTooLong } from '../../query-print';
import { SliceDirection } from '../../../core/steps/all/static-slicing/00-slice';

/** Calculates and returns all clusters encountered in the dataflow graph. */
export interface StaticSliceQuery extends BaseQueryFormat {
Expand All @@ -22,6 +23,8 @@ export interface StaticSliceQuery extends BaseQueryFormat {
readonly noReconstruction?: boolean;
/** Should the magic comments (force-including lines within the slice) be ignored? */
readonly noMagicComments?: boolean
/** The direction to slice in. Defaults to backward slicing if unset. */
readonly direction?: SliceDirection
}

export interface StaticSliceQueryResult extends BaseQueryResult {
Expand Down Expand Up @@ -65,6 +68,7 @@ export const StaticSliceQueryDefinition = {
type: Joi.string().valid('static-slice').required().description('The type of the query.'),
criteria: Joi.array().items(Joi.string()).min(0).required().description('The slicing criteria to use.'),
noReconstruction: Joi.boolean().optional().description('Do not reconstruct the slice into readable code.'),
noMagicComments: Joi.boolean().optional().description('Should the magic comments (force-including lines within the slice) be ignored?')
noMagicComments: Joi.boolean().optional().description('Should the magic comments (force-including lines within the slice) be ignored?'),
direction: Joi.string().valid(...Object.values(SliceDirection)).optional().description('The direction to slice in. Defaults to backward slicing if unset.')
}).description('Slice query used to slice the dataflow graph')
} as const satisfies SupportedQuery<'static-slice'>;
14 changes: 13 additions & 1 deletion src/slicing/static/static-slicer.ts
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ import type { NodeId } from '../../r-bridge/lang-4.x/ast/model/processing/node-i
import { VertexType } from '../../dataflow/graph/vertex';
import { shouldTraverseEdge, TraverseEdge } from '../../dataflow/graph/edge';
import { getConfig } from '../../config';
import { invertDfg } from '../../dataflow/graph/invert-dfg';

export const slicerLogger = log.getSubLogger({ name: 'slicer' });

Expand All @@ -29,7 +30,7 @@ export const slicerLogger = log.getSubLogger({ name: 'slicer' });
* @param threshold - The maximum number of nodes to visit in the graph. If the threshold is reached, the slice will side with inclusion and drop its minimal guarantee. The limit ensures that the algorithm halts.
* @param cache - A cache to store the results of the slice. If provided, the slice may use this cache to speed up the slicing process.
*/
export function staticSlicing(
export function staticBackwardSlice(
graph: DataflowGraph,
{ idMap }: NormalizedAst,
criteria: SlicingCriteria,
Expand Down Expand Up @@ -142,3 +143,14 @@ export function updatePotentialAddition(queue: VisitingQueue, id: NodeId, target
}]);
}
}

export function staticForwardSlice(
graph: DataflowGraph,
ast: NormalizedAst,
criteria: SlicingCriteria,
threshold = getConfig().solver.slicer?.threshold ?? 75,
cache?: Map<Fingerprint, Set<NodeId>>
): Readonly<SliceResult> {
const useGraph = invertDfg(graph);
return staticBackwardSlice(useGraph, ast, criteria, threshold, cache);
}
106 changes: 56 additions & 50 deletions test/functionality/_helper/shell.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,24 +4,16 @@ import { NAIVE_RECONSTRUCT } from '../../../src/core/steps/all/static-slicing/10
import { guard, isNotUndefined } from '../../../src/util/assert';
import { PipelineExecutor } from '../../../src/core/pipeline-executor';
import type { TestLabel, TestLabelContext } from './label';
import { dropTestLabel , modifyLabelName , decorateLabelContext } from './label';
import { decorateLabelContext, dropTestLabel, modifyLabelName } from './label';
import { printAsBuilder } from './dataflow/dataflow-builder-printer';
import { RShell } from '../../../src/r-bridge/shell';
import type { NoInfo, RNode } from '../../../src/r-bridge/lang-4.x/ast/model/model';
import type { fileProtocol, RParseRequests } from '../../../src/r-bridge/retriever';
import { requestFromInput } from '../../../src/r-bridge/retriever';
import type {
AstIdMap, IdGenerator, NormalizedAst,
RNodeWithParent
} from '../../../src/r-bridge/lang-4.x/ast/model/processing/decorate';
import {
deterministicCountingIdGenerator
} from '../../../src/r-bridge/lang-4.x/ast/model/processing/decorate';
import { TREE_SITTER_SLICE_AND_RECONSTRUCT_PIPELINE,
DEFAULT_SLICE_AND_RECONSTRUCT_PIPELINE,
DEFAULT_DATAFLOW_PIPELINE,
DEFAULT_NORMALIZE_PIPELINE, TREE_SITTER_NORMALIZE_PIPELINE
} from '../../../src/core/steps/pipeline/default-pipelines';
import type { AstIdMap, IdGenerator, NormalizedAst, RNodeWithParent } from '../../../src/r-bridge/lang-4.x/ast/model/processing/decorate';
import { deterministicCountingIdGenerator } from '../../../src/r-bridge/lang-4.x/ast/model/processing/decorate';
import type { DEFAULT_SLICE_AND_RECONSTRUCT_PIPELINE, TREE_SITTER_SLICE_AND_RECONSTRUCT_PIPELINE } from '../../../src/core/steps/pipeline/default-pipelines';
import { createSlicePipeline, DEFAULT_DATAFLOW_PIPELINE, DEFAULT_NORMALIZE_PIPELINE, TREE_SITTER_NORMALIZE_PIPELINE } from '../../../src/core/steps/pipeline/default-pipelines';
import type { RExpressionList } from '../../../src/r-bridge/lang-4.x/ast/model/nodes/r-expression-list';
import { diffOfDataflowGraphs } from '../../../src/dataflow/graph/diff-dataflow-graph';
import type { NodeId } from '../../../src/r-bridge/lang-4.x/ast/model/processing/node-id';
Expand All @@ -31,10 +23,10 @@ import type { SlicingCriteria } from '../../../src/slicing/criterion/parse';
import { normalizedAstToMermaidUrl } from '../../../src/util/mermaid/ast';
import type { AutoSelectPredicate } from '../../../src/reconstruct/auto-select/auto-select-defaults';
import { resolveDataflowGraph } from '../../../src/dataflow/graph/resolve-graph';
import { assert, test, describe, afterAll, beforeAll } from 'vitest';
import { afterAll, assert, beforeAll, describe, test } from 'vitest';
import semver from 'semver/preload';
import { TreeSitterExecutor } from '../../../src/r-bridge/lang-4.x/tree-sitter/tree-sitter-executor';
import type { PipelineOutput , Pipeline } from '../../../src/core/steps/pipeline/pipeline';
import type { Pipeline, PipelineOutput } from '../../../src/core/steps/pipeline/pipeline';
import type { FlowrSearchLike } from '../../../src/search/flowr-search-builder';
import { runSearch } from '../../../src/search/flowr-search-executor';
import type { ContainerIndex } from '../../../src/dataflow/graph/vertex';
Expand All @@ -46,6 +38,8 @@ import { extractCFG } from '../../../src/control-flow/extract-cfg';
import { cfgToMermaidUrl } from '../../../src/util/mermaid/cfg';
import type { CfgProperty } from '../../../src/control-flow/cfg-properties';
import { assertCfgSatisfiesProperties } from '../../../src/control-flow/cfg-properties';
import type { KnownParser } from '../../../src/r-bridge/parser';
import { SliceDirection } from '../../../src/core/steps/all/static-slicing/00-slice';

export const testWithShell = (msg: string, fn: (shell: RShell, test: unknown) => void | Promise<void>) => {
return test(msg, async function(this: unknown): Promise<void> {
Expand Down Expand Up @@ -229,6 +223,7 @@ export function assertAst(name: TestLabel | string, shell: RShell, input: string
}
// the ternary operator is to support the legacy way I wrote these tests - by mirroring the input within the name
return describe.skipIf(skip)(`${decorateLabelContext(name, labelContext)} (input: ${input})`, () => {
const ts = !skipTreeSitter ? new TreeSitterExecutor() : undefined;
let shellAst: RNode | undefined;
let tsAst: RNode | undefined;
beforeAll(async() => {
Expand All @@ -237,6 +232,7 @@ export function assertAst(name: TestLabel | string, shell: RShell, input: string
tsAst = await makeTsAst();
}
});
afterAll(() => ts?.close());
test('shell', function() {
assertAstEqual(shellAst as RNode, expected, !userConfig?.ignoreAdditionalTokens, userConfig?.ignoreColumns === true,
() => `got: ${JSON.stringify(shellAst)}, vs. expected: ${JSON.stringify(expected)}`);
Expand All @@ -250,25 +246,25 @@ export function assertAst(name: TestLabel | string, shell: RShell, input: string
assertAstEqual(tsAst as RNode, shellAst as RNode, true, userConfig?.ignoreColumns === true,
() => `tree-sitter ast: ${JSON.stringify(tsAst)}, vs. shell ast: ${JSON.stringify(shellAst)}`, false);
});
});

async function makeShellAst(): Promise<RNode> {
const pipeline = new PipelineExecutor(DEFAULT_NORMALIZE_PIPELINE, {
parser: shell,
request: requestFromInput(input)
});
const result = await pipeline.allRemainingSteps();
return result.normalize.ast;
}
async function makeShellAst(): Promise<RNode> {
const pipeline = new PipelineExecutor(DEFAULT_NORMALIZE_PIPELINE, {
parser: shell,
request: requestFromInput(input)
});
const result = await pipeline.allRemainingSteps();
return result.normalize.ast;
}

async function makeTsAst(): Promise<RNode> {
const pipeline = new PipelineExecutor(TREE_SITTER_NORMALIZE_PIPELINE, {
parser: new TreeSitterExecutor(),
request: requestFromInput(input)
});
const result = await pipeline.allRemainingSteps();
return result.normalize.ast;
}
async function makeTsAst(): Promise<RNode> {
const pipeline = new PipelineExecutor(TREE_SITTER_NORMALIZE_PIPELINE, {
parser: ts as TreeSitterExecutor,
request: requestFromInput(input)
});
const result = await pipeline.allRemainingSteps();
return result.normalize.ast;
}
});
}

/** call within describeSession */
Expand Down Expand Up @@ -442,15 +438,28 @@ function testWrapper(skip: boolean | undefined, shouldFail: boolean, testName: s

export type TestCaseFailType = 'fail-shell' | 'fail-tree-sitter' | 'fail-both' | undefined;

export function assertSlicedF(
name: TestLabel,
shell: RShell,
input: string,
criteria: SlicingCriteria,
expected: string,
userConfig?: Partial<TestConfigurationWithOutput> & { autoSelectIf?: AutoSelectPredicate, skipTreeSitter?: boolean, skipCompare?: boolean, cfgExcludeProperties?: readonly CfgProperty[], sliceDirection?: SliceDirection },
testCaseFailType?: TestCaseFailType,
getId: () => IdGenerator<NoInfo> = () => deterministicCountingIdGenerator(0)
) {
return assertSliced(name, shell, input, criteria, expected, { ...userConfig, sliceDirection: SliceDirection.Forward }, testCaseFailType, getId);
}

export function assertSliced(
name: TestLabel,
shell: RShell,
input: string,
criteria: SlicingCriteria,
expected: string,
userConfig?: Partial<TestConfigurationWithOutput> & { autoSelectIf?: AutoSelectPredicate, skipTreeSitter?: boolean, skipCompare?: boolean, cfgExcludeProperties?: readonly CfgProperty[] },
userConfig?: Partial<TestConfigurationWithOutput> & { autoSelectIf?: AutoSelectPredicate, skipTreeSitter?: boolean, skipCompare?: boolean, cfgExcludeProperties?: readonly CfgProperty[], sliceDirection?: SliceDirection },
testCaseFailType?: TestCaseFailType,
getId: () => IdGenerator<NoInfo> = () => deterministicCountingIdGenerator(0),
getId: () => IdGenerator<NoInfo> = () => deterministicCountingIdGenerator(0)
) {
const fullname = `${JSON.stringify(criteria)} ${decorateLabelContext(name, ['slice'])}`;
const skip = skipTestBecauseConfigNotMet(userConfig);
Expand All @@ -459,34 +468,23 @@ export function assertSliced(
dropTestLabel(name);
}
describe.skipIf(skip)(fullname, () => {
const ts = !userConfig?.skipTreeSitter ? new TreeSitterExecutor() : undefined;
let shellResult: PipelineOutput<typeof DEFAULT_SLICE_AND_RECONSTRUCT_PIPELINE> | undefined;
let tsResult: PipelineOutput<typeof TREE_SITTER_SLICE_AND_RECONSTRUCT_PIPELINE> | undefined;
beforeAll(async() => {
shellResult = await new PipelineExecutor(DEFAULT_SLICE_AND_RECONSTRUCT_PIPELINE, {
getId: getId(),
request: requestFromInput(input),
parser: shell,
criterion: criteria,
autoSelectIf: userConfig?.autoSelectIf,
}).allRemainingSteps();
shellResult = await executePipeline(shell);
if(!userConfig?.skipTreeSitter) {
tsResult = await new PipelineExecutor(TREE_SITTER_SLICE_AND_RECONSTRUCT_PIPELINE, {
getId: getId(),
request: requestFromInput(input),
parser: new TreeSitterExecutor(),
criterion: criteria,
autoSelectIf: userConfig?.autoSelectIf
}).allRemainingSteps();
tsResult = await executePipeline(ts as TreeSitterExecutor);
}
});
afterAll(() => ts?.close());

testWrapper(
false,
testCaseFailType === 'fail-both' || testCaseFailType === 'fail-shell',
'shell',
() => testSlice(shellResult as PipelineOutput<typeof DEFAULT_SLICE_AND_RECONSTRUCT_PIPELINE>, testCaseFailType !== 'fail-both' && testCaseFailType !== 'fail-shell'),
);

testWrapper(
userConfig?.skipTreeSitter,
testCaseFailType === 'fail-both' || testCaseFailType === 'fail-tree-sitter',
Expand Down Expand Up @@ -524,6 +522,15 @@ export function assertSliced(
});
handleAssertOutput(name, shell, input, userConfig);

async function executePipeline(parser: KnownParser): Promise<PipelineOutput<typeof DEFAULT_SLICE_AND_RECONSTRUCT_PIPELINE | typeof TREE_SITTER_SLICE_AND_RECONSTRUCT_PIPELINE>> {
return await createSlicePipeline(parser, {
getId: getId(),
request: requestFromInput(input),
criterion: criteria,
autoSelectIf: userConfig?.autoSelectIf,
direction: userConfig?.sliceDirection
}).allRemainingSteps();
}
function testSlice(result: PipelineOutput<typeof DEFAULT_SLICE_AND_RECONSTRUCT_PIPELINE | typeof TREE_SITTER_SLICE_AND_RECONSTRUCT_PIPELINE>, printError: boolean) {
try {
assert.strictEqual(
Expand All @@ -538,7 +545,6 @@ export function assertSliced(
throw e;
} /* v8 ignore stop */
}
handleAssertOutput(name, shell, input, userConfig);
}

function findInDfg(id: NodeId, dfg: DataflowGraph): ContainerIndex[] | undefined {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,16 +2,16 @@ import { afterAll, assert, beforeAll, describe, test } from 'vitest';
import {
findSource,
setSourceProvider
} from '../../../../src/dataflow/internal/process/functions/call/built-in/built-in-source';
import type { RParseRequest } from '../../../../src/r-bridge/retriever';
import { requestProviderFromFile, requestProviderFromText } from '../../../../src/r-bridge/retriever';
} from '../../../../../src/dataflow/internal/process/functions/call/built-in/built-in-source';
import type { RParseRequest } from '../../../../../src/r-bridge/retriever';
import { requestProviderFromFile, requestProviderFromText } from '../../../../../src/r-bridge/retriever';
import {
amendConfig,
defaultConfigOptions,
DropPathsOption,
InferWorkingDirectory,
setConfig
} from '../../../../src/config';
} from '../../../../../src/config';
import path from 'path';

describe('source finding', () => {
Expand Down
Loading