End To End Test Suite extended

jplag.endToEndTesting/README.md

@@ -2,18 +2,17 @@
 With the help of the end-to-end module, changes to the detection of JPlag are to be tested.
 With the help of elaborated plagiarisms, which have been worked out from suggestions in the literature on the topic of "plagiarism detection and avoidance", a wide range of detectable change can be covered. The selected plagiarisms are the decisive factor here as to whether a change in recognition can be perceived. 
 
-### References
+## References
 These elaborations provide basic ideas on how a modification of the plagiarized source code can look like or be adapted.
 These code adaptations refer to a wide range of changes starting from
 adding/removing comments to architectural changes in the deliverables.
 
 The following elaborations were used to be able to create the plagiarisms with the largest coverage:
-- [Mossad: defeating software plagiarism detection](https://dl.acm.org/doi/abs/10.1145/3428206 "Mossad: defeating software plagiarism detection")
 - [Detecting Source Code Plagiarism on Introductory Programming Course Assignments Using a Bytecode Approach - Oscar Karnalim](https://ieeexplore.ieee.org/abstract/document/7910274 "Detecting Source Code Plagiarism on Introductory Programming Course Assignments Using a Bytecode Approach - Oscar Karnalim")
 - [Detecting Disguised Plagiarism - Hatem A. Mahmoud](https://arxiv.org/abs/1711.02149 "Detecting Disguised Plagiarism - Hatem A. Mahmoud")
 - [Instructor-centric source code plagiarism detection and plagiarism corpus](https://dl.acm.org/doi/abs/10.1145/2325296.2325328 "Instructor-centric source code plagiarism detection and plagiarism corpus")
 
-### Steps Towards Plagiarism
+## Steps Towards Plagiarism
 The following changes were applied to sample tasks to create test cases:
 <ul type="1">
 	<li>Inserting comments or empty lines (normalization level)</li>
@@ -47,59 +46,129 @@ More detailed information about the create as well as about the subject to the i
 
 Software is according to [§ 2 of the copyright law](https://www.gesetze-im-internet.de/urhg/__2.html "§ 2 of the copyright law") a protected work which may not be plagiarized. 
 
-### JPlag - End To End TestSuite Structure
+## JPlag - End To End TestSuite Structure
 The construction of an end to end test is done with the help of the JPlag api. 
+The tests are generated dynamically according to the existing test data and allow the creation of endToEnd tests for all supported languages of JPlag without having to make any changes to the code.
+The helper loads the existing test data from the designated directory and creates dynamic tests for the individual directories. It is therefore possible to create different test classes for the different languages.
+- JPlagTestSuiteHelper:
 
 ``` java 
-[...]
-    /**
-     * This method creates the necessary results as well as models for a test run and summarizes them for a comparison.
-     * @param testClassNames Plagiarized classes names in the resource directorie which are needed for the test
-     * @param testIdentifier name of the testId to load and identify the stored results
-     * @throws IOException is thrown in case of problems with copying the plagiarism classes
-     * @throws ExitException in case the plagiarism detection with JPlag is preemptively terminated would be of the test.
-     */
-    private void runJPlagTestSuite(String[] testClassNames, int testIdentifier) throws IOException, ExitException {
-        String functionName = StackWalker.getInstance().walk(stream -> stream.skip(1).findFirst().get()).getMethodName();
-        TestCaseModel testCaseModel = jplagTestSuiteHelper.createNewTestCase(testClassNames, functionName);
-        JPlagResult jplagResult = new JPlag(testCaseModel.getJPlagOptionsFromCurrentModel()).run();
-
-        for (JPlagComparison jPlagComparison : jplagResult.getAllComparisons()) {
-            assertEquals(testCaseModel.getCurrentJsonModel().getResultModelById(testIdentifier).getResultSimilarity(), jPlagComparison.similarity(),
-                    "The JPlag results [similarity] do not match the stored values!");
-        }
-    }
-[...]
+public static Map<LanguageOption, Map<String, Path>> getAllLanguageResources()
+```
+
+The list of languages created in this way and their associated data are dynamically generated into test cases in the Test Suite.
+
+``` java
+ Collection<DynamicTest> dynamicOverAllTest()
 ```
-The created plagiarisms are copied with the JPlagTestSuiteHelper into temporary directories, which can then be checked with JPlag. 
 
 In order to be able to distinguish in which domain of the recognition changes have occurred, fine granular test cases are used. These are composed of the changes already mentioned above. The plagiarism is compared with the original delivery and thus it is possible to detect and test small sections of the recognition. 
 
-The values compared so far to detect a match are limited to the similarity of the JPlag scan. It is already planned to expand the comparative values in order to have more indications of a change. 
+The comparative values were discussed and tested. The following results of the JPlag scan are used for the comparison:
+1. minimal similarity as `float`
+2. maximum similarity as `float`
+3. matched token numbe as `int`
 
-The Current Discussion is already taking place and can be found at [End to end testing - "comparative values"](https://github.com/jplag/JPlag/issues/548 "End to end testing - \"comparative values\""). 
+The comparative values were disscussed and elaborated in the issue [End to end testing - "comparative values"](https://github.com/jplag/JPlag/issues/548 "End to end testing - \"comparative values\""). 
+
+Additionally it is possible to create several options for the test data. More information about the test options can be found at [JPlag - option variants for the endToEnd tests #590](https://github.com/jplag/JPlag/issues/590 "JPlag - option variants for the endToEnd tests #590"). Currently, various settings are supported by the `minimumTokenMatch`.  This can be extended as desired in the record class `Options`.
 
 The current JPlag scans will be compared with the stored ones.
-This was done by storing the data in a *.json file which is read at the beginning of each test run. 
+This was done by storing the data in a *.json file which is read at the beginning of each test run.
 
 ``` json  
-{
 [...]
-    "function_name": "normalizationLevelTest",
-    "test_results": [
-      {
-        "result_similarity": 100,
-        "test_identifier": 0
-      },
-      {
-        "result_similarity": 100,
-        "test_identifier": 1
-      },
-      {
-        "result_similarity": 100,
-        "test_identifier": 2
-      }
-    ]
-  }
+{
+  "options" : {
+    "minimum_token_match" : 1
+  },
+  "tests" : {
+    "SortAlgo-SortAlgo5" : {
+      "minimal_similarity" : 82.14286,
+      "maximum_similarity" : 82.14286,
+      "matched_token_number" : 46
+    },
+    "SortAlgo-SortAlgo6" : {
+      "minimal_similarity" : 83.58209,
+      "maximum_similarity" : 100.0,
+      "matched_token_number" : 56
+    },
+    "SortAlgo-SortAlgo7" : {
+      "minimal_similarity" : 96.42857,
+      "maximum_similarity" : 100.0,
+      "matched_token_number" : 54
+    },
 [...]
 ```
+--- 
+
+## Create New Language End To End Tests
+
+This section explains how to create new end to end tests in the existing test suite. 
+### Creating The Plagiarism
+Before you add a new language to the end to end tests i would like to point out that the quality of the tests depends dreadfully on the plagiarism techniques you choose wicht were explaint in sechtion [Steps Towards Plagiarism](#steps-towards-plagiarism).
+If you need more information about the creation of plans for this purpose, you can also read the elaborations that can be found under [References](#references).
+The more and varied changes you apply, the more accurate the end-to-end tests for the language will be.
+
+In the following an example is shown which is in the JavaEndToEnd tests and is used.
+
+**Changing control structures for(…) to while(…):**
+
+``` java
+//base class
+public class SortAlgo {
+//[...]
+public void BubbleSortWithoutRecursion(Integer arr[]) {
+		for(int i = arr.length; i > 1 ; i--) {
+			for(int innerCounter = 0; innerCounter < arr.length-1; innerCounter++)
+			{
+				if (arr[innerCounter] > arr[innerCounter + 1]) {
+					swap(arr, innerCounter, (innerCounter + 1));
+				}
+			}
+		}
+	}
+//[...]
+}
+```
+
+``` java
+//created plagiarism
+public class SortAlgo5{
+//[...]
+public void BubbleSortWithoutRecursion(Integer arr[]) {
+		int i = arr.length;
+		while(i > 1)
+		{
+			int innerCounter = 0;
+			while(innerCounter < arr.length -1)
+			{
+				if (arr[innerCounter] > arr[innerCounter + 1]) {
+					swap(arr, innerCounter, (innerCounter + 1));
+				}
+				innerCounter++;
+			}
+			i--;
+		}
+	}
+//[...]
+}
+```
+### Copying Plagiarism To The Resources
+
+The plagiarisms created in [Creating The Plagiarism](#creating-the-plagiarism) must now be copied to the corresponding resources folder. It is important not to mix the languages of the plagiarisms or to copy the data into bottle resource paths.
+
+- At the path `JPlag\jplag.endToEndTesting\src\test\resources\languageTestFiles` a new folder for the language should be created if it does not already exist. For example `[...]\resources\languageTestFiles\JAVA`. If you have plagiarized several different code samples, you can also create additional subfolders under the newly created folder for example `[...]\resources\languageTestFiles\JAVA\sortAlgo`.
+
+It is important to note that the resource folder name must be exactly the same as the language identifier name in JPlag/Language. Otherwise the language option cannot be parsed correctly to the enum-type.
+ - c++ with "cpp"
+ - c# with "csharp"
+ - GO with "go"
+ - Java with "java"
+ - Kotlin with "kotlin"
+ - Python3 with "python3"
+ - R with "rlang"
+ - Rust with "rust"
+ - Scala with "scala"
+
+Once the tests have been run for the first time, the information for the tests is stored in the folder `..\target\testing-directory-submission\LANGUAGE`.  This data can be copied to the path `[...]\resources\results\LANGUAGE`. Each subdirectory gets its own result json file as `[...]\resources\results\JAVA\sortAlgo.json`. Once the test data has been copied, the endToEnd tests can be successfully tested. As soon as a change in the detection takes place, the results will differ from the stored results and the tests will fail if the results have changed.

jplag.endToEndTesting/src/main/java/de/jplag/end_to_end_testing/constants/TestDirectoryConstants.java

@@ -12,21 +12,24 @@ private TestDirectoryConstants() {
     }
 
     /**
-     * Name for the folder to copy the submissions to in order to test them with JPlag
+     * Create the complete path to the temporary result files. Here the temporary system path is extended with the
+     * "RESULT_DIRECTORY_NAME", which is predefined in this class.
      */
-    public static final String TEMPORARY_DIRECTORY_NAME = "testing-directory-submission";
+    public static final Path TEMPORARY_RESULT_DIRECTORY_NAME = Path.of("target", "testing-directory-temporary-result");
+
     /**
-     * Create the complete path to the submission files. Here the temporary system path is extended with the
-     * "TEMPORARY_DIRECTORY_NAME", which is predefined in this class.
+     * Base path to the saved results
      */
-    public static final String TEMPORARY_SUBMISSION_DIRECTORY_NAME = Path.of("target", TEMPORARY_DIRECTORY_NAME).toString();
+    public static final Path BASE_PATH_TO_RESULT_JSON = Path.of("src", "test", "resources", "results");
 
     /**
-     * Base path to the created plagiarism and the main file located in the project resources.
+     * Base path to the endToEnd testing resources
      */
-    public static final Path BASE_PATH_TO_JAVA_RESOURCES_SORTALGO = Path.of("src", "test", "resources", "java", "sortAlgo");
+    public static final Path BASE_PATH_TO_LANGUAGE_RESOURCES = Path.of("src", "test", "resources", "languageTestFiles");
+
     /**
-     * Base path to the saved results of the previous tests in a *.json file
+     * Create the complete path to the submission files. Here the temporary system path is extended with the
+     * "SUBMISSION_DIRECTORY_NAME", which is predefined in this class.
      */
-    public static final Path BASE_PATH_TO_JAVA_RESULT_JSON = Path.of("src", "test", "resources", "results", "JavaResult.json");
+    public static final Path TEMPORARY_SUBMISSION_DIRECTORY_NAME = Path.of("target", "testing-directory-submission");
 }

jplag.endToEndTesting/src/main/java/de/jplag/end_to_end_testing/helper/FileHelper.java

@@ -0,0 +1,143 @@
+package de.jplag.end_to_end_testing.helper;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.StandardCopyOption;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+import java.util.Objects;
+import java.util.stream.Collectors;
+
+import de.jplag.end_to_end_testing.constants.TestDirectoryConstants;
+
+/**
+ * Helper class to perform all necessary operations or functions on files or folders.
+ */
+public class FileHelper {
+
+    private FileHelper() {
+        // private constructor to prevent instantiation
+    }
+
+    /**
+     * Merges all contained filenames together without extension
+     * @param files whose names are to be merged
+     * @return merged filenames
+     */
+    public static String getEnclosedFileNamesFromCollection(Collection<File> files) {
+
+        return files.stream().map(File::getName).map(fileName -> fileName.substring(0, fileName.lastIndexOf('.'))).collect(Collectors.joining());
+    }
+
+    /**
+     * Load all possible languages in resource path
+     * @param directoryNames folder names for which the language options should be listed.
+     * @return list of all LanguageOptions included in the resource path
+     */
+    public static List<String> getLanguageOptionsFromPath(String[] directoryNames) {
+        return Arrays.stream(directoryNames).map(language -> language).filter(Objects::nonNull).toList();
+    }
+
+    /**
+     * @param directorieRoot path from which all folders should be loaded
+     * @return all folders found in the specified path
+     */
+    public static String[] getAllDirectoriesInPath(Path directorieRoot) {
+        return directorieRoot.toFile().list((dir, name) -> new File(dir, name).isDirectory());
+    }
+
+    /**
+     * Copies the passed filenames to a temporary path to use them in the tests
+     * @param classNames for which the test case is to be created
+     * @return paths created to the test submissions
+     * @throws IOException Exception can be thrown in cases that involve reading, copying or locating files.
+     */
+    public static String[] createNewTestCaseDirectory(String[] classNames) throws IOException {
+        // Copy the resources data to the temporary path
+        String[] returnSubmissionPath = new String[classNames.length];
+        for (int counter = 0; counter < classNames.length; counter++) {
+            Path originalPath = Path.of(classNames[counter]);
+            returnSubmissionPath[counter] = Path
+                    .of(TestDirectoryConstants.TEMPORARY_SUBMISSION_DIRECTORY_NAME.toString(), "submission" + (counter + 1)).toAbsolutePath()
+                    .toString();
+            Path copyPath = Path.of(TestDirectoryConstants.TEMPORARY_SUBMISSION_DIRECTORY_NAME.toString(), "submission" + (counter + 1),
+                    originalPath.getFileName().toString());
+
+            File directory = new File(copyPath.toString());
+            if (!directory.exists()) {
+                directory.mkdirs();
+            }
+            Files.copy(originalPath, copyPath, StandardCopyOption.REPLACE_EXISTING);
+        }
+        return returnSubmissionPath;
+    }
+
+    /**
+     * Delete directory with including files
+     * @param folder Path to a folder or file to be deleted. This happens recursively to the path
+     * @throws IOException if an I/O error occurs
+     */
+    public static void deleteCopiedFiles(File folder) throws IOException {
+        if (!folder.exists()) {
+            return;
+        }
+        File[] files = folder.listFiles();
+        if (files == null) { // some JVMs return null for empty dirs
+            return;
+        }
+        for (File file : files) {
+            if (file.isDirectory()) {
+                deleteCopiedFiles(file);
+            } else {
+                Files.delete(file.toPath());
+            }
+        }
+        Files.delete(folder.toPath());
+    }
+
+    /**
+     * Creates directory if it dose not exist
+     * @param directory to be created
+     * @throws IOException if the directory could not be created
+     */
+    public static void createDirectoryIfItDoesNotExist(File directory) throws IOException {
+        if (!directory.exists() && !directory.mkdirs()) {
+            throw new IOException(createNewIOExceptionStringForFileOrFOlderCreation(directory));
+        }
+    }
+
+    /**
+     * Creates file if it dose not exist
+     * @param file to be created
+     * @throws IOException if the file could not be created
+     */
+    public static void createFileIfItDoesNotExist(File file) throws IOException {
+        if (!file.exists() && !file.createNewFile()) {
+            throw new IOException(createNewIOExceptionStringForFileOrFOlderCreation(file));
+        }
+    }
+
+    /**
+     * @param resourcenPaths list of paths that lead to test resources
+     * @return all filenames contained in the paths
+     */
+    public static String[] loadAllTestFileNames(Path resourcenPaths) {
+        var files = resourcenPaths.toFile().listFiles();
+        String[] fileNames = new String[files.length];
+        for (int i = 0; i < files.length; i++) {
+            fileNames[i] = files[i].getName();
+        }
+        return fileNames;
+    }
+
+    /**
+     * @param file for which the exception text is to be created
+     * @return exception text for the specified file
+     */
+    private static String createNewIOExceptionStringForFileOrFOlderCreation(File file) {
+        return "The file/folder at the location [" + file.toString() + "] could not be created!";
+    }
+}