To merge changes from 6.6.1 into master (digital-preservation#907)

* [maven-release-plugin] prepare release droid-6.6.0-rc2 * [maven-release-plugin] prepare for next development iteration * CXF dependency update update due to vulnerability identification * New screenshots and a few corrections in the droid help (digital-preservation#864) * Fix for the problem of no profile output having changed unexpectedly (digital-preservation#890) * Fix for the problem of no profile output having changed unexpectedly since previous version. * Added fixes to result output to ProfileResourceNode toString * converted tabs to whitespaces to keep checkstyle happy * Removed file:/ from the windows tests. --------- Co-authored-by: Paul Wells <[email protected]> Co-authored-by: sparkhi <[email protected]> * Correct filename for bzip archive (digital-preservation#891) * bzip filename correctly identified * unit test added for the bzip prefix and some code improvements * Do not process archives by default * added image types so we can use in URI scheme rather than hardcoding the scheme * No profile mode should not expand archives by default * [maven-release-plugin] prepare release droid-6.6.0-rc3 * [maven-release-plugin] rollback the release of droid-6.6.0-rc3 * [maven-release-plugin] prepare release droid-6.6.0-rc3 * [maven-release-plugin] rollback the release of droid-6.6.0-rc3 * [maven-release-plugin] prepare release droid-6.6.0-rc3 * [maven-release-plugin] rollback the release of droid-6.6.0-rc3 * [maven-release-plugin] prepare release droid-6.6.0-rc3 * [maven-release-plugin] rollback the release of droid-6.6.0-rc3 * [maven-release-plugin] prepare release droid-6.6.0-rc3-A * [maven-release-plugin] prepare for next development iteration * cater for possible blank space in launch path of droid on Windows * [maven-release-plugin] prepare release droid-6.6.0 * [maven-release-plugin] prepare for next development iteration * issue-900 fix for hash column missing from the export (digital-preservation#901) * issue-900 fix for hash column missing from the export * fixed checkstyle violation errors * Adjusted alignment * issue-898 path separator slashes to work as per OS (digital-preservation#902) * issue-898 path separator slashes to work as per OS * issue-897 update bundled JRE for Windows version * issue-886 terminology corrected in preferences dialog * removed commented block of code * documentation updates for 6.6.1 (digital-preservation#903) * documentation updates for 6.6.1 * correction in the sentence phrasing * capitalisation of the product name * [maven-release-plugin] prepare release 6.6.1-rc * [maven-release-plugin] prepare for next development iteration * Added info about including filepath for files inside a container in... (digital-preservation#904) * Added info about including filepath for files inside a container in the csv export * changed terminology to use "archive" * [maven-release-plugin] prepare release droid-6.6.1-rc2 * [maven-release-plugin] prepare for next development iteration * [maven-release-plugin] prepare release 6.6.1 * [maven-release-plugin] prepare for next development iteration --------- Co-authored-by: paulwellsagilej <[email protected]> Co-authored-by: Paul Wells <[email protected]>
NEONScience · Mar 17, 2023 · f4f5f81 · f4f5f81
1 parent f78a437
commit f4f5f81
Show file tree

Hide file tree

Showing 27 changed files with 164 additions and 65 deletions.
diff --git a/droid-api/pom.xml b/droid-api/pom.xml
@@ -4,7 +4,7 @@
 	<parent>
 		<artifactId>droid-parent</artifactId>
 		<groupId>uk.gov.nationalarchives</groupId>
-		<version>6.6.1-SNAPSHOT</version>
+		<version>6.6.2-SNAPSHOT</version>
 		<relativePath>../droid-parent</relativePath>
 	</parent>
 

diff --git a/droid-binary/bin/ChangeLog b/droid-binary/bin/ChangeLog
@@ -1,11 +1,26 @@
-Version 6.6
-* Introduced internal API for making droid easier to use as a dependency
+Version 6.6.1
+* #898 Bugfix: Use backslash on Windows OS in no profile output
+* #900 Bugfix: Failure to consistently output HASH column in CSV
+* Updated to use latest version of Java 11 bundled JRE
+* #886 Bugfix: Terminology corrected on the preferences dialog
+
+Version 6.6.0
+* Updated the bundled JRE to a newer version
 * Improvement to ?? signature pattern
+* Ability to specify columns to export with the -co option, e.g. -co NAME PUID
+* Configure CSV to only quote if there is a comma in a field, use the -qc option
+* The GUI export dialog also includes column and quoting configuration options.
+* Write a profile straight to a CSV file, not writing to a database, using the -o output option, e.g. -o "/home/user/results.csv"
+* Write a profile straight to the console, by not specifying an output file or a profile to write to.
+* Ability to filter profiles written to CSV file or console just as for an export.
+* Ability to filter a more than one extensions, e.g. -F "file_ext any doc docx xls xlsx".
+* Filter files before they are identified on file metadata (name, extension, last modified, size), using the -ff and -FF filter options.
+* Set both binary and container signature files to use to override profile defaults.
+* Update profile defaults for both binary and container signature files
 * Fix for inverted syntax when the bytes or ranges are inverted
 * Introduced semantic versioning
 * Improved consistency across profile and no profile mode
-* Replaced iTextPdf with OpenHTMLToPDF to address vulnerability
-* Updated to use Java 11 as bundled JRE
+* Introduced internal API for making DROID easier to use as a dependency
 * Code refactoring
 * Bug fixes
 

diff --git a/droid-binary/pom.xml b/droid-binary/pom.xml
@@ -4,7 +4,7 @@
     <parent>
         <artifactId>droid-parent</artifactId>
         <groupId>uk.gov.nationalarchives</groupId>
-        <version>6.6.1-SNAPSHOT</version>
+        <version>6.6.2-SNAPSHOT</version>
   	    <relativePath>../droid-parent</relativePath>
     </parent>
 
@@ -35,7 +35,7 @@
                         </goals>
                         <configuration>
                             <!-- To update, retrieve link for windows jdk on x86 architecture from https://adoptium.net -->
-                            <url>https://github.com/adoptium/temurin11-binaries/releases/download/jdk-11.0.16.1%2B1/OpenJDK11U-jre_x86-32_windows_hotspot_11.0.16.1_1.zip</url>
+                            <url>https://github.com/adoptium/temurin11-binaries/releases/download/jdk-11.0.18%2B10/OpenJDK11U-jre_x86-32_windows_hotspot_11.0.18_10.zip</url>
                             <unpack>false</unpack>
                             <outputDirectory>${project.build.directory}</outputDirectory>
                         </configuration>
@@ -56,10 +56,9 @@
                         <configuration>
                             <tasks>
                                 <!-- Update binary from https://adoptopenjdk.net/releases.html -->
-                                <unzip src="${project.build.directory}/OpenJDK11U-jre_x86-32_windows_hotspot_11.0.16.1_1.zip" dest="${project.build.directory}/jre_tmp/" />
-
+                                <unzip src="${project.build.directory}/OpenJDK11U-jre_x86-32_windows_hotspot_11.0.18_10.zip" dest="${project.build.directory}/jre_tmp/" />
                                 <move todir="${project.build.directory}/jre-windows/">
-                                    <fileset dir="${project.build.directory}/jre_tmp/jdk-11.0.16.1+1-jre/">
+                                    <fileset dir="${project.build.directory}/jre_tmp/jdk-11.0.18+10-jre/">
                                         <include name="**/*" />
                                     </fileset>
                                 </move>

diff --git a/droid-build-tools/pom.xml b/droid-build-tools/pom.xml
@@ -4,7 +4,7 @@
     <parent>
         <artifactId>droid-parent</artifactId>
         <groupId>uk.gov.nationalarchives</groupId>
-        <version>6.6.1-SNAPSHOT</version>
+        <version>6.6.2-SNAPSHOT</version>
         <relativePath>../droid-parent</relativePath>
     </parent>
 

diff --git a/droid-command-line/pom.xml b/droid-command-line/pom.xml
@@ -4,7 +4,7 @@
     <parent>
         <artifactId>droid-parent</artifactId>
         <groupId>uk.gov.nationalarchives</groupId>
-        <version>6.6.1-SNAPSHOT</version>
+        <version>6.6.2-SNAPSHOT</version>
         <relativePath>../droid-parent</relativePath>
     </parent>
 

diff --git a/droid-container/pom.xml b/droid-container/pom.xml
@@ -4,7 +4,7 @@
     <parent>
         <artifactId>droid-parent</artifactId>
         <groupId>uk.gov.nationalarchives</groupId>
-        <version>6.6.1-SNAPSHOT</version>
+        <version>6.6.2-SNAPSHOT</version>
   	    <relativePath>../droid-parent</relativePath>
     </parent>
 

diff --git a/droid-core-interfaces/pom.xml b/droid-core-interfaces/pom.xml
@@ -4,7 +4,7 @@
     <parent>
         <artifactId>droid-parent</artifactId>
         <groupId>uk.gov.nationalarchives</groupId>
-        <version>6.6.1-SNAPSHOT</version>
+        <version>6.6.2-SNAPSHOT</version>
         <relativePath>../droid-parent</relativePath>
     </parent>
 

diff --git a/droid-core/pom.xml b/droid-core/pom.xml
@@ -4,7 +4,7 @@
     <parent>
         <artifactId>droid-parent</artifactId>
         <groupId>uk.gov.nationalarchives</groupId>
-        <version>6.6.1-SNAPSHOT</version>
+        <version>6.6.2-SNAPSHOT</version>
         <relativePath>../droid-parent</relativePath>
     </parent>
 

diff --git a/droid-export-interfaces/pom.xml b/droid-export-interfaces/pom.xml
@@ -4,7 +4,7 @@
     <parent>
         <artifactId>droid-parent</artifactId>
         <groupId>uk.gov.nationalarchives</groupId>
-        <version>6.6.1-SNAPSHOT</version>
+        <version>6.6.2-SNAPSHOT</version>
    <relativePath>../droid-parent</relativePath>
     </parent>
 

diff --git a/droid-export/pom.xml b/droid-export/pom.xml
@@ -4,7 +4,7 @@
     <parent>
         <artifactId>droid-parent</artifactId>
         <groupId>uk.gov.nationalarchives</groupId>
-        <version>6.6.1-SNAPSHOT</version>
+        <version>6.6.2-SNAPSHOT</version>
         <relativePath>../droid-parent</relativePath>
     </parent>
 

diff --git a/droid-help/pom.xml b/droid-help/pom.xml
@@ -4,7 +4,7 @@
     <parent>
         <artifactId>droid-parent</artifactId>
         <groupId>uk.gov.nationalarchives</groupId>
-        <version>6.6.1-SNAPSHOT</version>
+        <version>6.6.2-SNAPSHOT</version>
   	    <relativePath>../droid-parent</relativePath>
     </parent>
 

diff --git a/droid-help/src/main/resources/Images/Profile Defaults.png b/droid-help/src/main/resources/Images/Profile Defaults.png
diff --git a/droid-help/src/main/resources/Web pages/Exporting profiles.html b/droid-help/src/main/resources/Web pages/Exporting profiles.html
@@ -135,9 +135,9 @@ <h3>
     </h3>
     <p>
        The file system&nbsp;location of the resource being profiled, if the resource was directly
-      in a file system. &nbsp;Some files are not inside a file system - for example, files inside a
-      zip file. In this case, the file path will be blank, as there is no file path to the
-      resource. &nbsp;
+      in a file system. &nbsp;Some files are not on the file system - for example, files inside a
+      zip file. In this case, the file paths are written relative to the parent file which
+      exists on the file system. Such paths also include the type of archive as a prefix.
     </p>
     <p>
        Please note that file paths are platform dependent (they are different on Windows and unix).

diff --git a/droid-help/src/main/resources/Web pages/Whats New.html b/droid-help/src/main/resources/Web pages/Whats New.html
@@ -35,7 +35,7 @@
   <head>
     <meta http-equiv="Content-Type" content="text/html; charset=Cp1252">
     <meta name="generator" content="Helen">
-    <title>Whats New</title>
+    <title>What's New</title>
   </head>
   <body>
     <table width="100%" cellpadding="0" cellspacing="0" bgcolor="#555555" border="0">
@@ -64,18 +64,40 @@ <h5 align="center">
         </td>
       </tr>
     </table>
+    <p>
+      Droid 6.6.1 offers the following fixes and improvements:
+    </p>
+    <ul>
+      <li>#898 Bugfix: Use backslash on Windows OS in no profile output</li>
+      <li>#900 Bugfix: Failure to consistently output HASH column in CSV</li>
+      <li>Updated to use latest version of Java 11 bundled JRE</li>
+      <li>#886 Bugfix: Terminology corrected on the preferences dialog</li>    </ul>
     <p>
       Droid 6.6.0 offers the following new features:
     </p>
     <ul>
-      <li>Internal API for making droid easier to use as a dependency</li>
-      <li>Updated bundled jre to Java 11</li>
+      <li>Updated the bundled JRE to a newer version</li>
       <li>Improvement to ?? signature pattern</li>
+      <li>Ability to specify columns to export with the -co option, e.g. -co NAME PUID</li>
+      <li>Configure CSV to only quote if there is a comma in a field, use the -qc option</li>
+      <li>The GUI export dialog also includes column and quoting configuration options.</li>
+      <li>Write a profile straight to a CSV file, not writing to a database, using the -o output option, e.g. -o "/home/user/results.csv"</li>
+      <li>Write a profile straight to the console, by not specifying an output file or a profile to write to.</li>
+      <li>Ability to filter profiles written to CSV file or console just as for an export.</li>
+      <li>Ability to filter a more than one extensions, e.g. -F "file_ext any doc docx xls xlsx".</li>
+      <li>Filter files before they are identified on file metadata (name, extension, last modified, size), using the -ff and -FF filter options.</li>
+      <li>Set both binary and container signature files to use to override profile defaults.</li>
+      <li>Update profile defaults for both binary and container signature files</li>
       <li>Fix for inverted syntax when the bytes or ranges are inverted</li>
+      <li>Introduced semantic versioning</li>
       <li>Improved consistency across profile and no profile mode</li>
+      <li>File paths inside an archive are included in the export, consistent with the no-profile mode</li>
+      <li>Introduced internal API for making DROID easier to use as a dependency</li>
+      <li>Code refactoring</li>
+      <li>Bug fixes</li>
     </ul>
     <p>
-      For a full list of the included changes with this version, please check the <a href="http://www.github.com/digital-preservation/droid/releases/tag/droid-6.6/">changelog</a>
+      For a full list of the included changes with this version, please check the <a href="http://www.github.com/digital-preservation/droid/releases/tag/droid-6.6.0/">changelog</a>
       <img src="../Images/Icon_External_Link.png"> on github
     </p>
   </body>

diff --git a/droid-help/src/main/resources/index.xml b/droid-help/src/main/resources/index.xml
@@ -37,7 +37,7 @@
 
 <index version="2.0">
 	<indexitem text="Welcome to DROID" target="Welcome to DROID">
-		<indexitem text="Whats New" target="Whats New" />
+		<indexitem text="What's New" target="Whats New" />
 		<indexitem text="License" target="License" />
 		<indexitem text="Third-party components.html" target="Third-party components" />
 		<indexitem text="FAQ" target="FAQ" />

diff --git a/droid-help/src/main/resources/toc.xml b/droid-help/src/main/resources/toc.xml
@@ -37,7 +37,7 @@
 
 <toc version="2.0">
 	<tocitem text="Welcome to DROID" target="Welcome to DROID">
-		<tocitem text="Whats New" target="Whats New" />
+		<tocitem text="What's New" target="Whats New" />
 		<tocitem text="License" target="License" />
 		<tocitem text="Third-party components" target="Third-party components" />
 		<tocitem text="FAQ" target="FAQ" />

diff --git a/droid-parent/pom.xml b/droid-parent/pom.xml
@@ -11,7 +11,7 @@
 
     <groupId>uk.gov.nationalarchives</groupId>
     <artifactId>droid-parent</artifactId>
-    <version>6.6.1-SNAPSHOT</version>
+    <version>6.6.2-SNAPSHOT</version>
     <packaging>pom</packaging>
 
     <name>droid-parent</name>

diff --git a/droid-report-interfaces/pom.xml b/droid-report-interfaces/pom.xml
@@ -4,7 +4,7 @@
     <parent>
         <artifactId>droid-parent</artifactId>
         <groupId>uk.gov.nationalarchives</groupId>
-        <version>6.6.1-SNAPSHOT</version>
+        <version>6.6.2-SNAPSHOT</version>
         <relativePath>../droid-parent</relativePath>
     </parent>
 

diff --git a/droid-report/pom.xml b/droid-report/pom.xml
@@ -4,7 +4,7 @@
     <parent>
         <artifactId>droid-parent</artifactId>
         <groupId>uk.gov.nationalarchives</groupId>
-        <version>6.6.1-SNAPSHOT</version>
+        <version>6.6.2-SNAPSHOT</version>
         <relativePath>../droid-parent</relativePath>
     </parent>
 

diff --git a/droid-results/pom.xml b/droid-results/pom.xml
@@ -4,7 +4,7 @@
 	<parent>
 		<artifactId>droid-parent</artifactId>
 		<groupId>uk.gov.nationalarchives</groupId>
-		<version>6.6.1-SNAPSHOT</version>
+		<version>6.6.2-SNAPSHOT</version>
 		<relativePath>../droid-parent</relativePath>
 	</parent>
 

diff --git a/droid-results/src/main/java/uk/gov/nationalarchives/droid/profile/CsvItemWriter.java b/droid-results/src/main/java/uk/gov/nationalarchives/droid/profile/CsvItemWriter.java
@@ -46,7 +46,10 @@
 import uk.gov.nationalarchives.droid.export.interfaces.ItemWriter;
 import uk.gov.nationalarchives.droid.profile.referencedata.Format;
 
+import java.io.File;
 import java.io.Writer;
+import java.net.URI;
+import java.nio.file.Paths;
 import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.Date;
@@ -86,6 +89,8 @@ public class CsvItemWriter implements ItemWriter<ProfileResourceNode> {
             "FORMAT_VERSION",
     };
 
+    private static final String FILE_URI_SCHEME = "file";
+
     /*
      * Indexes of the headers used in the CSV output.
      */
@@ -107,6 +112,7 @@ public class CsvItemWriter implements ItemWriter<ProfileResourceNode> {
     private static final int MIME_TYPE_ARRAY_INDEX          = 15;
     private static final int FORMAT_NAME_ARRAY_INDEX        = 16;
     private static final int FORMAT_VERSION_ARRAY_INDEX     = 17;
+    private static final String BLANK_SPACE_DELIMITER       = " ";
 
     private final Logger log = LoggerFactory.getLogger(getClass());
 
@@ -212,7 +218,7 @@ public void open(final Writer writer) {
         csvWriterSettings.setFormat(format);
         csvWriter = new CsvWriter(writer, csvWriterSettings);
         if (headers == null) {
-            headers = HEADERS;
+            headers = Arrays.copyOf(HEADERS, HEADERS.length) ;
         }
         String[] headersToWrite = getHeadersToWrite(headers);
         csvWriter.writeHeaders(headersToWrite);
@@ -246,7 +252,29 @@ private static String nullSafeDate(Date date, FastDateFormat format) {
     private static String toFileName(String name) {
         return FilenameUtils.getName(name);
     }
-
+
+    private String toFilePath(URI uri) {
+        if (uri == null) {
+            log.warn("[URI not set]");
+            return "";
+        }
+        if (FILE_URI_SCHEME.equals(uri.getScheme())) {
+            return Paths.get(uri).toAbsolutePath().toString();
+        }
+
+        // for URIs that have other than "file" scheme
+        String result = java.net.URLDecoder.decode(uri.toString()).replaceAll("file://", "");
+        result = result.replace("/", File.separator);
+
+        // Handle substitution of 7z
+        final String sevenZedIdentifier = "sevenz:";
+        if (result.startsWith(sevenZedIdentifier)) {
+            result = "7z:" + result.substring(sevenZedIdentifier.length());
+        }
+
+        return result;
+    }
+
     /**
      * No config is needed by this class, but it's retained temporarily for backwards compatibility purposes.
      * @param config the config to set
@@ -261,9 +289,10 @@ public void setConfig(DroidGlobalConfig config) {
     public void setHeaders(Map<String, String> headersToSet) {
 
         if (this.headers == null) {
-            this.headers = HEADERS;
+            this.headers = Arrays.copyOf(HEADERS, HEADERS.length);
         }
 
+        // The header for hash is modified based on algorithm used to generate the hash
         String hashHeader = headersToSet.get("hash");
         if (hashHeader != null) {
             this.headers[HASH_ARRAY_INDEX] = hashHeader;
@@ -301,10 +330,7 @@ public void setColumnsToWrite(String columnNames) {
                 numColumnsToWrite = numberToWrite;
                 // If there are some columns specified left over, they aren't valid columns - log a warning:
                 if (headersToWrite.size() > 0) {
-                    String invalidHeaders = "";
-                    for (String invalidColumn : headersToWrite) {
-                        invalidHeaders = invalidHeaders + invalidColumn + ' ';
-                    }
+                    String invalidHeaders = String.join(BLANK_SPACE_DELIMITER, headersToWrite);
                     log.warn("-co option - some CSV columns specified were invalid: " + invalidHeaders);
                 }
             }
@@ -313,7 +339,7 @@ public void setColumnsToWrite(String columnNames) {
 
     private Set<String> getColumnsToWrite(String columnNames) {
         if (columnNames != null && !columnNames.isEmpty()) {
-            String[] columns = columnNames.split(" ");
+            String[] columns = columnNames.split(BLANK_SPACE_DELIMITER);
             if (columns.length > 0) {
                 Set<String> set = new HashSet<>();
                 for (String column : columns) {
@@ -344,7 +370,7 @@ private void addNodeColumns(List<String> row, ProfileResourceNode node) {
         addColumn(row, ID_ARRAY_INDEX, nullSafeNumber(node.getId()));
         addColumn(row, PARENT_ID_ARRAY_INDEX, nullSafeNumber(node.getParentId()));
         addColumn(row, URI_ARRAY_INDEX, DroidUrlFormat.format(node.getUri()));
-        addColumn(row, FILE_PATH_ARRAY_INDEX, node.toString());
+        addColumn(row, FILE_PATH_ARRAY_INDEX, toFilePath(node.getUri()));
         addColumn(row, FILE_NAME_ARRAY_INDEX, toFileName(metaData.getName()));
         addColumn(row, ID_METHOD_ARRAY_INDEX, nullSafeName(metaData.getIdentificationMethod()));
         addColumn(row, STATUS_ARRAY_INDEX, metaData.getNodeStatus().getStatus());