Improve S3 Documentation and Support

## Improvements to S3 Documentation
* Create a new document *quickstart_paths.md* that give a summary of the legal path formats used by netcdf-c. This includes both file paths and URL paths.
* Modify *nczarr.md* to remove most of the S3 related text.
* Move the S3 text from *nczarr.md* to a new document *cloud.md*.
* Add some S3-related text to the *byterange.md* document.

Hopefully, this will make it easier for users to find the information they want.

## Rebuild NCZarr Testing
In order to avoid problems with running make check in parallel, two changes were made:
1. The *nczarr_test* test system was rebuilt. Now, for each test.
any generated files are kept in a test-specific directory, isolated
from all other test executions.
2. Similarly, since the S3 test bucket is shared, any generated S3 objects
are isolated using a test-specific key path.

## Other S3 Related Changes
* Add code to ensure that files created on S3 are reclaimed at end of testing.
* Used the bash "trap" command to ensure S3 cleanup even if the test fails.
* Cleanup the S3 related configure.ac flag set since S3 is used in several places. So now one should use the option *--enable-s3* instead of *--enable-nczarr-s3*, although the latter is still kept as a deprecated alias for the former.
* Get some of the github actions yml to work with S3; required fixing various test scripts adding a secret to access the Unidata S3 bucket.
* Cleanup S3 portion of libnetcdf.settings.in and netcdf_meta.h.in and test_common.in.
* Merge partial S3 support into dhttp.c.
* Create an experimental s3 access library especially for use with Windows. It is enabled by using the options *--enable-s3-internal* (automake) or *-DENABLE_S3_INTERNAL=ON* (CMake). Also add a unit-test for it.
* Move some definitions from ncrc.h to ncs3sdk.h

## Other Changes
* Provide a default implementation of strlcpy and move this and similar defaults into *dmissing.c*.

.github/workflows/run_tests_ubuntu.yml

@@ -17,12 +17,11 @@ jobs:
         hdf5: [ 1.10.8, 1.12.2, 1.14.0 ]
 
     steps:
-
       - uses: actions/checkout@v3
 
       - name: Install System dependencies
         shell: bash -l {0}
-        run: sudo apt update && sudo apt install -y libaec-dev zlib1g-dev automake autoconf libcurl4-openssl-dev libjpeg-dev wget curl bzip2 m4 flex bison cmake libzip-dev doxygen
+        run: sudo apt update && sudo apt install -y libaec-dev zlib1g-dev automake autoconf libcurl4-openssl-dev libjpeg-dev wget curl bzip2 m4 flex bison cmake libzip-dev doxygen openssl
 
         ###
         # Installing libhdf5
@@ -168,7 +167,7 @@ jobs:
 
       - name: Configure
         shell: bash -l {0}
-        run: CFLAGS=${CFLAGS} LDFLAGS=${LDFLAGS} LD_LIBRARY_PATH=${LD_LIBRARY_PATH} ./configure --enable-hdf5 --enable-dap --disable-dap-remote-tests  --enable-doxygen  --enable-external-server-tests
+        run: CFLAGS=${CFLAGS} LDFLAGS=${LDFLAGS} LD_LIBRARY_PATH=${LD_LIBRARY_PATH} ./configure --enable-hdf5 --enable-dap --disable-dap-remote-tests  --enable-doxygen  --enable-external-server-tests --enable-s3 --enable-s3-internal --enable-nczarr-s3-tests
         if: ${{ success() }}
 
       - name: Look at config.log if error
@@ -192,7 +191,14 @@ jobs:
 
       - name: Run Tests
         shell: bash -l {0}
-        run: CFLAGS=${CFLAGS} LDFLAGS=${LDFLAGS} LD_LIBRARY_PATH=${LD_LIBRARY_PATH} make check -j
+        env:
+          AWS_PROFILE: ${{ secrets.DEFAULT_PROFILE }}
+        run: |
+          mkdir -p ~/.aws
+          echo "" > ~/.aws/config
+          chmod go-rwx ~/.aws/config
+          echo "${AWS_PROFILE}" >> ~/.aws/config
+          CFLAGS=${CFLAGS} LDFLAGS=${LDFLAGS} LD_LIBRARY_PATH=${LD_LIBRARY_PATH} make check -j
         if: ${{ success() }}
 
   ##

.github/workflows/run_tests_win_cygwin.yml

@@ -30,6 +30,7 @@ jobs:
             libhdf4-devel zipinfo libxml2-devel perl zlib-devel
             libzstd-devel libbz2-devel libaec-devel libzip-devel
             libdeflate-devel gcc-core libcurl-devel libiconv-devel
+            libssl-devel libcrypt-devel
 
       - name: (Autotools) Run autoconf and friends
         run: |
@@ -40,10 +41,11 @@ jobs:
 
       - name: (Autotools) Configure in-tree build
         run: >-
-          /bin/dash ./configure --enable-hdf5 --enable-shared
-          --disable-static --enable-dap --disable-dap-remote-tests
-          --enable-plugins --disable-nczarr-filters
-          --disable-nczarr-s3 --disable-nczarr-s3-tests --disable-nczarr
+           /bin/dash ./configure --enable-hdf5 --enable-shared
+           --disable-static --enable-dap --disable-dap-remote-tests
+           --enable-plugins --enable-nczarr
+           --enable-s3 --enable-s3-internal
+
 
       - name: Look at config.log if error
         if: ${{ failure() }}
@@ -66,4 +68,11 @@ jobs:
 
       - name: (Autotools) Build and run tests
         timeout-minutes: 30
-        run: make check -j8 SHELL=/bin/dash
+        env:
+          AWS_PROFILE: ${{ secrets.DEFAULT_PROFILE }}
+        run: |
+          mkdir -p ~/.aws
+          echo "" > ~/.aws/config
+          chmod go-rwx ~/.aws/config
+          echo "${AWS_PROFILE}" >> ~/.aws/config
+          make check -j8 SHELL=/bin/dash

.github/workflows/run_tests_win_mingw.yml

@@ -37,7 +37,7 @@ jobs:
         run: autoreconf -if
 
       - name: (Autotools) Configure Build
-        run: ./configure --enable-hdf5 --enable-dap --disable-dap-remote-tests --disable-static --disable-byterange --disable-dap-remote-tests --disable-logging --enable-plugins --disable-nczarr-filters --disable-nczarr-s3 --disable-nczarr-s3-tests
+        run: ./configure --enable-hdf5 --enable-dap --disable-dap-remote-tests --disable-static --disable-byterange --disable-dap-remote-tests --disable-logging --enable-plugins --disable-nczarr-filters --disable-s3
         if: ${{ success() }}
 
       - name: (Autotools) Look at config.log if error
@@ -54,7 +54,7 @@ jobs:
       - name: Check for plugins
         run: |
           dir ./plugins
-          if test -e ./plugins/.libs ; then dir ./plugins/.libs ; fi
+          if test -f ./plugins/.libs ; then dir ./plugins/.libs; fi
 
       - name: (Autotools) Build and Run Tests
         run: make check -j 8 LDFLAGS="-Wl,--export-all-symbols"

CMakeLists.txt

@@ -1270,6 +1270,7 @@ ENDIF()
 
 # Options for S3 Support
 OPTION(ENABLE_S3 "Enable S3 support." OFF)
+OPTION(ENABLE_S3_INTERNAL "Enable S3 Internal support." OFF)
 OPTION(ENABLE_NCZARR_S3 "Enable NCZarr S3 support; Deprecated in favor of ENABLE_S3" OFF)
 OPTION(ENABLE_NCZARR_S3_TESTS "Enable NCZarr S3 tests." OFF)
 
@@ -1279,11 +1280,6 @@ if (NOT ENABLE_S3 AND ENABLE_NCZARR_S3)
 ENDIF()
 UNSET(ENABLE_NCZARR_S3)
 
-IF(ENABLE_NCZARR_S3_TESTS AND NOT ENABLE_S3)
-  message(FATAL_ERROR "S3 support is disabled; please specify option -DENABLE_NCZARR_S3_TESTS=NO")
-  SET(ENABLE_NCZARR_S3_TESTS OFF CACHE BOOL "NCZARR S3 TESTS" FORCE)
-ENDIF()
-
 # Note we check for the library after checking for enable_s3
 # because for some reason this screws up if we unconditionally test for sdk
 # and it is not available. Fix someday
@@ -1293,22 +1289,35 @@ IF(ENABLE_S3)
   IF(AWSSDK_FOUND)
     SET(service s3;core)
     AWSSDK_DETERMINE_LIBS_TO_LINK(service AWS_LINK_LIBRARIES)
-    SET(ENABLE_S3_SDK ON CACHE BOOL "S3 SDK" FORCE)
+    SET(ENABLE_S3_AWS ON CACHE BOOL "S3 AWS" FORCE)
   ELSE()
-    SET(ENABLE_S3_SDK OFF CACHE BOOL "S3 SDK" FORCE)
+    SET(ENABLE_S3_AWS OFF CACHE BOOL "S3 AWS" FORCE)
   ENDIF()
 ELSE()
-  SET(ENABLE_S3_SDK OFF CACHE BOOL "S3 SDK" FORCE)
+  SET(ENABLE_S3_AWS OFF CACHE BOOL "S3 AWS" FORCE)
 ENDIF()
 
-IF(NOT ENABLE_S3_SDK)
-  IF(ENABLE_S3)
+# Unless/until we get aws-sdk-cpp working for Windows, force use of internal
+IF(ENABLE_S3 AND MSVC)
+  SET(ENABLE_S3_INTERNAL ON CACHE BOOL "S3 Intern" FORCE)
+ENDIF()
+
+IF(ENABLE_S3)
+  IF(NOT ENABLE_S3_AWS AND NOT ENABLE_S3_INTERNAL)
     message(FATAL_ERROR "S3 support library not found; please specify option -DENABLE_S3=NO")
-    SET(ENABLE_S3 OFF CACHE BOOL "S3 support" FORCE)
-    SET(ENABLE_NCZARR_S3_TESTS OFF CACHE BOOL "S3 tests" FORCE)
+    SET(ENABLE_S3 OFF CACHE BOOL "S3 support" FORCE)        
+  ENDIF()
+  IF(ENABLE_S3_AWS AND ENABLE_S3_INTERNAL)
+    message(WARNING "Both aws-sdk-cpp and s3-internal enabled => use s3-internal")
+    SET(ENABLE_S3_AWS OFF CACHE BOOL "S3 AWS" FORCE)        
   ENDIF()
 ENDIF()
 
+IF(ENABLE_NCZARR_S3_TESTS AND NOT ENABLE_S3)
+  message(FATAL_ERROR "S3 support is disabled; please specify option -DENABLE_NCZARR_S3_TESTS=NO")
+  SET(ENABLE_NCZARR_S3_TESTS OFF CACHE BOOL "NCZARR S3 TESTS" FORCE)
+ENDIF()
+
 IF(ENABLE_NCZARR_S3_TESTS)
   message(WARNING "**** DO NOT ENABLE_NCZARR_S3_TESTS UNLESS YOU HAVE ACCESS TO THE UNIDATA S3 BUCKET! ***")
 ENDIF()
@@ -1890,6 +1899,7 @@ ENDIF()
 # Check for various functions.
 CHECK_FUNCTION_EXISTS(fsync HAVE_FSYNC)
 CHECK_FUNCTION_EXISTS(strlcat   HAVE_STRLCAT)
+CHECK_FUNCTION_EXISTS(strlcpy  HAVE_STRLCPY)
 CHECK_FUNCTION_EXISTS(strdup  HAVE_STRDUP)
 CHECK_FUNCTION_EXISTS(strndup HAVE_STRNDUP)
 CHECK_FUNCTION_EXISTS(strtoll HAVE_STRTOLL)
@@ -2298,6 +2308,10 @@ ENDIF()
 IF(ENABLE_DAP4)
   ADD_SUBDIRECTORY(libdap4)
   ADD_SUBDIRECTORY(libncxml)
+ELSE()
+  IF(ENABLE_S3_INTERNAL)
+    ADD_SUBDIRECTORY(libncxml)
+  ENDIF()
 ENDIF()
 
 IF(ENABLE_PLUGINS)
@@ -2540,7 +2554,9 @@ is_enabled(USE_CDF5 HAS_CDF5)
 is_enabled(ENABLE_ERANGE_FILL HAS_ERANGE_FILL)
 is_enabled(HDF5_HAS_PAR_FILTERS HAS_PAR_FILTERS)
 is_enabled(ENABLE_S3 HAS_S3)
-is_enabled(ENABLE_S3_SDK HAS_S3_SDK)
+is_enabled(ENABLE_S3_AWS HAS_S3_AWS)
+is_enabled(ENABLE_S3_INTERNAL HAS_S3_INTERNAL)
+is_enabled(HAS_HDF5_ROS3 HAS_HDF5_ROS3)
 is_enabled(ENABLE_NCZARR HAS_NCZARR)
 is_enabled(ENABLE_NCZARR_S3_TESTS DO_NCZARR_S3_TESTS)
 is_enabled(ENABLE_NCZARR_ZIP HAS_NCZARR_ZIP)
@@ -2554,7 +2570,18 @@ is_enabled(HAVE_SZ HAS_SZLIB_WRITE)
 is_enabled(HAVE_ZSTD HAS_ZSTD)
 is_enabled(HAVE_BLOSC HAS_BLOSC)
 is_enabled(HAVE_BZ2 HAS_BZ2)
-
+if(ENABLE_S3_AWS)
+SET(WHICH_S3_SDK "aws-sdk-cpp")
+SET(NC_WHICH_S3_SDK "aws-sdk-cpp")
+else()
+if(ENABLE_S3_AWS)
+SET(WHICH_S3_SDK "internal")
+SET(NC_WHICH_S3_SDK "internal")
+else()
+SET(WHICH_S3_SDK "none")
+SET(NC_WHICH_S3_SDK "none")
+endif()
+endif()
 
 # Generate file from template.
 CONFIGURE_FILE("${CMAKE_CURRENT_SOURCE_DIR}/libnetcdf.settings.in"

Makefile.am

@@ -7,9 +7,19 @@
 
 # Ed Hartnett, Ward Fisher
 
+# Put together AM_CPPFLAGS and AM_LDFLAGS.
+include $(top_srcdir)/lib_flags.am
+
 # This directory stores libtool macros, put there by aclocal.
 ACLOCAL_AMFLAGS = -I m4
 
+TESTS_ENVIRONMENT =
+TEST_EXTENSIONS = .sh
+#SH_LOG_DRIVER = $(SHELL) $(top_srcdir)/test-driver-verbose
+#sh_LOG_DRIVER = $(SHELL) $(top_srcdir)/test-driver-verbose
+#LOG_DRIVER = $(SHELL) $(top_srcdir)/test-driver-verbose
+#TESTS_ENVIRONMENT += export SETX=1;
+
 ##
 # Turn off plugin directory during distcheck, see
 # comment thread at https://github.com/Unidata/netcdf-c/pull/2348
@@ -74,10 +84,14 @@ endif
 
 if ENABLE_DAP4
 DAP4 = libdap4
-XML = libncxml
 NCDAP4TESTDIR = dap4_test
+XML = libncxml
 endif #DAP4
 
+if ENABLE_S3_INTERNAL
+XML = libncxml # Internal S3 requires XML
+endif #ENABLE_S3_INTERNAL
+
 # Build PnetCDF
 if USE_PNETCDF
 LIBSRCP = libsrcp

RELEASE_NOTES.md

@@ -7,19 +7,19 @@ This file contains a high-level description of this package's evolution. Release
 
 ## 4.9.3 - TBD
 
+* Improve S3 documentation, testing, and support See [Github #????](https://github.com/Unidata/netcdf-c/pull/????).
+* Remove obsolete code. See [Github #2680](https://github.com/Unidata/netcdf-c/pull/2680).
 * [Bug Fix] Add a crude test to see if an NCZarr path looks like a valid NCZarr/Zarr file. See [Github #2658](https://github.com/Unidata/netcdf-c/pull/2658).
 
 ## 4.9.2 - March 14, 2023
 
 This is the maintenance release which adds support for HDF5 version 1.14.0, in addition to a handful of other changes and bugfixes.  
 
-* Fix 'make distcheck' error in run_interop.sh. See [Github #2549](https://github.com/Unidata/netcdf-c/pull/2549).
+* Fix 'make distcheck' error in run_interop.sh. See [Github #2631](https://github.com/Unidata/netcdf-c/pull/2631).
 * Fix a minor bug in reporting the use of szip. See [Github #2679](https://github.com/Unidata/netcdf-c/pull/2679).
 * Simplify the handling of XGetopt. See [Github #2678](https://github.com/Unidata/netcdf-c/pull/2678).
-* Fix 'make distcheck' error in run_interop.sh. See [Github #2631](https://github.com/Unidata/netcdf-c/pull/2631).
 * Update `nc-config` to remove inclusion from automatically-detected `nf-config` and `ncxx-config` files, as the wrong files could be included in the output.  This is in support of [GitHub #2274](https://github.com/Unidata/netcdf-c/issues/2274).
 * Update H5FDhttp.[ch] to work with HDF5 version 1.13.2 and later. See [Github #2635](https://github.com/Unidata/netcdf-c/pull/2635).
-* Fix 'make distcheck' error in run_interop.sh. See [Github #2631](https://github.com/Unidata/netcdf-c/pull/2631).
 * [Bug Fix] Update DAP code to enable CURLOPT_ACCEPT_ENCODING by default. See [Github #2630](https://github.com/Unidata/netcdf-c/pull/2630).
 * [Bug Fix] Fix byterange failures for certain URLs. See [Github #2649](https://github.com/Unidata/netcdf-c/pull/2649).
 * [Bug Fix] Fix 'make distcheck' error in run_interop.sh. See [Github #2631](https://github.com/Unidata/netcdf-c/pull/2631).

config.h.cmake.in

@@ -136,12 +136,6 @@ are set when opening a binary file on Windows. */
 /* if true, do remote tests */
 #cmakedefine ENABLE_DAP_REMOTE_TESTS 1
 
-/* if true, enable S3 support */
-#cmakedefine ENABLE_S3 1
-
-/* if true, S3 SDK is available */
-#cmakedefine ENABLE_S3_SDK 1
-
 /* if true, enable NCZARR */
 #cmakedefine ENABLE_NCZARR 1
 
@@ -157,6 +151,15 @@ are set when opening a binary file on Windows. */
 /* if true, Allow dynamically loaded plugins */
 #cmakedefine ENABLE_PLUGINS 1
 
+/* if true, enable S3 support */
+#cmakedefine ENABLE_S3 1
+
+/* if true, AWS S3 SDK is available */
+#cmakedefine ENABLE_S3_AWS 1
+
+/* if true, Force use of S3 internal library */
+#cmakedefine ENABLE_S3_INTERNAL 1
+
 /* if true, run extra tests which may not work yet */
 #cmakedefine EXTRA_TESTS 1
 
@@ -379,6 +382,9 @@ are set when opening a binary file on Windows. */
 /* Define to 1 if you have the `strlcat' function. */
 #cmakedefine HAVE_STRLCAT 1
 
+/* Define to 1 if you have the `strlcpy' function. */
+#cmakedefine HAVE_STRLCPY 1
+
 /* Define to 1 if you have the `strtoll' function. */
 #cmakedefine HAVE_STRTOLL 1