* Remove copy of GDAL_DATA files in main source tree
* Copy both GDAL and PROJ helper files from deps
* Copy to sibling gdal-data and proj-data dirs
* Use CPL API to set config, instead of pushing to the process environment
* Add importer-additional-proj-data-path config
* Set either PROJ_DATA or PROJ_LIB config, dependent on GDAL/PROJ version

Signed-off-by: Misiu Godfrey <[email protected]>

…that is not a multiple of the page size

Signed-off-by: Misiu Godfrey <[email protected]>

* Fixup check logic

* Avoid duplicated expr in the

Signed-off-by: Misiu Godfrey <[email protected]>

…ure (#7657)

Signed-off-by: Misiu Godfrey <[email protected]>

* Enable the device extension
* Enable the validation layer feature
* Add compile option (default OFF)

Signed-off-by: Misiu Godfrey <[email protected]>

Signed-off-by: Misiu Godfrey <[email protected]>

* Fix path of cpack install of gdal and proj from deps to ThirdParty
Keep gdal and proj directory names for simplicity
Revert to setenv() as CPLSetConfigOption() does NOT work as documented

* Add OSDependent/heavyai_env
Use heavyai::setenv()

* Add heavyai::env_path_separator()

Signed-off-by: Misiu Godfrey <[email protected]>

Signed-off-by: Misiu Godfrey <[email protected]>

… cases (#7618)

* Modify isDirectColumnarConversionPossible logic

* Simplify op

* make auto_parallel_row_count_threshold be a global constant

* Add logs

* Add test

* Fixup dist test

Signed-off-by: Misiu Godfrey <[email protected]>

* Add artificial project node between filter and left-join nodes

* Change rewriter function name

* Add comments

* Add test

* Cover more edge cases

* Address comment

* Pass reference of the node_list to the query rewriting function

Signed-off-by: Misiu Godfrey <[email protected]>

* Remove num tuple threshold

* Address nullptr issue of EXPECT_TRUE

* Remove global g_is_test_env from PerfectJoinHashTable

* Modify comment

* Fixup join column buckettized entry count

* Fixup test failures: re-introduce g_is_test_env to control the behavior

Signed-off-by: Misiu Godfrey <[email protected]>

Signed-off-by: Misiu Godfrey <[email protected]>

…arning. (#7666)

Signed-off-by: Misiu Godfrey <[email protected]>

Signed-off-by: Misiu Godfrey <[email protected]>

…ault

Signed-off-by: Misiu Godfrey <[email protected]>

Signed-off-by: Misiu Godfrey <[email protected]>

* Add deps include path when compiling extensions functions with nvcc

Change the `custom_command` for extension function compilation to include
CMAKE_PREFIX_PATH/include to allow nvcc to find (and prefer) the custom
build of boost on Ubuntu (required to update boost to 1.84)

* Disable boost::serialization support for std::optional

Define BOOST_NO_CXX17_HDR_OPTIONAL before including boost/serialization/optional.hpp
to disable boost's new native std::optional serialization as it results in compile
errors when using boost::serialization::split_free

Signed-off-by: Misiu Godfrey <[email protected]>

Signed-off-by: Misiu Godfrey <[email protected]>

* Add row-level llm_transform UDF WIP

* Support LLM_TRANSFORM

* Return response string, not an entire response JSON

* Add newline after answer keyword

* Improve error handling logic

* Support dictionary type

* Support llm_transform as filter expr

* Move nlohmann json to thirdparty folder

* Use iq-url instead of hard-coded url

* Convert LLM_TRANSFORM to string function

* Add test

* Allow controlling # concurrent request

* Check input operands

* Use translation_cache to avoid double translation

* Remove duplicated json impl

* Fixup compilation failure

* Add missing close character for prompt

* Add missing newline

* Use RapidJSON

* Use robinhood unordered map

* Allow none-encoded string input

* Update llm call endpoint

* Change the default value of g_max_concurrent_llm_transform_call to 16

* Modify the exception msg

* Use shared/unique lock to access translation_cache

* Address comments

* Add cpr lib

* Make an individual test

* Address comments v2

---------

Co-authored-by: Todd Mostak <[email protected]>
Signed-off-by: Misiu Godfrey <[email protected]>

Signed-off-by: Misiu Godfrey <[email protected]>

* Add log

* Remove unnecessary logging about row_size

* Address comments

* Address comments

Signed-off-by: Misiu Godfrey <[email protected]>

BE-6491 Geospatial Dependencies Update for 8.0

Update geospatial libs as follows:

GDAL 3.7.3
PROJ 9.3.1
TIFF 4.5.1
GeoTIFF 1.7.1
GEOS 3.12.1

Adds GDAL support for JPEG2000/JP2 raster, using OpenJPEG library. Also include additional dependent libs LittleCMS2 and WebP. Removed now-superfluous GDAL patch.

Disabled ImportExportTest ExportTest/Shapefile/MULTIPOLYGON for now, as the new GDAL seems to write Shapefiles slightly differently, which breaks a comparison. New comparison Shapefiles are included in this PR, but the test is disabled. The test can be re-enabled once the deps transition is complete.

Fix the GEOS version check which enables ST_ConcaveHull, and fix the test of ST_ConcaveHull, both broken due to not actually including the GEOS version header, so the pre-processor logic was silently failing.

Changed use of boost::process::system() to call ogrinfo to use the more explicit form, as the simple form does not work with the new GDAL.

Added PDAL patch for GDAL API const change. PDAL version is unchanged.

Added fix for static GDAL data destruction (avoids ASAN error on exit in GeospatialTest).

Added separate shared build of PROJ and GDAL tools, so they are still present on a CentOS build.

Install PROJ and GDAL run-time data files directly from deps.

Moved LLVM builds after geo libs build to allow faster iteration. Move back again once dust settles.

NOTE: There is a crash in GDAL FlatGeobuf geo export which we are still tracking down. Work on this is ongoing, but we are rolling the minimal deps change out anyway so that proper testing for 8.0 can proceed. For now, there is a new build option to disable FlatGeobuf export, defaulting to disabled. Interactive export will error, and the corresponding tests in ImportExportTest are disabled.

---------

Co-authored-by: Steve Blackmon <[email protected]>
Signed-off-by: Misiu Godfrey <[email protected]>

… in RelWithDebInfo mode w/ GCC 13.2 and LLVM 14 (#7621)

Signed-off-by: Misiu Godfrey <[email protected]>

Signed-off-by: Misiu Godfrey <[email protected]>

…e geometry projection (#7678)

* Throw an appropriate exception to avoid the crash

* Add test

Signed-off-by: Misiu Godfrey <[email protected]>

* Add Calcite query rewrite rule to pushdown any expr on a column expr used in a join qual

* Add tests

* Address comment

Signed-off-by: Misiu Godfrey <[email protected]>

* Allow the `THREADS` key to be passed to data wrappers.

Signed-off-by: Misiu Godfrey <[email protected]>

… python cleanup, and more (#7683)

* Bump centos gcc from 11.1 to 11.4 to match Ubuntu

* Add baseline support for 22.04 / 23.10 and remove pre-20.04 support

 - Use 22.04 prebuilts for 23.10

* Remove legacy python and use modern cmake find_package

  - Remove python, python2 and python-yaml from the Ubuntu package
    installs, and install python3 and python3-yaml

  - Remove python-devel from CentoOS

  - Replace the cmake find_package(PythonInterp) with the new
    find_package(Python). The latter defines slightly different vars
    (e.g. PYTHON_EXECUTABLE becomes Python_EXECUTABLE)

* Bump boost to 1.84

 - Build boost on ubuntu and remove libboost-all-dev from packages

 - Add libnuma-dev (for memkind)

* Bump folly to 2023.01.16.00 and FMT to 9.1.0

 - Disable exception tracer in folly, required for static linking libstdc++
   with the new version

* Update to OpenSSL 3.0.10

 - Build openssl and curl on all ubuntu versions

* Misc dependency script cleanup

 - Remove LLVM 9 patches from scripts

 - Change prebuilt script to always overwrite mapd-deps.sh and not append.
   Fixes the double entries

 - Fix several interactive prompts

Signed-off-by: Misiu Godfrey <[email protected]>

Signed-off-by: Misiu Godfrey <[email protected]>

* Add ASAN suppression for ForeignTableDmlTest shutdown leak

Leak in CRYPTO_zalloc during test shutdown

* Add TSAN suppression for lock inversion in GeospatialTest

* Add TSAN suppression for gdal lock inversion (RasterImporter)

* TSAN - Disable GeoJoinTest SingleAndMultipleBins

* TSAN - Disable MLRegressionTests

Signed-off-by: Misiu Godfrey <[email protected]>

…(#7688)

* Remove unnecessary input expr type checking

* Add test

Signed-off-by: Misiu Godfrey <[email protected]>

* Log cuda driver version at startup

* Address comment

Signed-off-by: Misiu Godfrey <[email protected]>

…t expression down to a child project node (#7686)

* Fixup logic issue

* Add test

* Address comment

* Address comment v2

* Address comment v3

Signed-off-by: Misiu Godfrey <[email protected]>

…cuda{malloc/free} (#7690)

* Add logs

* Address comments

Signed-off-by: Misiu Godfrey <[email protected]>

Signed-off-by: Misiu Godfrey <[email protected]>

* Add logs for cuda d2h, h2d, and d2d

* Add logs for d2h and h2d query data

* Address comments

* Revise cudamgr d2h, h2d, d2d function to printout tag

* allow flag to control logging behavior

* Rebase and allow controlling logging behavior

* Change size to value

Signed-off-by: Misiu Godfrey <[email protected]>

* Build and static link OpenLDAP 2.5.16 on Centos

* Bump ncurses to 6.4, libedit to 20230829-3.1

Remove install of ncurses-devel

* Cleanup prebuilt script installs

* Remove patch_old_thrust_tb

* Fix extension function parse_ast for python 3

Signed-off-by: Misiu Godfrey <[email protected]>

Signed-off-by: Misiu Godfrey <[email protected]>

Signed-off-by: Misiu Godfrey <[email protected]>

Temporarily skip this test pending investigation

Signed-off-by: Misiu Godfrey <[email protected]>

* Update SDS Dockerfiles

* Always set RPATH

* Fix supported Ubuntu versions in install-heavyai.sh

Signed-off-by: Misiu Godfrey <[email protected]>

Signed-off-by: Misiu Godfrey <[email protected]>

Signed-off-by: Misiu Godfrey <[email protected]>

Signed-off-by: Misiu Godfrey <[email protected]>

* Improve cardinality cache key gen logic

* Add test

* Address comment

* Address  comment v2

Signed-off-by: Misiu Godfrey <[email protected]>

…r (#7698)

* Scans entire provided ResultSet and extracts Geo*TargetValuePtr "coords" datums
* Validates each one with CudaMgr to ensure it falls within a known device memory allocation
* Constructs the set of device IDs associated with those allocations
* If the resulting set contains just one device ID, we force the Thrust and render to that GPU
* If the resulting set contains more than one device ID, we switch to the slower render path
* Logs important info to VLOG(1) including timing

Signed-off-by: Misiu Godfrey <[email protected]>

 remove generated files from repository and TTypeInfo dictionary test fix

Signed-off-by: Misiu Godfrey <[email protected]>

Signed-off-by: Misiu Godfrey <[email protected]>

…eStat" exception (#7696)

* Add fallback logic when request alloc error occurred

* Add test

* Address comment

* Address comments v2

Signed-off-by: Misiu Godfrey <[email protected]>

Signed-off-by: Misiu Godfrey <[email protected]>

Signed-off-by: Misiu Godfrey <[email protected]>

Signed-off-by: Misiu Godfrey <[email protected]>

… when an exception is thrown. (#7714)

Signed-off-by: Misiu Godfrey <[email protected]>

Signed-off-by: Misiu Godfrey <[email protected]>

* Update returned compute capability for Ada and Hopper

* Update CMAKE_CUDA_ARCHITECTURES/CUDA_COMPILATION_ARCH

Signed-off-by: Misiu Godfrey <[email protected]>

* Downgrade deps GEOS from 3.12.1 (too new) to 3.11.1
* Build GEOS with CMake
* Install GEOS 3.11.1 run-time DSOs from public ubuntugis PPA

Signed-off-by: Misiu Godfrey <[email protected]>

Signed-off-by: Misiu Godfrey <[email protected]>

* Bump oneDAL to 2024.1.0

* Patch issues when using c++20

- Remove c++ standard override from oneDALConfig.cmake

- Remove redundant template params from ctor declarations

Signed-off-by: Misiu Godfrey <[email protected]>

apt-key add was deprecated in 22.04 and is no longer usable

Signed-off-by: Misiu Godfrey <[email protected]>

…y_for_range_compressed` runtime function (#7712)

* Add null handling logic

* Add test

* Return null sentinel depending on dimension

* Address comments

Signed-off-by: Misiu Godfrey <[email protected]>

* Allow debug timer log iff single-threaded topK sort

* Improve logging

* Remove unnecessary debug timer for topPermute function

* Address comments

Signed-off-by: Misiu Godfrey <[email protected]>

* Link CPU_RESULT_MEM resource (sub)type to appropriate one depending on the flag

* Add tests

* Address comments

Signed-off-by: Misiu Godfrey <[email protected]>

@mattpulver okay, I will try next time.

Signed-off-by: Misiu Godfrey <[email protected]>

Signed-off-by: Misiu Godfrey <[email protected]>

* Added more verbose logging for locks.  Specificallty, ExecutorLocks, Catalog Locks, and SessionsStore locks.

---------

Co-authored-by: Misiu Godfrey <[email protected]>
Signed-off-by: Misiu Godfrey <[email protected]>

* Support fast count distinct allocator for unserialize resultset

* Protect rsmo during fast allocator initialization

* Address comment

* Address comments v2

* Revert the change to call

Signed-off-by: Misiu Godfrey <[email protected]>

* Replace boost::multi_index with AliasInfoMap class

* Move SQLType <-> BufferAttrType conversions to QueryRenderer/TypeUtils

* Move table id and col id extraction to AnalyzerUtils

* Use SQLTypeInfo equality operator for AttrAliasInfo

Signed-off-by: Misiu Godfrey <[email protected]>

* Changed catalog function to return table names instead of table desciptors

Signed-off-by: Misiu Godfrey <[email protected]>

Signed-off-by: Misiu Godfrey <[email protected]>

* Remove exitHandler again and put GDAL::init() inside static map accessor
* Also fix a couple of other unrelated TSAN warning sources

Co-authored-by: Matt Pulver <[email protected]>
Signed-off-by: Misiu Godfrey <[email protected]>

* Replace LOG(FATAL) with just return kNULLT in low-level comparison
* Abort function match attempt on receipt of kNULLT indicating type incompatibility
* Add second variant of ST_Contains_MultiPolygon_Point that does not require bounds
* Add simple tests
This may be replaced by a fix to carry bounds with temp geo, after discussion with Pearu

Signed-off-by: Misiu Godfrey <[email protected]>

* Fixup (incorrect) assumption that reading batches from parquet files
always guarantees a fixed number of levels read

* Improve logging during detect in the case where no valid data is
obtained

Signed-off-by: Misiu Godfrey <[email protected]>

This reverts commit 51ebdc38485b5208762ce6b046cff19b30301bfe.

Signed-off-by: Misiu Godfrey <[email protected]>

* Add buffer holders for GPU execution

* Rename structures used to codegen

* Introduce WindowFunctionCtx namespace

* Add preparation for GPU execution in window ctx

* Cleanup & improve WindowFunctionContext::compute()

* Improve a logic to build aggregate tree w/ supporting reusing

* Improve segment tree constructor

* Rebase

* Address comments #1

* Address comments #2: refactor bool param functions

* Address comments #3

* Address comments #4: tbb

* Address comments #5

* Address comments #6

* Fixup test failures

Signed-off-by: Misiu Godfrey <[email protected]>

* Remove DeviceClock class and unnecessary timing event

* Add trailing underscore for variables

* Refactor query timing logic defined in QueryExecutionContext

* Address comments

Signed-off-by: Misiu Godfrey <[email protected]>

…742)

Signed-off-by: Misiu Godfrey <[email protected]>

* Register new threads created during S3 import with debug timers.

Signed-off-by: Misiu Godfrey <[email protected]>

* Equivalent of OSGeo/gdal#9313 which will be in the next GDAL
* Resolves clash with the other default-namespaced flatbuffers symbols in Arrow 9.0.0

Signed-off-by: Misiu Godfrey <[email protected]>

* Reinstate fatal error which is useful in future development to catch unhandled types
* Check for pointer type to avoid getting to the fatal error in the first place

Signed-off-by: Misiu Godfrey <[email protected]>

Extract what was BaseQueryDataTableSQL into a standalone
concrete class
--
2.43.0

Signed-off-by: Misiu Godfrey <[email protected]>

…(#7732)

* Skip logging D2H for varlen resulset row value fetching while resultset iteration.

* Address comment

Signed-off-by: Misiu Godfrey <[email protected]>

* Move GfxContext creation to DBHandler
* Plumb GfxContext to TableFunctionExecutionContext
* DeviceContexts now owned by GfxContext
* Implement example graphics TF (headless DrawOneTriangle)
also (unrelated)
* Remove a self-#include

---------

Co-authored-by: Steve Blackmon <[email protected]>
Signed-off-by: Misiu Godfrey <[email protected]>

First iteration of raster data wrapper.

Signed-off-by: Misiu Godfrey <[email protected]>

Signed-off-by: Misiu Godfrey <[email protected]>

Signed-off-by: Misiu Godfrey <[email protected]>

* Add a mutex to VulkanQueue submission

The lock is also invoked prior to checking the internal device_lost_
state flag

* Add CommandExecutionContext and Vulkan implementation

Holds command components that must be unique per execution thread:
CommandList, CommandExecutor, VulkanCommandPool

* includes cleanup

Signed-off-by: Misiu Godfrey <[email protected]>

Signed-off-by: Misiu Godfrey <[email protected]>