Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ignore features ordering in test #1675

Merged
merged 16 commits into from
Feb 2, 2025
Merged

Conversation

sharkAndshark
Copy link
Collaborator

@sharkAndshark sharkAndshark commented Feb 2, 2025

Try to fix #1651

  • Ignore *.pbf as the features ordering has changed
  • Use ogrmerge of gdal to generate the description text of *.pbf files
  • Add "gid" to function_Mixed_Name.sql for jq sorting
  • Upgrade to PostGIS:15-3.4
  • Add verification of gdal-bin in test.sh
  • Just bless
  • Update doc

@@ -410,7 +414,7 @@ jobs:
tests/test.sh
- name: Compare test output results (Linux)
if: matrix.target == 'x86_64-unknown-linux-gnu'
run: diff --brief --recursive --new-file tests/output tests/expected
run: diff --brief --recursive --new-file --exclude='*.pbf' tests/output tests/expected
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Features order changed. So no need to diff the *.pbf

@sharkAndshark sharkAndshark marked this pull request as ready for review February 2, 2025 14:51
@sharkAndshark
Copy link
Collaborator Author

sharkAndshark commented Feb 2, 2025

Or maybe It should be splited to two PRs: Ignore features order and upgrade to PostGIS:15-3.4

Copy link
Collaborator

@CommanderStorm CommanderStorm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for tackling this huge ammount of work.

I have left one nit below, where a better error message would help new devs a lot and a sync-comment ^^

@@ -95,7 +95,10 @@ test_pbf() {

if [[ $OSTYPE == linux* ]]; then
./tests/fixtures/vtzero-check "$FILENAME"
./tests/fixtures/vtzero-show "$FILENAME" > "$FILENAME.txt"
# see https://gdal.org/en/stable/programs/ogrmerge.html#ogrmerge
ogrmerge.py -o "$FILENAME.geojson" "$FILENAME" -single -src_layer_field_name "source_mvt_layer" -src_layer_field_content "{LAYER_NAME}" -f "GeoJSON" -overwrite_ds
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for users which don't have the right dependency installed this would look like the tests not being able to run.
For the other dependencys, we can reasonably assume that they are already insstalled ^^

As a fresh dev, a missing python file might look like a bug and lead to me skipping testing..

=> Could you add a check at the start of the test that bails if ogrmerge.py is not installed and advises the dev to do so?

@@ -28,7 +28,7 @@ jobs:
PGPASSWORD: postgres
services:
postgres:
image: postgis/postgis:14-3.3
image: postgis/postgis:15-3.4
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To be on the same page after merging:
Would there something blocking us from updating to 3.5 and isssuing warnings to <3.5?
From what I have seen, the 3.5 results seem more correct..

(lets do this in a sepaate PR, if desired)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The feature order changed in v3.4 and MBTiles content changed(related issue here) in v3.5.
I planned to fix them one by one ( for simplicity) rather than directly upgrading to v3.5 in one single step.
But you inspired me, on second thought, I think we could upgrade to 3.5 directly since we have aligned the PostGIS versions in both docker-compose.yml and ci.yml, so the results of local and ci should be same.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For possibly excessive caution and make this PR dedicated, let's address this in a separate PR? ^^

tests/test.sh Outdated
Comment on lines 30 to 37
# Verify the tools used in the tests are available
# todo add more verification for other tools like jq file curl sqlite3...
if ! command -v ogrmerge.py > /dev/null; then
echo "gdal-bin is required for testing"
echo "For Ubuntu, you could install it with sudo apt update && sudo apt install gdal-bin -y."
echo "see more at https://gdal.org/en/stable/download.html#binaries"
exit 1
fi
Copy link
Collaborator Author

@sharkAndshark sharkAndshark Feb 2, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah you are right. And maybe we should add more. @CommanderStorm

@sharkAndshark sharkAndshark enabled auto-merge (squash) February 2, 2025 17:47
@sharkAndshark sharkAndshark merged commit 6984f3a into maplibre:main Feb 2, 2025
15 of 16 checks passed
@sharkAndshark sharkAndshark deleted the ordering branch February 2, 2025 18:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Ensure PG 16+ generate same result as PG15 and before
2 participants