Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor: merge the products and product images directories for OFF, OBF, OPF and OPFF #10959

Merged
merged 43 commits into from
Nov 18, 2024

Conversation

stephanegigandet
Copy link
Contributor

@stephanegigandet stephanegigandet commented Oct 31, 2024

In the last few weeks, we normalized product barcodes and deduplicated products that existed in multiple flavors (Open Food Facts, Open Beauty Facts, Open Product Facts and Open Pet Food Facts).

We are now going to merge the directories that contain product and product revision data (the .sto files) and the product images. We will keep separate MongoDB databases for each flavor.

This will:

  • make it much easier to move products from one product type to another
  • remove the possibility of having duplicate products on multiple flavors
  • make it much easier to have read and write APIs that can be used to retrieve / update products of any type.

Deployment plan:

  • stop OBF, OPF, OPFF for the duration of the migration (a couple of hours)
  • update OFF code
  • move products and product images dirs from OBF, OPF, OPFF to the OFF directory structure
  • change the links to products and products of images on OBF, OPF, OPFF to use the OFF dirs
  • restart OBF, OPF, OPFF with the new code

In this PR:

  • code updates to check product_type when reading / editing a product for both the website and API. Redirect to the right flavor if the product type is different than the one of the server the request was made to.
  • changed the product edit form to allow moderators to change the product_type
  • keep support for moderators to put "obf" etc. in the change code field
  • tests for the above
  • small refactor to put into Config.pm things that are the same for all flavors and that are currently duplicated in Config_off.pm etc.

Will solve:

@stephanegigandet stephanegigandet requested a review from a team as a code owner October 31, 2024 16:12
@github-actions github-actions bot added ✏️ Editing API WRITE WRITE API to allow sending product info and image API READ All READ APIs include Product, Search… 🧪 tests Template::Toolkit The templating toolkit used by product opener. The starting point for HTML/JS/CSS fixes. Display Products URL config multilingual products 🌐 Translations labels Oct 31, 2024
@stephanegigandet stephanegigandet marked this pull request as draft October 31, 2024 16:18
@github-actions github-actions bot added the 💥 Merge Conflicts 💥 Merge Conflicts label Oct 31, 2024
@teolemon
Copy link
Member

teolemon commented Nov 4, 2024

@stephanegigandet small test conflict

@github-actions github-actions bot removed the 💥 Merge Conflicts 💥 Merge Conflicts label Nov 4, 2024
@stephanegigandet stephanegigandet changed the title refactor: merge the products and product images directories for OFF, OBF, OPF and OPFF - WIP refactor: merge the products and product images directories for OFF, OBF, OPF and OPFF Nov 12, 2024
@github-actions github-actions bot removed the 💥 Merge Conflicts 💥 Merge Conflicts label Nov 12, 2024
@stephanegigandet stephanegigandet marked this pull request as ready for review November 13, 2024 13:04
Copy link

sonarcloud bot commented Nov 18, 2024

Comment on lines +487 to +488
if ( (defined $request_body_ref->{product}{code})
and ($product_ref->{code} ne $request_body_ref->{product}{code}))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm a bit disappointed that we have to do those checks here and in product_jqm_multilingual… could'nt we factor this out ?

autoload("ProductOpener::Config_$flavor");

#
# Add values common to all flavors
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for simplifying this !

@@ -10599,6 +10614,24 @@ sub display_product_api ($request_ref) {
$response{jqm} .= $html;
}
}
elsif ((defined $product_ref->{product_type}) and ($product_ref->{product_type} ne $options{product_type})) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there again we have a complete duplication for product_multilingual.pl, that's a bit frustrating…

Comment on lines +109 to +110
&change_product_code
&change_product_type
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's great we de-entangled both actions !

Comment on lines 1063 to 1065
$product_ref->{old_code} = $code;
$code = $new_code;
$product_ref->{code} = $code;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
$product_ref->{old_code} = $code;
$code = $new_code;
$product_ref->{code} = $code;
# we only register the code change,
# file modification will happen as we save the product sto
$product_ref->{old_code} = $code;
$code = $new_code;
$product_ref->{code} = $code;

# Update the product_type from the server
if (defined $options{flavors_product_types}{$new_server}) {
my $errors_ref = {};
change_product_type($product_ref, $options{flavors_product_types}{$new_server}, $errors_ref);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think scripts should call store_product with a try / catch !
Or maybe we could add a parameter to say we wan't a fail-safe call.

@@ -1303,15 +1362,15 @@ sub store_product ($user_id, $product_ref, $comment) {
"moving product images",
{
source => "$BASE_DIRS{PRODUCTS_IMAGES}/$old_path",
destination => "$new_www_root/images/products/$path"
destination => "$www_root/images/products/$path"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In this whole sections you should not use $www_root and $data_root any more…
Please use %BASE_DIRS values …


sub move_product_dir_to_off ($dir, $dir2, $dir3, $dir4) {
# move .sto files
print STDERR "moving $dir/$dir2/$dir3/$dir4\n";
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For me, it's incredible that in this whole function you never put $dir/$dir2/$dir3/$dir4 in a variable to avoid copying it everywhere around !

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's just a one off script that will never be used again, so I mainly wanted to get this thing done as quickly as possible.

die;
}
}
die;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why do you need this die ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I launched the script product by product for the first products to verify everything was working as expected as I couldn't replicate everything in the dev environment. I removed the die() in opf etc. after that to do all the products.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
API READ All READ APIs include Product, Search… API WRITE WRITE API to allow sending product info and image API Issues related to the Open Food Facts API. More specific labels exist & should be used (API WRITE…) config Data import Display ✏️ Editing 👮 Moderation multilingual products 🧴 Open Beauty Facts Our cosmetic analysis project https://world.openbeautyfacts.org 🐾 Open Pet Food Facts Our pet food analysis project https://world.openpetfoodfacts.org 📸 Open Products Facts Our project to increase the lifespan of objects. https://world.openproductsfacts.org 🏭 Producers Platform https://wiki.openfoodfacts.org/Platform_for_producers Products REDIS Template::Toolkit The templating toolkit used by product opener. The starting point for HTML/JS/CSS fixes. 🧪 tests 🌐 Translations URL 👥 Users
Projects
Status: Done
Status: Done
Status: Done
Status: Done
Status: Done
Development

Successfully merging this pull request may close these issues.

3 participants