[Question]: How to get more than 20 search results back?

[shopapps]

What happened?

Searching only ever returns 20 results.

How do i add pagination to the search results?

How to reproduce the bug

search in table search box always returns 20 (or less) results.

Package Version

0.3.0

PHP Version

8.2

Laravel Version

10.00

Which operating systems does with happen with?

Linux

Notes

This is using Scout & Meilisearch

[kainiklas]

I assume you use the InteractsWithScout trait for table search?

Have you tried customizing the pagination settings on the table itself? https://filamentphp.com/docs/3.x/tables/advanced#pagination

[shopapps]

Hi, Thanks for the response.

Yeah I have the trait added, the search itself works it's just that i only ever get 20 results back (which i understand to be the meilisearch default). My understanding is that you can pass additional method calls to the search using

$model->search($query)->paginate(5) ;

and i was just wondering if you have that running to pass in the current tables pagination info (results per page, current page etc) or if you had an example of which method to override to achieve it :-)

[shopapps]

Ok, for others I managed to get something working:

adding this to my Resource/Pages/ListXXX.php file ( the one you added the use IntercactsWithScout; trait) ...

protected function applySearchToTableQuery(Builder $query): Builder
    {
        $this->applyColumnSearchesToTableQuery($query);

        $search = $this->getTableSearch();

        if (blank($search)) {
            return $query;
        }
        
// sets the hard limit in meilisearch to a huge figure
        resolve($this->getModel())->updateScoutSettings([
            'pagination' => [
                'maxTotalHits' => 100000000
            ]
        ]);

        $allIds = collect();
        $page = 1;
        $perPage = 1000; // Define how many items you want per page

        do {
            $results = $this->getModel()::search($search)->query(function ($query) {
                return $query->select('id');
            })->paginate($perPage, 'page', $page);

            $allIds = $allIds->concat($results->pluck('id'));

            $page++;
            
        } while ($results->count() > 0);

        return $query->whereIn('id', $allIds->toArray());
    }

I then needed to create a method on my model:

in App\Models\XXX.php

use Meilisearch\Client;

class XXXX extends Model
{

... existing model code

public function updateScoutSettings($settings = null) {


        $settings = $settings ?? [
            'pagination' => [
                'maxTotalHits' => 10000000  // Set to the maximum hits you require
            ]
        ];

        // Initialize the MeiliSearch client
        $client = new Client(config('scout.meilisearch.host'), config('scout.meilisearch.key'));

        // Update settings for the specific index
        $res = $client->index($this->searchableAs())->updateSettings($settings);
    }
}

ideally you would fire the $model->updateScoutSettings(); only once as i think this gets set "permanently" in the meilisearch engine server (dont quote me on that), but for now this got me working.

[kainiklas]

This looks quite complex. I drafted PR #11 which should do what you want.

Meilisearch has a limit of 20 items per search request (https://www.meilisearch.com/docs/reference/api/search#limit). By setting it explicitly on each request to a higher number, the issue can be resolved.

The maximum of hits can be configured per index (https://www.meilisearch.com/docs/reference/api/settings#pagination). Default is 1000.

I will merge the PR in the comming days. Perhaps you can check it as well 🙂

[shopapps]

looking at the source code of the PR it looks a lot cleaner thanks. :-)

There will still be an issue that by default meillisearch limits your searching (even pagination) to 1000 records by default. as per: https://www.meilisearch.com/docs/reference/api/settings#pagination. regardless of what you set your limits and offset to.

maxTotalHits takes priority over search parameters such as limit, offset, hitsPerPage, and page.

For example, if you set maxTotalHits to 100, you will not be able to access search results 

beyond 100 no matter the value configured for offset.

so even if you set a search_limit of 500 you would only get a max of 2 sets of results even if you have 5000 records that should be returned. :-(

You are also missing the need to continue looping until you have no results left on your search to get all 5000 results back and then let the filament query actually do the pagination within it's subsequent pagination query , otherwise, you will always return only the first 100 results (using your default 100 as an example)

The only way (i can see) to fix this is to change the value of pagination. maxTotalHits in meillisearch engine itself. Which is done via the updateSettings call

What would be cool is to create an artisan console command in your package that you can run once, something like:

php artisan meillisearch:maxhits 10000000

So your PR just to gather all the results (but still limited to 1000 unless you change maxTotalHits) but in smaller chunks should read:

        $searchLimit = config('kainiklas-filament-scout.search_limit'); // default 100
        $primaryKeyName = app($this->getModel())->getKeyName();

        $keys = collect();
        $page_count = 1;

        do {
                $search_results = $this->getModel()::search($search)->query(function ($query) use ($primaryKeyName) {
                           $query->select($primaryKeyName);
                    })->paginate($searchLimit, page:  $page_count)
                      ->pluck($primaryKeyName);
              
                  $keys = $keys->concat($search_results);
                  $page_count++;
    
            } while (count($search_results) > 0);

            return $query->whereIn($primaryKeyName, $keys->toArray());

[shopapps]

actually my console command idea would also need to include the model

php artisan meillisearch:settings --maxTotalHits=10000000 --model="App\Models\YourModel"

[shopapps]

If it helps others i created this: App\Console\Commands\MeillisearchSettings.php. feel free to add it into your package if you want.

php artisan meillisearch:settings --index_name="aws_files_index" --maxTotalHits=2000000

you can either use options of --index_name=your_index or --model="\App\Models\YouModel"

<?php

namespace App\Console\Commands;

use Illuminate\Console\Command;
use Meilisearch\Client;

class MeillisearchSettings extends Command
{
    /**
     * The name and signature of the console command.
     *
     * @var string
     */
    protected $signature = 'meillisearch:settings {--index_name=? : index to apply settings} {--maxTotalHits=? : Change pagination.maxTotalHits value} {--model= : model to apply settings to}';

    /**
     * The console command description.
     *
     * @var string
     */
    protected $description = 'Update Meilisearch settings for a model';

    /**
     * Execute the console command.
     */
    public function handle()
    {
        $index_name = $this->option('index_name');
        if(empty($index_name)) {
            $model = resolve($this->option('model'));
            if (empty($model)) {
                $this->error('Index or Model is required');
                return 1;
            }
            $index_name = $model->searchableAs();
        }

        $maxTotalHits = (int) $this->option('maxTotalHits');


        $settings = [];
        if($maxTotalHits) {
            data_set($settings, 'pagination.maxTotalHits', $maxTotalHits);
        }


        $client = new Client(config('scout.meilisearch.host'), config('scout.meilisearch.key'));

        $client->index($index_name)->updateSettings($settings);

        $this->info('Settings updated for ' . $index_name . ' index to ' .  $maxTotalHits);
        return 0;
    }
}

[shopapps]

also added it as a GIST here, in case others want to improve on it:

https://gist.github.com/shopapps/e6ea29c4e208388e3f381703165a07ec

[kainiklas]

You can also add the maxTotalHits to your scout config:

'index-settings' => [
  MyClass::class => [
    'pagination' => [
      'maxTotalHits' => 10000
    ],
  ],
],

And then call php artisan scout:sync-index-settings.

[shopapps]

ahh i did not know that - brilliant thanks