Cover Article: Custom Plugins, Part 4: Finder Plugin

In episode 4 of our series about building custom plugins we will look at finder plugins. With a finder plugin you make the content of a component findable by Joomla's Smart Search. In this article we use the example of a custom finder plugin that indexes recipes from a recipes extension.

Indexing Instead of Searching

The component that is responsible for search results on your Joomla site is com_finder and is called Smart Search. Until Joomla 4 came out, we shipped another, more primitive search component: com_search. That old component asked every search plugin to run a query at the moment a visitor pressed the search button. Every plugin returned its own result set, and the component glued those together. That is simple to write but it doesn’t scale: the work was done again for every visitor.

Smart Search turns that around. All content is chopped into terms and stored in an index (stored in tables #__finder_terms, #__finder_taxonomy, #__finder_links, etcetera) before anyone searches. The finder plugins are called when indexing the content, not during the search anymore. At search time only com_finder is involved: it looks up terms, weighs them and renders the results.

A finder plugin boils down to: "Here is a row from a table with some content. What should end up in the index?" Almost every peculiarity of this plugin type follows from that one sentence. Core Joomla ships five of them, one per indexable core content type: categories, contacts, content, newsfeeds and tags. You can find them in the /plugins/finder folder. 

Adapter

A finder plugin is an adapter between the Smart Search indexer and some content type. You don’t directly extend your plugin from Joomla\CMS\Plugin\CMSPlugin, but from Joomla\Component\Finder\Administrator\Indexer\Adapter, an abstract class that itself extends CMSPlugin. This Adapter class lives in com_finder. 

Say, you want to create a finder plugin for some component that handles recipes. The plugin class declaration would then look like this:

final class Recipes extends Adapter implements SubscriberInterface

The fields plugins that we will look at in the next episode,  do something comparable: they extend FieldsPlugin, which lives in com_fields. 

The Properties are The Configuration

Most of what your adapter does is not written as code but declared as properties. This is our configuration for the recipes plugin:

   protected $context      = 'Recipes';      // must be unique: it keys the indexer state
   protected $extension    = 'com_recipes';
   protected $layout       = 'recipe';       // com_finder sublayout for a result
   protected $type_title   = 'Recipe';       // the content type in the Smart Search filters
   protected $table        = '#__recipes';
   protected $state_field  = 'published';    // the default is 'state'

Some explanation on these:

  • $context is used to identify this specific adapter, so it really has to be unique. The convention is the plugin Extension class name: plg_finder_content uses Content.
  • $type_title is the name of the content type. The parent constructor calls Helper::addContentType() when it does not exist yet, so the moment you enable the plugin, "Recipe" appears in the Smart Search type filter. You don't write a line for that.
  • $state_field exists because not every component calls its published column state. Our recipes table uses published, and so does the #__tags table.
  • $layout selects the sublayout com_finder uses to render a result. More about that at the end of this article.
  • There is also a $mime property, used by the adapters that index files rather than database rows.

Mapping Columns With The getListQuery Method

This method gives a query for a list of content items that you want to index. But here is the trick: you adapt the names of some columns to the names the indexer uses, by giving them an alias. In that way columns of different content types are all standardised to properties for the indexer. 

Here is an overview of all the properties that are used in the five core plugins with the table columns that are mapped to it. If a column name is different from the property name, you have to use an alias.

1. Search fields, taken straight from the item's own table

These four names are searched without you asking.
Alias your columns to them and the text is findable; call them something else and it is not.
Property Column in the query
title a.title
a.name (contacts, newsfeeds)
summary a.introtext (content)
a.description (categories, tags)
a.misc (contacts)
body a.fulltext (content only)
alias a.alias
id a.id (indexed at title weight, so an item can be found by its number)

2. Taken from a joined table

An item in an unpublished or restricted category must not turn up in a search. These two carry the category’s own state and access so the indexer can combine them with the item’s. Only for content that lives in categories; the categories and tags plugins have neither.
Property Column in the query
cat_state c.published (join #__categories on c.id = a.catid)
cat_access c.access

3. Known to the indexer, but not searched

These decide whether an item appears at all, who may see it and where it sits in time. None of their text is matched by a query.
Property Column in the query
state a.state (content)
a.published (categories, contacts, newsfeeds, tags)
access a.access
language a.language
start_date a.created (content, contacts, newsfeeds)
a.created_time (categories, tags)
publish_start_date a.publish_up
publish_end_date a.publish_down

In the core contacts finder plugin there is something special going on: #__contact_details has its own 'state' column meaning 'province', so the plugin aliases a.state AS region and a.published AS state. 

There are five declared properties no core plugin uses: subtitle, meta, comments, list_price and sale_price. They're weighted slots waiting for a finder plugin with something to put in them; worth knowing if you're writing one, because they cost nothing but an alias.

Adding Indexer Properties

With the addInstruction() method of the Indexer class you can add properties to the Indexer. You add them to one of the five “context groups” in the indexer. A different context group has a different weight.

The five context groups, their built-in properties and the default weight multiplier.
Context Properties Default multiplier
Indexer::TITLE_CONTEXT title, subtitle, id 1.7
Indexer::TEXT_CONTEXT summary, body 0.7
Indexer::META_CONTEXT meta, list_price, sale_price 1.2
Indexer::PATH_CONTEXT path, alias 2.0
Indexer::MISC_CONTEXT comments 0.3

This addInstruction() is a method of a so-called Result object, which is the item that you want to index. It is called from the index() method (see next section).

The setup and index methods

The Adapter class has two abstract methods that you have to implement:

  • setup(): is run once before indexing and returns a boolean; returning false aborts. In all five core adapters it is literally only: return true;
  • index(Result $item): this is the main method where you set how you want to index an item (= a row in the list of items that you want to index). To add a new search property you use the addInstruction() method. For instance in the core Content finder plugin the name of the user whose id is in the created_by column is aliased as ‘author’. In the index() method this author property is added to the META_CONTEXT group:
    $item->addInstruction(Indexer::META_CONTEXT, 'author');

Except for adding properties with which you index an item, you can also add taxonomies, ways to categorise data. See the core plugins for the use of $item->addTaxonomy(), for instance relating all articles of an author,  and
$item->addNestedTaxonomy(), for instance relating nested categories to an article.

Two properties you set in index() deserve care. $item->url is not really a URL but the identity of the item in the index: index.php?option=com_recipes&view=recipe&id=7. Garbage collection matches on exactly that string to find entries whose items no longer exist, so don’t get creative with it. $item->route is the link the visitor follows, and that is where your component’s RouteHelper belongs.

One call you cannot leave out is $item->state = this->translateState(item->state);, which maps your component’s state values onto the two the index knows: visible or not. It also reads the search_archived parameter, which is why all five core plugins carry that option in their form. If your content lives in categories there is a second argument; see the section on categories further on.

So the whole plugin is: those two methods, the getListQuery() from the previous section, and four thin event handlers, which the next section is about. Everything else is inherited. If your component has no categories there is one inherited method you do have to override; that gets a section of its own further on.

Events That Trigger The Plugin

There are four events your plugin handles: onFinderBeforeSave, onFinderAfterSave, onFinderAfterDelete and onFinderChangeState. There is a fifth, onFinderCategoryChangeState, which only concerns content that lives in categories. Those are not core CMS events, and nothing in com_content or in your own component dispatches them. They come from a separate plugin, Content – Smart Search (plg_content_finder), which listens to the real content events and re-dispatches them:

onContentBeforeSave   →  onFinderBeforeSave
onContentAfterSave    →  onFinderAfterSave
onContentAfterDelete  →  onFinderAfterDelete
onContentChangeState  →  onFinderChangeState
onCategoryChangeState → onFinderCategoryChangeState

If you use the core’s AdminModel, those standard content events are fired automatically. If editing content does not update the index, check whether that Smart Search content plugin is enabled before looking in your own code. 

Handling the event is first checking if we are dealing with the right content for this plugin. For instance for a recipe in com_recipes you’d get something like this in the onFinderAfterSave Event:

      if ($event->getContext() !== 'com_recipes.recipe') {
           return;
       }

To get the changed item and reindex it: 

      $item = $event->getItem();
      $this->reindex($item->id);

And some extra check if the access permissions were changed for an existing item, if applicable for your custom component. You can see an example of that in the core Content finder plugin.

The parent Adapter class already handles some Events. See in the parent getSubscribedEvents() method:

   'onStartIndex'              => 'onStartIndex',
   'onBeforeIndex'             => 'onBeforeIndex',
   'onBuildIndex'              => 'onBuildIndex',
   'onFinderGarbageCollection' => 'onFinderGarbageCollection',

You don’t handle them yourself. But of course in the plugin’s getSubscribedEvents() method you’ll have to merge that parent array with the one in your plugin. The getSubscribedEvents() method in your finder plugin will therefore look like this (mind the array_merge() function):

/**
* Returns an array of events this subscriber will listen to.
*
* @return  array
*/
public static function getSubscribedEvents(): array
{
   return array_merge(parent::getSubscribedEvents(), [
       'onFinderChangeState'         => 'onFinderChangeState',
       'onFinderAfterDelete'         => 'onFinderAfterDelete',
       'onFinderBeforeSave'          => 'onFinderBeforeSave',
       'onFinderAfterSave'           => 'onFinderAfterSave',
   ]);
}

The One Override a Component Without Categories Needs

Our recipes have no categories, and that is the simpler case, but it is also the one that needs a line of code you would never think to write, because everything appears to work until someone unpublishes a single item.

Adapter::getStateQuery() builds the query that the state and access handlers use, and it hard-codes a join:

  ->from($this->table . ' AS a')
  ->join('LEFT', '#__categories AS c ON c.id = a.catid');

It also selects c.published AS cat_state and c.access AS cat_access, and itemStateChange() and itemAccessChange() read those two properties back. Both run on the most ordinary path there is: one item unpublished, or one item's access level changed. With no catid column the full index run is perfectly happy and that single change gives you a database error.

So you override the method and keep the two properties present, which is exactly what the core tags adapter does for the same reason:

  protected function getStateQuery()
   {
       $db = $this->getDatabase();

       return db->createQuery()->select(db->quoteName('a.id'))
           ->select($db->quoteName('a.' . this->statefield,'state'))->select(db->quoteName('a.access'))
           ->select('NULL AS ' . $db->quoteName('cat_state'))
           ->select('NULL AS ' . db->quoteName('cataccess'))->from(db->quoteName($this->table, 'a'));
   }

The nulls are the whole point. Nothing here supports categories; the override exists so that inherited code written for content that has them finds the two properties it expects to read. Note the shape of the rule: nothing is required of you because you have categories, but one override is required because you have none.

While looking at this class I also noticed that Adapter still carries getUpdateQueryByTime() and getUpdateQueryByIds(), and that several core adapters dutifully override them, but nothing in Joomla 6.1 calls either method any more. They are leftovers from the incremental indexing of an earlier version. Don't copy them into your own plugin, as I first did; there is no scheduled or command line indexing in core at the moment, and these methods will not bring it back.

What Changes When Your Component Has Categories

Everything so far is a complete, working plugin. If your content does live in categories, nothing of it has to be undone: you add, and in one place you remove:

  • Drop the getStateQuery() override from the previous section and let the inherited one do its work. The join it hard-codes is now the join you want.
  • Select the category in your list query: a.catid, a left join on #__categories, and c.published AS cat_state with c.access AS cat_access.
  • Pass the category state as the second argument of translateState($item->state, $item->cat_state). An item in an unpublished category should not be findable, and this is what makes that true. The same holds for access: the inherited code takes the stricter of the two.
  • Subscribe to the fifth event, onFinderCategoryChangeState, and handle it the way the core content adapter does. Without it, unpublishing a whole category leaves its items in the index.
  • Consider addNestedTaxonomy() instead of a flat category taxonomy, so that filtering on a parent category also finds the items in its children.
  • Add the catslug CASE block to your list query only if your router actually wants it (see the note earlier in this article).

Which is a summary of this plugin type as a whole: the inherited class assumes content shaped like an article, and the work is either to meet that assumption or to say plainly where your content differs. 

Inject Dependencies in The Service Provider

A finder plugin uses the Application and the Database objects. Hence, after setting the instantiated plugin in the container, you setter inject the Application and the Database:

    $plugin->setApplication(Factory::getApplication());
    $plugin->setDatabase($container->get(DatabaseInterface::class));

By the way, since Joomla 6.1 all core plugins are so called “lazy” instantiated in the container: only at the moment you use the plugin it is instantiated. The register() method in the service provider for our recipes finder plugin will then look like:

/**
* Registers the service provider with a DI container.
*
* @param   Container  $container  The DI container.
*
* @return  void
*/
public function register(Container $container):void
{
   $container->set(
       PluginInterface::class,
       $container->lazy(Recipes::class, function (Container $container) {
           $plugin = new Recipes(
               (array) PluginHelper::getPlugin('finder', 'recipes')
           );
           $plugin->setApplication(Factory::getApplication());
           $plugin->setDatabase($container->get(DatabaseInterface::class));

           return $plugin;
       })
   );
}

Something to Index: A Demo Component

A finder plugin without a component is a lamp without electricity, so I made a deliberately small demo component, com_recipes (it is in the example repository together with the plugin), with a #__recipes table holding a title, a description, instructions, an image, a preparation time, a number of servings and a difficulty.

The interesting part is how little it has to do to become indexable. There is no Smart Search code in the component at all. The only three things that matter are:

  1. A table with the columns the adapter selects. That's it for the schema.
  2. The standard content events, with the context com_recipes.recipe. You get these for free from core: AdminModel::save() dispatches onContentBeforeSave and onContentAfterSave, AdminController::publish() dispatches onContentChangeState, and AdminModel::delete() dispatches onContentAfterDelete. The context is composed of $this->option . '.' . $this->getName(), so a RecipeModel in com_recipes produces exactly the context our plugin filters on. Build your component the ordinary way and this is simply true.
  3. A typeAlias on the table class, com_recipes.recipe, which is what tags and custom fields key off.

In other words: a component becomes searchable by being conventional. If your component does not get indexed on save, the first question is not "what does the finder plugin miss?" but "does my component fire the standard events?".

The plugin’s own list query is where those names become concrete. For recipes it is four aliases:

protected function getListQuery($query = null)
{
    $db = $this->getDatabase();

    return $query instanceof QueryInterface ? $query : $db->createQuery()
        ->select($db->quoteName(
            ['a.id', 'a.title', 'a.alias', 'a.description', 'a.instructions',
             'a.published', 'a.access', 'a.language', 'a.params', 'a.metadata',
             'a.metakey', 'a.metadesc', 'a.created', 'a.modified',
             'a.image', 'a.prep_time', 'a.servings', 'a.difficulty'],
            [null, null, null, 'summary', 'body',
             'state', null, null, null, null,
             null, null, 'start_date', null,
             null, null, null, null]
        ))
        ->from(db->quoteName(this->table, 'a'));
}

So description becomes summary, instructions becomes body, published becomes state and created becomes start_date. The image, prep_time, servings and difficulty columns are aliased to nothing: they are not names the indexer knows, and they are there for the result layout. 

index() is the only method that is genuinely specific to your content. It receives a Result object, one row from that query, enriches it, and hands it to the indexer: 

protected function index(Result $item)
{
    if (ComponentHelper::isEnabled($this->extension) === false) {
        return;
    }

    $item->setLanguage();
    $item->context = 'com_recipes.recipe';

    registry   =newRegistry(item->params);
    item->params=cloneComponentHelper::getParams(this->extension, true);
    item->params->merge(registry);

    item->metadata=newRegistry(item->metadata);

    // Runs onContentPrepare over the text before it is tokenised
    item->summary=Helper::prepareContent(item->summary, $item->params, $item);
    item->body  =Helper::prepareContent(item->body, $item->params, $item);

    $item->url   = this->getUrl(item->id, $this->extension, $this->layout);
    $item->route = $item->url;

    $item->addInstruction(Indexer::META_CONTEXT, 'metakey');
    $item->addInstruction(Indexer::META_CONTEXT, 'metadesc');

    $item->state = this->translateState(item->state);

    $item->addTaxonomy('Type', 'Recipe');
    $item->addTaxonomy('Language', $item->language);

    Helper::getContentExtras($item);
    Helper::addCustomFields($item, 'com_recipes.recipe');

    this->indexer->index(item);
}

From The Index to The Page

There is one more thing you need to know before you can show anything of your own in the search results, and it took me a while to find it.

At search time your table is never read again. Everything shown comes out of the serialised object in #__finder_links, and Result::__serialize() stores a fixed list of properties: access, description, elements, language, route, start_date, state, taxonomy, title, type_id, url and a few more. Your prep_time is not in that list.

But elements is. And Result::__set() writes every property that is not a declared class property into that elements array. So anything you invent in index() survives:

   $item->imageUrl   = $item->image ?: '';
   $item->prep_time  = (int) $item->prep_time;
   $item->servings   = (int) $item->servings;
   $item->difficulty = $item->difficulty;

This is how core smuggles imageUrl and imageAlt through to the layout, and it is the mechanism you use for your own fields. In the result layout you read them back as plain properties:

$prepTime = (int) $this->result->prep_time;

Which layout? That's the $layout property from the top of this article. com_finder's search view calls getLayoutFile('recipe'), checks whether default_recipe.php exists, and quietly falls back to default_result.php if it doesn't. Quietly: no error, no warning, just the default rendering. So for a custom presentation you copy com_finder's default_result.php into your template as a template override and adapt it:

templates/cassiopeia/html/com_finder/search/default_recipe.php

Note that this is a template override, not part of the plugin. A plugin manifest cannot install one. You either ship it as a reference copy and let the site builder move it, or you wrap plugin and override together in a package extension. The one from this article is in the example repository, under template-override/.

One detail: at the moment results are rendered, your plugin is not instantiated, so $autoloadLanguage has done nothing for that request. If your layout uses language strings from the plugin, load them yourself:

$lang = Factory::getApplication()->getLanguage();
$lang->load('plg_finder_recipes', JPATH_ADMINISTRATOR);

Disabling the Plugin Empties its Part of the Index

A last characteristic that has no equivalent in the other plugin groups. Look again at the state handler:

  if ($context === 'com_plugins.plugin' && $value === 0) {
       this->pluginDisable(event->getPks());
   }

Disabling a finder plugin does not only stop future indexing, it removes everything that adapter has indexed. Re-enabling it does not bring the entries back either; you need a new index run. For every other plugin type, switching a plugin off is a reversible act with no consequences for stored data. Here it is a data operation. Worth knowing before you toggle a plugin on a live site to test something.

Related to this is onFinderGarbageCollection, added in Joomla 4.2, which removes index entries whose underlying items have disappeared without an event ever being fired (after a direct database manipulation, for instance).

Some Useful Snippets for Finder Plugins

Reindex a single item from your own code

this->reindex(id);   // inside the adapter

Weigh a field as metadata instead of body text

$item->addInstruction(Indexer::META_CONTEXT, 'metaauthor');

Add a whole category branch as a facet

$categories = $this->getApplication()->bootComponent('com_recipes')
   ->getCategory(['published' => false, 'access' => false]);
$category   = categories->get(item->catid);

$item->addNestedTaxonomy('Category', $category, this->translateState(category->published),
   $category->access, $category->language);

Pull in tags and custom fields

Helper::getContentExtras(item);Helper::addCustomFields(item, 'com_recipes.recipe');

Use the title of a menu item, when there is one

$title = this->getItemMenuTitle(item->url);

if (!empty($title) && $this->params->get('use_menu_title', true)) {
   $item->title = $title;
}

Generating Plugins

The plugin generator I mentioned in the previous episodes is growing along with this series. A finder plugin is a good candidate for generation: the properties, the four event handlers and the list query are almost mechanical once you know the table and the column mapping, and the parts that need thinking (the taxonomies and the extra fields for the result layout) are exactly the parts you want to fill in yourself. You can find the generator at github.com/HermanPeeren/plug-gen.

Next Episode

This was the fourth episode of this series about custom plugins. We went from a general overview via task plugins and workflow plugins to the finder plugins in this article. Three types out of the 24 plugin groups in Joomla 6.x. Next episode we will explore custom fields plugins: how to make your own "custom fields".

Resources

Articles in This Series About Custom Plugins

 

About the author

Independent, creative software builder. Been around from punch cards to cloud computing. Proponent of a model driven approach.

I like modelling. When creating software you first make a model of the problem and solution. That is what philosophers have been doing for ages: building a model in order to better understand the complex world around us. The model and the modelling language are abstractions, found in a creative process.

Visit website

Comments