How to make Drupal Search API index the full rendered HTML content

Posted in: Blog, Drupal | Mar 2, 2018 at 1:44 PM

No comments

INTRODUCTION

Drupal is somehow a selected content management system by a lot of government bodies in Western Australia. While it’s a great little system, I still feel that it needs a lot of improvements especially around the search area.

One thing that struck me was the fact that it did not index the final rendered HTML content. Yes it can index content fields fine, but it definitely does not index the final rendered HTML – which sucks in my opinion. The reason is, you would have had a lot of custom content blocks dropped onto the different content placeholders which would appear on your content pages. But, since the only content indexed is your main page content field, when you try to look for a text within your rendered custom blocks, the search results won’t return the corresponding page(s) where the blocks may be loaded to.

I’ve installed Search API and have tried using the default index or even SOLR! And yet both came to no avail. The main reason behind it is not so much about the search technology, but rather the content passed from Drupal to these search indexer does not contain the full rendered HTML content. Therefore, no matter what search technology you use, it won’t return the proper results since the content is never indexed in the first place.

RESOLUTION

Finally, after some fiddling around, I found a workable solution! And no, it’s not installing Google search but rather a modification to the search API file.

Search API introduced a search field called “Rendered Item”. You can add this field in the Fields area of your index, as well as your Search API Pages. The idea behind it is great, that is to render the actual HTML output of the field. Therefore, I hijack this field to index the actual page’s HTML output instead of just the field’s HTML output – if that makes sense?

Anyway, the Rendered Item column is contained within this file:

\modules\search_api\src\Plugin\search_api\processor\RenderedItem.php

This is the line that actually adds the field value into the index:

public function addFieldValues(ItemInterface $item) 

So, I hijack it to include the rendered page HTML. I’m using this method to download the rendered page HTML:

file_get_contents

My method now looks like below. Please look at the line in red and green.

function SiteURL()
{
$protocol = (!empty($_SERVER[‘HTTPS’]) && $_SERVER[‘HTTPS’] !== ‘off’ || $_SERVER[‘SERVER_PORT’] == 443) ? “https://” : “http://”;
$domainName = $_SERVER[‘HTTP_HOST’].’/’;
return $protocol.$domainName;
}

/**
* {@inheritdoc}
*/
public function addFieldValues(ItemInterface $item) {
$original_user = $this->currentUser->getAccount();

// Switch to the default theme in case the admin theme is enabled.
$active_theme = $this->getThemeManager()->getActiveTheme();
$default_theme = $this->getConfigFactory()
->get(‘system.theme’)
->get(‘default’);
$default_theme = $this->getThemeInitializer()
->getActiveThemeByName($default_theme);
$this->getThemeManager()->setActiveTheme($default_theme);

// Count of items that don’t have a view mode.
$unset_view_modes = 0;

$fields = $this->getFieldsHelper()
->filterForPropertyPath($item->getFields(), NULL, ‘rendered_item’);
foreach ($fields as $field) {

$idString = $item->getId() . ”;

if(strpos($idString, ‘node/’) != false)
{
$idString = str_replace(“entity:node/”, “”, $idString);
$idString = str_replace(“:en”, “”, $idString);

$html = \file_get_contents($this->SiteURL() . ‘/node/’ . $idString);
$field->addValue($html);
}
else
{
$configuration = $field->getConfiguration();

// Change the current user to our dummy implementation to ensure we are
// using the configured roles.
$this->currentUser->setAccount(new UserSession([‘roles’ => $configuration[‘roles’]]));

$datasource_id = $item->getDatasourceId();
$datasource = $item->getDatasource();
$bundle = $datasource->getItemBundle($item->getOriginalObject());
// When no view mode has been set for the bundle, or it has been set to
// “Don’t include the rendered item”, skip this item.
if (empty($configuration[‘view_mode’][$datasource_id][$bundle])) {
// If it was really not set, also notify the user through the log.
if (!isset($configuration[‘view_mode’][$datasource_id][$bundle])) {
++$unset_view_modes;
}
continue;
}
else {
$view_mode = (string) $configuration[‘view_mode’][$datasource_id][$bundle];
}

$value = (string) $this->getRenderer()->renderPlain($build);
if ($value) {
$field->addValue($value);
}

}
}

// Restore the original user.
$this->currentUser->setAccount($original_user);
// Restore the original theme.
$this->getThemeManager()->setActiveTheme($active_theme);

if ($unset_view_modes > 0) {
$context = [
‘%index’ => $this->index->label(),
‘%processor’ => $this->label(),
‘@count’ => $unset_view_modes,
];
$this->getLogger()->warning(‘Warning: While indexing items on search index %index, @count item(s) did not have a view mode configured for one or more “Rendered item” fields.’, $context);
}
}

Basically, I’m just telling Drupal, if you load a content node, download the full page HTML output and add it to the index.

if(strpos($idString, ‘node/’) != false)
{
$idString = str_replace(“entity:node/”, “”, $idString);
$idString = str_replace(“:en”, “”, $idString);

$html = \file_get_contents($this->SiteURL() . ‘/node/’ . $idString);
$field->addValue($html);
}

The addValue(string) is the method that adds any string to the index.

DONE!

Delete all item from the index, then do a full re-index 🙂

NOTE: As this is updating Search API’s system file, when you upgrade the module, ensure you re-apply the changes because it would have been overridden by the upgrade.

Hope this helps,

Tommy

No Comments Yet.