Setting up Prerendering in the Scrivito Example App

Today, many websites are fully fledged JS applications that, moreover, integrate third-party libraries and services, or remotely stored content, into their business logic or page layout. If such websites are accessed under poor bandwidth conditions, fetching scripts and assets as well as remotely stored CMS content may take some time, causing impatient visitors to leave the site prematurely. This is where prerendering comes in. It additionally lets crawlers not able to handle single page JavaScript apps see the structure and content of your site, which is a prerequisite for SEO to be effective.

Prerendering doesn’t solve the underlying performance problem but counters it by doing some of the time and bandwidth consuming work in advance, on the (cloud-based) server side. 

In the case of apps based on the Scrivito Example App and served by our hosting partner Netlify, prerendering can be made an integral step in the deployment process and can also be triggered each time changes to the content are published. In this tutorial, we’ll show you how.

With prerendering, static versions of all or specific pages are generated beforehand and delivered instead of their dynamic, fully JS-enabled versions. Afterwards, these static pages are given back their full functionality. This way, the visible parts of the pages (including all their content) are available to visitors much earlier compared to pages assembled by scripts running in a low performing client-side network environment.

That said, let’s take a look at how prerendering can be enabled.

Prerequisites

This guide is based on the Scrivito Example App and presupposes that continuous deployment is switched on for your app in the “Build and deploy” settings on Netlify. 

Continuous deployment frees you from having to manually build your app and upload it to Netlify. Instead, you are maintaining your app in a Git repository on GitHub, GitLab, or BitBucket, and connect it to your Netlify site. This enables Netlify to automatically build and deploy your app whenever you push to your Git repository. You might need to add Netlify as a third-party app to your account at your repository provider.

Configure the build and publishing details

First, edit your “netlify.toml” file located at the root of your repository, and specify the build command Netlify should issue, as well as the directory to be used for storing the files generated by the prerendering process, publish:

netlify.toml
Copy
# Global settings applied to the whole site.
# “command” is your build command.
# “publish” is the directory to publish (relative to root of your repo).

[build]
  command = "npm run prerender"
  publish = "buildPrerendered"

Netlify executes the build command whenever it detects that the specified branch of your repository has been pushed to.

Additionally, the command needs to be executed after a working copy of the Scrivito CMS involved has been published. So, next, let’s set up the hook for on-the-fly prerendering on publish. We will need an incoming build hook URL from Netlify and specify it as a publish web hook in the dashboard of the Scrivito CMS concerned.

Netlify lets you obtain a build hook URL only after the continuous deployment setup has been completed. Then, return to the “Continuous Deployment” of your site on Netlify, scroll down to the “Build hooks” section, and add a build hook:

After saving, the hook URL is displayed and should be copied to the clipboard:

Now, before adding the URL to your Scrivito CMS settings, scroll down a bit further and deselect Netlify’s prerendering option for your site (in the “Post processing” section) in case it is selected. Netlify’s prerendering is targeted at bots and doesn’t optimize your website for fast loading.

Coming back to the hook URL, open the “Settings” of your Scrivito CMS in your dashboard, edit the “Publish Webhooks” section, paste the URL to the corresponding form field, and click “Save”:

The URL is requested on every publish, causing the command specified in the “netlify.toml” file to be executed, i.e., the prerendering build to be generated and deployed.

You can inspect the build creation process via the “Deploys” section of your site on Netlify.

SEE SCREENSHOT

Configuring prerendering

Exempting content not suited for prerendering

Not all content is suited for being prerendered. Binaries, e.g. images, as well as special-purpose CMS objects such as redirects or invisible data objects, should be exempted to not waste time on producing output that is never used.

Since all CMS objects are based on object classes, it seems natural to offer a mechanism for excluding CMS objects by their object class. This is exactly what the PRERENDER_OBJ_CLASSES_BLACKLIST array in the “src/prerender_content.js” file is for. As a default, it includes a couple of names of standard object classes whose instances are to be ignored during prerendering:

src/prerender_content.js
Copy
const PRERENDER_OBJ_CLASSES_BLACKLIST = [
  "Download",
  "Image",
  "Redirect",
  "Video",
];

Configuring the sitemap contents

On prerendering your website, Scrivito automatically generates a sitemap for crawlers to see and properly index your content. The CMS objects to include can be specified analogously to the list of object classes above, however, the array, SITEMAP_OBJ_CLASSES_WHITELIST, is a whitelist. In the Example App it is set to:

src/prerender_content.js
Copy
const SITEMAP_OBJ_CLASSES_WHITELIST = [
  "Author",
  "Blog",
  "BlogPost",
  "Event",
  "Homepage",
  "Job",
  "LandingPage",
  "Page",
];

Reporting errors

The Example App outputs errors to the console. If prerendering a page causes an error, the page is simply skipped and reportError() in “storePrerenderedContent.js” is called:

storePrerenderedContent.js
Copy
function reportError(message, ...args) {
  // Report to your external error tracker here, like Honeybadger or Rollbar.
  console.log(`  ❌ [reportError] ${message}`, ...args);
}

You may want to replace this basic error logging with a call to your favorite application error tracking and health monitoring service like Honeybadger.

Adjusting the output template

The prerenderer fills in a template, “src/prerenderContent/generateHtml.js” to generate the pages of your website. If you want to adjust the contents of this file (e.g. add a meta tag or call additional scripts), the non-prerendering version, “public/catch_all_index.html”, should be checked for consistency.

Test it locally

For testing (and debugging) the prerendering procedure, simply execute from within the root of your app directory:

Copy
npm run prerender
cd buildPrerendered
npx serve -l 8080

The last of the above commands starts a webserver on localhost:8080 for you to check the prerendered pages using the browser of your choice.

If you have been experimenting with your local Example App installation, we recommend to remove odd ends, especially dysfunctional widgets, prior to prerendering as they cause the page concerned to be skipped and thus to be unavailable in the browser.

If your app isn’t prerendered as expected, switching to development mode for debugging should be helpful. From your project’s root directory run:

Copy
SCRIVITO_PRERENDER=true npm start

Then open http://localhost:8080/_prerender_content.html in your browser and execute prerenderContent() in the JS console. This allows you follow the prerendering process much more closely and find the source of error faster.

How it works

First, executing npm run prerender sets SCRIVITO_PRERENDER to true and then invokes npm run build. The SCRIVITO_PRERENDER flag causes webpack to also include “src/prerender_content.js” and “_prerender_content.html” in the “build” output folder.

In the second step, the command executes node storePrerenderedContent.js. This starts a local webserver on port 8080, which serves the contents of the  “build” directory. Then, a headless chrome browser (“puppeteer”) is started and pointed to http://localhost:8080/_prerender_content.html. Finally prerenderContent() (from “src/prerender_content.js”) is invoked in the browser to generate the prerendered files and store them in the “buildPrerendered” directory.

prerenderContent() first generates the “sitemap.xml” file and then for every page an HTML file as well as a “preloadDump-obj-id.js” file (where “obj-id” is the CMS object ID of the page). The latter file is included as a script in the generated HTML file. It contains all the data required to render the requested page without having to fetch the content from the Scrivito CMS service, reducing page load times to a minimum.