Prerender & cache your SPA pages for crawlers on Laravel

This package intends to make it easier to serve prerendered pages to crawlers for better SEO.

You could make use of it if

• you are running Laravel as backend for your single-page webapp or
• parts of your Laravel app are generated using Javascript.

It could be used as a

• middleware for third-party prerender service (like prerender.io) but
• it can also cache prerendered responses & keep cached responses up to date (for running your local prerender server).

Installation

Via Composer

$ composer require tsekka/prerender

Preparing the database

The package loads migrations

php artisan migrate

Publishing the config file

Publishing the config file is optional.

php artisan vendor:publish --provider="Tsekka\Prerender\PrerenderServiceProvider" --tag="config"

Registering prerender middleware

Enable prerendering for all routes by adding PRERENDER_REGISTER_GLOBALLY=true in your .env file; or add middleware to specific routes:

// app/Http/Kernel.php
  protected $routeMiddleware = [
    // ...
    'prerender' => 
        \Tsekka\Prerender\Http\Middleware\PrerenderMiddleware::class,
  ];

// routes/web.php
Route::get('/{path?}', 'SPAController')->where('path', '.*')
    ->middleware('prerender');

Prerendering by third party service

By using prerender.io or similar service, you don't have to install node server and headless chrome by yourself.

The primary use case of this package is to make it easier to run custom prerender server & cache it's responses. However, the easiest way to start prerendering the pages for crawlers is by using third-party service like prerender.io.

1. Register at prerender.io or at another similar service and follow their instructions.
2. Set prerenderer's url PRERENDER_URL=https://service.prerender.io and token PRERENDER_TOKEN=YOUR-THIRD-PARTY-TOKEN (add this in your .env file)
3. Prerender.io already caches the pages for speed, so you can turn off local cache PRERENDER_CACHE_TTL=null (.env)
4. Register the middleware and you're good to go!

Running your own prerender service

To run your own prerender service, you must have Node, headless Chrome and their dependencies installed in your webserver.

Prerender.io has open-sourced node server that you can use to prerender the pages at your server.

Here's how you can make use of it:

1. Install and run prerenderer's node server.
   • Clone it from this package's directory (cp -r ./vendor/tsekka/prerender/prerenderer ./prerenderer) and install dependencies cd prerenderer && npm install.
   • You can also follow this quick tutorial which includes instructions how to install headless Chrome browser on Debian-based Linux distributions.
2. Set prerenderer's url to url of your prerenderer's service. Eg. if you're running it locally, then add PRERENDER_URL=http://localhost:3000 to your .env file.
3. Decide if you will keep the prerender server constantly running or if you would rather start the server for the duration of schedule command.
   • If you will keep the server constantly running, then start the prerendering server node server.js and make sure that the node server will re-start even after webserver is rebooted.
   • If you would rather start the server only for the duration of prerender command, then set PRERENDER_RUN_LOCAL_SERVER=true in your .env file.
4. Register the middleware
5. Prerendering the page on-demand can be slow and therefore, by default the pages will be cached.
6. It's recommended that you set up the schedule to re-cache the prerendered pages.

Caching prerendered responses

Prerendering the page can take up to few seconds or even more.

Therefore the pages will be cached by default for 1 week.

You can change cache time-to-live and cache driver by setting .env variables PRERENDER_CACHE_TTL and PRERENDER_CACHE_DRIVER or by publishing and modifying the config file.

If you're using third-party service like Prerender.io, then the responses are probably already cached, so you can turn off local cache (add PRERENDER_CACHE_TTL=null to your .env).

Running cache command

You can run php artisan prerender:cache command to cache all pages that are defined in array of cacheable urls.

By default, the cache command only caches urls that have not been cached yet or whose cache ttl have already expired. You can run cache command with --force option (php artisan prerender:cache --force) to re-cache all urls.

Keeping prerendered pages up to date

Prerendering the page on-demand can be slow and it's therefore recommended to keep fresh copy of pages constantly in cache.

You could set up event listener to keep the prerendered page in sync

    // 1. Set up event-listener
    // 2. Inside your listener:
    public function handle($event)
    {
       return \Artisan::call('prerender:cache', [
                'url' => '/your-model-resource-url',
                '--force' => true,
                '--log' => false,
        ]);
    }

Or schedule prerendering and recaching on specified time

    // app/Console/Kernel.php
    protected function schedule(Schedule $schedule)
    {
        // Daily re-cache all urls that's cache-time-to-live is expired
         $schedule->command('prerender:cache')->dailyAt("02:00");
         
        // Daily re-cache all urls
         $schedule->command('prerender:cache --force')->dailyAt("02:00");
    }

Providing list of urls to cache

Each time crawler visits the url that matches all requirements for it to be prerendered, the prerendered response will be cached and the url of request will be recorded in database.

So by default, before you actually start using the package, the list will be empty and the urls will be prerendered at the time of request (and therefore the request time could be quite slow, as prerendering takes time).

If you would like to cache the pages only on demand or you would like to keep response time low even on first crawler visit, then you should provide a class & method name that returns array of of urls by publishing config file and modifying it's cacheable_urls value.

Pruning old entries

The package logs all crawler visits into database.

Use php artisan prerender:prune command to clear old entries.

You can also schedule the artisan command:

    // app/Console/Kernel.php
    protected function schedule(Schedule $schedule)
    {
        // Daily prune all crawler visit entries older than 1 month
        $schedule->command('prerender:prune "1 month"')->daily();
    }

Whitelisting urls

Whitelist paths or patterns. You can use asterix syntax. If a whitelist is supplied, only url's containing a whitelist path will be prerendered. An empty array means that all URIs will pass this filter. Note that this is the full request URI, so including starting slash and query parameter string.

// prerender.php:
'whitelist' => [
    '/frontend/*' // only prerender pages starting with '/frontend/'
],

Blacklisting urls

Blacklist paths to exclude. You can use asterix syntax. If a blacklist is supplied, all url's will be prerendered except ones containing a blacklist path. By default, a set of asset extentions are included (this is actually only necessary when you dynamically provide assets via routes). Note that this is the full request URI, so including starting slash and query parameter string.

// config/prerender.php 
'blacklist' => [
    '/api/*' // do not prerender pages starting with '/api/'
    // ...
],

Other resources

If you don't know why or when you should prerender your SPA apps, then there are some resources for you to check out:

• blog post at netlify.com,
• prerender.io
• stackoverflow question.

Contributing

This package is under development. Contributions are appreciated and will be credited.

Security

If you discover any security related issues, please email kristjan@pintek.ee instead of using the issue tracker.

Credits

• Forked from Jeroen Noten's Laravel-Prerender

License

MIT. Please see the license file for more information.