Replace middleman-search with lunr with Algolia DocSearch #706

tnir · 2022-07-18T13:58:41Z

What was the end-user problem that led to this PR?

As middleman-search (RubyGems, GitHub) has not been maintained for more than 5.5 years (for example manastech/middleman-search#38), bundler-site maintainers/contributors cannot upgrade lunr to the latest without their additional efforts (or other community's efforts than maintainers/contributors of this repo).

Closes #691

What was your diagnosis of the problem?

UI can be replaced by https://github.com/algolia/docsearch or its forks. Backend (current /search/lunr-index.json) can be replaced by Algolia DocSearch with automated crawling or Typesense cloud with manual crawling by us.

What is your fix for the problem, implemented in this PR?

🎉 Removes middleman-search gem and relevant gems (mini_racer and libv8-node).
🎉 Removes JS/CSS codes and Middleman configuration for middleman-search.
🎉 Removes HTML snippets thanks to new UI from DocSeach v3.
🎉 Removes (the load of) Popover CSS from Bootstrap 5 along with the above JS code removal.
🏗️ Also the load of /application.min.js is moved to outside of <head>.

The crawler is manually run by @tnir at any time.

Screenshots

PC (as-is)

PC (searching)

Mobile (as is)	Mobile (searching)

Old screenshots for previous changeset

Mobile (as is)	Mobile (searching)

Checklist

--docsearch-searchbox-background is now #ebedf0 (default), which is away from "Bunlder color scheme", which does not exist 😁
(optional) searchbox has 750px-breakpoint for mobile while the current site powered by Bootstrap 5 has 768px for navbar.
- cf. [docsearch-css] Breakpoint is to be more de facto value algolia/docsearch#1444
  - was addressed in feat(styles): set medium breakpoint to standarized value algolia/docsearch#1446

Why did you choose this fix out of the possible options?

Algolia's DocSearch for open source projects gets ready in a few days. If Typesense offers

a some tiny instance to us at no cost
and the permission to the UI
DocSearch v3-ready UI for enduser,

we might be able to use #702, but this change looks clear at this moment.

Signed-off-by: Takuya Noguchi takninnovationresearch@gmail.com

shortcuts · 2022-07-18T21:36:06Z

Hey, small feedback to improve your DocSearch results, you can update the Crawler config to better scope the lvl0 (e.g. h1), so you have more sections appearing in the modal (like on https://docsearch.algolia.com/)

tnir · 2022-07-19T09:06:56Z

@shortcuts Thanks, but I could not see any good element for lvl0 at this moment. I planned to update them by just adding section name after information architecting 😁 What do you think?

shortcuts · 2022-07-19T09:10:43Z

@shortcuts Thanks, but I could not see any good element for lvl0 at this moment. I planned to update them by just adding section name after information architecting 😁 What do you think?

The main title of the page, the navbar section/active subsection is what we recommend

deivid-rodriguez

Looks nice!

I agree that Algolia Docsearch seems easier for the moment.

Is it possible to recrawl the website on every deploy instead of daily?

While testing I noticed some extra search results which should not be needed. For example, when looking for "docker", one of the results is a link to the sidebar: https://bundler.io/guides/bundler_docker_guide.html#sidebar-wrapper. Can we avoid generating these redundant results?

assets/javascripts/docsearch.js

deivid-rodriguez · 2022-07-19T10:10:10Z

source/layouts/base.haml

@@ -30,3 +30,6 @@

    .footer
      = partial 'layouts/footer'
+    = javascript_include_tag "https://cdn.jsdelivr.net/npm/@docsearch/js@3.1.1/dist/umd/index.min.js"
+    = javascript_include_tag "https://cdn.jsdelivr.net/npm/bootstrap@5.2.0-beta1/dist/js/bootstrap.min.js"


Why is this? Is it related to not longer needing Popover?

I used to think this is not needed. By removing the final load import { Popover } from 'bootstrap'; from assets/javascripts/search.js, it looks like bootstrap esm JS will not loaded any more, which will break navbar collapse. So with the current situation of slowness of asset (https://bundler.io/application.min.js) downloading from GitHub Pages.

The below image shows how fast JavaScripts are delivered from CDN (DocSearch JS, Bootstgrap JS, and our own JS from top to bottom):

See also https://getbootstrap.com/docs/5.1/getting-started/webpack/#importing-javascript

simi · 2022-07-19T10:44:14Z

Is the search branded right now? That would be big 👎 for me.

tnir · 2022-07-19T10:53:53Z

@simi Yes I see you, but Typesense will cause the same result...

tnir · 2022-07-19T11:14:42Z

@simi As the next step to this PR, we can prepare our own cluster on Algolia (or Typesense cloud). We can estimate how much it will be if some sponsors pay for it.

If this is not possible for you, you're free to open your own Algolia account and run DocSearch on your own without this limitation. In that case, though, depending on the size of your documentation, you might need a paid account (free accounts can hold as much as 10k records).

https://docsearch.algolia.com/docs/DocSearch-program

deivid-rodriguez · 2022-07-19T11:29:47Z

In any case, it seems best for us to be in full control of when we run the crawler (after every deploy), so running our own cluster would seem like the way to go.

shortcuts · 2022-07-19T11:59:11Z

Is it possible to recrawl the website on every deploy instead of daily?

Sure, we have a GitHub action available: https://github.com/algolia/algoliasearch-crawler-github-actions (which works with DocSearch :))

tnir · 2022-07-19T12:01:51Z

In any case, it seems best for us to be in full control of when we run the crawler (after every deploy), so running our own cluster would seem like the way to go.

Is it possible to recrawl the website on every deploy instead of daily?

My previous comment is at least now incorrect (my memory might be from years ago...). As described in the doc below, we can schedule it more frequently and trigger it manually. Not sure we can do it via API/CLI though:

Crawls are scheduled at a random time once a week. You can configure this schedule from the config file or trigger one manually from the Crawler interface.

https://www.algolia.com/doc/tools/crawler/apis/configuration/schedule/ still says every 24 hours: schedule: 'every 1 day at 3:00 pm',

While testing I noticed some extra search results which should not be needed. For example, when looking for "docker", one of the results is a link to the sidebar: https://bundler.io/guides/bundler_docker_guide.html#sidebar-wrapper. Can we avoid generating these redundant results?

Crawler configuration can be modified at https://crawler.algolia.com/admin/crawlers/7f4b6579-bba3-4c1f-a0ff-462d9f408281/configuration/edit (I sent an invitation to you). Just hours ago, I updated it as follows:

new Crawler({
  rateLimit: 8,
  startUrls: ["https://bundler.io"],
  renderJavaScript: false,
  sitemaps: [],
  ignoreCanonicalTo: false,
  discoveryPatterns: ["https://bundler.io/**"],
  schedule: "at 5:44 PM on Sunday",
  actions: [
    {
      indexName: "bundler",
      pathsToMatch: ["https://bundler.io/**"],
      recordExtractor: ({ helpers }) => {
        return helpers.docsearch({
          recordProps: {
            lvl1: [
              ".commands h2#NAME",
              "h1",
              "#page-content-wrapper h2",
              "head > title",
            ],
            content: [
              "#page-content-wrapper p, #page-content-wrapper li",
              "p",
              ".team .card-body",
              "span.contributor",
            ],
            lvl0: {
              selectors: "",
              defaultValue: "Documentation",
            },
            lvl2: ["h3"],
            lvl3: ["h4"],
            lvl4: ["h5"],
            lvl5: ["article h5", "main h5", "h5"],
            lvl6: ["article h6", "main h6", "h6"],
          },
          aggregateContent: true,
          recordVersion: "v3",
        });
      },
    },
  ],
  initialIndexSettings: {
    bundler: {
      attributesForFaceting: ["type", "lang"],
      attributesToRetrieve: [
        "hierarchy",
        "content",
        "anchor",
        "url",
        "url_without_anchor",
        "type",
      ],
      attributesToHighlight: ["hierarchy", "content"],
      attributesToSnippet: ["content:10"],
      camelCaseAttributes: ["hierarchy", "content"],
      searchableAttributes: [
        "unordered(hierarchy.lvl0)",
        "unordered(hierarchy.lvl1)",
        "unordered(hierarchy.lvl2)",
        "unordered(hierarchy.lvl3)",
        "unordered(hierarchy.lvl4)",
        "unordered(hierarchy.lvl5)",
        "unordered(hierarchy.lvl6)",
        "content",
      ],
      distinct: true,
      attributeForDistinct: "url",
      customRanking: [
        "desc(weight.pageRank)",
        "desc(weight.level)",
        "asc(weight.position)",
      ],
      ranking: [
        "words",
        "filters",
        "typo",
        "attribute",
        "proximity",
        "exact",
        "custom",
      ],
      highlightPreTag: '<span class="algolia-docsearch-suggestion--highlight">',
      highlightPostTag: "</span>",
      minWordSizefor1Typo: 3,
      minWordSizefor2Typos: 7,
      allowTyposOnNumericTokens: false,
      minProximity: 1,
      ignorePlurals: true,
      advancedSyntax: true,
      attributeCriteriaComputedByMinProximity: true,
      removeWordsIfNoResults: "allOptional",
    },
  },
  appId: "3JA5LRH987",
  apiKey: "masked",
});

tnir · 2022-07-19T12:15:18Z

Updated the primary color for box-shadow etc. in searchbox and modal as I have forgotten the adaption of Bundler branding color.

tnir · 2022-07-22T04:19:15Z

Admin role is now requested to Org members/Repo admins in #730 to set up periodic indexing on Actions 🙏

tnir

I will fix it soon.

source/layouts/base.haml

Signed-off-by: Takuya Noguchi <takninnovationresearch@gmail.com>

tnir · 2022-07-27T09:20:35Z

@deivid-rodriguez How can I trigger a review app for this (old) PR (just for development purpose)?

deivid-rodriguez · 2022-07-29T06:37:48Z

I do it manually from Heroku UI, I created it for you.

tnir self-assigned this Jul 18, 2022

deivid-rodriguez temporarily deployed to bundler-site-tnir-algol-j2zyh5 July 18, 2022 13:58 Inactive

tnir force-pushed the tnir-algolia-docsearch-691 branch from f34b2cf to 5351a54 Compare July 18, 2022 16:05

deivid-rodriguez temporarily deployed to bundler-site-tnir-algol-j2zyh5 July 18, 2022 16:05 Inactive

tnir force-pushed the tnir-algolia-docsearch-691 branch from 5351a54 to bb85a7e Compare July 18, 2022 16:23

deivid-rodriguez temporarily deployed to bundler-site-tnir-algol-j2zyh5 July 18, 2022 16:23 Inactive

tnir force-pushed the tnir-algolia-docsearch-691 branch from bb85a7e to 9abd7c2 Compare July 18, 2022 16:29

deivid-rodriguez temporarily deployed to bundler-site-tnir-algol-j2zyh5 July 18, 2022 16:30 Inactive

tnir requested a review from deivid-rodriguez July 18, 2022 16:55

tnir force-pushed the tnir-algolia-docsearch-691 branch from 9abd7c2 to ba24618 Compare July 19, 2022 01:17

deivid-rodriguez temporarily deployed to bundler-site-tnir-algol-j2zyh5 July 19, 2022 01:17 Inactive

deivid-rodriguez reviewed Jul 19, 2022

View reviewed changes

tnir force-pushed the tnir-algolia-docsearch-691 branch from ba24618 to eff461e Compare July 19, 2022 12:12

deivid-rodriguez temporarily deployed to bundler-site-tnir-algol-j2zyh5 July 19, 2022 12:13 Inactive

tnir force-pushed the tnir-algolia-docsearch-691 branch from eff461e to 846f7c4 Compare July 19, 2022 14:42

deivid-rodriguez temporarily deployed to bundler-site-tnir-algol-j2zyh5 July 19, 2022 14:42 Inactive

tnir force-pushed the tnir-algolia-docsearch-691 branch from 846f7c4 to e85d9be Compare July 19, 2022 14:44

deivid-rodriguez temporarily deployed to bundler-site-tnir-algol-j2zyh5 July 19, 2022 14:44 Inactive

tnir force-pushed the tnir-algolia-docsearch-691 branch from e85d9be to dd60648 Compare July 19, 2022 15:28

deivid-rodriguez temporarily deployed to bundler-site-tnir-algol-j2zyh5 July 19, 2022 15:28 Inactive

tnir force-pushed the tnir-algolia-docsearch-691 branch from dd60648 to d2fdc7e Compare July 22, 2022 00:16

deivid-rodriguez temporarily deployed to bundler-site-tnir-algol-j2zyh5 July 22, 2022 00:16 Inactive

tnir force-pushed the tnir-algolia-docsearch-691 branch from d2fdc7e to 2b5f07e Compare July 22, 2022 02:02

deivid-rodriguez temporarily deployed to bundler-site-tnir-algol-j2zyh5 July 22, 2022 02:02 Inactive

tnir requested a review from deivid-rodriguez July 22, 2022 02:27

tnir mentioned this pull request Jul 22, 2022

Attach Maintain role to @tnir #730

Closed

tnir mentioned this pull request Jul 22, 2022

Evaluate Typesense cloud (poc) #731

Open

6 tasks

tnir marked this pull request as draft July 23, 2022 23:43

tnir removed the request for review from deivid-rodriguez July 23, 2022 23:43

tnir added this to the Architecture overhaul milestone Jul 23, 2022

tnir force-pushed the tnir-algolia-docsearch-691 branch from 2b5f07e to 093b187 Compare July 27, 2022 08:54

tnir commented Jul 27, 2022

View reviewed changes

source/layouts/base.haml Outdated Show resolved Hide resolved

tnir force-pushed the tnir-algolia-docsearch-691 branch 2 times, most recently from 3f2bf0e to 5238184 Compare July 27, 2022 09:08

Replace middleman-search with lunr with Algolia DocSearch

2c82de3

Signed-off-by: Takuya Noguchi <takninnovationresearch@gmail.com>

tnir force-pushed the tnir-algolia-docsearch-691 branch from 5238184 to 2c82de3 Compare July 27, 2022 09:10

tnir marked this pull request as ready for review July 27, 2022 09:10

deivid-rodriguez temporarily deployed to bundler-site-tnir-algol-qcaxiy July 29, 2022 06:37 Inactive

tnir modified the milestones: Architecture overhaul H2 2022, Re-architecturing H1 2023 Dec 30, 2022

tnir modified the milestones: Re-architecturing H1 2023, Re-architecturing H2 2023 Jul 10, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Replace middleman-search with lunr with Algolia DocSearch #706

Replace middleman-search with lunr with Algolia DocSearch #706

tnir commented Jul 18, 2022 •

edited

shortcuts commented Jul 18, 2022 •

edited

tnir commented Jul 19, 2022

shortcuts commented Jul 19, 2022

deivid-rodriguez left a comment •

edited

deivid-rodriguez Jul 19, 2022

tnir Jul 19, 2022

simi commented Jul 19, 2022

tnir commented Jul 19, 2022 •

edited

tnir commented Jul 19, 2022

deivid-rodriguez commented Jul 19, 2022

shortcuts commented Jul 19, 2022 •

edited

tnir commented Jul 19, 2022

tnir commented Jul 19, 2022

tnir commented Jul 22, 2022 •

edited

tnir left a comment

tnir commented Jul 27, 2022 •

edited

deivid-rodriguez commented Jul 29, 2022

Replace middleman-search with lunr with Algolia DocSearch #706

Are you sure you want to change the base?

Replace middleman-search with lunr with Algolia DocSearch #706

Conversation

tnir commented Jul 18, 2022 • edited

What was the end-user problem that led to this PR?

What was your diagnosis of the problem?

What is your fix for the problem, implemented in this PR?

Screenshots

PC (as-is)

PC (searching)

Checklist

Why did you choose this fix out of the possible options?

shortcuts commented Jul 18, 2022 • edited

tnir commented Jul 19, 2022

shortcuts commented Jul 19, 2022

deivid-rodriguez left a comment • edited

Choose a reason for hiding this comment

deivid-rodriguez Jul 19, 2022

Choose a reason for hiding this comment

tnir Jul 19, 2022

Choose a reason for hiding this comment

simi commented Jul 19, 2022

tnir commented Jul 19, 2022 • edited

tnir commented Jul 19, 2022

deivid-rodriguez commented Jul 19, 2022

shortcuts commented Jul 19, 2022 • edited

tnir commented Jul 19, 2022

tnir commented Jul 19, 2022

tnir commented Jul 22, 2022 • edited

tnir left a comment

Choose a reason for hiding this comment

tnir commented Jul 27, 2022 • edited

deivid-rodriguez commented Jul 29, 2022

tnir commented Jul 18, 2022 •

edited

shortcuts commented Jul 18, 2022 •

edited

deivid-rodriguez left a comment •

edited

tnir commented Jul 19, 2022 •

edited

shortcuts commented Jul 19, 2022 •

edited

tnir commented Jul 22, 2022 •

edited

tnir commented Jul 27, 2022 •

edited