Searching for the term C++ is broken

[hofmandl1]

Describe the bug

Instead of searching for C++ sphinx searches for C (two spaces after the C)

I've done some investigation on the matter:

sphinx/sphinx/themes/basic/static/doctools.js

Line 54 in af62fa6

var key = jQuery.urldecode(tmp[0]);

These two lines

    var key = jQuery.urldecode(tmp[0]);
    var value = jQuery.urldecode(tmp[1]);

do too much decoding. Instead I suppose that https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/decodeURIComponent should be used.

To Reproduce

Go to https://www.sphinx-doc.org/en/master/search.html?q=C%2B%2B

On the sphinx search page In your browser's Javascript Console, type

jQuery.urldecode('C%2B%2B') // e.g. a user wants to search for the language `C++`, leading for the URI to be search.html?q=C%2B%2B

-> C
so sphinx will actually search for C instead of C++.

Expected behavior

I would like sphinx to search for C++

[hofmandl1]

Hmm .. I just saw, that this happens on purpose, somewhat above:

jQuery.urldecode = function(x) {
  return decodeURIComponent(x).replace(/\+/g, ' ');
};

[hofmandl1]

When trying to fix it locally, I ran into:

jquery.js:2 Uncaught SyntaxError: Invalid regular expression: /c++/: Nothing to repeat
    at String.match (<anonymous>)
    at Object.performTermsSearch (searchtools.js:396)
    at Object.query (searchtools.js:205)
    at Object.performSearch (searchtools.js:146)
    at Object.init (searchtools.js:79)
    at HTMLDocument.<anonymous> (searchtools.js:503)
    at e (jquery.js:2)
    at t (jquery.js:2)

which I fixed using the attached patch

searchtools.js.escape_search_regex.patch.txt

[hofmandl1]

Ok, Now I see, why

jQuery.urldecode = function(x) {
  return decodeURIComponent(x).replace(/\+/g, ' ');
};

was done.
It needs to be just the other way round:

jQuery.urldecode = function(x) {
  return decodeURIComponent(x.replace(/\+/g, ' '));
};

also see https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/decodeURIComponent#Decoding_query_parameters_from_a_URL

[tk0miya]

Thank you for reporting. But, unfortunately, our indexer does not support a word containing "+". So it will be failed even if we fixed the decoding logic because search-index does not contain "C++" as a keyword.

[hofmandl1]

we have defined a custom search-language with a custom splitter that is very simple and doesn't throw away punctutations, thus in our (commercial) project we can now nicely search for C++:

    def split(self, content_snippet: str) -> List[str]:
        """Especially the "splitting" of "en" is way too agressive for
           technical documentation. It splits search terms on all kinds
           of punctuations, that all too often are parts of search-terms
           our customers use."""

        # !!! This must match the javascript implementation as closely as possible !!!
        return [x for x in content_snippet.split(' ') if x]

we also are locally applying 2 more patches to searchtools.js removing the feature that excludes searchterms with a leading - and the one that excludes digit-only search-terms in order to make the search more useful for us.

The leading - feature bit us hard, because we want to document things like command-line switches. Digit-only search terms are also quite common I guess e.g. when searching for a keyword in combination with a version number or similar.

I would be willing to start a PR dumping in these changes if someone is interested, however I don't think I'd have the time to get this merge-ready.