Is there a more detailed guide to creating a new bridge?

[arozbiz]

I'm interested in creating a new bridge (for ssrn.com, specifically). I've read the documentation and have looked at some example bridges, but I'm still fundamentally not sure where to get started. I don't have a good understanding of rss-bridge actually works under the hood. E.g., how does it crawl the website to find new posts (to then process into an RSS feed)? This is clearly a very basic question, but I can't seem to find enough of an answer to give me the conceptual overview I need to start hacking on a new bridge.

[dvikan]

You have to manually scrape and parse out items. Not much automatic here. Example:

<?php

class BearBlogBridge extends BridgeAbstract
{
    const NAME = 'BearBlog (bearblog.dev)';

    public function collectData()
    {
        $dom = getSimpleHTMLDOM('https://herman.bearblog.dev/blog/');
        foreach ($dom->find('.blog-posts li') as $li) {
            $a = $li->find('a', 0);
            $this->items[] = [
                'title' => $a->plaintext,
                'uri' => 'https://herman.bearblog.dev' . $a->href,
            ];
        }
    }
}

You also need to learn css-style traversing of the DOM. Example: $dom->find('div.items > item > p')

[arozbiz]

This is immensely helpful, thank you. To make sure I understand, rss-bridge will, on some pre-determined interval, scrape based on the collectData() function and then, on the basis of some parsing magic, create RSS etc.? But how does this work for much larger sites? E.g., if 'https://herman.bearblog.dev/blog/' had a million posts, would rss-bridge process each of them every time it scrapes? Is that what the cache is for? If so, how does rss-bridge know before scraping what it needs to scrape or not?

[dvikan]

The default behavior is that rss-bridge scrapes everything for each request a feed reader does. Yes a million posts each time.
On top of this there is caching so that later requests don't scrape those million posts each time a feed reader requests the feed.
It scrapes based off of a timeout which default to 24h.

[arozbiz]

Got it. So if you want to have a max number of items returned in the RSS feed, that has to be handled from within the function itself. OK, I think I get it!