Would it make sense to parse the HTML to only "analyze" the text?

[rasulkireev]

I was thinking would it make sense to add to scrapeghost the ability to parse HTML into text (or markdown even) and let GPT analyze that? That would prob reduce the cost even further right?

The logic should be simple. Something like this:

import requests
import html2text

def html_to_markdown(url):
    # Get HTML content from URL
    response = requests.get(url)
    html_content = response.content.decode('utf-8')

    # Convert HTML to markdown
    markdown = html2text.html2text(html_content)

    return markdown

Add as an option in the Parser. @jamesturk if you think this is something that will help Scrapeghost, I can try making a PR.

[jamesturk]

Hi Rasul,

Converting to text loses a lot of context that can be used to determine page structure. Imagine a page like

<span>John Smith</span>
<span class="party">D</span>
<span class="district-number">33</span>

When converted to text or markdown you'd wind up with John Smith D 33 & the helpful information is lost.

[rasulkireev]

That makes a lot of sense, did not think about it.