From the outside it looks like every site is its own special case, so adding a site to a feed really means writing a small crawler for that site. I don't think it can be solved for ANY site from scratch. If I were building this, I'd limit the source list (a local newsletter probably needs a few dozen sources, not the whole web) and write a dedicated bot for each site. With AI help that part isn't hard anymore. Then it becomes a question of support: noticing when a site changes its layout and fixing that one bot.
@onerinas, no, it is to get the new content on any site when it's published.
I need it to get new content across all kinds of web and social channels so that
#heynews can curate the best stories for a user's newsletter.
Some common issues I face:
I'm using RSS.app to resolve these issues, so HeyNews can follow all kinds of sources.
Oh I see. I was curious as I had worked on a rss reader app recently. Your use case sounds like way more complicated and a fun problem to solve.
Unfortunately I don't have any experience on that.
From the outside it looks like every site is its own special case, so adding a site to a feed really means writing a small crawler for that site. I don't think it can be solved for ANY site from scratch. If I were building this, I'd limit the source list (a local newsletter probably needs a few dozen sources, not the whole web) and write a dedicated bot for each site. With AI help that part isn't hard anymore. Then it becomes a question of support: noticing when a site changes its layout and fixing that one bot.