Possible to Create a Web Crawler That Searches Other Websites?

I want to build an app that runs searches (manually or automated) of job websites looking for specific information within their job listings and then reports that information back to me, rather than having to search all those websites manually for specific job listings. Is this possible with Bubble?

1 Like

Hi, although I’m a new user of Bubble and may be incorrect, I don’t think this is possible. What you describe is definitely something I would run as a background process or push into a worker queue, something that requires custom development. If you’re comfortable with JavaScript and Node.js, Express etc. then check out Hackathon Starter where we have a web scraper example using Cheerio.

Other than that, I’d be careful about scraping job sites and reposting the jobs without permission (if that’s your intent). Employers usually pay to post on these sites and if you’re then scraping them and listing them for free (possibly making ad revenue) then you could find yourself facing legal action.
Sorry I can’t be of any more help.

This app would be for my own personal job search, but thanks for the input!

No worries. Have you looked into Google Alerts? If you’re seeking a job, it might be a great way of being notified of job posts containing certain keywords etc.

1 Like

There are scrapers and crawlers as Node libraries that you could run server side. Of course they require some setting up effort.

Hi @calicass83

This tips from @Jici can make the job

-Read data from an external webpage

3 Likes

Also found an easy way plugin:

3 Likes

Is it possible to scrape social media (Instagram/Facebook for example) when they sign in with their Facebook login? I’d like to gather information based on their activity to help personalize their search experience in my app, showing them more relevant results.

Hi @austinjarts, did you get a soluition for this?

@calicass83 , I have a like need. Did you find a solution?