Automated Data-Extraction & Website Integration Developer
Project Description
We are looking for an experienced developer to build a reliable automated data-extraction system for 8 third-party websites and integrate the results into our website/plugin through an API.
The system should:
* Extract specific publicly/permissibly accessible data from 8 websites automatically.
* Provide the extracted data through a clean API for integration with our website/plugin.
* Support proxy management and IP rotation where permitted.
* Handle sessions, cookies, rate limits, retries, timeouts and temporary website failures.
* Support JavaScript/dynamic websites using tools such as Playwright, Puppeteer or Selenium where required.
* Include monitoring/logging so failed requests can be identified and retried.
* Be designed for high reliability and easy maintenance if a website changes.
* Store/cache relevant results to reduce unnecessary requests.
* Include documentation for the API and extraction system.
Required experience: web scraping/data extraction, API development, browser automation, proxy infrastructure, backend development, databases, error handling and scalable automation. Show More
The system should:
* Extract specific publicly/permissibly accessible data from 8 websites automatically.
* Provide the extracted data through a clean API for integration with our website/plugin.
* Support proxy management and IP rotation where permitted.
* Handle sessions, cookies, rate limits, retries, timeouts and temporary website failures.
* Support JavaScript/dynamic websites using tools such as Playwright, Puppeteer or Selenium where required.
* Include monitoring/logging so failed requests can be identified and retried.
* Be designed for high reliability and easy maintenance if a website changes.
* Store/cache relevant results to reduce unnecessary requests.
* Include documentation for the API and extraction system.
Required experience: web scraping/data extraction, API development, browser automation, proxy infrastructure, backend development, databases, error handling and scalable automation. Show More
Attachments
Skills
Freelancers Bidding (0)
-
This project has no proposals yet.
Be the first to place a bid on this project!
Budget
$750 - $1,125
Client Info
-
- Demo USER
-
Project Summary
-
Status
-
Category
Web Development
-
Project Type
Fixed Price
-
Budget
$750 - $1,125
-
Date Posted
16 days ago
-
Project ID
4900
-
Project Views
306
