In modern e-commerce and retail, product prices and availability change rapidly. Retailers who track competitor prices manually face serious delays, high labor costs, and frequent human errors.
Nexora is an automated data pipeline that continuously and ethically collects product data from permitted public e-commerce websites. The system cleans and standardizes the incoming text, detects duplicate products using smart approximate text matching, and reconciles external listings with the company's internal product catalog.
All clean data is loaded into a relational database (PostgreSQL), which tracks full price history over time (historical snapshots) and logs product updates (such as category or name modifications). The entire workflow is automated on a regular schedule using Apache Airflow, providing business stakeholders with clear analytical SQL reports and visual interactive dashboards.
| In-Scope (What We Are Building) | Out-of-Scope (Project Boundaries) |
|---|---|
| • Extracting product title, category, price, rating, stock status, and URL from permitted public websites. | • Scraping private data behind login portals, password protections, or paywalls. |
• Checking robots.txt rules and applying polite delays between web requests. |
• Bypassing CAPTCHAs, bot blockers, or anti-scraping security perimeters. |
| • Cleaning text, normalizing currencies, and matching products using fuzzy algorithms. | • Sub-second real-time streaming architectures (e.g., Apache Kafka / Apache Flink). |
| • PostgreSQL relational database with historical price snapshots and attribute version tracking. | • Non-relational NoSQL document or graph databases. |
| • Scheduled batch execution and automated failure retries using Apache Airflow. | • High-frequency continuous trading or automated dynamic repricing execution bots. |
| • Pre-built analytical SQL queries and visual business dashboards. | • Distributed multi-cloud Kubernetes clusters or mobile application interfaces. |
The project is structured into 5 iterative phases. Delivery windows are intentionally kept flexible to accommodate university schedules and mid-term examinations occurring between mid-to-late November. Exact internal target delivery dates will be finalized collaboratively by the team.
| Phase Block | Phase Name | Core Scope & Key Deliverables |
|---|---|---|
| Phase 1 | Planning, Research & System Architecture | Architectural blueprint, GitHub repository setup, literature review, requirements specification, and project management plan. |
| Phase 2 | Web Ingestion & Storage Foundations | Configured Scrapy spiders, robots.txt compliance validator, Dockerized PostgreSQL database, and table schema scripts. |
| Phase 3 | Data Cleaning, Deduplication & Historical Tracking | Regular expression (regex) cleaning modules, currency converters, RapidFuzz entity matching engine, and historical data tracking logic. |
| Phase 4 | Orchestration, Data Quality & SQL Analytics | Apache Airflow Directed Acyclic Graphs (DAGs), automated retry policies, data quality validation gates, and analytical SQL queries. |
| Phase 5 | Dashboards, Testing & Final Capstone Delivery | Power BI / Streamlit interactive dashboard, complete unit and integration test suite, final project report, and demonstration presentation. |
To ensure maximum ownership and clear task accountability, each team member is directly responsible and accountable for executing their assigned technical module. As Team Leader, Saif Wael coordinates sprint milestones, leads team meetings, manages the repository and documentation submissions, while directly architecting and executing the web crawling and data ingestion pipelines.
| Core Work Package | Saif Wael (Lead) | Salma Mohamed | Youtham Romany | Hazem Osama | Reem Hassan | Rana Samy |
|---|---|---|---|---|---|---|
| Web Ingestion & Robots.txt Compliance | A / R | I | I | I | I | I |
| Data Cleansing & Currency Normalization | C | A / R | C | C | C | C |
| Architecture Design & Data Modeling | C | C | A / R | C | I | I |
| Fuzzy Matching & Catalog Reconciliation | C | I | C | A / R | I | C |
| Airflow Orchestration & Docker Setup | C | I | I | I | A / R | I |
| Analytical SQL Queries & BI Dashboard | C | I | I | I | C | A / R |
| Quality Assurance & Testing | C | R | R | R | R | R |
| Project Coordination & Submissions | A / R | I | I | I | I | I |
| Identified Risk | Severity | Likelihood | Plain-Language Explanation & Mitigation Strategy |
|---|---|---|---|
| Target Website IP Rate Limiting / Blocking | High | Medium | Explanation: If a scraper sends too many requests too fast, the website may temporarily block the IP address. Mitigation: Implement polite request delays (1.5 seconds between pages), randomized intervals, rotate user-agent headers, and test primarily against allowed demo/sandbox websites. |
| Target Website HTML Structure Updates (DOM Changes) | High | High | Explanation: When an e-commerce website updates its design or changes its HTML tags/classes, scraping selectors can fail and return empty fields. Mitigation: Keep all CSS and XPath selectors isolated in external YAML configuration files (so spiders don't need code changes) and add automated validation checks that trigger alerts immediately if expected fields are missing. |
| Inconsistent Price Formats & Currency Symbols | Medium | High | Explanation: Product pages use mixed formats (e.g., "$1,250.00", "EGP 1.250,00", "LE 500"), which can cause errors when stored as numbers. Mitigation: Build robust regular expression (regex) cleaners that handle different thousand/decimal separators and standardize all currencies into uniform numeric decimals with ISO codes. |
| Fuzzy Matching False Positives | High | Medium | Explanation: When matching external products to our internal catalog, approximate text algorithms might mistakenly match two completely different items (e.g., "iPhone 15 Case" matched to "iPhone 15 Phone"). Mitigation: Always perform exact Stock Keeping Unit (SKU) matching first. For remaining items, use fuzzy string matching with a strict similarity threshold (above 88%) and exclude common marketing buzzwords. |
| System Resource Limits (Airflow & Docker) | Medium | Low | Explanation: Running multiple containers simultaneously on developer laptops can consume high RAM and CPU. Mitigation: Use lightweight Python workers, limit task concurrency in Docker Compose, and clean up temporary files after each batch run. |
In the retail industry, competitive pricing is one of the most critical factors driving customer purchase decisions. Retailers who regularly track market trends can optimize their profit margins and avoid pricing themselves out of the market.
However, collecting this data manually by having staff browse websites is slow, incomplete, and impossible to scale. Modern data engineering pipelines solve this problem by converting unstructured public web data into clean, structured records ready for business analysis.
| Evaluation Criteria | SaaS Scraper Tools (e.g., Octoparse) | Enterprise Pricing Platforms (e.g., Price2Spy) | Custom In-House Solution (Nexora) |
|---|---|---|---|
| Licensing & Total Cost of Ownership (TCO) | Requires recurring monthly subscriptions based on volume. | Expensive enterprise software contracts. | 100% open-source tools with zero software licensing costs. |
| Internal Catalog Reconciliation | None; only exports raw CSV or JSON files. | Rigid, pre-built matching with limited customization. | Custom fuzzy text matching designed to link directly with internal company SKUs. |
| Historical Data Modeling | Saves basic flat files without database versioning. | Pre-aggregated vendor dashboards; no direct database access. | Full relational PostgreSQL database with historical price snapshots and attribute change tracking. |
| Workflow Automation | Basic built-in timers with limited custom error handling. | Proprietary closed-source scheduler. | Apache Airflow Directed Acyclic Graphs (DAGs) with automated retries and Data Quality (DQ) checks. |
| Data Ownership & Privacy | Data is stored on third-party cloud servers. | Vendor lock-in; data stays in vendor systems. | Complete private control and ownership over all company data. |
1. Retail Pricing Analyst: Needs daily reports and visual charts showing competitor price changes to recommend price adjustments.
2. Category & Sourcing Manager: Needs alerts when competitors launch brand-new items or discontinue products to identify gaps in our own inventory.
3. Data Engineer / DevOps: Needs an automated, reliable pipeline with comprehensive logs, data validation gates, and clear failure alerts.
| Req ID | Feature Module | Requirement Specification |
|---|---|---|
| FR-01 | Automated Ingestion | The system shall crawl designated e-commerce websites and extract product title, category, price, rating, stock availability, and product URL. |
| FR-02 | Compliance Enforcement | The scraper must verify robots.txt permissions before crawling and enforce polite request delays. |
| FR-03 | Text Sanitization | The cleaning module shall strip HTML tags, remove unwanted symbols, and normalize whitespace in product names. |
| FR-04 | Currency Normalization | The pipeline shall extract numeric price values from text, handle comma/dot decimal separators, and assign standard ISO currency codes. |
| FR-05 | Deduplication & Entity Matching | The system shall detect duplicate listings and link scraped products to internal catalog items using exact Stock Keeping Unit (SKU) checks and approximate text matching (RapidFuzz with a similarity cutoff of 88% or higher). |
| FR-06 | Historical Metadata & Price Tracking | The warehouse shall track product updates (such as name or category changes) using Slowly Changing Dimensions (SCD Type 2), keeping past records intact while marking current records as active. |
| FR-07 | Fact Price Snapshots | Every pipeline run shall record timestamped price and stock availability snapshots into a dedicated historical table. |
| FR-08 | Analytical SQL Queries | The system shall include pre-built SQL analytical queries for business insights, including price change calculations, new product detection, discontinued item detection, and category drift analysis. |
| FR-09 | Airflow Orchestration | The end-to-end pipeline shall be managed by an Apache Airflow DAG that controls task execution order, handles retries, and logs run history. |
| FR-10 | Visualization Dashboard | The final data shall be displayed in an interactive dashboard (Power BI or Streamlit) showing key pricing metrics and trends. |