Course 40 - Web Scraping with Python | Episode 5: From Environment Setup to Pandas DataFrames — CyberCode Academy

In this lesson, you’ll learn about: setting up a professional Python scraping environment, extracting web data step-by-step, and transforming raw HTML into structured datasets1. Setting Up Your Development Environment🔹 Python Version ManagementUse pyenv

Install and switch between Python versions بسهولة
Avoid compatibility issues across projects

🔹 Virtual Environments & DependenciesUse pipenv

Create isolated environments
Manage dependencies like:
- requests
- BeautifulSoup4
- pandas

👉 Key Insight
Clean environment = fewer bugs + reproducible projects🔹 Interactive DevelopmentUse JupyterLab

Run code in cells step-by-step
Inspect outputs instantly
Explore files and HTML visually

2. Downloading & Inspecting Web Content🔹 Fetching HTML PagesUse Requestsimport requests url = "https://example.com" response = requests.get(url) html = response.text 🔹 Why Save Locally?

Work offline
Avoid repeated requests
Debug faster

🔹 Inspecting the PageUse:

JupyterLab HTML viewer
Browser DevTools (Elements tab)

👉 Goal:
Locate the exact HTML structure of your target data (e.g., tables, divs)3. Extracting Data with BeautifulSoup🔹 Parsing HTMLUse BeautifulSoupfrom bs4 import BeautifulSoup soup = BeautifulSoup(html, "html.parser") 🔹 Using CSS Selectorstable = soup.select("table.wikitable")[0] rows = table.select("tr") 👉 This allows precise targeting of elements4. Cleaning the Data🔹 Fix Column Names

Remove whitespace
Replace spaces with _

clean_header = header.text.strip().replace(" ", "_") 🔹 Remove Unwanted Patterns (Regex)Use Regular Expressionimport re clean_text = re.sub(r"\[.*?\]", "", raw_text) 👉 Removes things like:

[1], [citation needed]

5. Structuring the Data🔹 Build a “List of Lists”data = [] for row in rows: cols = [col.text.strip() for col in row.select("td")] data.append(cols) 👉 Structure becomes:[ ["Name", "Age", "City"], ["John", "25", "NY"], ] 6. Creating a DataFrame🔹 Use PandasUse pandasimport pandas as pd df = pd.DataFrame(data[1:], columns=data[0]) 🔹 Why DataFrames Matter

Easy filtering
Data analysis
Export to CSV/Excel

7. Full Workflow (Big Picture)

Setup environment (pyenv + pipenv)
Fetch HTML (Requests)
Inspect structure (DevTools / Jupyter)
Extract data (BeautifulSoup)
Clean data (Regex + string ops)
Structure data (lists)
Analyze (Pandas DataFrame)

Mental ModelRaw HTML → Parsed DOM → Extracted Elements → Clean Data → Structured Dataset → Analysis👉 Final Takeaway
A successful scraping project is not just about extraction—
it’s about building a clean, repeatable pipeline that turns messy web content into usable data.

You can listen and download our episodes for free on more than 10 different platforms:
https://linktr.ee/cybercode_academy