Course 40 - Web Scraping with Python | Episode 13: Mastering Scrapy Shell, CSS, and XPath Selectors — CyberCode Academy

In this lesson, you’ll learn about: how to use Scrapy Shell for interactive crawling, how CSS selectors work for fast extraction, and how XPath enables advanced and flexible data targeting1. What is Scrapy Shell?🔹 Interactive Prototyping ToolScrapy Shell is a live testing environment where you can:

Test selectors before writing spiders
Inspect HTML responses instantly
Experiment with scraping logic

🔹 Key Objects Inside Shell

response → HTML content of the page
request → HTTP request details
spider → scraper context

👉 Key Insight
You can test everything before writing real crawling logic2. Working with Live URLs and FilesScrapy Shell supports:

🌐 Live websites
📄 Local HTML files

👉 This makes it ideal for debugging broken or complex pages3. CSS Selectors (Fast & Simple)🔹 Basic Extraction🔹 Common SyntaxSelectorMeaning#idSelect by ID.classSelect by classtagSelect by tag🔹 Scrapy Shell Methodsresponse.css("title").get() response.css("p").get_all() 🔹 Extract Attributesresponse.css("img::attr(src)").get() 👉 Key Insight
CSS is perfect for quick, readable extraction4. Important Behavior: Cached Responses🔹 One Hidden DetailScrapy Shell:

Works on cached HTML
Won’t reflect live changes unless restarted

👉 Key Insight
Always restart shell when debugging updated pages5. XPath Selectors (Advanced Power)🔹 Full DOM NavigationXPath lets you navigate HTML like a tree structure6. Absolute vs Relative XPath🔹 Absolute Path/html/body/div/p

Starts from root
Very strict

🔹 Relative Path//div/p

Searches anywhere
More flexible

7. Attribute Matching in XPath🔹 Using @response.xpath("//img[@id='logo']").get() 🔹 Using Wildcardsresponse.xpath("//*[@id='main']").get() 8. Using contains()🔹 Pattern Matchingresponse.xpath("//p[contains(text(), 'news')]").get() 👉 Key Insight
XPath is powerful for uncertain or messy HTML structures9. CSS vs XPathFeatureCSSXPathSimplicity✔️ Easy❌ More complexPower❌ Limited✔️ Very powerfulFlexibilityMediumVery high10. Mental ModelThink of Scrapy Shell as:

A laboratory
CSS = quick filters
XPath = surgical precision tools

Final TakeawayScrapy Shell bridges the gap between:👉 “guessing selectors”
and
👉 “engineered extraction logic”Once you master CSS + XPath inside the shell, you can confidently build spiders that work on even the most complex websites.

You can listen and download our episodes for free on more than 10 different platforms:
https://linktr.ee/cybercode_academy