docs.scrapy.orgScrapy 1.8 documentation — Scrapy 1.8.0 documentation

-- Scrapy latest First steps Scrapy at a glance Installation guide Scrapy Tutorial Examples Basic concepts Command line tool Spiders Selectors Items Item Loaders Scrapy shell Item Pipeline Feed exports Requests and Responses Link Extractors Settings Exceptions Built-in services Logging Stats Collection Sending e-mail Telnet Console Web Service Solving specific problems Frequently Asked Questions Debugging Spiders Spiders Contracts Common Practices Broad Crawls Using your browser’s Developer Tools for scraping Selecting dynamically-loaded content Debugging memory leaks Downloading and processing files and images Deploying Spiders AutoThrottle extension Benchmarking Jobs: pausing and resuming crawls Extending Scrapy Architecture overview Downloader Middleware Spider Middleware Extensions Core API Signals Item Exporters All the rest Release notes Contributing to Scrapy Versioning and API Stability Scrapy Docs » Scrapy 1.8 documentation Edit on GitHub Scrapy 1.8 documentation ¶ Scrapy is a fast high-level web crawling and web scraping framework, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining to monitoring and automated testing. Getting help ¶ Having trouble? We’d like to help! Try the FAQ – it’s got answers to some common questions. Looking for specific information? Try the Index or Module Index . Ask or search questions in StackOverflow using the scrapy tag . Ask or search questions in the Scrapy subreddit . Search for questions on the archives of the scrapy-users mailing list . Ask a question in the #scrapy IRC channel , Report bugs with Scrapy in our issue tracker . First steps ¶ Scrapy at a glance Understand what Scrapy is and how it can help you. Installation guide Get Scrapy installed on your computer. Scrapy Tutorial Write your first Scrapy project. Examples Learn more by playing with a pre-made Scrapy project. Basic concepts ¶ Command line tool Learn about the command-line tool used to manage your Scrapy project. Spiders Write the rules to crawl your websites. Selectors Extract the data from web pages using XPath. Scrapy shell Test your extraction code in an interactive environment. Items Define the data you want to scrape. Item Loaders Populate your items with the extracted data. Item Pipeline Post-process and store your scraped data. Feed exports Output your scraped data using different formats and storages. Requests and Responses Understand the classes used to represent HTTP requests and responses. Link Extractors Convenient classes to extract links to follow from pages. Settings Learn how to configure Scrapy and see all available settings . Exceptions See all available exceptions and their meaning. Built-in services ¶ Logging Learn how to use Python’s builtin logging on Scrapy. Stats Collection Collect statistics about your scraping crawler. Sending e-mail Send email notifications when certain events occur. Telnet Console Inspect a running crawler using a built-in Python console. Web Service Monitor and control a crawler using a web service. Solving specific problems ¶ Frequently Asked Questions Get answers to most frequently asked questions. Debugging Spiders Learn how to debug common problems of your scrapy spider. Spiders Contracts Learn how to use contracts for testing your spiders. Common Practices Get familiar with some Scrapy common practices. Broad Crawls Tune Scrapy for crawling a lot domains in parallel. Using your browser’s Developer Tools for scraping Learn how to scrape with your browser’s developer tools. Selecting dynamically-loaded content Read webpage data that is loaded dynamically. Debugging memory leaks Learn how to find and get rid of memory leaks in your crawler. Downloading and processing files and images Download files and/or images associated with your scraped items. Deploying Spiders Deploying your Scrapy spiders and run them in a remote server. AutoThrottle extension Adjust crawl rate dynamically based on load. Benchmarking Check how Scrapy performs on your hardware. Jobs: pausing and resuming crawls Learn how to pause and resume crawls for large spiders. Extending Scrapy ¶ Architecture overview Understand the Scrapy architecture. Downloader Middleware Customize how pages get requested and downloaded. Spider Middleware Customize the input and output of your spiders. Extensions Extend Scrapy with your custom functionality Core API Use it on extensions and middlewares to extend Scrapy functionality Signals See all available signals and how to work with them. Item Exporters Quickly export your scraped items to a file (XML, CSV, etc). All the rest ¶ Release notes See what has changed in recent Scrapy versions. Contributing to Scrapy Learn how to contribute to the Scrapy project. Versioning and API Stability Understand Scrapy versioning and API stability. Next © Copyright 2008–2018, Scrapy developers Revision be2e910d . Built with Sphinx using a theme provided by Read the Docs . Read the Docs v: latest Versions master latest stable 1.8 1.7 1.6 1.5 1.4 1.3 1.2 1.1 1.0 0.24 0.22 0.20 0.18 0.16 0.14 0.12 0.10.3 0.9 xpath-tutorial Downloads pdf html epub On Read the Docs Project Home Builds Free document hosting provided by Read the Docs ....