Downloader

Receives a Request object from a http middleware before cycle and creates an HTTP request to a page. HTTP response is processed into a Response object, passed into http middleware after cycle and further into a spider for processing page data.

Override DOWNLOADER class when you wish to define custom downloader functionality.