We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
G:\DaguerreSpider-master\Daguerre>scrapy crawl superspider 2021-03-21 14:17:09 [scrapy.utils.log] INFO: Scrapy 2.4.1 started (bot: Daguerre) 2021-03-21 14:17:09 [scrapy.utils.log] INFO: Versions: lxml 4.6.2.0, libxml2 2.9.5, cssselect 1.1.0, parsel 1.6.0, w3lib 1.22.0, Twisted 21.2.0, Python 3.8.8 (tags/v3.8.8:024d805, Feb 19 2021, 13:18:16) [MSC v.1928 64 bit (AMD64)], pyOpenSSL 20.0.1 (OpenSSL 1.1.1j 16 Feb 2021), cryptography 3.4.6, Platform Windows-10-10.0.19041-SP0 2021-03-21 14:17:09 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.selectreactor.SelectReactor 2021-03-21 14:17:09 [scrapy.crawler] INFO: Overridden settings: {'BOT_NAME': 'Daguerre', 'CONCURRENT_REQUESTS': 32, 'DOWNLOAD_DELAY': 1, 'NEWSPIDER_MODULE': 'Daguerre.spiders', 'ROBOTSTXT_OBEY': True, 'SPIDER_MODULES': ['Daguerre.spiders']} 2021-03-21 14:17:09 [scrapy.extensions.telnet] INFO: Telnet Password: 8853f12029b3d17c 2021-03-21 14:17:09 [scrapy.middleware] INFO: Enabled extensions: ['scrapy.extensions.corestats.CoreStats', 'scrapy.extensions.telnet.TelnetConsole', 'scrapy.extensions.logstats.LogStats'] 2021-03-21 14:17:10 [scrapy.middleware] INFO: Enabled downloader middlewares: ['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware', 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware', 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware', 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware', 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware', 'scrapy.downloadermiddlewares.retry.RetryMiddleware', 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware', 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware', 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware', 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware', 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware', 'scrapy.downloadermiddlewares.stats.DownloaderStats'] 2021-03-21 14:17:10 [scrapy.middleware] INFO: Enabled spider middlewares: ['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware', 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware', 'scrapy.spidermiddlewares.referer.RefererMiddleware', 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware', 'scrapy.spidermiddlewares.depth.DepthMiddleware'] 2021-03-21 14:17:11 [scrapy.middleware] INFO: Enabled item pipelines: ['Daguerre.pipelines.DaguerrePipeline'] 2021-03-21 14:17:11 [scrapy.core.engine] INFO: Spider opened 2021-03-21 14:17:11 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2021-03-21 14:17:11 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023 2021-03-21 14:17:13 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://cl.286x.xyz/robots.txt> (referer: None) 2021-03-21 14:17:14 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://cl.286x.xyz/thread0806.php?fid=16&search=&page=1> (referer: None) 2021-03-21 14:17:14 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://cl.286x.xyz/thread0806.php?fid=16&search=&page=2> (referer: https://cl.286x.xyz/thread0806.php?fid=16&search=&page=1) 2021-03-21 14:17:16 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://cl.286x.xyz/thread0806.php?fid=16&search=&page=3> (referer: https://cl.286x.xyz/thread0806.php?fid=16&search=&page=2) 2021-03-21 14:17:16 [scrapy.core.engine] INFO: Closing spider (finished) 2021-03-21 14:17:16 [scrapy.statscollectors] INFO: Dumping Scrapy stats: {'downloader/request_bytes': 1282, 'downloader/request_count': 4, 'downloader/request_method_count/GET': 4, 'downloader/response_bytes': 37408, 'downloader/response_count': 4, 'downloader/response_status_count/200': 4, 'elapsed_time_seconds': 5.242255, 'finish_reason': 'finished', 'finish_time': datetime.datetime(2021, 3, 21, 6, 17, 16, 313344), 'log_count/DEBUG': 4, 'log_count/INFO': 10, 'request_depth_max': 2, 'response_received_count': 4, 'robotstxt/request_count': 1, 'robotstxt/response_count': 1, 'robotstxt/response_status_count/200': 1, 'scheduler/dequeued': 3, 'scheduler/dequeued/memory': 3, 'scheduler/enqueued': 3, 'scheduler/enqueued/memory': 3, 'start_time': datetime.datetime(2021, 3, 21, 6, 17, 11, 71089)} 2021-03-21 14:17:16 [scrapy.core.engine] INFO: Spider closed (finished)
The text was updated successfully, but these errors were encountered:
No branches or pull requests
G:\DaguerreSpider-master\Daguerre>scrapy crawl superspider
2021-03-21 14:17:09 [scrapy.utils.log] INFO: Scrapy 2.4.1 started (bot: Daguerre)
2021-03-21 14:17:09 [scrapy.utils.log] INFO: Versions: lxml 4.6.2.0, libxml2 2.9.5, cssselect 1.1.0, parsel 1.6.0, w3lib 1.22.0, Twisted 21.2.0, Python 3.8.8 (tags/v3.8.8:024d805, Feb 19 2021, 13:18:16) [MSC v.1928 64 bit (AMD64)], pyOpenSSL 20.0.1 (OpenSSL 1.1.1j 16 Feb 2021), cryptography 3.4.6, Platform Windows-10-10.0.19041-SP0
2021-03-21 14:17:09 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.selectreactor.SelectReactor
2021-03-21 14:17:09 [scrapy.crawler] INFO: Overridden settings:
{'BOT_NAME': 'Daguerre',
'CONCURRENT_REQUESTS': 32,
'DOWNLOAD_DELAY': 1,
'NEWSPIDER_MODULE': 'Daguerre.spiders',
'ROBOTSTXT_OBEY': True,
'SPIDER_MODULES': ['Daguerre.spiders']}
2021-03-21 14:17:09 [scrapy.extensions.telnet] INFO: Telnet Password: 8853f12029b3d17c
2021-03-21 14:17:09 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
'scrapy.extensions.telnet.TelnetConsole',
'scrapy.extensions.logstats.LogStats']
2021-03-21 14:17:10 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
'scrapy.downloadermiddlewares.retry.RetryMiddleware',
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
'scrapy.downloadermiddlewares.stats.DownloaderStats']
2021-03-21 14:17:10 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
'scrapy.spidermiddlewares.referer.RefererMiddleware',
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
'scrapy.spidermiddlewares.depth.DepthMiddleware']
2021-03-21 14:17:11 [scrapy.middleware] INFO: Enabled item pipelines:
['Daguerre.pipelines.DaguerrePipeline']
2021-03-21 14:17:11 [scrapy.core.engine] INFO: Spider opened
2021-03-21 14:17:11 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2021-03-21 14:17:11 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2021-03-21 14:17:13 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://cl.286x.xyz/robots.txt> (referer: None)
2021-03-21 14:17:14 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://cl.286x.xyz/thread0806.php?fid=16&search=&page=1> (referer: None)
2021-03-21 14:17:14 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://cl.286x.xyz/thread0806.php?fid=16&search=&page=2> (referer: https://cl.286x.xyz/thread0806.php?fid=16&search=&page=1)
2021-03-21 14:17:16 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://cl.286x.xyz/thread0806.php?fid=16&search=&page=3> (referer: https://cl.286x.xyz/thread0806.php?fid=16&search=&page=2)
2021-03-21 14:17:16 [scrapy.core.engine] INFO: Closing spider (finished)
2021-03-21 14:17:16 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 1282,
'downloader/request_count': 4,
'downloader/request_method_count/GET': 4,
'downloader/response_bytes': 37408,
'downloader/response_count': 4,
'downloader/response_status_count/200': 4,
'elapsed_time_seconds': 5.242255,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2021, 3, 21, 6, 17, 16, 313344),
'log_count/DEBUG': 4,
'log_count/INFO': 10,
'request_depth_max': 2,
'response_received_count': 4,
'robotstxt/request_count': 1,
'robotstxt/response_count': 1,
'robotstxt/response_status_count/200': 1,
'scheduler/dequeued': 3,
'scheduler/dequeued/memory': 3,
'scheduler/enqueued': 3,
'scheduler/enqueued/memory': 3,
'start_time': datetime.datetime(2021, 3, 21, 6, 17, 11, 71089)}
2021-03-21 14:17:16 [scrapy.core.engine] INFO: Spider closed (finished)
The text was updated successfully, but these errors were encountered: