Description
Description
I am running splash by the first time in Ubuntu (virtual Machine) with python3 in PyCharm. I am following this video: https://www.youtube.com/watch?v=rofkkuSf9iA&t=168s
First I set up de settings.py:
BOT_NAME = 'scrapy_splash'
SPIDER_MODULES = ['scrapy_splash.spiders']
NEWSPIDER_MODULE = 'scrapy_splash.spiders'
ROBOTSTXT_OBEY = True
SPLASH_URL = 'http://localhost:8050'
DOWNLOADER_MIDDLEWARES = {
'scrapy_splash.SplashCookiesMiddleware': 723,
'scrapy_splash.SplashMiddleware': 725,
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 810,
}
SPIDER_MIDDLEWARES = {
'scrapy_splash.SplashDeduplicateArgsMiddleware': 100,
}
DUPEFILTER_CLASS = 'scrapy_splash.SplashAwareDupeFilter'
HTTPCACHE_STORAGE = 'scrapy_splash.SplashAwareFSCacheStorage'
Then
import scrapy
from scrapy.spiders import Spider
from scrapy_splash.items import GameItem
from scrapy_splash import SplashRequest
and following the same structure described in the video.
When running scrapy crawl scrapy_splash I got this error: No module named 'scrapy_splash.settings'
I've tried to look for info on .settings without succeed.
Any idea what can it ca be?
This is the answer in the terminal.
carlos@carlos:~/Scrapy_scripts/scrapy_splash$ scrapy crawl ss
Traceback (most recent call last):
File "/home/carlos/.local/bin/scrapy", line 11, in <module>
sys.exit(execute())
File "/home/carlos/.local/lib/python3.6/site-packages/scrapy/cmdline.py", line 114, in execute
settings = get_project_settings()
File "/home/carlos/.local/lib/python3.6/site-packages/scrapy/utils/project.py", line 68, in get_project_settings
settings.setmodule(settings_module_path, priority='project')
File "/home/carlos/.local/lib/python3.6/site-packages/scrapy/settings/__init__.py", line 294, in setmodule
module = import_module(module)
File "/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 994, in _gcd_import
File "<frozen importlib._bootstrap>", line 971, in _find_and_load
File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'scrapy_splash.settings'
Versions
Scrapy : 1.7.4
lxml : 4.4.1.0
libxml2 : 2.9.9
cssselect : 1.1.0
parsel : 1.5.2
w3lib : 1.21.0
Twisted : 19.7.0
Python : 3.6.8 (default, Oct 7 2019, 12:59:55) - [GCC 8.3.0]
pyOpenSSL : 19.0.0 (OpenSSL 1.1.1d 10 Sep 2019)
cryptography : 2.8
Platform : Linux-5.0.0-32-generic-x86_64-with-Ubuntu-18.04-bionic
Metadata
Metadata
Assignees
Labels
Type
Projects
Milestone
Relationships
Development
Participants
Activity
lopuhin commentedon Nov 1, 2019
I'm not sure, but it could be a conflict of the library name (scrapy_splash) and your project name (the same) - so could you try re-doing with another project name? Unless this resolves the problem, https://github.com/scrapy-plugins/scrapy-splash/ issue tracker or stackoverflow would likely be a better place to ask for help with this issue, as this does not look to be a scrapy bug.
crianopa commentedon Nov 1, 2019
OK Thanks. I will check and ask in stackoverflow and in /issue tracker.