Welcome to the Treehouse Community

Want to collaborate on code errors? Have bugs you need feedback on? Looking for an extra set of eyes on your latest project? Get support with fellow developers, designers, and programmers of all backgrounds and skill levels here with the Treehouse Community! While you're at it, check out some resources Treehouse students have shared here.

Looking to learn something new?

Treehouse offers a seven day free trial for new students. Get access to thousands of hours of content and join thousands of Treehouse students and alumni in the community today.

Start your free trial

Python Scraping Data From the Web A World Full of Spiders Installing Scrapy

Posted by Nathan English
.a{fill-rule:evenodd;}techdegree
Nathan English
Front End Web Development Techdegree Student 10,817 Points

How do you create a scrapy project?

I'm using pycharm. I have scrapy installed on my project I go to the terminal. I type "scrapy startproject AraneaSpider"

I'm fairly new to using pycharm, I was following the directions in this course. Also I'm using windows.

Then I get this error.

  File "C:\Users\ID611\PycharmProjects\web_scraping\venv\Scripts\scrapy.exe\__main__.py", line 5, in <module>
ModuleNotFoundError: No module named 'scrapy.cmdline'; 'scrapy' is not a package
Eric M
Eric M
11,546 Points
Eric M
Eric M
11,546 Points

Hi Nathan,

What happens if you type pip show scrapy in your venv terminal?

3 Answers

Eric M
Eric M
11,546 Points
Eric M
Eric M
11,546 Points

Although Scrapy is showing as installed it's likely something has gone wrong with the install. Perhaps with one of Scrapy's dependencies or path.

If you've only installed via the video's instructions you should only have Scrapy as a project install. You can verify this by running pip show scrapy from a command line outside of Pycharm's venv. Such as a regular powershell terminal. It will be blank.

This means you can try creating a brand new PyCharm project and installing Scrapy again there (the same way as from the video). Be sure to wait for the green install successful message as appears in the video (though it may take a bit longer than in the video).

Nathan English
seal-mask
.a{fill-rule:evenodd;}techdegree
Nathan English
Front End Web Development Techdegree Student 10,817 Points
Nathan English
.a{fill-rule:evenodd;}techdegree
Nathan English
Front End Web Development Techdegree Student 10,817 Points

(venv) C:\Users\ID611\PycharmProjects\web_scraping>pip show scrapy Name: Scrapy Version: 1.5.1 Summary: A high-level Web Crawling and Web Scraping framework Home-page: https://scrapy.org Author: Scrapy developers Author-email: None License: BSD Location: c:\users\id611\pycharmprojects\web_scraping\venv\lib\site-packages Requires: pyOpenSSL, cssselect, queuelib, PyDispatcher, service-identity, parsel, six, w3lib, lxml, Twisted Required-by: You are using pip version 10.0.1, however version 18.1 is available. You should consider upgrading via the 'python -m pip install --upgrade pip' command.

Nathan English
seal-mask
.a{fill-rule:evenodd;}techdegree
Nathan English
Front End Web Development Techdegree Student 10,817 Points
Nathan English
.a{fill-rule:evenodd;}techdegree
Nathan English
Front End Web Development Techdegree Student 10,817 Points

Thank you for the fix, I just created a new project and ran the same commands.

Eric M
Eric M
11,546 Points
Eric M
Eric M
11,546 Points

No worries Nathan, glad it worked!