Welcome to the Treehouse Community

Want to collaborate on code errors? Have bugs you need feedback on? Looking for an extra set of eyes on your latest project? Get support with fellow developers, designers, and programmers of all backgrounds and skill levels here with the Treehouse Community! While you're at it, check out some resources Treehouse students have shared here.

Looking to learn something new?

Treehouse offers a seven day free trial for new students. Get access to thousands of hours of content and join thousands of Treehouse students and alumni in the community today.

Start your free trial

Python Scraping Data From the Web Additional Scraping Tasks Using Scrapers for Site Testing

Posted by Stanisław Dudzik
Stanisław Dudzik
9,731 Points

Can you use selenium for scrapy spiders sraping js-based websites?

I am wondering how could you implement similar solution shown here for beautiful soup with scrapy. I am tinkering with a spider of my own and having a hardtime figuring out how to make my spider wait for js-cripts to load.

Stanisław Dudzik
Stanisław Dudzik
9,731 Points

I disabled js in my browser while entering website of my choice and it turns out it doesn't show up so I guess this is the issue I am having. Not sure though. I saw people using not selenium but splash with scrapy. Anyone have any recommendations? Each and every comment would mean world to me :)