3. Run a spider using the hoaxly-scraping-container
Author: Luis Rosenstrauch
Last updated
Author: Luis Rosenstrauch
Last updated
Respository:
This is useful for testing your spider locally before using it to retrieve data regularly.
For portia spiders: portiacrawl command (footnote 1). For spiders created programmatically: scrapy crawl cli command
you will get a list of spiders if you run this command
docker exec portia portiacrawl [SPIDER] [OPTIONS] docker exec portia portiacrawl /app/data/projects/Hoaxlyspiders
For example, to run the climatefeedback.org crawler and save its output into /app/data/example-output/output.json using the hoaxly settings, you would run:
docker exec portia portiacrawl /app/data/projects/Hoaxlyspiders climatefeedback.org -o /app/data/example-output/output.json --settings=hoaxly
the more lowlevel command using scrapy looks like
scrapy crawl -s PROJECT_DIR=./ -s SPIDER_MANAGER_CLASS=slybot.spidermanager.SlybotSpiderManager snopes.com
You can also locally deploy exported spiders to the scrapingdaemon and schedule a run there to test what would happen in production environment there is a cli container supplied so you dont need to install any dependencies on your host
Run:
docker exec -ti cli /bin/bash
Now you are in container and can tell you local scrapydaemon container to run these spiders
scrapyd-client deploy local scrapyd-client -t schedule -p Hoaxlyspiders climatefeedback.org
and view your results in the storage container:
Footnotes
1