> For the complete documentation index, see [llms.txt](https://hoaxly.gitbook.io/documentation/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://hoaxly.gitbook.io/documentation/developer-documentation/adding-new-sites-to-the-database/technical-steps-to-create-spiders/4.-deploy-spiders.md).

# 4. Deploy spiders

This will be normally just for internal use.

Scrapyd is a daemon that can be started to schedule runs

<https://doc.scrapy.org/en/latest/index.html>

<http://scrapyd.readthedocs.io/en/latest/>

configure your live instance hostname in scrapy.cfg once you tested everything locally you can deploy to live scrapyd and schedule crawls using scrapyd-client

`docker exec -ti cli bash`&#x20;

`scrapyd-deploy live`&#x20;

once deployed you can interact directly with scrapyd through the webapi, either using the client

`docker exec -ti cli bash scrapyd-client -t` [`https://htaccessusername:htaccesspassword@scrapyd.hoax.ly/`](https://htaccessusername:htaccesspassword@scrapyd.hoax.ly/) `schedule -p Hoaxlyspiders climatefeedback.org`&#x20;

or from anywhere else.

`curl` [`https://htaccessusername:htaccesspassword@scrapyd.hoax.ly/schedule.json`](https://htaccessusername:htaccesspassword@scrapyd.hoax.ly/schedule.json) `-d project=Hoaxlyspiders -d spider=climatefeedback.org curl` [`https://htaccessusername:htaccesspassword@scrapyd.hoax.ly/listprojects.json`](https://htaccessusername:htaccesspassword@scrapyd.hoax.ly/listprojects.json)&#x20;

`curl` [`https://htaccessusername:htaccesspassword@scrapyd.hoax.ly/listspiders.json?project=Hoaxlyspiders`](https://htaccessusername:htaccesspassword@scrapyd.hoax.ly/listspiders.json?project=Hoaxlyspiders)

A crawl can be scheduled to run regularly by deploying it to a dedicated server.

For portia spiders deployment should work normally but currently requires a workaround in our settings.