> For the complete documentation index, see [llms.txt](https://hoaxly.gitbook.io/documentation/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://hoaxly.gitbook.io/documentation/developer-documentation/adding-new-sites-to-the-database/technical-steps-to-create-spiders/2.-create-spider.md).

# 2. Create spider

## Create a new branch

![](/files/-LGRsEgG7mInKPJqLAI0)

Respository: <https://github.com/hoaxly/hoaxly-scraping-container>

## After setting up the environment visit <http://hoaxly.docksal:9001/#/projects/hoaxlyPortia>

## Enter url you want to scrape

![](/files/-LGRsK4HpFy_70ImulJd)

## Using the portia interface: visit the page where you want to start crawling through links

![Example start url](/files/-LGRs_OTUXolSTDkmGW-)

## Create a new spider

![](/files/-LGRsgChVKkFIvph5WVw)

## Follow a link to a sample item you want to scrape

![Sample item link](/files/-LGRslG4Z8m1jm5MdP1p)

![Sample item page](/files/-LGRsu5Q7llSB63FpAzH)

## Create a new sample annotation

![](/files/-LGRtI4SH_RP8irAV2wP)

## Select the appropriate schema (hoaxly)

TODO: screenshot of new schema

![](/files/-LGRtN6559OS8nUjK6zX)

![](/files/-LGRtQEzSVEASGVG5dhm)

## Annotate the first element by clicking on the visible project headline

## Select the appropriate field from schema

![](/files/-LGRu4Pswb-zXCeTsz5S)

## Repeat for all fields in the schema

![](/files/-LGRtiE45AfP8KRW1sjy)

![](/files/-LGRtxo8sWINUIVvneTU)

## Close sample

![](/files/-LGRuEHsxw1k4t-qhuJq)

## Configure url crawling schema

![](/files/-LGRuLyQOc5aTBJr_sff)

using regex:

![](/files/-LGRuPALp8W9UqecQPWc)

## Export spider as scrapy spider (python code)

## Add the new spider to the scrapy\_projects directory and commit the new spider

![](/files/-LGRuZuxzV3Pm3UEZgiZ)

&#x20;`% git add scrapy_projects/hoaxlyPortia/spiders/ -p`&#x20;

`% git commit scrapy_projects/hoaxlyPortia/spiders/`&#x20;

use a commit message that tells us what spider you are adding using which schema

## Create a merge request