Adrian Malacoda 4cd9b22eb9 Use a loop to iterate thread/board pages, not recursion. For large threads this can cause a stack overflow. Also, since we're no longer doing the http request in the same function that does the scraping, we need to limit the @retry to the function that actually does the http call as that's what we want to be retrying.

2016-11-27 17:42:46 -06:00

..

__init__.py

if can_scrape_url isn't available don't call it

2016-11-27 01:58:25 -06:00

pickle.py

add pickle scraper which can load serialized pickle files

2016-11-27 01:58:39 -06:00

yuku.py

Use a loop to iterate thread/board pages, not recursion. For large threads this can cause a stack overflow. Also, since we're no longer doing the http request in the same function that does the scraping, we need to limit the @retry to the function that actually does the http call as that's what we want to be retrying.

2016-11-27 17:42:46 -06:00