Adrian Malacoda 4cd9b22eb9 Use a loop to iterate thread/board pages, not recursion. For large threads this can cause a stack overflow. Also, since we're no longer doing the http request in the same function that does the scraping, we need to limit the @retry to the function that actually does the http call as that's what we want to be retrying.

2016-11-27 17:42:46 -06:00

..

outputters

__sanitize_title -> sanitize_title

2016-11-27 02:04:54 -06:00

scrapers

Use a loop to iterate thread/board pages, not recursion. For large threads this can cause a stack overflow. Also, since we're no longer doing the http request in the same function that does the scraping, we need to limit the @retry to the function that actually does the http call as that's what we want to be retrying.

2016-11-27 17:42:46 -06:00

__init__.py

redo main() so it can work with either the local filesystem or urls. rename --url to --in for consistency.

2016-11-27 02:04:44 -06:00

model.py

add extra user metadata such as title and avatar

2016-11-27 00:43:11 -06:00

util.py

commas and quotes too

2016-11-27 02:06:38 -06:00