Adrian Malacoda
|
77775ae0be
|
need to pass url to scrape_board_from_document
|
2016-11-27 17:52:47 -06:00 |
|
Adrian Malacoda
|
acdf659e4a
|
dry up pagination logic using a generator
|
2016-11-27 17:50:25 -06:00 |
|
Adrian Malacoda
|
4cd9b22eb9
|
Use a loop to iterate thread/board pages, not recursion. For large threads this can cause a stack overflow. Also, since we're no longer doing the http request in the same function that does the scraping, we need to limit the @retry to the function that actually does the http call as that's what we want to be retrying.
|
2016-11-27 17:42:46 -06:00 |
|
Adrian Malacoda
|
b67ab06b55
|
Add exponential backoff for retrying
|
2016-11-27 13:03:13 -06:00 |
|
Adrian Malacoda
|
d54f3ec21c
|
there's multiple h1's on the page and the one we want is like .eq(2) or something. But once you start addressing nodes by index like that you get real brittle and can break easily. I don't think we have a problem with just selecting all h1's here.
|
2016-11-27 01:19:59 -06:00 |
|
Adrian Malacoda
|
39b8bfff30
|
grab board description from forum index (we can't get it from the board index)
|
2016-11-27 01:13:45 -06:00 |
|
Adrian Malacoda
|
3f4eecc238
|
use dateutil to parse rfc3339 datetime strings in <time> elements, if they are present.
|
2016-11-27 01:10:04 -06:00 |
|
Adrian Malacoda
|
5bcb6e8884
|
add extra post & user info
|
2016-11-27 00:48:55 -06:00 |
|
Adrian Malacoda
|
c83d4a9916
|
for now, limit to forumer forums (fr.yuku.com) as I'm not sure if this scraper will support non-forumer ones
|
2016-11-27 00:18:39 -06:00 |
|
Adrian Malacoda
|
741573d30a
|
only want first h1/h2 etc
|
2016-11-27 00:16:21 -06:00 |
|
Adrian Malacoda
|
ea46ae8853
|
.text() not text
|
2016-11-27 00:14:16 -06:00 |
|
Adrian Malacoda
|
9c401cbfb1
|
need to use .items() grumble grumble
|
2016-11-27 00:11:42 -06:00 |
|
Adrian Malacoda
|
b304297019
|
fix signature parsing, use html instead of text. Unfortunately there's a lot of garbage here we'll have to clean up
|
2016-11-27 00:03:30 -06:00 |
|
Adrian Malacoda
|
eabf099f47
|
fix for yuku's broken postbit markup
|
2016-11-26 23:42:30 -06:00 |
|
Adrian Malacoda
|
c04c030540
|
add user object
|
2016-11-26 23:14:09 -06:00 |
|
Adrian Malacoda
|
933e178ce5
|
initial commit for the-great-escape yuku scraper
|
2016-11-26 23:09:12 -06:00 |
|