Static Wiki Website Archive #5

New Issue

kuschelyagi · 2020-07-13T04:09:17Z

kuschelyagi commented

2020-07-13 04:09:17 +00:00

Create a static HTML/CSS archive of the wiki. This archive is HTML/CSS only (no javascript) and can be hosted anywhere.

kuschelyagi commented

2020-08-24 05:55:31 +00:00

Progress was made. Main pain points right now:

Template transclusion is half-baked. Note that the library we're using (mwparserfromhell) doesn't actually do this so we have to roll our own.
- Magic words (e.g. PAGENAME, NUMBEROFARTICLES etc) are not yet implemented - they would be implemented like any other template
Category parsing/generation is also half-baked, doesn't seem to take into account template transclusion (which it should, since transclusion happens before we look for categories)
Images are not yet included. We could make a script to pull images or just use the dump from archive.org for this
Page name normalization is funky. Notable in that sometimes a slash (/) indicates a subpage (e.g. GlitchDex/RB:000) and sometimes it's just a slash (e.g. Gold/Silver/Crystal). We have some special logic in both the archive builder and redirector to know the difference.
- nginx normalizes URLs before passing them into our handlers so e.g. Gold%2FSilver%2FCrystal (which is what we would generate) is interpeted as Gold/Silver/Crystal (i.e. a directory structure), which obviously fails. It's probably possible to configure nginx to get around this but the goal with these archives is to avoid having special server configuration for them, and they should be hostable anywhere. Most likely we need to come up with a new character to represent / (+ is the current candidate)

Progress was made. Main pain points right now: * Template transclusion is half-baked. Note that the library we're using (mwparserfromhell) doesn't actually do this so we have to roll our own. * Magic words (e.g. PAGENAME, NUMBEROFARTICLES etc) are not yet implemented - they would be implemented like any other template * Category parsing/generation is also half-baked, doesn't seem to take into account template transclusion (which it should, since transclusion happens before we look for categories) * Images are not yet included. We could make a script to pull images or just use the dump from archive.org for this * Page name normalization is funky. Notable in that sometimes a slash (/) indicates a subpage (e.g. GlitchDex/RB:000) and sometimes it's just a slash (e.g. Gold/Silver/Crystal). We have some special logic in both the archive builder and redirector to know the difference. * nginx normalizes URLs before passing them into our handlers so e.g. Gold%2FSilver%2FCrystal (which is what we would generate) is interpeted as Gold/Silver/Crystal (i.e. a directory structure), which obviously fails. It's probably possible to configure nginx to get around this but the goal with these archives is to avoid having special server configuration for them, and they should be hostable anywhere. Most likely we need to come up with a new character to represent / (+ is the current candidate)

kuschelyagi commented

2020-09-03 21:57:22 +00:00

URL issue was resolved by replacing '/' with '+' as in: https://archives.glitchcity.info/wiki/Mimic_glitch_(Japanese_Diamond+Pearl).html

I'll make separate tickets for other issues with the wiki archival script.

URL issue was resolved by replacing '/' with '+' as in: https://archives.glitchcity.info/wiki/Mimic_glitch_(Japanese_Diamond+Pearl).html I'll make separate tickets for other issues with the wiki archival script.

kuschelyagi added the

Wiki archiver

label 2020-09-03 22:01:46 +00:00

kuschelyagi added a new dependency 2020-09-03 22:03:55 +00:00

#7 Archive linker not correctly handling fragments in URLs

kuschelyagi added a new dependency 2020-09-03 22:04:01 +00:00

#8 Wiki archiver not generating heading anchors

kuschelyagi added a new dependency 2020-09-03 22:09:35 +00:00

#10 Add images to wiki archive

kuschelyagi commented

2024-06-10 08:15:30 +00:00

Wiki archive is deprecated since we redirect all traffic to the successor wiki.