Map Archive Import Pipeline
How the r2010 collection was brought from FileUniverse into TA Archive: 1,674 maps downloaded, virus-scanned, metadata-extracted, minimap-rendered and integrated — including every encoding trap the internet set along the way.
Results at a glance
Download: skip the browser
FileUniverse sits behind Cloudflare, but individual zip files can be fetched with plain urllib + a Chrome User-Agent:
req = urllib.request.Request(url, headers={"User-Agent":
"Mozilla/5.0 (X11; Linux x86_64) ... Chrome/131.0.0.0 Safari/537.36"})
data = urllib.request.urlopen(req, timeout=180).read()
curl and HEAD requests get 403; a full GET with the Chrome UA gets through. No Playwright required.
Validating a map zip
A playable TA map must contain at least one of:
| Extension | Meaning |
|---|---|
.ufo | r2010 package (HPI container, magic bytes HAPI) — usually contains TNT + OTA internally |
.tnt | Terrain data (may need a separate .ota) |
.ota | Mission parameters (players, size, AI profiles) |
Virus scanning: clamscan --no-summary -i <zip> — exit 0 = clean, exit 1 = infected. Infected files are quarantined, never integrated.
Integration order (get this right or it breaks)
- Move zips into
static/downloads/maps/(spaces → underscores). - Render minimaps from TNT heightmap indices through the 256-color
PALETTE.PALintostatic/minimaps_color/. - Extract metadata — size, players, description. Two sources:
- OTA text (loose file or hidden inside the UFO, see below)
- README
.txtinside the zip ("Map Size: 15 x 15","Size:14 x 4","numplayers=Two or Four;")
- Percent-encode all URL fields in
data/maps_all.json(#→%23,[/]→%5B/%5D, non-ASCII→UTF-8-percent). - Build to a staging directory first; only deploy to live after verification.
maps_all.json, the new maps silently get no minimap and no metadata. Always run extraction after the entry-add step, or re-run it.Where the OTA hides
The r2010 packagers stored the OTA inside the first 65536-byte SQSH blob of the UFO — mission parameters followed by zero-padding, not as its own small chunk. A scanner that only inspects “small blobs” will wrongly conclude the OTA is missing. The OTA text (missionname=, numplayers=, size=) can be found by decompressing every SQSH blob (zlib for flag 2, LZ77 for flag 1) and searching for missionname in the result.
Consequence: maps without an obviously visible OTA are still playable — TA reads the OTA from the blob.
URL-encoding traps
# hash 20x20) had raw # in their URLs. Browsers treat # as a fragment marker and cut the URL: the minimap request becomes /minimaps_color/ → 404, downloads fail. Fix: %23 everywhere, in zip, minimap and minimap_color.Other cases found in the wild:
| Raw | Encoded | Cause |
|---|---|---|
# | %23 | HTML fragment trap |
[ / ] | %5B / %5D | IPv6-host parser confusion |
é í Î | %82 %a2 %8c | CP850/DOS encoding — FileUniverse stores filenames in DOS codepage |
& | %26 | HTML-entity escapes from scraped listings (&) |
mü | stored as m%fc | CP850 again; rename the local file to mu_... for a safe filesystem path |
When a Wayback-derived URL fails with 404, check whether the listing page stored the filename as raw CP850 percent-bytes — substitute those exact bytes and the download works.
Ghost entries
static/downloads/maps/ contains subdirectories (cavedog/, tam/, incoming/, …). An importer that runs over os.listdir without filtering creates ghost map-cards with empty names, no minimap, no details — they appear sorted first on the page. Only .zip files count as maps.
Source collection status
| Source | State |
|---|---|
FileUniverse maps-2 (r2010) | fully integrated |
FileUniverse maps-1 (author folders, ~69 dirs) | mostly not archived anywhere — top candidate for a future import run |
| Smaller sites (Poosticks, TA-Power, TA-Zone, DesignX, M.A.D.) | dozens of maps each; useful gap fillers |
Tools
The import pipeline scripts live in ~/ta-maps-fu/ (_fu_*.py), all resumable with state in _state.json. Metadata source of truth: ~/ta-halle/data/maps_all.json.