Map Archive Import — FileUniverse Pipeline

Results at a glance

Download: skip the browser

FileUniverse sits behind Cloudflare, but individual zip files can be fetched with plain urllib + a Chrome User-Agent:

req = urllib.request.Request(url, headers={"User-Agent":
    "Mozilla/5.0 (X11; Linux x86_64) ... Chrome/131.0.0.0 Safari/537.36"})
data = urllib.request.urlopen(req, timeout=180).read()

curl and HEAD requests get 403; a full GET with the Chrome UA gets through. No Playwright required.

Validating a map zip

A playable TA map must contain at least one of:

Virus scanning: clamscan --no-summary -i <zip> — exit 0 = clean, exit 1 = infected. Infected files are quarantined, never integrated.

Integration order (get this right or it breaks)

1. Move zips into static/downloads/maps/ (spaces → underscores).
2. Render minimaps from TNT heightmap indices through the 256-color PALETTE.PAL into static/minimaps_color/.
3. Extract metadata — size, players, description. Two sources:
   • OTA text (loose file or hidden inside the UFO, see below)
   • README .txt inside the zip ("Map Size: 15 x 15", "Size:14 x 4", "numplayers=Two or Four;")
4. Percent-encode all URL fields in data/maps_all.json (#→%23, [/]→%5B/%5D, non-ASCII→UTF-8-percent).
5. Build to a staging directory first; only deploy to live after verification.

Where the OTA hides

The r2010 packagers stored the OTA inside the first 65536-byte SQSH blob of the UFO — mission parameters followed by zero-padding, not as its own small chunk. A scanner that only inspects “small blobs” will wrongly conclude the OTA is missing. The OTA text (missionname=, numplayers=, size=) can be found by decompressing every SQSH blob (zlib for flag 2, LZ77 for flag 1) and searching for missionname in the result.

Consequence: maps without an obviously visible OTA are still playable — TA reads the OTA from the blob.

URL-encoding traps

Other cases found in the wild:

When a Wayback-derived URL fails with 404, check whether the listing page stored the filename as raw CP850 percent-bytes — substitute those exact bytes and the download works.

Ghost entries

static/downloads/maps/ contains subdirectories (cavedog/, tam/, incoming/, …). An importer that runs over os.listdir without filtering creates ghost map-cards with empty names, no minimap, no details — they appear sorted first on the page. Only .zip files count as maps.

Source collection status

Updated 2026-09-17 after the September sourcing sweep. Live ledger: ~/ta-maps-fu/SOURCING_LEDGER.md (19 tracked sources with status).

Sourcing workflow notes

Things that worked (or bit us) during the sweep, beyond what the integration order above covers:

• Content fingerprints before integration. Every candidate zip gets a SHA-256 of its minimap block (or the whole TNT for loose pairs) and is checked against _fingerprints.json. In practice about half of the “new” files from any source turn out to be re-zipped duplicates — the Mayhem Map Packs appeared byte-different at zip level while their UFO payloads were identical, so compare archive contents, not the wrapper.
• Metadata lives with the download. The ModDB scrape carried uploader, upload date, MD5 and description per file, which made verification (re-hash on arrival) and provenance records free. Scraping sources without metadata costs that forever.
• The browser detour. Cloudflare-managed challenges reject headless browsers outright. The working setup: a real desktop browser on a second machine solves the challenge once, then plain rsync moves the files to the server. Files arrive with an MD5 manifest and get re-verified server side.
• Hidden maps inside unit collections. Unit archives routinely carry map packs under mod-specific types (ufo-container, HAPI-container, nested-zip, map-pack). A fingerprint sweep over every UFO in the units tree found 55 previously unknown maps in one mod’s ten .ccx containers — everything else was already integrated. Worth re-running whenever new unit collections land.

Tools

The import pipeline scripts live in ~/ta-maps-fu/ (_fu_*.py), all resumable with state in _state.json. Metadata source of truth: ~/ta-halle/data/maps_all.json.

Minimap integrity audit (Sep 2026)

Every map card’s minimap can be verified against ground truth: the 252×252 palette-index block at ptrMm inside the map’s .tnt. Three lessons from a full 3,784-card audit:

• Never extract TNTs by scanning for chunk markers. A scan-and-merge heuristic that concatenates every decompressed chunk of an archive builds a byte stream that straddles file boundaries. For multi-map packs (Zombieland Map Pack V2: 9 maps in one UFO) the misaligned stream renders tile graphics as the minimap — a diagonal stripe pattern, diagnostic because 1,024-byte tiles sliced at 252 bytes per row drift 16 pixels per tile. Always read the HPI directory and extract the first .tnt file properly.
• A “wrong” minimap is not always corrupt. On pack cards the image may be a real minimap of a sibling map in the same pack (2 of 57 mismatches). Classify by comparing the live PNG against every TNT minimap in the archive before calling it garbage.
• Audit helpers must resolve URLs before failing. Percent-encoded (%C3%B3) and renamed (%23 → _hash_) filenames made 41 cards look “missing” to a naive os.path.exists check; all but the encoding issue were fine. urllib.parse.unquote the URL field first.

Verification tooling: ~/ta-maps-fu/_mm_audit.py (all cards vs TNT truth), _mm_triage.py (sibling vs garbage classification), _mm_fix.py (re-render through the directory-based extraction chain), _mm_7z_check.py (zips with exotic compression methods that python’s zipfile cannot read — extract via 7z e). Pixel-perfect reference for the renderer: author-shipped PCX captures (252×252) inside the Zombieland pack match the TNT minimap block byte for byte.

Exploded index (~/ta-extract)

Almost everything in a TA download is hidden inside containers (zip → UFO/HPI → SQSH chunks) with mixed compression, and every pipeline pass used to pay the full decompress cost again. ~/ta-maps-fu/_explode.py therefore builds a regenerable cache:

• ~/ta-extract/<section>/<zip-stem>/ — small metadata files (.fbi, .tdf, .ota, .gaf, .txt, .bos, .cob, < 4 MB) with their inner container paths preserved; big payloads (.tnt, .3do, .wav) stay manifest-only
• __pack.json per pack (full inner file list) and _index.jsonl globally
• the canonical zips under ~/ta-live/downloads are NEVER touched — the cache is disposable and rebuilt with python3 _explode.py maps units

Consumers use ~/ta-maps-fu/_exstore.py (lookup/lookup_bytes): it returns the extracted files or None, in which case the caller falls back to full container parsing. The lookup also returns None when a pack’s manifest holds “too big” meta files, so the cache can never silently hide candidates. Measured: an OTA lookup drops from 100–300 ms (decompress) to ~3 ms.

Trap: gadget GAFs (unit interface icons, entry version 3) report frame-count 0 while holding icon triples — _pack_pics.gaf_icons scans frame items instead of trusting the count field.