HPI/GAF Feature Extraction — Technical Reference (ta-archive.com)

Status: Sep 2026, internal KB. Related: Feature-Library project, TNT format skill.

HPI v1 archive format (validated against 591/591 files, TA_Features_2013.ccx; encrypted HPIs like `Martian Scream!.ufo`)

Header:
  0x00  char[4]  magic 'HAPI'
  0x04  uint32   version (0x00010000)
  0x08  uint32   DirectorySize   (end of directory, file-absolute)
  0x0C  uint32   HeaderKey       (encryption key field; 0 = none)
  0x10  uint32   Start           (directory begins here, usually 20)

Key transform (TAUtil TransformKey): key = (kf*4 | kf>>6) & 0xFF; if key >= 128: key -= 256.
Directory decryption (the gotcha): dec[i] = ((fileOffset + i) & 0xFF) ^ key ^ raw[i] — XOR against the file-absolute position & 0xFF, NOT a subtract transform. The encrypted region is file[Start:DirectorySize]. TAUtil allocates a DirectorySize-byte buffer and writes the decrypted bytes at buffer offset Start, so all directory offsets are file-absolute and index directly into the decrypted buffer.
DirectoryData: (uint32 NumberOfEntries, uint32 EntryListOffset) at file offset Start.
Entry (9 bytes): (uint32 NameOffset, uint32 DataOffset, uint8 IsDirectory). Names are NUL-terminated inside the decrypted directory buffer.
DirectoryData of a subdirectory = same structure at DataOffset.
FileData triple at DataOffset: (DataOffset, FileSize, Compression) — offset FIRST, size second (TAUtil struct order misleads; field order verified empirically).

File data

DataOffset points to a list of ceil(FileSize/65536) int32 values; each value is the full chunk footprint csize + 19 (header included). Trust the chunk header’s csize for the payload, advance by the list value.
Chunk header (19 bytes): uint32 marker ('SQSH' / 0x48535153), uint8 unknown1, uint8 CompMethod (1=LZ77, 2=zlib, 0=stored), uint8 Encrypt, uint32 CompressedSize, uint32 DecompressedSize, uint32 Checksum.
Checksum = sum of bytes of the decrypted-after-header-key payload (TAUtil computes it after ReadAndDecrypt; the earlier “checksum over raw” note was wrong for key!=0 archives — with the correct XOR transform the checksum matches after decryption).
If Encrypt==1: payload[i] = ((b - i) & 0xFF) ^ (i & 0xFF).
CompMethod 2 = plain zlib (78 9e visible). CompMethod 1 = TA LZ77.
TA LZ77: 4096-byte circular window, position starts at 1; tag byte, LSB first: bit 0 = literal, bit 1 = back-reference (pos = word>>4, len = (word&0xF)+2, pos==0 terminates).

GAF format

Header: uint32 IDVersion (0x00010100), int32 Entries, int32 Unknown1
Then `Entries` × int32 pointer → GAFENTRY:
  int16 Frames, int16 Unknown1, int32 Unknown2, char Name[32]
Then frame-pointer list per entry; frame data: int16 Width, Height, XPos, YPos, ...

TNT feature table (for map feature scanning)

Feature records: 132 bytes = 4-byte prefix + 128-byte NUL-terminated name, located between attribute data and the 252×252 minimap block.
Attribute feature index: bytes 1+2 of each 4-byte attribute record (layout height(1), index(2), pad(1)); sentinels 0xFFFF / 0xFFFC / 0xFFFE = no feature.
Working scanner: ~/ta-maps-fu/_feature_scan.py (output features_scan.json).

Feature-Library data pipeline (as built Sep 2026)

File	Content
`~/ta-maps-fu/_feature_scan.py`	extracts feature tables from all map zips
`~/ta-maps-fu/features_scan.json`	map → feature-name list (611 vmaps)
`~/ta-maps-fu/_hpi_v2.py`	full HPI parser (open-source candidate)
`~/ta-maps-fu/ccx_feature_index.json`	TDF section name → tdf/filename
`~/ta-maps-fu/ccx_gaf_index.json`	GAF entry name → gaf files
`~/ta-maps-fu/features_still_missing.json`	unresolvable feature names

Sources integrated (Sep 2026): TA_Features_2013.ccx (591 files), TA_Features_2011.zip (544), CCDATA.CCX from real TA:CC CD ISO via archive.org (585: acid/creon/slate sets), Tamec2004final.zip → tamechpi2004.ccx (418), AtlantisMB Features, Frenzy MFP. Coverage: 1346 plausible feature names in corpus; 576 direct + 247 Cavedog-stock covered; 544 remain (Archipelago-transition / Aquatic tileset packs: AquaOre, ArchiTree, Barrier01-23, ArchMetal) — distributed as standalone UFOs by The Pack community (ModDB Cavedog addon pages).

Gotchas

Tropical Paradise-style SFX installers embed ONE repacked HPI (tnt+ota only, no features).
Cavedog DL maps use stock/CC features only (checked The Bayou.tnt in CCMAPS.CCX: Lush+lushvine).
worlds.hpi (TAE on the CC disc) = map-editor tile sections (.sct), not features.
ModDB blocks urllib (403); files.tauniverse.com directory listings work with a Chrome UA.
archive.org advancedsearch.php + metadata/<id> API is reliable for finding game ISOs.