Reverse Engineering TA Files

This article documents how we reverse engineered parts of Total Annihilation’s file formats — not from leaked source code, but the old-fashioned way: hex editors, educated guesses, checksum verification and a lot of trial and error. Everything below has been verified against real files: 96 of 97 third-party map archives on this site decode cleanly with this method. It is based on the classic community documentation (Hpi-fmt.txt, HPIDump.c) but corrects and completes several details those sources glossed over.

You don’t need to be an expert to follow along — but you should be comfortable with hex numbers and a scripting language like Python.

Why bother?

TA is from 1997, its formats are documented only partially and in scattered, half-dead community pages. If you want to extract anything — previews, textures, unit data — or verify that a download is intact, you need to parse HPI archives yourself. And parsing something yourself is also the best way to understand it.

The tools

You need surprisingly little:

• A hex editor (ImHex, HxD, or xxd in a terminal)
• Python with struct and zlib (both in the standard library)
• A couple of known-good files to test against — ideally one map you can open in the game, so you can tell “my decode is wrong” apart from “the file is broken”

The general method for every format mystery:

1. Look for signatures. Open several files of the same type side by side and search for repeated ASCII strings. TA formats are full of them: HAPI, SQSH, UNITNAME, etc.
2. Guess the surrounding fields. Once you find a marker, the bytes around it are usually lengths, offsets or flags. Interpret them as little-endian int32 first — that’s TA’s default everywhere.
3. Validate every guess. A length field should point within the file; a checksum field should match the bytes it covers. If it doesn’t, your interpretation is wrong — try other offsets before inventing a new theory.
4. Test on many files, not one. One file can mislead you; 97 files don’t.

HPI archives: the container

Every archive starts with an 20-byte header:

At Start lives the directory: an int32 entry count, an int32 offset to the entry list, then entries of 9 bytes each — name offset (int32), data offset (int32), and a flag byte (1 = subdirectory, 0 = file). Names are NUL-terminated strings, stored somewhere else in the file and referenced by offset. Directories nest; you simply recurse.

A file entry points to another 9-byte record: DataOffset, FileSize, and a compression flag. The data itself is split into chunks of at most 65536 bytes: first an array of int32 chunk sizes (one per chunk), then the chunk records one after another.

The encryption layer

Here’s where it gets interesting. If HeaderKey at 0x0C is non-zero, the entire file from Start onwards is encrypted — including the directory, all file records and all chunk payloads. Original Cavedog archives are usually encrypted; many third-party files are not. That’s also why some old community parsers “work on some files and not others”: they ignore the key.

The cipher is a simple position-dependent XOR. For every byte b at absolute file position p:

key  = NOT((HeaderKey * 4) | (HeaderKey >> 6)) & 0xFF
out  = ((p ^ key) ^ (NOT b)) & 0xFF

Two things that took us a while to confirm:

• The position p is the absolute position in the file, not relative to the chunk or the directory. Get this wrong and the first byte decodes fine but nothing else does.
• The shift HeaderKey >> 6 on a 32-bit signed int must be masked to the low bits before the NOT — in Python terms, treat it as 26 bits. Sloppy implementations that skip this produce keys that work for some archives and silently corrupt others.

(Cross-check: TAUtil, the most widely used community library, implements the same cipher as (key << 2) | (key >> 6) on a single byte, XORed without the complement. That is algebraically identical to the formula above — the complement of the data plus the complement of the key cancel out. Two independent implementations agreeing is exactly the kind of validation you want.)

Because of the position dependence, encryption and decryption are the same operation — apply it twice and you get the original back. Handy for testing.

Chunk records: SQSH

Each chunk (after decryption, if applicable) begins with:

Then the checksum gotcha: it is the plain sum of the payload bytes (& 0xFFFFFFFF), but — and this is the detail that cost us a debugging session — in UFO archives with a HeaderKey, the checksum covers the payload after the archive-wide decryption but before the per-chunk decryption and decompression. Our verification loops therefore: decrypt archive-wide, sum the payload bytes, compare — only then decode the chunk itself.

The checksum is your best friend while reverse engineering: it tells you immediately whether you’ve read the chunk boundary correctly, before you even attempt to decompress. Any parser should check it and fail loudly rather than produce garbage.

Compression 2: zlib (the easy one)

Flag 2 is ordinary zlib — zlib.decompress() and done. Many third-party archives use this exclusively, presumably because the community tools of the era wrapped zlib rather than reimplementing Cavedog’s scheme.

Compression 1: the Cavedog LZ77 variant

Flag 1 is a custom LZ77, ported from the classic HPIDump.c. It’s an LZSS-style scheme with a 4096-byte sliding window:

• A control byte precedes up to 8 items; its bits (LSB first) tell whether each item is a literal byte or a back-reference.
• A back-reference is two bytes, little-endian: the high 12 bits are the position in the window, the low 4 bits plus 2 are the run length (so runs of 2–17 bytes).
• Positions wrap with & 0xFFF — the window is circular.

A minimal decoder in Python fits on a postcard:

def lz77_decompress(data, expected):
    out = bytearray()
    win = bytearray(4096)
    w1 = 1                      # write cursor in the window
    ctrl, i = 1, 0
    flags = data[i]; i += 1
    while True:
        if not (ctrl & flags):              # literal
            if i >= len(data) or len(out) >= expected: break
            b = data[i]; i += 1
            out.append(b); win[w1] = b
            w1 = (w1 + 1) & 0xFFF
        else:                               # back-reference
            if i + 2 > len(data): break
            ref = data[i] | (data[i+1] << 8); i += 2
            src = ref >> 4
            if src == 0: break              # terminator
            n = (ref & 0x0F) + 2
            for _ in range(n):
                if len(out) >= expected: break
                b = win[src]
                out.append(b); win[w1] = b
                src = (src + 1) & 0xFFF
                w1 = (w1 + 1) & 0xFFF
        ctrl *= 2
        if ctrl & 0x100:                    # control byte exhausted
            ctrl = 1
            if i >= len(data): break
            flags = data[i]; i += 1
    return bytes(out)

Always decompress against the expected dsize and compare lengths afterwards — silent overrun is the classic bug here.

Case study: the hidden 252×252 minimap

Our favorite find. Many map downloads circulate as “pictureless” archives — the author never included a preview image, so map browsers show nothing. But it turns out every TNT (map terrain) file carries a built-in minimap: the exact 252×252-pixel grayscale preview the original game renders in its map selection screen. It’s just buried in the terrain blob.

Where to find it:

1. Parse the archive (HPI/UFO) and extract the .tnt file.
2. In the TNT header, read the int32 pointer at offset 0x28.
3. At that offset: two int32s (width, height) — always 252 × 252 — followed by 252 × 252 = 63,504 bytes of 8-bit grayscale pixels.
4. Wrap those bytes in any image library and you have a preview.

Caveat: the TNT is stored as a stream of 65536-byte chunks, so the minimap data usually straddles a chunk boundary — you must concatenate the decompressed chunks in order before reading. We also found one archive (“Luschie”) whose TNT header is internally inconsistent and defeats this method; out of 97 map archives, 96 yield clean minimaps.

That’s exactly how all preview images on this site’s map pages were produced — no image editing, just format archaeology.

The colors are in the palette, not the map

Our first renders were grayscale — a mistake worth documenting. We treated the 252×252 bytes as gray levels, and the result looked plausible: bright highlands, dark water. Plausible-looking is dangerous; it took a second look to realize the bytes make more sense as indices into TA’s 256-color palette (the same fixed palette used by all TA bitmaps, famously shipped as PALETTE.PCX with the game’s asset files).

The test is simple: render both interpretations side by side for a few maps. Grayscale shows smooth terrain-like gradients; palette-index rendering shows hard color boundaries between green terrain, blue water and gray rock — colors that exactly match what the in-game minimap shows. When the same index (e.g. 8) maps to the same color across all 96 maps, you’re looking at a shared palette, not per-map gray data.

With a community-documented copy of the palette (the MIT-licensed kbot-io project ships PALETTE.PAL as 1024 bytes — 256 × RGBA, alpha unused), every minimap renders in full color: index 0 is pure black, 8 the pale green of TA’s default terrain, 255 pure white. All map previews on this site now use this path.

Third-party archive variants

Cavedog’s own archives are consistent; the third-party ecosystem is not. When we ingested ~400 community maps from an unpacked 1990s TA installation, the parser that handled every original file started rejecting some of them. The differences we found are worth knowing before you trust a “working” HPI parser:

• Compression meets encryption. Community packers produced chunks with compression = 2 (zlib) and encrypted = 1 — a combination that does not occur in original Cavedog data. If your decoder assumes these fields are mutually exclusive, those archives fail silently. The correct order stays the same as everywhere else: archive-wide decrypt, verify checksum against the (now-decrypted) payload, then per-chunk decrypt, then decompress.
• The chunk-size table is usually not encrypted — check before you trust it. The int32 array of chunk sizes referenced by each file record reads as plain little-endian in most archives, ours included at first. But we later hit archives (a 1999-era map pack) where the size table is encrypted with the usual archive-wide layer. Practical heuristic: read the table raw; if any entry is implausible (≤ 0 or

  ~70000, i.e. bigger than the 65536 maximum chunk payload allows), decrypt it with the archive key and try again. If both fail, your directory walk is wrong — the table itself is a good anchor for testing key correctness, precisely because correct output is so obviously plausible.

• Multi-map HPIs exist. Map collections were commonly shipped as a single .ufo containing dozens of maps/*.ota + maps/*.tnt pairs. Enumerate the directory and split by basename instead of assuming one map per archive — otherwise 45 of 46 maps in a pack silently disappear.
• Loose map files. Unpacked installations store .ota/.tnt pairs as bare files. They parse like any other TNT; just repackage them together, since the OTA is worthless without its terrain and vice versa.

Practical advice

• Checksums and magic numbers are your ground truth. If the SQSH signature and the checksum both validate, your parsing is right; if either fails, don’t “fix” it by relaxing checks — you’ll corrupt output silently.
• Little-endian everywhere. We never found a big-endian field in TA.
• Write a strict parser first, a lenient one later. Our first strict parser rejected broken files loudly — which is how we discovered the one broken archive instead of shipping garbage.
• Document as you go. Half of what you read above exists because we wrote comments into the extraction scripts the same day we figured each detail out. Details you don’t write down get re-derived painfully later.

What’s still unknown

Honest list of the gaps we haven’t closed:

• The unknown byte at chunk offset 8 — the community library TAUtil names it “Version” but assigns no semantics to it either; it is constant in everything we’ve seen.
• The exact semantics of the SaveMarker version field.
• 3DO model internals (we’ve only consumed community tools’ output, not parsed 3DO ourselves).
• Whether every TNT really embeds the minimap or only maps from a certain era/toolchain — 96/97 says “every”, but one exception means stay careful.

If you decode any of these, we’d genuinely like to hear about it — see the contact address in the Imprint.

If you’d rather not write a parser

Standing on the shoulders of others is allowed. TAUtil is a solid C# library that reads and writes HPI archives, TNT maps and other TA formats — the same territory this article covers, but battle-tested and packaged. For checking unit/weapon definitions rather than raw archives, CheckTdf validates TDF-family files, even inside HPI archives. More tools on our Tools page.