WiiPax

WiiPax
General
Author(s)	fail0verflow
Type	PC utility
Version	0.2
Links
Source

WiiPax is the tool used to obfuscate the Homebrew Channel, CEIL1NG_CAT, and the HackMii Installer. The open source version uses LZMA to compress ELF files, with a loader stub added to decompress it.

Closed source unpacking algorithm

The standalone version copies itself to MEM2, and has a total of 33 functions (3662 instructions).

Key generation

3 keys exist: an index key (32-bit entries), a value key (32-bit entries), and a finalization value key (8-bit entries).

The value key is generated by first building a two-way lookup table for a function that leftshifts a byte by one bit and XORs the result with 0x1B if the removed bit is 1. Another two-way lookup table is then built where each value is obtained by finding lookup1[255 - inverseLookup1[index]] and XORing the result with the result leftrotated 1, 2, 3, and 4 bits, as well as the constant 99; the reverse lookup table is the finalization value key. Finally, each entry in the value key is the joining of 4 bytes, each of which is found by applying a function where the first argument is the respective value in [11, 13, 9, 14], and the second argument is the entry in the finalization value key. This function performs a reverse lookup in the first table on both arguments, then adds the two results together and performs a division-like operation, before looking up the result in forward table 1.

The index key is generated by creating a temporary buffer with space for 44 words where the first 4 words are an input seed key that may vary between binaries. The remaining words are initialized with tempBuf[index] = tempBuf[index-4] ^ func(tempBuf[index-1]) ^ xorList[index], where func is a rotation 8 bits to the right for indices that are multiples of 4 or the identity for other indices, and xorList is a list of results from successive computations of the first function involved in value key computation. After this, the first 4 words of the temporary buffer are copied to the resulting key, and the remaining words are computed through a similar division process as above, with the same 4 constants being rotated. Finally, the last 4 words in the temporary buffer are copied to the beginning of the key to prevent the seed from being discovered.

Unpacking

For each 16-byte chunk of the ciphertext, the 16 bytes are XORed with the first 16 bytes of the index key. Then, a transformation is repeated 9 times where each word is mapped to indexKey[4 + 4*transformationIteration + index] ^ valueKey[preimage[4*index]] ^ (valueKey[preimage[4*C]] r<< 24) ^ (valueKey[preimage[4*A+2]] r<< 8) ^ (valueKey[preimage[4*B+1]] r<< 16) (the array (A,B,C) rotates between (3,2,1), (0,3,2), (1,0,3), and (2,1,0) to reduce consistency). Finally, each word in the ciphertext is mapped to indexKey[index] ^ finalizationValueKey[preimage[index]] ^ (finalizationValueKey[preimage[4*C]] << 24) ^ (finalizationValueKey[preimage[4*A+2]] << 8) ^ (finalizationValueKey[preimage[4*C+1]] << 16).

After this algorithm is applied to individual chunks, the state is XORed with the plaintext from 1 or 2 chunks before depending on the WiiPax version.