In the widescreen post the native port of Saint Seiya: The Hades ended up with a 16:9 picture. Since then I spent the week on a less visible kind of work: making fights lighter, hunting the one crash that still happened, and filling in what was missing in sound and video. I will go through it in the order it happened, because each step set up the next.
Where the time went
The port has a thread that plays the part of the PS2’s graphics chip (the GS) and runs the VU1, the vector unit that transforms the 3D models. In a fight between two Mu, that thread used 88% of a CPU core. Not a problem for a fast machine, but with no room to spare.
The room matters for a reason that took me a while to see. Every frame, the game waits for the GS thread to finish the previous frame before going on. If that takes more than 16.7 ms, the game misses the vblank and the fight drops to 30 fps. There is no 30 fps latch in the game: as soon as the thread is back under 16.7 ms, it returns to 60. So that thread’s cost per frame is what decides between 60 and 30, not how much CPU is free overall.
I started by measuring. perf showed the VU1 programs at about 30% of that thread, with the rest spread over the GS emulation, so there was no single culprit. I went hotspot by hotspot, one commit each:
- VU1 program lookup. An overlay seen before needs no walk of the code again. Lookup time per frame went from 0.67 to 0.08 ms.
- Texture decode. One loop per format, the palette expanded once and the row worked out once per line. I checked it against the old decode on about 9,700 textures from a fight, and no texel differs.
- Texture lookups. The hash of each 8 KB page is kept until something writes into it, so a lookup no longer walks hundreds of page versions.
- UNPACK and 8-bit uploads with SSE4.1 and
pshufb. - VIF commands whose data is all in the packet run from there, without the copy into the VIF’s buffer.
The port’s threads also got names (game, gs, ee-N, iop-N, vblank, vu-compiler). It sounds minor, but it is what lets perf filter by thread and lets top and gdb tell who is doing what.
That first round took the GS thread from 88% to 59% of a core. One caveat I wrote in the commit and keep here: the fight is different every run, so whole-fight totals vary as much between two runs of the same build as between builds. The shares above come from the profiles, not from one lucky run.
VU1 programs compiled before the game starts
The port already translated VU1 programs to native code, but it did that in the background while you played, and a program the game needed right away could run in the interpreter first (about 5 times slower).
All the game’s VU1 code lives in its executable, in DMA packets the game sends to the VU1’s code memory: a resident part and 56 overlays loaded at the same address (0x33B0). So instead of waiting to see which ones the game runs, the port now finds every point where a program can start: the first instruction pair, the pair after an E bit (the end of a program) or a JR, the return of each JALR, and the addresses built with IADDIU. That gives 189 programs. sstheh --compile-vu compiles all of them (build.sh runs it), and with a warm cache the check at startup takes about 13 ms. From an empty cache, in a Mu vs Mu fight and in Seiya vs Shiryu, no program was compiled or interpreted during the game. To make sure the rule left nothing out, I ran all 44 fighters of versus mode (22 fights, mashing buttons) and two story mode runs, and the cache stayed at exactly the same 189.
The next step was making the generated C better. The overlays call the resident subroutines with IADDIU plus JALR, to constant addresses, so the translator can follow the call and keep it and its return in one native function: about 7 hops per program run became none. The MAC and status flags are now computed only where something can read them: on the VU an instruction sees the flags from four pairs back, so looking at the program tells which ones are never read. The Q and P countdowns (the results of a division and of the EFU functions, which arrive a few cycles later) also only advance where a result is on its way. FTOI and CLIP were vectorized, because they went through memory lane by lane and stalled on store forwarding.
Faster code is only worth something if it is still the same code, so I wrote a bench (tools/vubench.c). It replays 11,573 program runs recorded in a fight and compares registers and data memory against the game’s own result at the end of each run, also with the inherited flags scrambled. Cycles for those runs went from 62.6 M to 29.9 M. In the game, VU1 time per frame including its drawing went from 7.8-8.6 ms to 6.3-6.4 ms. As a bonus, the 189 programs now compile in about 20 s instead of 35.
For this post I recorded a new Mu vs Mu fight with the current build and ran the bench again:
12059 runs, 0 not native, 0 end differently from the game12059 runs, 0 not native, 26.7 Mcycles from 000: 4299 runs, 0.16 Mcycles from 3B8: 1453 runs, 0.79 Mcycles from 2188: 1071 runs, 4.56 Mcycles from 33B0: 2390 runs, 20.39 Mcycles ...The bottom line says where to look next: the programs starting at 0x33B0, the overlays’ address, are 20% of the runs and 76% of the cycles. That is where the heavy model transforms live; the resident ones are short routines.
A note on method: measuring the VU1 inside the game misleads. Recording the runs slows the game so much that 20 vblanks become about 3 frames, and the cost changes a lot with what is on screen (a cosmo aura weighs more). One change that seemed to gain nothing in the game, following constant JRs, gained 13% in the bench.
15,000 draws that were really 525
The GS thread still had a large share of its time in sending draws to the GPU. A fight frame had about 15,000 of them, and many were the same triangle strip split in pieces with identical state.
The reason for so many pieces is in the PS2. The GS has no draw calls: the VU1 sends vertices in packets (the XGKICK), and each packet has to fit in the VU1’s 16 KB of data memory, so a model arrives in several packets. On the console that is free. On a PC GPU, each draw is a command with state to check and bind, and the cost per draw is what adds up.
A primitive with the same state as the draw before it, arriving right after its vertices, now grows that draw instead of starting a new one. Draws that read their own render target are not merged, because the target is copied before each one. A fight frame went from about 15,000 draws to about 525, and the render thread samples in a Mu vs Mu fight dropped from 7-8k to 3.2k.
To trust that, I added a recorder for the GS thread: SSTHEH_GSREC saves what it is given for 150 pictures, and sstheh --gs-replay runs it again without the game and prints the time per frame and a hash of what was recorded for the GPU. Draws with the same state hash as one, so the hash stays the same with or without merging. The three builds I compared gave the same hash, and the time per frame went from 8.66 to 7.14 to 6.95 ms. Over the 40 timed frames, the render thread went from 608,759 draws to 21,028.
The hash ignoring the merge is on purpose: it is what proves that merging changed nothing of what reaches the screen. If a bad merge joined two draws with different state, the hash would change. In the same commit XGKICK started reading the packet straight from VU1 memory, without copying 16 KB per kick. Today’s recording, on the current build, gives this:
[vujit 0] 189 VU1 programs (resident code and 56 overlays), 0 to compile150 pictures shown (120 timed): 40 frames, 16814 draws, 6.935 ms per frame, all 425 ms, hash c2e9093a967c6e3fThat is about 420 draws per frame in that stretch of the fight. The replay has to run as ./build/sstheh: from anywhere else it does not find the compiled VU1 programs, falls back to the interpreter and gives another time and another hash, silently.
The crash that was not a use after free
About 1 fight in 25 crashed inside func_00155540, always the same function. Without symbols, a crash report with only registers is not much to go on, so I made it print the calls under way: the words above sp that look like return addresses, right after a JAL or JALR. It is a heuristic: the game’s code keeps no frame pointer, so the stack cannot be unwound with certainty, but a value on the stack pointing right after a call is almost always a return.
Then I wrote a tool for the suspect I had: SSTHEH_POISON_FREE=1 fills each freed block with 0xA5 and remembers who freed it, so a pointer read from a freed block stops the program there. It found a candidate: a move transition releases the outgoing move’s slots, and a later one instantiated a move with a pointer into one of those blocks. I wrote it down as a use after free, with “cause not confirmed yet”.
It was wrong, and the commit that fixed it says so. No slot was ever freed while another still pointed into it. The poison had found noise: blocks that really were freed, read by code unrelated to the crash.
What changed course was to stop looking at the code and look at the data. I saved the game’s 32 MB of RAM at the moment of the crash (SSTHEH_RAM_DUMP), decoded the move archive outside the game and compared it with the same archive read from the ISO. They differed in 295 bytes, always at the same absolute address, on every run. A use after free spreads garbage wherever the heap says; a fixed address is someone always writing to the same place. What actually broke was the data: the special-move archive of Aiacos was corrupted in RAM, and move 23 decoded to garbage whose script crashed.
The corruption was mine. Interrupt handlers ran on a stack at 0x01F00000, 1 MB below the end of RAM. When I wrote the kernel, I took that for free space below the main thread’s stack. It was not: the game gives out heap up to 0x01F80000, where that stack starts. Each interrupt wrote about 0x140 bytes at 0x01EFFEC0..0x01EFFFF7, over whatever the game had there, and in fights that was the archive. Since the stack grows down, the bytes landed right below 0x01F00000. On the console the first 512 KB of RAM (0x00000000 to 0x0007FFFF) belong to the kernel, and the game loads from 0x00100000, so I moved the stack to 0x00080000, the top of that area. With it, the archive in RAM is byte-identical to the one on the ISO, and 18 fights ran without a crash.
Make the crash report say what was running.
The call chain from the stack narrowed it to a script function.Test the suspect with a tool, not a guess.
The poison found something that looked like the cause.Compare the data with the source.
The archive in RAM had to match the one on the ISO, and it did not, which a use after free would not explain.Ask who else writes there.
The only writer was my own interrupt stack.
The same code, different results
While checking the translated programs against the interpreter, vubench check reported 997 of 8,771 runs ending differently. The compiler was the cause: with -march=native, GCC contracts _mm_add_ps(_mm_mul_ps(...)) into a fused multiply-add. One rounding instead of two, and the VU does two. The two paths gave different results for the same program.
This is not a GCC bug. The C standard allows contracting a * b + c, and in GNU mode (GCC’s default) the option is -ffp-contract=fast, which contracts whenever the CPU has FMA. With -march=native on a modern CPU, it does. Worse, it happened in the translated programs and in parts of the interpreter, so neither one was a trustworthy reference. The game’s recompiled MIPS code had no FMA at all, because each of its floating point operations goes through a function of the port’s own.
-ffp-contract=off for the port, the translated programs and the bench fixed it: 0 of 8,220 runs ending differently, and about 2% more VU1 cycles. The generator’s version went up (GEN_VERSION 12) so the programs already compiled with FMA were rebuilt. It also made me change how the cache of compiled programs works. They are named by a hash, so nothing told which ones an older generator had left, and the cache only grew (54 MB and 1,546 files for 189 programs in use). They now live in ~/.cache/sstheh/vu/<version>, where the version also covers the compiler flags, and the old versions are removed at startup.
Reverb and the opening video
Two things were missing from sound and video, and both were things the game does on its own.
The game turns the SPU2’s effect on for stages and menus, and the port mixed only the dry voices. The effect area now runs as the PS2’s reverb: the comb and all-pass filters and the reflections of the PSX formula at half rate, with the SDK’s ten presets, the effect sends of the voices and the wet mix into the output when the core enables it.
The SPU2’s reverb is not a separate effect: it uses a stretch of the sound RAM itself as a delay line. The game says where that stretch is (from ESA to EEA) and which preset to use through sceSdSetEffectAttr, and each preset is a table of offsets inside it, in units of 8 bytes. The hardware runs the formula every other sample, reading and writing that buffer, hence “half rate”. Mode 0x100 just clears the area.
The only video in the game is the opening, an 85 second MPEG-2 file that plays after about 80 seconds on the title screen. Sofdec decodes it with the PS2’s IPU, which the port does not have, so the screen went black and the video never ended. The scene’s step is now replaced: the file is read from the disc image and decoded with libavformat and libavcodec on a thread of its own. The picture takes the place of the game’s, the sound is mixed with the game’s, START skips it, and at the end the scene ends the way the game ends it. Without FFmpeg at build time the video is skipped.
The file is DATA/GX_JP.SFD: MPEG-2 at 512×448, 25 frames per second, with ADX sound. Looking for it I also found a 1.9 GB GX_MAIN.SFD, which looked like the high quality version. It is all zeros, a filler file that takes up the rest of the disc. In the port, the scene’s step (0x002C3740) is swapped for a function of ours that talks to the rest of the game the same way the original does.
Two follow-ups
Widescreen had a gap: in frames with 3D reaching the 16:9 sides, all the 2D was kept at 4:3, so a white fade over a 3D scene whitened only the middle and the bands of the VS screen stopped short. A 2D draw that spans the frame’s whole width and is a band (at most 35% of the height), either untextured or textured with about one color, is now stretched with the 3D. Pictures, like the main menu’s Seiya, keep their shape, and the fight HUD stays at 4:3.
“One color” has a concrete definition: every texel within 8 levels of the first, in each channel. It is worked out once, when the texture is decoded, and kept with it, so it costs nothing per frame. The tolerance is there because a “white” fade is almost never exact white. To take the before picture, I built the current port with only that rule turned off.
And there is a second gamepad now. Only the first was opened and only port 1 was read. The first (with the keyboard) plays port 1 and the second plays port 2, so the game’s modes offer 1P vs 2P when two are connected.
The game itself decides this. It asks the pad driver whether anything is in port 2, and the port now answers yes while a second gamepad is connected. If the gamepad is removed, the port is freed and goes back to empty. To test without two gamepads, SSTHEH_PAD2=1 pretends there is one, which is how I took the picture above.
A training mode
The week’s last item did not exist in the game. In a fighting game, learning a character means repeating the same combo against someone standing still and seeing how much it takes, and The Hades has no mode for that. The port does now: F5 (or Select + Start on a gamepad) opens a menu over the game, and with training on, in any fight:
- The computer’s opponent stands still. Moves it already started finish, but it decides nothing else.
- Life is infinite. It comes back to full a second after the last hit taken, and no hit knocks out.
- Damage is on screen: each side’s life, the last hit’s damage and the combo’s, with the number of hits.
- Player 1’s inputs are on screen, newest first, in numpad notation (5 is neutral, 6 is forward), with how many frames each was held.
For that, the port needed three things the game says nowhere: where each fighter’s life is, where a hit takes life away, and where the computer decides what to do. None of them has a name in the executable.
Life was found through the picture. With SSTHEH_RAM_DUMP=1, each --shot also saves the 32 MB of RAM. I took pictures at different moments of a fight, measured the life bar in each one and looked in the dumps for a value that changed in the same proportion. Two showed up. One is life divided by 100, which the HUD uses to draw the bar, and writing to it changes nothing. The other is the real life, a float from 0 to 100: writing 100 to it every vblank, the fighter would not die. What was left was finding who wrote there, and for that I used a hardware watchpoint in gdb on that float in the emulated RAM. It stopped in 0x0031E5F0, which applies every hit: it takes the fighter’s life block and the base damage. One detail of it explains “no hit knocks out”: life is clamped to at most 100, but not to at least 0.
Heap addresses change from one fight to the next, so the port keeps none. There are three hooks, native functions that run on entry to one of the game’s functions and then let the original code go on:
- at
0x002F9140, the battle’s step, once a frame: reads both fighters and both controllers, and puts life back to 100 when a second has gone by without a hit; - at
0x0031E5F0, the damage: notes the life before each hit (the damage is the difference) and prevents the knock-out; - at
0x002FF490, the controllers’ common step: if the controller is the computer’s, it clears the decision it just made.
The last one was the nicest to find. Both kinds of controller, the player’s (which reads the pad) and the computer’s (the AI), end by calling the same function. The method table pointer tells which is which, and clearing the AI’s decision there leaves the fighter standing still without touching anything else. Against a second player nothing is cleared, because that controller is the player kind.
The panel is drawn by SDL’s software renderer into a 640×360 RGBA picture (480×360 in 4:3), which the GPU stretches over the game’s picture. Since it sits on top of the finished frame, it does not interfere with anything the GS draws, and widescreen and the render scale do not need to know it exists.
What I learned
- Measure before optimizing, and measure with something that does not change the result. The replay bench and the recorded VU runs let me change a lot of code while comparing the output to the original.
- Write the suspect down as a suspect. My first diagnosis was wrong and the note said “not confirmed” for that reason.
- A bug that looks like a bug in the game can be in the platform layer. The game never freed anything it should not have; my kernel wrote on its memory.
- The compiler is part of the hardware you emulate. A fused multiply-add is a valid optimization for the C and a wrong result for the VU.
Not done yet: the AI-upscaled texture pack I mentioned at the end of the last post. The code is in the port’s repository, which ships nothing from the game: everyone builds it from their own disc.