the snickerblog

Aerowings

9-11-2018 10:42 AM PDT

Aerowings works now. This game necessitated the implementation of 32-bit ARGB paletted textures.

The House of the Dead 2

8-19-2018 8:53 PM PDT

I just discovered that HotD2 is in-game. As with Zombie Revenge and Crazy Taxi 2, this is a game which worked the first time I tried it, meaning that it has probably been working for a while and I just didn't know it.

The House of the Dead 2 is a lightgun shooter. I don't have the lightgun implemented in yet but this game also lets you play with the controller by using the analog stick to move an on-screen cursor, and that does work in WashingtonDC.

Things have been going a little slower this month due to some IRL things getting in the way: a really tough project at work, Bay Area gaming/anime conventions, and most recently a crippling Monster Hunter World addiction. I think I'll have more time for WashingtonDC in September but even now I'm making sure to get some progress made every day even if it's something minor like fixing bugs in the remote gdb stub. My current focus is on the AICA, which is Dreamcast's audio system. I already have AICA's CPU, the ARM7 working but the actual audio hardware is mostly unimplemented. I suspect that most games which aren't working aren't working because the AICA implementation is incomplete.

Evolution: The World of Sacred Device

8-10-2018 1:53 AM PDT

Evolution is now in-game. This is a JRPG featuring a cast of cute characters and randomly-generated dungeons. It has some really memorable music, too (not that you'd know playing it on WashingtonDC...I still haven't implemented sound yet). Some of the enemies look like they belong in a PSX game, but the main party is pretty high-polygon and the lighting model this game uses looks amazing.

The technological improvement that made this game playable in WashingtonDC was the implementation of the PowerVR2's YUV420 converter. What this does is implement part of the MPEG/JPEG image compression in hardware. Specifically, it doubles the vertial chrominance sampling and interleaves the luminance and chrominance channels together into a format that PowerVR2 can sample from as a texture. Evolution uses this YUV converter to show an intro cinematic of Mag and Linear entering the dungeon at the beginning of the game. Unfortunately, the still movie doesn't actually get displayed becase the game sees that there's something wrong with the audio hardware (which there is, because I haven't implemented it yet) and gives up on showing it right after it begins. Nevertheless, I still had to get the YUV converter working so the game can proceed because the game does actually try to show movie before it realises there's no audio hardware.

BTW, one interesting fact about this game: it got ported to the Neo Geo Pocket Color, of all platforms. In the waning years of SEGA's hardware days, they gave up on trying to beat the Game Boy and threw their weight behind SNK's handheld platform instead. Neo Geo Pocket Color received this, a 2D Sonic game, and Puyo Puyo Tsu (and probably more SEGA stuff too, those three are just all I can name off the top of my head). I haven't had the pleasure of trying out the NGPC port of Evolution, but from what I've seen it does a great job of remaking Evolution's 3D graphics as old-school pixel art.

Crazy Taxi 2 and Zombie Revenge

8-2-2018 10:55 PM PDT

Looks like we got ourselves a twofer in today's snickerblog update! Crazy Taxi 2 and Zombie Revenge are both games which I tried to run for the first time in the past week and both of them got in-game without needing any changes to WashingtonDC, so probably they've both been working for a while now and I just didn't know it.

Both of these games remain Dreamcast exclusives to this day. Zombie Revenge is a beat-emu-up game set in the world of House of the Dead. Crazy Taxi 2 is largely similar to its predecessor, except it's set in New York City instead of San Francisco and there are some new game mechanics thrown in.

I also managed to get SoulCalibur and Crazy Taxi (1) working on the same branch at the same time, so SoulCalibur support has been merged to master now (unfortunately, there are still mysteries surrounding all the weird things SoulCalibur tries to do with TAFIFO). That brings the total number of games known to work (by which I mean able to get in-game) to 7. A few weeks ago, there were only 3 known working games, so this is exciting progress.

SoulCalibur

7-27-2018 11:24AM PDT

Here's SoulCalibur running on the snickerbockers/soulcalibur branch. This game does some strange things with the PowerVR2's TAFIFO register, so to get it working I had to make some changes which *might* not be correct. One of those changes ended up breaking Crazy Taxi, so I need to do some more research and development before this can get merged into master.

Namco Museum

7-26-2018 10:06AM PDT

Namco Museum has joined Crazy Taxi, Daytona USA, and Power Stone to become the fourth (known) game to get in-game in WashingtonDC. This game was previously unbootable because WashingtonDC was exposing a bug in the game itself. All of WashingtonDC's IRQs happen instantly, and it seems that was causing this game to miss a PowerVR2 interrupt and hang forever.

Now I have PowerVR2 IRQs running on a 1 millisecond delay, and as a result this game can boot and go in-game. Other IRQs are still happening instantly for now because I haven't bothered to get good measurements for how long they generally take on real hardware. I don't really even know if 1 millisecond is too long or too short for the PowerVR2 IRQs.

This game exposed some bugs in WashingtonDC that had to be fixed. There are two remaining problems with this game that need to be fixed. One of them is a soft hang that only happens with the JIT enabled. I think that's a self-modifying code problem, caused by the game not clearing out the instruction after overwriting program memory. The other bug involves the spinning logos at the beginning disappearing when they stop spinning. That's being caused by the screen getting cleared when it shouldn't have. I need to run some more hardware tests to figure out exactly what's supposed to be happening there, but it looks like I don't have the tile rendering implemented correctly.

There were a couple other games that had the same IRQ timing bug as Namco Museum. Since those are finally unblocked, I'm going to get some of them in-game as well so I have more test-cases before I move on to finishing AICA. The next game will probably be either Soul Calibur or Star Wars Episode I Racer.

WashingtonDC now running on Windows via cygwin

7-18-2018 12:49AM PDT

This blog update is actually about a month and half late because I got this working way back in late May, but WashingtonDC is finally workin in Windows via Cygwin.

It runs really slow (which I think is because OpenGL might be using a software rasterizer under cygwin) but it does have a working x86_64 backend for the dynarec so I can use this as a starting point to gradually get a native Windows port running.

Going forward, I think the roadmap is going to be to get WashingtonDC to build in mingw and then later get it working under MSVC. The main roadblock is going to be that MSVC doesn't implement C11 and it doesn't really even implement C99 very well. I don't really know what the solution to this is going to be. I'm rather fond of C11, so if I can get things running fast enough in mingw then I just might never bother with MSVC. The dynarec is going to emit the same code regardless of what compiler it was built with itself, so I might find there is a negligible difference between MSVC and mingw for my purposes.

I still hope to have WashingtonDC ready for a release in time for the Dreamcast's 20th birthday in November. That only leaves me a little over four months to get it to a usable state, so I'm thinking Windows support might not be available in the first release. Build systems and GUIs tend to be the two biggest time-sinks in software development even though there's no reason why they should, so I'm worried that the effort needed to get the Windows port boostrapped might cause me to miss my target date.

T-minus six months to liftoff!

5-9-2018 1:41AM PDT

Wow, it's already been five months since I wrote the post about regression testing! This site might not have changed much, but there's been a lot going on with WashingtonDC in that time. When I wrote that last update, WashingtonDC was barely rendering the Dreamcast's spiral-swirl bootup at a mere 8% speed (on my PC). Back then WashingtonDC only had a simple interpreter. Now it has a dynamic recompiler (sometimes called a dynarec) which crushes the spiral-swirl at an average speed of 140%. That's faster than a real Dreamcast!

In-game performance isn't quite as perfect as bootup performance; you can generally expect that to swing between 30% and 60%. I think the main bottleneck there is the FPU; all of its opcodes are implemented by calling into the interpreter's FPU opcode handlers. This brings in all the overhead of a function call, plus the overhead of not being able to hold stuff in registers for long periods of time, plus the overhead of having to branch based on whether the FPU is configured for single or double precision on every instruction. I imagine that can add up and turn into a bottleneck even though it's just one subset of the SH4's instruction-set.

Most importantly, I just got Daytona USA booting a couple nights ago! A lot of the textures are being rendered with the wrong colors, but Three-Seven Speedway still has its classic charm. I haven't gotten any new games working since Crazy Taxi started working way back in September (or was it October?), so I'm excited to finally have a third game in my testing set alongside Crazy Taxi and Power Stone.

Beyond that, there are a bunch of games which are booting, but can't make it ingame. Puyo Puyo~n shows a message in Japanese (which I think is it complaining about the VMU not being plugged in) before failing due to unimplemented functionality. Evolution: The World of Sacred Device bitches about the VMU not being plugged in and then flashes the logos of its developers before ultimately failing due to unipmlemented functionality. Shenmue and Jet Set Radio both display "Now Loading" screens before failing (due to unimplemented functionality, of course). Twinkle Star Sprites technically gets in-game, but the graphics are heavily corrupted.

I didn't quite get around to finishing the regression tests I was talking about in my last post; I don't think I've touched them since January. I need to keep my time balanced between auxiliary stuff like that and exciting stuff like making progress. Naturally, the boring auxiliary stuff gets way less attention compared to exciting things like a faster recompiler and booting new games. I'll get back around to it some day.

The primary thing that keeps WashingtonDC from booting most games is the ARM7 CPU. This is a relatively unadvanced 45MHz RISC machine which is wired into the Dreamcast's audio system (AICA). Games can load whatever code they want onto the ARM7 to control AICA. Most games will end up hanging indefinitely without this because the SH4 spins forever waiting for the program it just loaded onto ARM7 to respond. I get around this by manually hacking the AICA memory to show games what they want to see (so they think the ARM7 program is responding) but this is only feasible for a very small set of games, and if you actually want the audio to work then it's not feasible for any games at all.

I don't anticipate that it will be difficult to implement ARM7 emulation, but I do expect it will be difficult to keep this and the SH4 in sync with each other. I've been making a lot of infrastructure changes to support this and I still have more work to do before I can begin implementing the ARM7 in earnest.

After that the obvious next steps are to get the audio working, and to get WashingtonDC running at a smooth 60FPS on typical games. I want to have WashingtonDC ready for a release (insomuch as you can "release" something that's been sitting in a publicliy-accessible github repo since 2016) on 11/27/2018. That's the twentieth anniversary of Dreamcast's Japanese launch, so it's the perfect date to launch a Dreamcast emulator. That gives me 202 days, which is a tight schedule but I ought to be able to get WashingtonDC to a level of basic functionality by then.

I also intend to overhaul this site and have a real regularly-updated development blog going like other emulator projects. I might end up moving to a dedicated blogging platform or I might overhaul this webpage into something less spartan. Whatever I do, I probably won't get around to it until after I have the ARM7 working at the earliest. Ever since I got started way back in October 2016 I've been treating WashingtonDC like my hobby project, but going forward I'm going to have to start treating it as a product (albeit an open-source product that doesn't make any money).

To that end, I think I need a new name. "WashingtonDC" has always been a placeholder name which I chose because it has DC in it, and also because I was watching the 2016 Presidential Debates when I got started. I don't actually live in Washington DC or even Washington state so it feels a bit dishonest to keep calling it that. In my head, I tend to abbreviate it to "wash", so I'm thinking of rebranding WashingtonDC as washDC, with the logo being a wave (of water) curled over to vaguely resemble a Dreamcast logo. The other benefit of this is that I'll get better search-engine results if my emulator doesn't share a name with the capital of the United States (as somebody on /r/emulation astutely pointed out back in February).

Regression Testing WashingtonDC

12-10-2017 4:17PM PST

One of the difficult things about starting my emulator was trying to test it without having anything to test against. During the early stages there wasn't any software I could boot, and I didn't even have any form of visual feedback from the emulator until I had been working on it for six months. By my count there are 236 opcodes in the SuperH-4 instructin set, and most of them get used during the boot process. If one of those opcodes is wrong, the emulated software might loop forever (best-case scenario), or it might call the wrong function, or it might write a garbage-value somewhere and corrupt its own state. In the latter two cases it's very difficult to figure out what went wrong because the root-cause of a bug could be very far removed from its symptons. Worse still, without any form of feedback from the emulated software I might not even know that something did go wrong.

The obvious solution is to write test programs that can detect when something is wrong, but in order for that to work I'd need a way to load them into the emulator and a way to get feedback from them. Loading them can be accomplished by writing a test program that replaces the firmware image, but then I still need a way to get feedback from them; there's no way to get feedback from emulated software until the emulator is functional enough to support the mechanisms that software would use to send feedback, and I needed test programs to help get the emulator to that state.

My solution to this was to make a couple of unit-test programs which would link against WashingtonDC's .o files, initialize some of the hardware, artificially inject sh4 code into the memory via an sh4 assembler library I wrote, execute that code, and then inspect the CPU's registers to see if they matched the intended final values. This resulted in three separate tests: sh4mem_test, sh4inst_test, sh4div_test, sh4tmu_test, and sh4asm_test (which tested the assembler library). These four tests helped root out several bugs so I could move WashingtonDC along.

Since they all needed to link against WashingtonDC as a library, they had to be refactored every time WashingtonDC's internal APIs changes. This was an especially arduous task in the case of sh4inst_test, which had swollen to be more than 10000 lines of code as I kept adding more and more testcases. Sometime after I got the firmware booting, I decided the work needed to keep them running was greater than the benefit I got from them so I let them die of bitrot, and eventually I deleted them from git.

Recently, I've been getting ready to make some major changes to the sh4 to support a JIT recompiler, and also to make the existing interpreter faster. This motivated me to bring back the unit_tests, only this time I have the infrastructure needed to build real test roms for the Dreamcast. My current approach is to have the roms do complicated operations and print the results to the serial port. The tests are designed so that if anything goes wrong the output will change. I have a couple perl scripts which launch WashingtonDC with the given test roms and compare the output to captures taken from the same roms running on a real Dreamcast. If there are any discrepancies then the perl script knows something is wrong and it raises an error.

This scheme sounds primitive but it works much better than the old unit_tests scheme did because it's actually comparing WashingtonDC against a real Dreamcast instead of comparing it against my estimations of what a real Dreamcast would have done. I'm just getting started but I've already found and fixed two bugs which the old sh4inst_test was missing. I was worried initially that this would turn out to be a pedantic waste of time since WashingtonDC can already run a few games and I didn't think there would be any basic instruction bugs which wouldn't prevent something complicated like Crazy Taxi from booting; this assumption could not have been more incorrect.

I think the best part of the new test roms is that since they're real Dreamcast programs, they'll never die of bitrot like the old unit_tests did. I can keep making whatever major architectural changes I want to WashingtonDC and that will never require a major refactor of the test roms. I don't really even need to keep the source code around (not that I would ever get rid of it).

Bootstrapping WashingtonDC

10-9-2017 8:56PM PDT

WashingtonDC is an in-development SEGA Dreamcast emulator I started working on almost a year ago as a pet project. It's my first attempt at emulation development, and I'm quite pleased with the progress I've made so far. This is something I started last year as a pet project. It's the most complicated program I've ever worked on, and a year ago I didn't know if it was going to work out, or if it would turn out to be a waste of time.

On October 14 2016, I started with nothing and begin implementing some basic CPU opcode handlers and a simple memory device. I didn't have enough infrastructure at that point to run real Dreamcast programs, so I wrote a test function for each opcode which would send it a randomized input, execute an instruction, and check the output to see if it matches expectations. These test functions constituted more than half of WashingtonDC's source for the first few months, and ultimately grew to become a 10000+ line file before they outlived their usefulness. This testing system also included its own SH4 assembler so that I could specify the assembler directives in plain-text.

The assembler proved to be surprisingly difficult to implement, and I remember spending an entire weekend panicking over it because I had never written this much code before and I was worried it would come to dominate the entire project. In retrospect it wasn't that big of a deal because it only came out to approximately 3k-lines and got mostly written over the course of that single weekend, but at the time it felt like a major undertaking.

Eventually I had approximately half of the opcodes implemented, so I used the opcode handlers to implement a simple CPU interpreter. This interpreter would load a firmware dump (dc_bios.bin) and a flash dump (dc_flash.bin) from a file and begin decoding/executing the firmware one instruction at a time. It's designed to print an error-dump and exit every time it sees something it doesn't recognize so I know to go figure out what just failed and implement it. To make forward progress, I would run the firmware through the interpreter and observe the output when it failed. Usually the failure cause would be some unimplemented functionality, so I'd implement it and then re-run the firmware; it would then make it slightly further before failing again on some other thing that I would need to implement. The goal was to continue this cycle until I could boot the firmware.

Not all of the problems I encountered were related to unimplemented opcodes or unimplmented memory-mappings. Some of them were the result of mistakes I had made. Trying to debug a CPU interpreter when something goes wrong can be difficult when you don't even have access to the source code, so I decided to skip the firmware for now and make my emulator boot directly to a homebrew program (which it does by loading the program into memory where the firmware would have put it and pointing the SH4's program counter at the beginning). The Dreamcast has the best homebrew community of any classic game console, and part of that community is an open-source SDK called KallistiOS.

KallistiOS is a sort of lightweight OS kernel that provides high-level APIs for interacting with the Dreamcast hardware, as well cooperative multithreading, filesystems, networking and a host of other features. Having access to the source code made it a lot easier for me to debug issues caused by bugs in WashingtonDC. KallistiOS also gives me an easy way to run code on a real Dreamcast system so I can test for consistency between WashingtonDC and Dreamcast.

What's really great is that since KallistiOS' cross-compiler toolchain is based off of GNU, it generates standard ELF binaries complete with GDB-compatible debugger symbols. To leverage this, I wrote a remote GDB backend for WashingtonDC; it connects with GDB using GDB's remote debugging protocol over TCP via localhost, and it provides GDB with a way to interact with the state of the emulated Dreamcast. On the GDB-side of this connection, there's a normal GDB client (with SH4 support enabled), and I can control the execution of the program runnign inside of WashingtonDC's virtual machine just like it's running on my local machine; that includes breakpoints, watchpoints, single-stepping, memory-access, register access and even ELF symbol lookup (which the GDB client implements entirely independently from my GDB stub using the lower-level functionality my GDB stub provides it with).

Eventually WashingtonDC could boot through KallstiOS' initialization and get to the homebrew program's main function. I didn't have graphics at this time, so I implemented the SCIF, which is a UART on the SH4 which connects to the serial port on the back of the Dreamcast. KallistiOS sends its printf output to the SCIF so you can connect a null modem from your Dreamcast to your PC and view the output on your PC. My SCIF implementation sends that output over TCP to a telnet session so I can use telnet to watch the nessages KallistiOS prints when it boots.

The examples directory in KallistiOS' source includes several homebrew programs which demonstrate how to use KallistiOS. One of them is a port of the classic BSD text adventure, Colossal Cave Adventure which became the first game to work on WashingtonDC when I implemented the serial port.

With KallistiOS booting, I had a way to run my own programs in WashingtonDC or on a real Dreamcast in a high-level environment; this was a huge boon for testing and reverse-engineering. I kept moving forward a little bit at a time by getting more and more of KallistiOS' example programs to work. I got framebuffer graphics working, which allowed me to see visual output from some of KallistiOS' example programs such as this:

More importantly, this meant I had enough infrastructure to run the SEGA bootstrap program that is included in every Dreamcast game. This program is commonly referred to as IP.BIN within the context of homebrew development. It resides in the ISO9660 filesystem's first 32-kilobytes. When the Dreamcast firmware loads a game, it first runs some simple authentication tests whcih are designed to stop software pirates and homebrew developers from running unauthorized code off of CD-R discs (hackers figured out how to circumvent these checks a long time ago) and then it loads IP.BIN from the disc. IP.BIN's job is to draw a SEGA logo to the framebuffer along with the words "PRODUCED BY OR UNDER LICENSE FROM SEGA ENTERPRISED, LTD" and then load the game off of the disc.

It wasn't actually loading the game because I didn't have the GD-ROM code working yet, but it was drawing that logo. This was the first time I had a "real" Dreamcast program running in my emulator. At this point in time, the date was April 14. 2017 and I was just now starting to get visual feedback from WashingtonDC after 6 months of work.

I kept moving forward with remaining unimplemented features such as the GD-ROM drive, the PowerVR2 GPU (which does 3D graphics rendering), and the remaining unimplemented opcodes (which at this point were mostly just the SH4's floating-point unit). For these, KallstiOS was once again a huge help because I could use its source to understand what exactly programs expected these components to do. MAME's PowerVR2 code was also a huge help. After another couple of months I was running some of the KallistiOS demos that made use of the PowerVR2, such as this port of one of the classic NeHe OpenGL tutorials:

Eventually, I felt like I had enough infrastructure to go back to trying to boot the BIOS, so I gave it a try and this happened:

I immediately recognized this as a distorted version of the Dreamcast's "spiral swirl" bootup animation. After about a day spent fixing up the PowerVR2 code, I got this image which confirmed I was watching the firmware's bootup animation:

And after another day, I actually had a working spiral-swirl:

From there it wasn't long until I had the Real-Time Clock reset screen and the main firmware menu working. This was in mid-July, so it took me a total of 9 months to get to the point where I finally had something which I could honestly consider a working Dreamcast emulator.

Over the past three months, I've been working to get actual games booting. The first game I got working was Power Stone. I don't have the AICA (audio hardware) implemented yet, and Power Stone (along with most other games) doesn't boot without working audio hardware. Actually implementing the audio hardware will be time-consuming because it has its own dedicated CPU I have not yet implemented. I was feeling impatient after 9+ months of development, so instead I implemented a hack to fool it into thinking the audio hardware is present by editing the AICA's memory to show Power Stone what it wants to see (thus fooling it into thinking the program it loaded onto the AICA's ARM7 processor is talking back to it). This hack works, and it allows me to get in-game in both Power Stone and Crazy Taxi (and maybe other games, too).

A year ago I started this project not knowing if I would be able to create a working Dreamcast emulator. Now I have one and I look forward to turning it into a usable platform in terms of both compatibility and performance. Currently WashingtonDC can run two games, and it runs both of them at approximately 8% of the speed of a real Dreamcast on my humble 3.1GHz AMD Bulldozer. I hope by this time next year WashingtonDC can be a truly viable Dreamcast emulator, able to run a significant portion of the Dreamcast library at full-speed.

For the immediate future, my goals are to add the analog stick and analog triggers to my controller implementation (which currently only supports the digital buttons on the Dreamcast's gamepad) and then get started on a JIT compiler which can convert sh4 machine-code into x86_64 machine-code. With the JIT compiler implemented, WashingtonDC should be running siginificantly faster than it is now and it will be feasible to get the audio hardware working (if I got the audio hardware working now, it's doubtful I'd be able to hear anything due to the pitch-shifting from running significantly slower than a real Dreamcast).