Wednesday 23 December 2015

DreamPi 1.0 Released!

A few years ago I purchased a cheap USB modem with the idea of building a "PC/DC server" and connecting my Dreamcast to the Internet. When it arrived I couldn't find the time (and didn't have the skill) to build the required line voltage inducer, so the modem sat in a drawer untouched.

Later, the Raspberry Pi was released, and I bought one. I figured it was time to finally build that PC/DC server! But alas, like most Raspberry Pi purchases it ended up in the drawer alongside the USB modem, like pieces of an incomplete jigsaw puzzle.

Fast-forward to 5 months ago. Since buying the modem my electronics skills had improved considerably so I finally built that line voltage inducer, and tried to configure the PC/DC server with the Raspberry Pi. It didn't work. I tried everything! Different resistors, different batteries (up to 12V) , I rebuilt the line voltage inducer twice. It still didn't work.

Then someone on a forum said it seemed like there wasn't enough voltage. I couldn't understand that - all the tutorials had said 9V was plenty, I was using 12! One day I got reckless and connected two 9V batteries rather than one, and straight away Phantasy Star Online connected! It was at that point that I realized that I'd never actually seen anyone with a PAL Dreamcast manage to get a PC/DC server working, I'd seen plenty of people with PAL Dreamcasts complaining it didn't work however. This was the mystery solved! PAL Dreamcasts need more voltage than the U.S. counterparts.

At this point connecting was a pain. I had to connect to the Pi, launch a program, and then dial on the Dreamcast, then answer manually on the Pi. "Surely there's a better way", I thought.

It occurred to me that the USB modem could "listen" for the Dreamcast dialling. If it could then I wouldn't need to answer manually, I could write some code for that. I opened up my editor and started learning HAYES commands to talk to the modem. I put it online and waited until the Dreamcast dialled, and then answered. Success! After making sure that I could repeat the process automatically I pushed out a release of the DreamPi image. People were happy, but there were still problems...

Most games look for a dial tone before they will try to connect to a remote server. Obviously there is no dial tone when you've just connected one modem to another. While browsing for information I discovered that someone had built an interesting solution by connecting the line-out of a computer to the phone line, and played a recording of a dial tone into the line until the Dreamcast dialled. Then there was another "eureka!" moment. I already had the line open while I was listening for the Dreamcast to dial numbers, I could definitely play the sound of a dial tone through the modem at the same time.

It took a lot of fiddling to find the right set of HAYES commands, but eventually I succeeded. Now all Dreamcast games would work! It took several more bug fixes and releases to get to where we are today. But here we are at version 1.0.

So, today I'm announcing 1.0. This release extends the timeout for waiting for command responses, and doesn't echo our commands back to the modem which was confusing things.

There is still more to be done. I'm not happy with the IP address allocation, and WiFi support has always been on the list. But I'm happy to call this release complete. You can now play your Dreamcast games online with nothing more than a USB modem, a Raspberry Pi and a simple electrical circuit.

My next challenge is to get a Dream Arena authentication server running. At that point the last remaining games (Toy Racer and PAL Q3 Arena) will work and the DreamPi will allow playing every online game that still has servers.

Merry Christmas everyone!

Wednesday 16 December 2015

DreamPi 0.99 Released

This is a major bug fix release.

  • Switched to using "arp -n" rather than "arp-scan". This command was used to detect IPs in use but would cause the DreamPi to hang sometimes. Hopefully this will go away now.
  • New upstream DNS server. The internal DNS server has had all its rules removed except for one pointing to a new hosted DNS server. This means that the DreamPi retains the DNS caching and allows overriding but configuration changes can be made without pushing a new release.
  • Better command sending. The serial modem communication no longer echoes back the commands it's sent. Hopefully this will fix the problems with command responses not being recognized.
You can download the new release here.

Sunday 6 December 2015

DreamPi 0.98 Released!

This is a bug-fix release:

  • More reliable modem command sending. Some modems insert a newline character occasionally midway through a response. This was causing the software to get stuck in an infinite loop. This should now be fixed.
  • Faster Starlancer connecting. Starlancer attempts a reverse DNS lookup several times during connection, this PTR record doesn't exist and so it eventually times out. The DNS server built into the DreamPi now responds to this reverse DNS lookup immediately making the connection happen instantly. See video below!

Download the new release here.

Sunday 29 November 2015

DreamPi 0.97 Released!

 This release is a major release!

  • Near complete rewrite. The code is now much cleaner
  • Fewer modem commands used. Fewer commands == fewer places to go wrong!
  • Starlancer now supported! Starlancer now works with 100% connection success rate!
Download it here!

Saturday 28 November 2015

DreamPi 0.96 released!

This version is a minor release:

  • IP address autoconfiguration. The DreamPi now tries to use IP addresses .98 (for the Dreamcast) and .99 (for the Pi). If these IP addresses are taken it adjusts downwards to find the next available IPs.
  • DNS now available to other machines. For testing the DNS server is now accessible by any machine on the network, not just the Dreamcast.
Download the new release here!

Thursday 26 November 2015

DreamPi 0.95 Released

This is a bug fix release:

  • Pi <> Modem communication now happens at a constant safe baud rate
  • The PPP peers file is now configured with the max speed of the modem, this increases performance and hopefully fixes connection issues that some people were having during PPP negotiation
  • The Starlancer DNS is now configured in the dnsmasq configuration directly rather than hitting an upstream nameserver. This should hopefully fix DNS issues
Go download here!

Monday 23 November 2015

DreamPi 0.94 Released

This is a minor bug fix release:
  • Fixed a bug in the DNS configuration
  • Cleaned up the log output
  • Added a pause before answering to hopefully make things more reliable

Saturday 7 November 2015

DreamPi 0.93 Released!

This release is a minor release.

  • New DNS servers added. Starlancer and Quake 3 should work now.
Download it here.

Friday 6 November 2015

DreamPi 0.92 Released

This release includes the following new features:
  • Dial-tone generation. The DreamPi software now plays an 8 bit audio recording of a dial tone across the phone line to trick the Dreamcast into dialling. Blind dialling is no longer a requirement!
  • Custom DNS server. There is a DNS server now running directly on the Pi. Not only does this cache DNS requests which makes things a little quicker, but also it allows us to convince the Dreamcast that old sites and game servers are back online! To see this in action, try entering the "Internet" section on Sonic Adventure and visit the website!
  • Bug Fixes! Lots of bug fixes to the order of AT commands etc. hopefully more modems will be able to work with the Pi
  • Auto-configuration. On boot the software will detect your network subnet and update its configuration accordingly. 
 You can download the new release here.

Friday 7 August 2015

SEGA Won't Build a Dreamcast 2. But Could We?

The SEGA Dreamcast was an iconic console. To many, it is regarded as the greatest console that has ever been released. Since its untimely demise many fans have hoped that SEGA would re-enter the hardware industry and release a Dreamcast 2 - an updated but truely backwards-compatible successor to the original hardware.

This is very unlikely to happen. Consoles are expensive, if SEGA did re-enter the game and it was a failure (which is pretty likely) it would probably kill SEGA entirely.

Image from the Dreamcast 2 Petition on by Ben Plato

There is however a petition online to design a Dreamcast 2 which would be built-to-order, nullifying much of the risk. This actually isn't a terrible idea. Designing an updated version of the original Dreamcast, with perhaps even the same specifications but with modern connections (LAN vs Dial-up, HDMI vs VGA, HDD vs VMU) probably could be done by a small internal team. The very people interested in this kind of hardware would be willing to pay above the cost of manufacture to own it. SEGA could make a profit on the hardware, and as suggested in the petition, provide an online store for purchasing digital versions of the classic games that already exist. If they opened that store up to indies then that would be a bonus.

But still, this is very unlikely to happen. SEGA is primarily a software company now, and most of the team that designed the Dreamcast probably moved on long ago. But that doesn't necessarily mean a Dreamcast 2 couldn't be built. The tools needed for a dedicated community to build a Dreamcast clone (let's call it OpenDC) aren't that far way.

Patents are Expiring


The Dreamcast was released at the tail end of 1998. Given that patents only last 20 years in most countries, and that most of the Dreamcast patents (e.g. for the GD-ROM) were patented in 1997, the Dreamcast will essentially be free to clone in a couple of years.

There's even better news though. The SH4 CPU which the Dreamcast uses will no longer be patented in 2016 and the Open Processor Foundation is already working on their own version. It's not unlikely that at some point in the next 3-4 years there will be a "J4" system-on-a-chip (SOC) available, it's also probable that any SH4-clone SOC will be higher spec'd than the Dreamcast was, with some kind of OpenGL ES based GPU built in.

If an SH4 based SOC was released, the rest of the problems to solve involve software, peripherals and packaging.



On the software front, we have a bit of a head start. Linux is free, open source and supports the SH4 architecture. The community would need to build a Linux based OS with a basic graphical launcher, and provide any libraries which the Dreamcast executables call into. It's a fair amount of work, but there shouldn't be anything impossible here.

We also already have a free fully featured Dreamcast SDK in KallistiOS, and new games are being released based on it even now. This means that the community could provide an online store for new homebrew games written for the new console. This could be accessed through the OpenDC launcher and download games directly to disk.



The biggest issue on the peripheral side is the GD-ROM. This will either have to be custom made when the patents expire, or alternatively, the community would need to develop new firmware for an existing CD-ROM model. Another option would be to forgo the GD-ROM drive entirely, but that wouldn't really fulfil the backwards compatibility that everyone would like.

The dial-up modem could be emulated in software. This just leaves the controllers. With 3D printing really taking off now, it's quite probable that over the next few years it will be possible to print electrical connections into plastic. When that happens, 3D printing a case for the console, complete with Dreamcast compatible controller ports would simply just be a case of sending a design to Shapeways.

If using old Dreamcast controllers isn't enough, a custom community-designed controller could be made using 3D printing and software like Fritzing.

The Dream is Still Alive!


So, even though SEGA may never release another console, a driven and dedicated community following open source practices could build another Dreamcast. We might just have to wait a few more years to see it.

Tuesday 16 June 2015

How Djangae's App Engine Datastore Connector Works

One of my many responsibilities is being the BDFL of the Djangae project. Djangae is a Django application which provides a compatibility layer to allow Django to integrate well with Google App Engine.

The most complex part of the Djangae project is the "datastore connector". This is a Django ORM backend which allows much (but importantly, not all!) of the Django ORM to function using App Engine's non-relational Datastore as the database.

In this post I'll summarize the key tricks we use to make the Django ORM work on the Datastore.

Tuesday 7 April 2015

My C++ wishlist

I regularly use C++ for my own personal projects, that's normally because my personal projects are gamedev related and the power and performance of C++ comes into its own here. My day job however is writing Python, and although C++11 has completely revolutionised the language, there are occasionally some things I miss when I'm coding at home.

Here's a quick list of things I would love to be added to C++, but I'm pretty sure won't be!

1.  Symbolic imports

Imports in Python are amazingly clean and flexible. The ability to just type:

from os.path import join

...knowing full well that it's not going to dump the entire contents of another header file into my code is wonderful. C++ sucks here, including headers slows down compilation, it means you must use header guard #defines, and forces you to do strange tricks (like pre-declaring classes) to make things work. It's painful. I hear that modules are coming in some future C++ version, but IMO they can't come soon enough!

2. Decorators

In Python, you can write a function that wraps another function, allowing you to perform pre/post operations and have them called automatically whenever that function is called. The original function seems unchanged. In Python this implementation of the decorator pattern looks like this:

def the_function_that_is_wrapped():

Since C++11 introduced lambdas defining such wrapper properties could be very similar to Python. For example:

std::function<void ()> some_property_function(std::function<void ()> the_func) {
    return []() {
        //Do pre-stuff
        return the_func();

All that's missing is some way to apply such a decorator to a function or method. Although I'm sure that's not as trivial as it sounds.

3. Context Managers

In C++, there is no try/finally. If you want to ensure that something is destroyed no matter what the solution comes in the form of RAII.

However, RAII means that you end up having code like this:

if(something) {
   { //<< Seamingly pointless scope block that needs a comment
       Lock some_lock_which_will_be_freed;

Brilliant. It would be amazing to have an actual syntax for this behaviour, a 'with' statement is perfect!

with Lock()as some_lock_which_will_be_freed {

Now there are no weird extra braces without explanation!

4. Overloading of the '.' operator

This would be tricky to implement, but I've lost track of the number of times I've had to rely on overloading '->' just because overloading '.' isn't possible.

5. Read-only properties

Properties in Python allow you to add apparently public attributes to a class, which actually implicitly return the result of a function. These kind of read-only properties make using your API nicer, and allow you to abstract away the implementation of how that property's value is calculated. For example:

auto class_size = my_school.pupil_count / my_school.room_count;

If you actually exposed pupil_count and room_count as public variables, and then decide actually, you can calculate pupil_count on the fly by summing the school_year.pupil_counts you are forced to make all calling code do this instead:

auto class_size = my_school.pupil_count() / my_school.room_count;

A minor change perhaps, but if your API is used by hundreds of third parties, you're a little bit in trouble. Of course, you could have just used a method in the first place - but it makes the calling code more cumbersome, and less readable.

It would just be nicer and more flexible if C++ allowed you to do this:

property int pupil_count() { return pupil_count_; }

6. Tuple/Pair support in the range-for loop

C++11 brought us range-based for loops which allow you to iterate any container with begin()/end() methods.

for(auto& something: my_vector) {}

Unfortunately, in the case of dictionary style classes like map and unordered_map, 'something' ends up as a std::pair. Meaning you have to do this:

for(auto & something: my_map) {
    auto key = something.first;
    auto value = something.second;

That quickly gets tiresome, it would be nice if you could do something like this:

for(auto key, auto value: my_map) {
      // yay!

Friday 27 February 2015

Snippet: Class Properties in C++11

Update: I discovered several bugs with the initial implementation, so I've updated this post with a more complete solution.

Properties as seen in Python and C# are a pretty nice language feature, they allow you to have the syntax of accessing a public member variable, yet have that access still go through a getter or setter function.

I have to say I'm not totally sold on RW properties (I feel that if you are intending to change the state of a class, you should be forced to make it clear by calling a setter function) but I like RO properties, they remove the need to call a function to just access a value, and still allow you to check pre/post conditions or log access.

C++ doesn't have properties, if you Google around there's a general consensus that they don't fit the language well. However, I got a bit tired of my game engine's API having an excessive number of function calls. Here's an example:

TextureID rtt = window().new_texture(false);
window().mesh(rect_mesh)->set_texture_on_material(0, rtt);

It would be far nicer to be able to write:

TextureID rtt = window->new_texture(false);
window->mesh(rect_mesh)->set_texture_on_material(0, rtt);

... and still have the control that a getter function provides. So I wrote a Property template class, it looks like this..

template<typename Container, typename T>
class Property {
    Property(Container* _this, T Container::* member):
        getter_([member](Container* self) -> T& {
            return self->*member;
        }) {

    Property(Container* _this, T* Container::* member):
        getter_([member](Container* self) -> T& {
            return *(self->*member);
        }) {

    Property(Container* _this, std::shared_ptr<T> Container::* member):
        getter_([member](Container* self) -> T& {
            return *(self->*member);
        }) {

    Property(Container* _this, std::function<T& (Container*)> getter):
        getter_(getter) {


     *  We can't allow copy construction, because 'this_' will never be initialized
    Property(const Property& rhs) = delete;

    Property operator=(const Property& rhs) {
        assert(this_); //Make sure that this_ was initialized

        getter_ = rhs.getter_;
        // Intentionally don't transfer 'this_'

    inline operator T&() const { return getter_(this_); }
    inline T* operator->() const { return &getter_(this_); }
    Container* this_ = nullptr;
    std::function<T& (Container*)> getter_;

Container is the class you are adding the property to, T is the type of the property.

Thanks to C++11 allowing initialization of members in the body of the class, and the awesome syntax of lambdas, we can use the Property template like so:

Property<WindowBase, Watcher> watcher = {
    this, [](const WindowBase* self) -> Watcher& {
        if(!self->watcher_) {
            throw LogicError("Watcher has not been initialized");
        } else {
            return *self->watcher_.get();

Or, if you just want to wrap a member variable:

Property<WindowBase, Console> console = { this, &WindowBase::console_ };

Unfortunately we can't override the '.' operator in C++ so we can't get the exact same syntax as C# or Python, but the -> operator is good enough (and as a lot of my engine's accessors return pointers, it actually made things more consistent).