Thanks, Amazon…

We’ve been using for years (since 2011 I think), and I can count the number of times they’ve had downtime on one hand…ish.

But today, the day after our alpha starts… naturally we get this:

Keith notes: “I’m particularly fond of We have now repaired the ability to update the service health dashboard.” From the AWS side.

Not sure how that’s going to impact our ability to get a new build out today; we’re still working away at things, but we can’t check in code or share it between us as of about 3 hours ago, and this outage is lasting longer than usual.

Working on icon colors and flicker.

Cinth has been complaining about the neon signs that are the main icon colors for some time now, and many others have also brought it up.  Just haven’t gotten around to it yet, until today, since he kind of prompted me to do some more work with it.

Ultimately he wound up making a goodly selection of new things down in the bottom rows there (the topmost row is mine, the rest are his).

These are easily moddable, by the way, so technically Cinth made the “first mod,” although that’s dubious since it’s just part of the main game now since he’s directly helping create the game… but he wanted to note that, heh.

Anyway, the topmost row is actually after a whole lot of work on my end, working on completely new shaders today, too.  So that row is a lot better than things were before.

That Stupid Flicker!

Problem is, we’re still getting an absolutely awful flicker at times with the icons in general.  The problem overall is z-fighting because of floating point precision issues on your GPU at far distances despite us scaling these things up.  The glow exacerbates that.

What I’m going to wind up doing is recoding these things so that they don’t exist in the same space, but instead so that they exist in a layered camera that uses fake distance, and uses no rotation whatsoever.  By faking the distance, I can minimize chances for floating point errors, and by avoiding the rotation I avoid any form of rotation-based strangeness.

This will mean that I need to translate from the target point of the icon in world space into screen space, then back over to “other camera world space, with fake distance applied,” but that’s a pretty cheap calculation and I don’t have to do it every frame.  In the end, it’s probably cheaper than some of the math that I have to do now in order to make them billboard, and it will solve the flicker.

Frustratingly, I have bigger fish to fry right now.  So I’m not sure if I’ll get to that this week.  But I wanted to let folks know I figured it out, at least.

Integrating the Forums and the Wiki

Hello! Its story time!

A long time ago in a galaxy far, far away…. Wait, that isn’t how this starts. A couple of years ago, I joined the Arcen team. Originally, I was just for managing the wiki. Apparently, Chris liked my talents with website administration, and I ended up being the official Server/Website guy. As a result of me originally being here for the wiki, it holds a special place for me. I always look for ways to improve it and make it a better experience that more people are going to use. Sadly, most people were put off by the need to register on there as well as the forums and Mantis. I mean, I get it, but it still sucked since the wiki was falling into disrepair. So, I came up with an idea! What if I could combine the login for the forums and the wiki? Turns out, a couple of years ago, that wasn’t really feasible. No one had made a bridge for SMF and Mediawiki that worked with our version of the Wiki.

Fast forward a couple of years. AI War 2 is an actual thing (and looks amazing). This inspired me to get a bridge between the forums and the wiki. After all, I wanted AI War 2’s wiki to be a helpful and up-to-date tool for everyone. Then, I saw it! It was practically glowing and begging for me to implement. I grabbed the bridge and set to work on making it work with our versions of Mediawiki and SMF.

In order to get it up and running, I made a fresh install of Mediawiki in our dev folder ( for anyone who wants to know and see what all is going on in my mind at a given time. Usually there is only one project in there at a time, but sometimes there are more). With this fresh copy, I set the database up to be shared with the forums, to make communication easier. I then let Mediawiki do the heavy lifting and create all the databases and all that jazz. Now, I have a functional copy of Mediawiki that can, in theory, communicate with the forums. At this point, I crossed my fingers and hoped that the bridge would work. To my surprise, it actually ended up working (Though it was a bit wonky)!

But wait! Our copy of Mediawiki isn’t fresh out of the box. I knew I had to somehow test if it would work with our copy . So, I went ahead and copied over all the needed folders and files. And…. It failed. It gave me a giant 500 Error code, which means that the server is looking at the install and going WTF (What the frack, don’t look at me like that) is that? So, I tried deleting the bridge code to see if the transfer was working, or if that was what was being screwy. Then, it worked! Guess we know what the problem was. To troubleshoot this, I tried enabling bits of the code at a time. Eventually, I found the culprit. Turns out, if you forget the file that the bridge calls for, it breaks everything. Note to self: MAKE SURE YOU ACTUALLY HAVE THE FILES YOU NEED! I added the file and bam! It worked like a charm.

Now, we have to break it from the users point of view. So, I spend a good hour or so just toying with it trying to get an error. Nada. Sweet, that means its ready to go.

Moving an entire directory and database while keeping everything live would be too much, I decided. So, I locked the database down, then copied it over to SMF’s database and gave it a fancy prefix. Now it can talk to SMF, and has the most up to date file information. I test the dev build, and it works still. On to the next stage. I pull up the config files from the production wiki and the dev wiki. I merge them (Manually of course, it wouldn’t work to just append them). I test it. Frack. It just broke. What’s the problem now? I delete the bridge code. It works again. Okay, so the bridge is breaking it. But I want the bridge to work, so that’s unacceptable. I check to make sure the files are all there. They are. I check that I copied it over correctly. I did. Then it hits me. I didn’t change the relative directory field for the bridge code! I change that to point to the right place, and presto, we have a fully functioning wiki with the bridge operational.

Now, the wiki with the bridge is live for the world to play with. I see a small group join, including some guy named Keith Lamothe. Turns out, he thinks he needs permissions. So, I consult with my fellow AI, and we come to the conclusion that he can help make the AI smarter, faster, stronger, and more killy… I mean cuddly. We give him the permissions. As it happens, he is also an AI. I combine the server’s AI with him, and WOW it became a deadly killing machine fast.

Moral of the story: Keith is a force to be reckoned with when it comes to making !!FUN!! toys. At this point, Cinth joins. My nice little code to give anyone with permissions works on him. No idea why it didn’t work on Keith, but whatever. At this point, I look back on it and smile, content that I finally, after years of work, managed to combine the log in systems.

If any of you break it, I swear I will make you enemy #1 of the AI Empire. 😉

Okay… what!? OSX not only works, the performance is ridiculous so far.

Last time I fired up AI War 2 on my Mac, which is a pretty outdated machine with a really really bad GPU, the results were pretty bad.  I wasn’t really surprised, because I was running it with all the settings turned up, and the music was throwing errors a bunch, and I knew it could get a lot better with some polish.  An old intel 4000 integrated GPU is not exactly at the middle end of the market anymore, even.

Just retested the latest build, which has the music swapped over to a different playback system, and which has a metric ton of performance optimizations from myself and mainly from Keith (the entire game simulation runs on a secondary thread, now).

Results? 300-some odd ships in the very opening battle, with all the settings turned up to maximum still, and it was buttery-smooth.  I have no idea what the framerate was, but I feel confident saying it was north of 60.

I just… whaaat?  That’s… not what I expected.  It’s fantastic, but it’s also a big surprise.

I’m not going to run out immediately and lower the minimum stated system requirements expectations, but that is certainly super duper positive as an indication thus far.

Reminds Me Of Something…

As kind of a fun aside, back when I was coding AI War Classic in the early days I was hoping to get 10k ships in a game, and was pretty sure I could do it.  The maximum any other RTS had at the time was about 1000, and it was super choppy and laggy with that many units in the other titles.  I felt like I could bump that up by a 10x partly thanks to going 2D, and partly just due to coding practices.

By the time the first public builds were around for AI War Classic, the typically number of units in the game was more like 30k, and it would start to chug around 45k units.  Within a year after that, 50k was more typical and it would start to chug at 75k units.

More Generalized Thoughts on Optimization

Sometimes it’s just really surprising how things can go when you throw everything and the kitchen sink at optimization.  It’s kinda-sorta working, it’s getting along okay, and then suddenly you pass this critical mass and whoosh the performance jumps by an order of magnitude.

I’m not remotely ready to say that’s happened here, yet, though.  The simulation is not remotely finished for the game, as there’s still a lot more AI to build out, and lots more ship functions.  The largest battles still have only involved just a few thousand ships at a time for me, whereas in classic sometimes north of 10k ships in one fight would happen.

So I want to see what happens with all those things.  Right now early indications are ridiculously, surprisingly good; but some monkey wrench could still very well appear between now and early access that makes me say “yeah, the minimum system requirements for a truly pleasant experience should still be more than a lousy intel 4000.”  Then again — maybe not. 🙂

Oh, One Last Thing

In AI War Classic, the simulation speed was locked to the framerate on the slowest computer in a multiplayer game, or to your framerate in solo play.  Kinda common for RTS titles, though not all of them.

In AI War 2, your visible framerate is completely unrelated to simulation speed.  We’re actually running the simulation speed at 10 cycles per second no matter what, which gives us a lot of muscle per cycle on a secondary group of threads.  It doesn’t need to be any more fine-grained than that.

Then for input and actual visual display, of course it can run at much higher framerates, basically up to whatever your hardware can handle.  If you’re running at a buttery-smooth 200fps and everything is just peachy, and I’m having more trouble on my machine at a hard-won 30fps or something, the simulation won’t slow down for either of us.

I’m a pretty happy guy about now. 🙂

A fantastic conversation.

Just general forum humor.

BadgerBadger: What does “lerping shots out of ships” mean? I know larping, not lerping.

Keith: Please, please don’t let the shots larp.

Cyborg: The AI will now say, “Lightning bolt!” before each attack.

Chris: “I’m attacking the darkness!”

Keith: 20,000 MkV Gazebo Guardians to Murdoch in 0:32…

Chris:  My favorite version of that skit:

Hey, do you remember the Gazebo monster we put into Valley 1 as a bit of an easter egg? Most of the Gazebos were harmless, but a tiny minority were absolutely deadly murder machines.

I completely forgot about that. 🙂

No real way to build asset bundles selectively in one unity project??

That’s pretty frustrating.  It’s not a dealbreaker, because for truly isolated things (music and sound effects, for instance), I can just make yet more unity projects.

But this means that in order to have some platform-agnostic asset bundles for music and sound separately from everything else, I now have to have FOUR unity projects for AI War 2 rather than just the two we had before.

Seems kinda like an oversight of something obvious on the Unity 3D advanced build pipeline process, but at least there’s a trivial workaround.  Spent more time on that today than I meant to, but it was legitimately puzzling.

Writing now while I wait for things to compile to their bundles for the first time, ever so slowly.

Definitely having to prioritize my time.

If you’ve looked at my todo list, you’ll see why.  There are so many things that I want do do, where I just think “oh, but I could get that done in just a couple of hours at most!”

And it’s true in each instance, but there’s a very real additive cost to all that.

I know that I can make the background starfields, which I’m growing tired of already from a visual sense, look a lot better and more stylized.  But is that more important than bugfixes?  No… sigh.

I want to get MasterAudio set up so that I can go ahead and have music ducking and so forth in place when I get to that point… but is that more important than just getting uAudio out and thus the game working on OSX properly, and then back to those bugfixes?  Again… no… sigh.

And so it goes.  Things move fast with Arcen, but there’s still just never enough hours in a day.  I’ve had a very unhealthy work/life balance for years, and in the last year or so I’ve been trying to get that more under control.

So… priorities.  Making sure I can load music in a cross-platform way and play it efficiently.  We’ll worry about audio ducking later when we’d even be doing that at all.

Among other things that will just have to wait.  At least with the todo list, you alpha folks can see where my head is at and what I am valuing over what in the short and middle term.

Hopefully everything on that entire list is done within the next month, knock on wood.  I have three in which to do it, but I’d rather be done in one.

Some fun performance notes from Keith today

I don’t think he’ll mind if I share these tidbits from his victories.  All of the below is written by him:

Commit Notes

– moved main sim execution to separate thread- moved most of the “ui reading the current gamestate” stuff to that thread too

– moved the generation of sync data to that thread as well

– moved the update-visual-object stuff outside of process-sim-step; still on the main thread of course but now it’s easier for you to profile it without sim stuff cluttering up the results

– various performance improvements, some of which are fairly significant


So right now there’s some volatility in the visualization code but it’s not causing problems right now since I think it’s mostly dealing in primitive types. Will need to be dealt with later but doesn’t appear pressing.

Performance-wise the sim’s burden on the main thread is almost gone during most frames. There’s still some spikes when a lot of entities die at once, as that’s a lot of list removal work. I can actually optimize that a bit more, but for now with the sim-on-the-main-thread almost always under 2ms (and often way under that) I figured this was ok for now 🙂


Oh, I should also mention that the main thread per-frame heap alloc is also almost gone, and most of that was eliminated rather than just shifted elsewhere.

One Hour Later

I’m currently working on the other-thread heap alloc, via the deep-profiler and setting SimPlanningLoop.IsOnMainThreadForProfilingPurposes to true. It just makes everything execute sequentially, rather than farming it out.

Another Hour Later

Just nuked the heap alloc of the short term planning threads (which do most of the work) from about 800k to about 40k; I can cut about 25k more, too, just needed to go now. The remaining looks like legitimate allocation on the partition list (used for find-everything-within-range checks), and that should in theory stop once the list is big enough that all future checks use less space.

The execution thread is still doing 300k+ per go, and I’ll take a look at that later. Not sure what the long term (AI) thread is doing heap-wise, haven’t seen it in the results yet.