HaloRank.com

[Kornman00]

Instead of directly integrating with server apps, a better route to take would have probably been creating an API, either binary or data based (but not strictly 'database'd). Processing the chat log is one example of what I mean by making something data based. After all, the one thing that stays the same between all server apps is Halo. That should be your only concern. Then you could have your own service running alongside the dedi for handling the API submissions to the site itself.

I don't really see what the problem is with operating over HTTP. Hell, it's what the Halo engine has been using for stats/file transfers since H2. Unless you're performing some kind of heartbeat, meaning all servers are performing communications every X period instead of just at the post game.

[Amit]

Didn't he mention in the video that it sends statistics every 30 seconds or so to have a mostly "real-time" update system? Otherwise, the Games tab of the site wouldn't work properly for showing stats of the games that are still in progress.

Unless I'm just confusing all of this.

[Btcc22]

No, that's right. It sends stats at 30s intervals.

[Sean Aero]

Instead of directly integrating with server apps, a better route to take would have probably been creating an API, either binary or data based (but not strictly 'database'd). Processing the chat log is one example of what I mean by making something data based. After all, the one thing that stays the same between all server apps is Halo. That should be your only concern. Then you could have your own service running alongside the dedi for handling the API submissions to the site itself.

I totally agree here and it is one of the lessons learned from undertaking this project.
When I find a bit more time I'll write a post regarding the lessons I learned from undertaking such a project and the recommendation I would give to anyone else willing to attempt it.

I don't really see what the problem is with operating over HTTP. Hell, it's what the Halo engine has been using for stats/file transfers since H2. Unless you're performing some kind of heartbeat, meaning all servers are performing communications every X period instead of just at the post game.

We make use of an heartbeat, which beats every 10 seconds.
This heartbeat approach was done for 2 reasons, first being able to display more Live data on the website, as I'm still a huge believer of a more dynamic interactive website.
Second was being able to process and check the full game on the HaloRank server, thus making it much more challenging to boost/cheat statistics through memory edit or such.


The 30 seconds I mentioned was just to be sure that the 10 seconds, which is the maximum time from a certain event occurring until a heartbeat being send, plus the process time of the heartbeat on the HaloRank server would be fully completed.
On average once a heartbeat is received it's processed within 5 seconds on the HaloRank server.

When I have a bit more time I'll edit my first post with more information on the reason why certain choices were made as well as what could have been done better.

[sehe]

The problem is with the HTTP is that if you have 300+ servers the php/java scripts rapes the CPU. AFAIK that's why HaloRank shut down for a long time, cuz it couldn't handle so much servers. + It also gives a lot of network overheat too, while with a server sided C++ app you can can do this with 1/1000 CPU usage and about 1/10 network usage.

[Sean Aero]

AFAIK that's why HaloRank shut down for a long time, cuz it couldn't handle so much servers.

Main reason was I wrote slow SQL queries, I learned a lot about optimizing the queries since then.
Would be interesting to see what happens to the CPU of the server when the amount of server rises.
Btcc22 knows I keep a close eye on the load as it has been a large focus with this new release.

[Crenshaw]

EDIT: Woke up on the wrong side of the bed this morning, so sorry if I offended anyone.
I'll have a look at the IPs displayed when I get home, might be a small script bug.
EDIT: EDIT: Wrong IP display bug has been fixed.

Ok thank you.

[Kornman00]

Are you base encoding and/or compressing your HTTP payloads?

Also, javascript is entirely client side if I'm not mistaken. It shouldn't even factor into his server's overhead.

What's the server setup? Linux, Windows?

[urbanyoung]

The problem is with the HTTP is that if you have 300+ servers the php/java scripts rapes the CPU. AFAIK that's why HaloRank shut down for a long time, cuz it couldn't handle so much servers. + It also gives a lot of network overheat too, while with a server sided C++ app you can can do this with 1/1000 CPU usage and about 1/10 network usage.

If HTTP is good enough for millions of websites and services, it's good enough for HaloRank. I assume database queries are what contributes most to the server load, and that won't change if you use a dedicated protocol. HTTP doesn't really have that much overhead, especially if you compress it..

[Btcc22]

If HTTP is good enough for millions of websites and services, it's good enough for HaloRank. I assume database queries are what contributes most to the server load, and that won't change if you use a dedicated protocol. HTTP doesn't really have that much overhead, especially if you compress it..

It's not HTTP/network traffic that's a problem, it's invoking PHP to handle the data that's the most expensive part of the process.

Another problem is that HTTP's inherently stateless which means the database has to be used to provide persistence. That comes at the price of executing some fairly expensive queries to keep track of the data properly.