One unannounced and unintended bug was released with Odyssey 1.1. Market cache scrapers, such as Contribtastic and Reverence, were broken when Odyssey 1.1 revamped the way the client generates cache files. This, in turn, broke the tools that feed EMDR, EVE-Central, EVE-Marketdata, goonmetrics, and countless marketeers and industrialists, resulting in a complete shutdown of exported price data for 3rd parties to process outside of the game.
As of this writing, feeds are still broken, but player devs are working hard to remedy the situation. Contribtastic, the scraper packaged with EVEmon, and primary feed to EMDR, has a player-contributed patch that is currently being debugged and rolled out (UPDATE: EVEmon updated). Reverence, the more complete toolset, will be longer to update since its developer, Entity, is on holiday.
How Market Feeds Work Today
Nearly all out of game market feeds are enabled by EMDR. This system, designed and maintained by player-devs, is essentially an echoing service that helps distribute updates to all subscribers. As services like Contribtastic generate updates and EVEmon pushes that data out, EMDR sends that data to all listeners, but does not save that information for later. This means that if an update comes, and your service isn’t listening, you miss the update completely. Then these updates are processed by other aggregators such as EVE-Central and EVE-Marketdata and served through their own APIs so other player-devs can get high quality information quickly and easily. Also, EMDR data can be used directly by your own app if you are so inclined, like DRK Industrial Tracker. Though the above should make the problem pretty clear, if the scrapers can’t deliver data, all that infrastructure is for naught.
With Contribtastic being packaged with EVEmon, this means a large number of players have their market queries automatically updated (if they have enabled the feature). Couple that with tools like goonmetrics market scanner, enabled by the in-game browser API, and it becomes very easy to get snapshots of EVE market data at often enough frequencies to make out-of-game estimates incredibly accurate. Without high quality price data, market participants tend to lose their minds, like what happened to NASDAQ last month.
Isn’t Cache Scraping Banned by the EULA?
Technically yes. After the EVE-Uni market bot scandal earlier this year, CCP was forced to make some clarifications to their EULA policies in regard to cache scraping. Unfortunately, the ruling was still gray and vague, resulting in a “It’s wrong, but we won’t prosecute [unless we do]”.
But there’s a problem: out of game market feeds rely solely on cache scraping, and CCP has no desire to put their developer resources on a new API feed, especially with the expected service traffic required to keep that pipeline moving. So, the player-dev sphere is left in a Mexican standoff with CCP to get data that enables the entire market to function smoothly. Without those feeds, heavy industry grinds to a halt as player tools break down, sometimes leading to hilarious outcomes:
CCP Should Fix It!
This is a not uncommon call. The current system leaves player-devs working in a hostile and gray environment, trying to pick up the slack CCP refuses to fix. Also, market and industry complaints have gone long unanswered due to a mix of “don’t fix what isn’t broken” and the fact those features lack sex appeal. It’s not like we will see “New Market API” featured in a dubstep trailer.
The immediate problem is CCP rolled out Odyssey 1.1 with this cache update undocumented and unannounced, leaving player-devs flat footed and scrambling, many still out on holiday. The new cache files changed the namespace, but didn’t change the data. Thankfully, this means player-devs need to only update the keywords required to traverse the files. The longer term problem is players generate and demand a firehose of market data, and the current API is notoriously hard to work with in large volume applications. Lastly, there is a legitimate desire by CCP to avoid providing instantaneous and perfect data. With too powerful a feed, they only serve to enable more bots and power-players edging out the more casual crowd.
Steve Ronuken has proposed a pretty reasonable fix for CCP to implement: Have CCP host an EMDR (or similar) endpoint. By hooking into existing infrastructure, and designing a feed to meet their game design goals, both server and client parties can be satisfied. Also, with a EULA update rumored in "the fall", the player-dev environment stands to get grayer.
Regardless of how CCP acts, this tenuous relationship around out of game market feeds shows no signs of becoming clearer. CCP will continue to take the position that cache scraping is wrong, and take steps to shore up client security. On the other hand, player-devs will continue to try and provide the data feeds that CCP refuses to and players demand. Though CCP Stillman recently made a public apology about breaking the feeds, it doesn’t seem that a sanctioned API feed is on the foreseeable horizon.
For those interested in sources or more information on this topic in particular, I have furnished more review on my own blog.