Out of Game Market Feeds In Odyssey 1.1

Avatar
My heart fair bleeds for the market-bots inconvenienced by this. Get fucked, space plutocrats!
Avatar
This's been inconvenient for me as I rely on eve-central prices in my manufacturing worksheets, and contribute via Contribtastic. Hopefully this gets fixed soon, so I can get back to making things with solid data backing profitability, rather than my less-than-optimal back of the napkin math.
Avatar
I thought CCP just waits for their CREST API to come of age? It took quite a while already but oh well, ~development resources~
Avatar
EVEmon rolled out the update to Contribtastic tonight, shortly before posting
Avatar
market-bots and bot aspirants.
Avatar
My market bot manually loads data from screen and still working! lol if ccp doesn't fix it i will take all the profits before new bots come out.
Avatar
Ha ha ccp... Love you dearly... But the shanagins needs to stop
Avatar
Oh dear lord what have they done!!! So like, I will have to physically fly to a place and see what the price is for myself? Are you insane CCP?! It's supposed to be 'Press button, receive bacon!' What next? Dress myself?This is madness!
Avatar
This is to industrialists as manually loading each of your bullets in your gun is to pvp.
Avatar
This is not about knowing prices remotely. Anyone with an alt can do it.It's about using the data and processing it. You can't extract data at all from the game client, only view it manually. And for most industrialists or traders, it's not enough to do anything.It's like if you wanted to design a pvp fit without any EFT-like tools.
Avatar
This only makes the universe smaller. Power projection over 5 regions in a matter of minutes, and market data from all across EVE. You guys only want the game to be easy ... well it should not be !
Avatar
GET FUCKED SPREADSHEET NERDS
Avatar
more like making the bullets yourself, then loading them.
Avatar
I hate json-based API's.
Avatar
Wrong.It's like if you wanted to design a PvP fit without any EFT-like tool AND every module's stats and fitting requirements change on an hour-by-hour basis, up until the moment you fit it to a hull.
Avatar
Sure there is. And Opera is poised to take a relevant market share of browsers
Avatar
You can see active scrapers here if your interested http://map.eve-emdr.com
Avatar
LibreOffice is coming at you OpenOffice
Avatar
Play harder.
Avatar
EVE would be a lot more fun if the universe seemed much larger, if travel would not be soo easy. And the market would actually a lot more interesting if nobody could ahve all market data from anywhere in a split of a second
Avatar
This might explain why Red Frog's contract queue seemed to drop significantly after Ody 1.1 was deployed... Far fewer people needing stuff moved.
Avatar
Can someone explain what is hilarious about the screenshot? I don't get the implied joke.
Avatar
There's no "hostile and gray environment". The environment doesn't exist. You're doing something that wasn't meant to be done and I don't understand why people have such a hard time grasping the concept of "Unsupported". All of these applications are doing something the developers don't intend for them to do. You'd sound a lot more reasonable if instead of insisting something you're not supposed to be doing is a right with politician-style phraseology you made simple requests for these features like reasonable human beings.
Avatar
this is just an extension of the sandbox nature of the game. there are countless mechanics and playstyles that were "unsupported" and not intended by the devs. hell, the CEO has stated numerous times that this is what makes EVE so great, players using the game in novel and "unsupported" ways.
Avatar
You have to be really deluded to believe that includes manipulation of the client that delivers the sandbox. What you do IN THE GAME is way different from what is done to the client out of it. There's absolutely no basis for comparison and that's a pretty terrible argument. It also doesn't obligate the developers to having to hamstring their development cycle due to your poorly coded hack widget ceasing to function like so many people have been crying about.
Avatar
Thousands of players did the wrong thing to get them. Nobody is disagreeing with you that CCP should use the technology they developed in CREST to expose this. The problem is a legacy issue from a tiny company who were working on something bigger than them many years ago. Today we build interfaces for these things as we've learned that letting people monkey around with the client is a terrible idea. That's what CREST is.I agree that it would be wise for CCP to expose this data properly from a business sense. Where I disagree is that I know they're doing the wrong thing turning a blind eye to their client being reversed as it stands today from a business perspective and I think that if they don't get into the business of enforcing their legal contracts they can find themselves in hot water in the future for no gain.Again... nobody's turning it off. Nobody's saying it should be turned off now, though... I do somewhat feel that way I also think players shouldn't be punished for CCP's past transgressions, WHICH IS WHY I DIDN'T PUNISH THEM. I also feel though that if you're going to be getting this data you should be getting it the right way and that CCP is under no obligation and shouldn't plan their development around third party tools that were created by violating the EULA and using the client in a way it is not meant to be used.Philosophy has nothing to do with it. It's a purely legal argument which is why you even heard their lawyer say the EULA would need to be rewritten to allow this activity. Opinion has absolutely nothing whatsoever to do with it.
Avatar
"Everything not compulsory is forbidden.Everything not forbidden is compulsory." -- The Ants, in T.H. White's The Once and Future King
Avatar
So let me get this straight, Eve Online is a game where the communications infrastructure exists to beam YOUR ENTIRE CONSCIOUSNESS to another clone in any known system, but something as hum-drum as a Bloomberg terminal does not exist?You people are all crazy.

Official response from CCP here

One unannounced and unintended bug was released with Odyssey 1.1. Market cache scrapers, such as Contribtastic and Reverence, were broken when Odyssey 1.1 revamped the way the client generates cache files. This, in turn, broke the tools that feed EMDR, EVE-Central, EVE-Marketdata, goonmetrics, and countless marketeers and industrialists, resulting in a complete shutdown of exported price data for 3rd parties to process outside of the game.  

As of this writing, feeds are still broken, but player devs are working hard to remedy the situation. Contribtastic, the scraper packaged with EVEmon, and primary feed to EMDR, has a player-contributed patch that is currently being debugged and rolled out (UPDATE: EVEmon updated). Reverence, the more complete toolset, will be longer to update since its developer, Entity, is on holiday.  

How Market Feeds Work Today

[]

Please refer to the official EMDR page for the technical breakdown.

[]

Nearly all out of game market feeds are enabled by EMDR. This system, designed and maintained by player-devs, is essentially an echoing service that helps distribute updates to all subscribers. As services like Contribtastic generate updates and EVEmon pushes that data out, EMDR sends that data to all listeners, but does not save that information for later. This means that if an update comes, and your service isn’t listening, you miss the update completely. Then these updates are processed by other aggregators such as EVE-Central and EVE-Marketdata and served through their own APIs so other player-devs can get high quality information quickly and easily. Also, EMDR data can be used directly by your own app if you are so inclined, like DRK Industrial Tracker. Though the above should make the problem pretty clear, if the scrapers can’t deliver data, all that infrastructure is for naught.

With Contribtastic being packaged with EVEmon, this means a large number of players have their market queries automatically updated (if they have enabled the feature). Couple that with tools like goonmetrics market scanner, enabled by the in-game browser API, and it becomes very easy to get snapshots of EVE market data at often enough frequencies to make out-of-game estimates incredibly accurate. Without high quality price data, market participants tend to lose their minds, like what happened to NASDAQ last month.

Isn’t Cache Scraping Banned by the EULA?

Technically yes. After the EVE-Uni market bot scandal earlier this year, CCP was forced to make some clarifications to their EULA policies in regard to cache scraping. Unfortunately, the ruling was still gray and vague, resulting in a “It’s wrong, but we won’t prosecute [unless we do]”. 

But there’s a problem: out of game market feeds rely solely on cache scraping, and CCP has no desire to put their developer resources on a new API feed, especially with the expected service traffic required to keep that pipeline moving. So, the player-dev sphere is left in a Mexican standoff with CCP to get data that enables the entire market to function smoothly. Without those feeds, heavy industry grinds to a halt as player tools break down, sometimes leading to hilarious outcomes:

[]

CCP Should Fix It!

This is a not uncommon call. The current system leaves player-devs working in a hostile and gray environment, trying to pick up the slack CCP refuses to fix. Also, market and industry complaints have gone long unanswered due to a mix of “don’t fix what isn’t broken” and the fact those features lack sex appeal. It’s not like we will see “New Market API” featured in a dubstep trailer.

The immediate problem is CCP rolled out Odyssey 1.1 with this cache update undocumented and unannounced, leaving player-devs flat footed and scrambling, many still out on holiday. The new cache files changed the namespace, but didn’t change the data. Thankfully, this means player-devs need to only update the keywords required to traverse the files. The longer term problem is players generate and demand a firehose of market data, and the current API is notoriously hard to work with in large volume applications.  Lastly, there is a legitimate desire by CCP to avoid providing instantaneous and perfect data. With too powerful a feed, they only serve to enable more bots and power-players edging out the more casual crowd.

Steve Ronuken has proposed a pretty reasonable fix for CCP to implement: Have CCP host an EMDR (or similar) endpoint.  By hooking into existing infrastructure, and designing a feed to meet their game design goals, both server and client parties can be satisfied.  Also, with a EULA update rumored in "the fall", the player-dev environment stands to get grayer.

Regardless of how CCP acts, this tenuous relationship around out of game market feeds shows no signs of becoming clearer. CCP will continue to take the position that cache scraping is wrong, and take steps to shore up client security. On the other hand, player-devs will continue to try and provide the data feeds that CCP refuses to and players demand. Though CCP Stillman recently made a public apology about breaking the feeds, it doesn’t seem that a sanctioned API feed is on the foreseeable horizon.

For those interested in sources or more information on this topic in particular, I have furnished more review on my own blog.

Electrical engineer by day, hard core industrial player, 10th level black belt with a spreadsheet, and aspiring app developer by night. Can be found in the #tweetfleet as @HLIBindustry