How to became No1 in stats and why we should avoid that

The race to be the best feeder on FA (or any else site) really doesn't make any sense,
in special way if you consider that lot of participants merge data coming from several sources, it is just stuff for suckers.
To compare results has a meaning only if related to just two stations of which hardware, place orography, etc, is well known,
otherwise is madness.

 

FA feeders number in the 10's of thousands comparing the length their dongles.

Speaking of size comparisons.

I noticed OpenSky Network is now advertising itself as a commercial data seller. I guess that explains why they were so rude when we offered to do a data exchange.

Shots fired. The student / graduate who wrote this used our public api and smashed on it, then complained the positions were 500 meters off. While complaining about the latency, he also complained that server times changed and something else - my german is Google Translate.

If he would have contacted us I could have given him a direct feed to make an actual comparison.


Whats I find amusing is Pi use dump1090, so he's comparing 'accuracy' when they all use the same source and timestamp. It's just the delay in processing that data and displaying it on a website is the factor.

Oh well.

https://twitter.com/OpenSkyNetwork/status/1047815485906214912

 

In what way is the public API less precize than a direct feed? He writes of varying server time within a very brief period of data sampling.

From the PDF: " Die Rangfolge von ADSBexchange hängt von der Genauigkeit der Zeitstempel ab. Sind diese korrekt gesetzt, könnte die Qualität sogar noch vor Opensky liegen."
This states that ADSBx would be even more precise than OpenSky, if it wasn´t for "unprecise" time stamps in the ADSBx data.

I am just interested in understanding. No urgency.

Mind you TU München is probably pretty bets pals with Bundeswher University, same city, Munich. Bundeswehr probably hates ADSBx, as the can not control it. However, they can probably much better control OpenSky…

 

OpenSky is a joke. They are selling data to governments, hedge funds, and corporations under the guise of being and opensource and open university supported project. They filter planes out if corporations pay them as a 'sponsor'. We've had organizations come to us and ask why they can't see x company corporate jet on OpenSky (which they are paying for) but ADSBx shows it.

The 'researcher' student used the public api feed, which can get overloaded and travels through 5 servers before it is load balanced across all the 30 global api servers. I believe his 'research' was done when we were doing the server conversion, he says the API didn't respond.

The entire paper is suspect, he used the public feeds and compared them to a direct opensky feed. He used timestamps from VRS without understanding what he was looking at.

There were times when ADSBx public api didn't respond, he says. Which is probably true if he was hammering it. HAProxy will block him from making too many concurrent requests in a minute.

It wasn't a fair comparison, I would have given them access to a non-public feed that would be not over loaded as the Public feed usually is.

You can compare you local Pi feed to the Public Map and it's identical, considering ADSBx Global Map is on a 5 second delay, this is pretty amazing.

There are times when things might not line up exactly, due to how VRS times out aircraft signals and the MLAT stuff is going to lag since that has to be triangulated on other servers before it's sent to globals.

So ADSBx isn't perfect, but it's better than what is out there for no cost and all thanks to the feeders.

Also the VRS public json api is meant to be a stable enough for the general public to use it and cover most applications of the data.

"This states that ADSBx would be even more precise than OpenSky, if it wasn´t for "unprecise" time stamps in the ADSBx data. "

Global UI is also VRS based, so VRS creates the timestamp. As long as map and api are VRS based, this will be the case. Each global web server will tag the data with a timestamp as it receives it.


A long time ago, ADSBx offered to do a data exchange with OpenSky. We offered to feed them data and they feed data to ADSBx. They have big gaps in coverage and we can always use more feeds.

Everyone would benefit and the community would get better data. ADSBX was rudely told to go pound sand and to go [email protected]# ourselves by the OpenSky ceo or whatever he calls himself.

OpenSky is more interested in selling the feeder data to the highest bidder and hiding behind the Universities and sponsors with basically unlimited funding.

 

Very interesting, thanks a lot for taking the time to explain in detail! Somebody should notify the researcher and the Uni on the this methodical glitch.

I think the blocking of aircraft on OpenSky would go contrary to their public statement and they should be confronted. Especially since they use public money and claim to be non-blocking, by what I read on their website. I am still awaiting an answer from OpenSky.

 

MDA, yes, but the thread actually sent me to the actual article in question. Apologies for subject changing.

 

500m isn´t so much of a problem. The average of 1000m in that paper - it´s good to know it can spike up to 3000m. Because such an offset will make a difference in perception to the observer on the ground, depending on the altitude of the aircraft in question. It´s good to know that because one can explain to others and maybe try to abstract the real position of that low flying fighter jet there…

 

Opensky is concentrating on Europe.

You'll notice that SkySquitter is a commercial operation using OpenSky Data.

http://www.skysquitter.com/en/start/

The more power to them, but they definitely misrepresent their intentions and operations.

Same with https://sero-systems.de/


====

From: OpenSky Network <[email protected]>
Sent: Tuesday, October 16, 2018 10:22 AM
Subject: OpenSky Workshop: Program Announcement

Call for Participation
_________________________________________

We are proud to announce the excellent program of year's OpenSky Workshop and we are looking forward to meeting you all for an exciting series of talks and discussions! The registration is open and available on our website.

• Date: 15/16th November 2018
• Place: Frankfurt Airport, Germany
• Website: https://workshop.opensky-network.org

Program:

Thursday, 15/11/2018
09:00 AM - 09:30 AM Welcome & Introduction
09:30 AM - 10:30 AM
Session: OpenSky: Past, Present, Future
OpenSky Development Review 2016-2018
Matthias Schäfer (SeRo Systems)
Update on OpenSky - FLARM Integration
Giorgio Tresoldi (ETH Zurich)
10:30 AM - 11:00 AM Break
11:00 AM - 12:00 AM
Session: Weather & Co
Using ADS-B and Satellite imagery to analyze weather avoidance strategies
Simon Proud (University of Oxford)
Radiosondes - New opportunities
Rui Pinheiro (Skysquitter)
12:15 AM - 01:30 PM Lunch
01:30 PM - 02:30 PM
Session: Security 1 - Location & Track Verification
ADS-B Track Verification Using Signal Reflections
Daniel Moser (ETH Zurich)
ADS-B Message Authentication Protocol Based on Location Verification
Ala' Darabseh (NYU Abu Dhabi)
02:30 PM - 03:00 PM Break
03:00 PM - 04:00 PM
Session: Security 2 - Defending against Data Injection
FDI-T: A Testing Framework for Air Traffic Control Systems Based on False Data Injection
Aymeric Cretin (Femto-ST Institute)
The Importance of Being Earnest, A Trivial (ADS-B) Comedy for Serious People
Mauro Leonardi (University of Rome "Tor Vergata")
04:00 PM - 04:30 PM Break
04:30 PM - 05:30 PM
Session: Special Topics
ADS-B Archaeology – Evolution of ADS-B adoption in Europe and future prospects
Enrico Spinelli (Eurocontrol)
Using aircraft tracking data to infer state and statistics for firefighting aircraft operations in Australia
Andrew Matthews (National Aerial Firefighting Centre Australia)
05:30 PM - 06:00 PM
OpenSky Awards
Vincent Lenders
07:30 PM - Dinner Frankfurt City

Friday, 16/11/2018
09:30 AM - 10:30 AM
Session: Understanding Air Traffic (Control)
Detecting controllers' actions on past ADS-B data
Xavier Olive (ONERA)
Understanding airline behaviours on trajectory choices through opendata
Paola Bassi (University of Trieste)
10:30 AM - 11:00 AM Break
11:00 AM - 12:00 AM
Session: Machine Learning Applications
Exploring density-based clustering for anomaly detection in air traffic data
Axel Tanner (IBM Zurich Research Lab)
Aircraft Classification through Movement Patterns
Martin Strohmeier (University of Oxford)
12:00 AM - 01:30 PM Lunch
01:30 PM - 02:30 PM
Session: Data Processing Challenges
Using a Kubernetes Cluster to generate High-Resolution Flight-Track-Images
Kai Renz (University of Applied Sciences Darmstadt)
Establishing a flight route database for Opensky Network
Jannis Lübbe (OpenSky Network)
02:30 PM - 03:00 PM
Farewell & Closing
Rui Pinheiro & Vincent Lenders

Organizers
_________________________________________

• General: Rui Pinheiro (Skysquitter)
• Program: Martin Strohmeier (University of Oxford)
• Publication/Web: Matthias Schäfer (SeRo Systems)
• Registration: Marco Meides (SeRo Systems)

 

Correct, but those people looking at the sky for a jet should have a feeder running and be relying on that feeder and not a public data api from ADSBx.

 

It seems that way and I really don't want to tell people who they should or shouldn't feed. That's up to the person feeding.

ADSBx exists because back in 2016 nobody was sharing this data back to feeders or the community. ADSBx has an open API, an open web ui, and ADSBx used to have all the data archives open to anyone.

We had to restrict it a little bit and I've told quite a few people 'No' that ask about AWS Redshift or the JSON downloads.

I don't think it is too much to ask someone to host a feeder and pay for what costs they generate for data download.

Those that could host a feeder or help us with infrastructure but refuse to do so annoy me a little bit. The response of 'oh it takes so much time and effort to get my university to approve a donation or approve a feeder unit. Can't you just give it to me for free?"

My response is always, "No, contact ADSBx when you think of something mutually beneficial."

I learned my lesson once, I gave access to the JSON to a university Phd who wanted to do some analytics or machine learning (I can't remember). I set him up a login and he downloaded 10TB of data in a day, and connected using 30 connections at a time. He doubled the AWS bill then refused to pay for the bandwidth.

I think hosting a feeder is the least someone can do if they want access to the data. It's not free to store it or archive it, or do all the stuff that is required to make all this work.

Am I wrong?


As a side note, we had to rent another server today to host 2 more consolidator ingestion servers. ADSBx is growing we're nearly to 2000+ feeders!

 

I should mention that ADSBx is doing really good things with all the feeder data.

People were able to track those jets out of Riyadh with the Saudi goon squad, we tracked a Senator in the US and caught her lying about her private plane, and ADSBx data has been used by just about every major news network and paper in the last year for various stories.

I don't think I can talk about it yet but ADSBx was the only ADS-B site willing to give data to a large multinational organization that deals with official national statistics for their member countries.

ADSBx is expanding across Africa this year, thanks to the help of a lot of organizations on the continent.

We'll just keep doing our thing and let the rest of them do their thing. I get a little fired up sometimes, but it is what it is.

 

Yeah, folks should run receivers. It´s a very hard business to actually achieve that, around here! And I am definitely on the ADSBx-mission! Two of those I convinced are tec novices. Their piaware have been put offline, possibly because of the recent Google API shutdown. Possibly by some other party. Prime location for the purpose they were actually initially set up (ADSBx MLAT coverage of ED-R 401 MVPA NE - an opaque new style military super zone, implementing various militarization concepts [Single European Sky, Flexible Use of Airspace, Military Variable Profile Area] of the EU and national forces, sprwaling like cancer above formerly tranquil natural demilitarized protected habitat zones and landscapes of north east Germany). Other receivers are down in the area, with those folks not really knowing why, not having resources to tend to those receivers in detail. So it´s some kind of a problem - people set up receivers but at a point they just fail.

 

Install Zerotier and join ADSBx network.

Have them email me if they have changed default username and password to login. I can fix all that is problematic in minutes.

https://www.zerotier.com/download.shtml

curl -s https://install.zerotier.com/ | sudo bash

sudo zerotier-cli join c7c8172af18c605f

Send output of these commands to me, james (at) adsbexchange.com

sudo zerotier-cli status
sudo zerotier-cli listnetworks