For commercial customers, ADSBexchange makes its data available via a “pull” method. We offer the ability to download data files via an Amazon S3-compatible API (Cloudflare R2). Using an S3 compatible API is a much more efficient way to deal with thousands (or millions) of files than traditional transport methods like HTTP, FTP, SFTP, etc.
Things to know:
- Your credentials will include an “access_key_id” and an “access_key_secret”.
- The endpoint URL is: https://6ff2cd7dae70306649e2c1e1500e2e0a.r2.cloudflarestorage.com/
There are many S3 tools that can be used to access the data. For example, aws-cli or rclone.
Full documentation on rclone can be found at rclone.org. Below are some quick examples. In order to configure the authentication keys in rclone, add the following to the configuration file at ~/.config/rclone/rclone.conf:
[cf-adsbx]
type = s3
provider = Cloudflare
env_auth = false
access_key_id = your_access_key_id
secret_access_key = your_access_key_secret
region = auto
endpoint = https://6ff2cd7dae70306649e2c1e1500e2e0a.r2.cloudflarestorage.com
acl = private
Everyone can access bucket “adsbx-sample-data” using the following credentials. This is useful for test or demo purposes:
access_key_id = 7b8d31d9d2f2d73ffb2208614db599fa
secret_access_key = 8bd955340e355418a5204e52a055005b6ca761f896abe0cccdd741638e790a76
Depending on your contracted level of access, you may have access to any of these buckets:
("recent" buckets contain last 90 days of specified data type, updated in real time)
adsbx-recent-hires-traces
adsbx-recent-readsb-hist
adsbx-recent-traces
(Historical data is stored in buckets by year, where YYYY is the year in question)
adsbx-YYYY-readsb-hist
adsbx-YYYY-traces
adsbx-YYYY-hires-traces
Based on the rclone documentation, for example, you can list files in one of the buckets:
[root@yourhost]# rclone lsd cf-adsbx:adsbx-recent-readsb-hist
0 2000-01-01 00:00:00 -1 2023
0 2000-01-01 00:00:00 -1 2024
[root@yourhost]# rclone lsd cf-adsbx:adsbx-recent-readsb-hist
0 2000-01-01 00:00:00 -1 2023
0 2000-01-01 00:00:00 -1 2024
[root@yourhost]# rclone lsd cf-adsbx:adsbx-recent-readsb-hist/2024
0 2000-01-01 00:00:00 -1 01
[root@yourhost]# rclone lsd cf-adsbx:adsbx-recent-readsb-hist/2024/01
0 2000-01-01 00:00:00 -1 01
0 2000-01-01 00:00:00 -1 02
0 2000-01-01 00:00:00 -1 03
0 2000-01-01 00:00:00 -1 04
0 2000-01-01 00:00:00 -1 05
0 2000-01-01 00:00:00 -1 06
0 2000-01-01 00:00:00 -1 07
0 2000-01-01 00:00:00 -1 08
0 2000-01-01 00:00:00 -1 09
0 2000-01-01 00:00:00 -1 10
0 2000-01-01 00:00:00 -1 11
0 2000-01-01 00:00:00 -1 12
0 2000-01-01 00:00:00 -1 13
0 2000-01-01 00:00:00 -1 14
0 2000-01-01 00:00:00 -1 15
[root@yourhost]#
In order to copy the “readsb-hist” format files to your local filesystem, execute the following command. Note that “transfers” and “checkers” can be tuned to your environment and network connectivity. “transfers” is the number of simultaneous file transfers. “checkers” should be equal or greater than transfers.
[root@yourhost]# rclone copy cf-adsbx:adsbx-recent-readsb-hist/2024/01/25 /yourlocaldrive/yourfolder --transfers 100 --checkers 100 -P --stats-one-line --stats-unit bits --use-mmap --fast-list
Many more options can be found at https://rclone.org