While ADSBexchange can push raw data files to customer buckets on Amazon S3, Azure, Google, SFTP, etc., some customers prefer to download the data themselves due to security restrictions or storage space issues.
For those customers, we offer the ability to download data files via an Amazon S3-compatible API (Cloudflare R2). Using an S3 compatible API is a much more efficient way to deal with thousands (or millions) of files than traditional transport methods like HTTP, FTP, SFTP, etc.
Things to know:
- Your credentials will include an “access_key_id” and an “access_key_secret”.
- The endpoint URL is: https://6ff2cd7dae70306649e2c1e1500e2e0a.r2.cloudflarestorage.com/
There are many S3 tools that can be used to access the data. For example, aws-cli or rclone.
Full documentation on rclone can be found at rclone.org. Below are some quick examples. In order to configure the authentication keys in rclone, add the following to the configuration file at ~/.config/rclone/rclone.conf:
[cf-adsbx]
type = s3
provider = Cloudflare
env_auth = false
access_key_id = your_access_key_id
secret_access_key = your_access_key_secret
region = auto
endpoint = https://6ff2cd7dae70306649e2c1e1500e2e0a.r2.cloudflarestorage.com
acl = private
Depending on your contracted level of access, you may have access to any of these buckets:
adsbx-recent-hires-traces
adsbx-recent-readsb-hist
adsbx-recent-traces
Based on the rclone documentation, for example, you can list files in one of the buckets:
[root@yourhost]# rclone lsd cf-adsbx:adsbx-recent-readsb-hist
0 2000-01-01 00:00:00 -1 2023
0 2000-01-01 00:00:00 -1 2024
[root@yourhost]# rclone lsd cf-adsbx:adsbx-recent-readsb-hist
0 2000-01-01 00:00:00 -1 2023
0 2000-01-01 00:00:00 -1 2024
[root@yourhost]# rclone lsd cf-adsbx:adsbx-recent-readsb-hist/2024
0 2000-01-01 00:00:00 -1 01
[root@yourhost]# rclone lsd cf-adsbx:adsbx-recent-readsb-hist/2024/01
0 2000-01-01 00:00:00 -1 01
0 2000-01-01 00:00:00 -1 02
0 2000-01-01 00:00:00 -1 03
0 2000-01-01 00:00:00 -1 04
0 2000-01-01 00:00:00 -1 05
0 2000-01-01 00:00:00 -1 06
0 2000-01-01 00:00:00 -1 07
0 2000-01-01 00:00:00 -1 08
0 2000-01-01 00:00:00 -1 09
0 2000-01-01 00:00:00 -1 10
0 2000-01-01 00:00:00 -1 11
0 2000-01-01 00:00:00 -1 12
0 2000-01-01 00:00:00 -1 13
0 2000-01-01 00:00:00 -1 14
0 2000-01-01 00:00:00 -1 15
[root@yourhost]#In order to copy the “readsb-hist” format files to your local filesystem, execute the following command. Note that “transfers” and “checkers” can be tuned to your environment and network connectivity. “transfers” is the number of simultaneous file transfers. “checkers” should be equal or greater than transfers.
[root@yourhost]# rclone copy cf-adsbx:adsbx-recent-readsb-hist/2024/01/25 /yourlocaldrive/yourfolder --transfers 100 --checkers 100 -P --stats-one-line --stats-unit bits --use-mmap --fast-list
Many more options can be found at https://rclone.org