Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

mcap cli: Add support for Azure Storage Accounts and AWS S3 remote files #957

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

mcap cli: Add support for Azure Storage Accounts and AWS S3 remote files #957

wants to merge 1 commit into from

Conversation

RileyEv
Copy link

@RileyEv RileyEv commented Aug 28, 2023

Public-Facing Changes

You can now use mcap cli with Azure Storage Accounts and AWS S3 remote files, as well as GCS.

Description

Uses gocloud.dev from Google to build generic clients for Azure, AWS and Google Cloud.

Note: I am yet to test all these changes on every cloud environment

  • Verify on Azure
  • Verify on AWS S3
  • Verify on GCS

Azure Usage

TODO

AWS Usage

  • You must have an authenticated AWS session in your environment.
  • You must set the aws region of the s3 bucket, either through ~/.aws/config or AWS_REGION=...

GSC Usage

Hopefully unchanged (need to check)

@wkalt
Copy link
Contributor

wkalt commented Aug 30, 2023

Hey @RileyEv, thanks! This will be useful.

I can confirm it works on GCS:

[~/work/mcap/go/cli/mcap] ((HEAD detached at riley/mcap-cli-s3-remote-file)) $ mcap info gs://dp-byoc-inbox-wyatt/demo.mcap
library: mcap go v1.0.1
profile: ros1
messages: 1606
duration: 7.780758504s
start: 2017-03-21T19:26:20.103843113-07:00 (1490149580.103843113)
end: 2017-03-21T19:26:27.884601617-07:00 (1490149587.884601617)
compression:
        zstd: [14/14 chunks] [119.10 MiB/58.73 MiB (50.69%)] [7.55 MiB/sec]
channels:
        (0) /diagnostics              52 msgs (6.68 Hz)    : diagnostic_msgs/DiagnosticArray [ros1msg]
        (1) /image_color/compressed  234 msgs (30.07 Hz)   : sensor_msgs/CompressedImage [ros1msg]
        (2) /tf                      774 msgs (99.48 Hz)   : tf2_msgs/TFMessage [ros1msg]
        (3) /radar/points            156 msgs (20.05 Hz)   : sensor_msgs/PointCloud2 [ros1msg]
        (4) /radar/range             156 msgs (20.05 Hz)   : sensor_msgs/Range [ros1msg]
        (5) /radar/tracks            156 msgs (20.05 Hz)   : radar_driver/RadarTracks [ros1msg]
        (6) /velodyne_points          78 msgs (10.02 Hz)   : sensor_msgs/PointCloud2 [ros1msg]
attachments: 0
metadata: 0

On Azure I'm not sure I'm getting the call right but I see this:

[~/work/mcap/go/cli/mcap] ((HEAD detached at riley/mcap-cli-s3-remote-file)) $ mcap info azblob://fgdemostorage.blob.core.windows.net/lake/gps__fix_7f6e605ad1e52d05162190ff17be80b6/device_id=dev_BDIlonk6XMUtsPUJ/date=20210322/8568660e2377d5621c17da43217c39bf
Failed to read file azblob://fgdemostorage.blob.core.windows.net/lake/gps__fix_7f6e605ad1e52d05162190ff17be80b6/device_id=dev_BDIlonk6XMUtsPUJ/date=20210322/8568660e2377d5621c17da43217c39bf: azureblob: Options.AccountName is required

I am not sure how it wants me to give it an account name. I do have an authenticated azure CLI session so my expectation would be I need no other config/params to do this.

@wkalt
Copy link
Contributor

wkalt commented Aug 30, 2023 •
edited

on AWS I see this - the object definitely does exist:

[~/work/mcap/go/cli/mcap] ((HEAD detached at riley/mcap-cli-s3-remote-file)) $ mcap info s3://foxglove-wyatt-test-bucket/gps.mcap
Failed to read file s3://foxglove-wyatt-test-bucket/gps.mcap: failed to build read seek closer: blob (key "gps.mcap") (code=NotFound): operation error S3: GetObject, https response error StatusCode: 301, RequestID: NV48YNWKN5EDMWHN, HostID: xJyRtsJqDtovu+5FKWpQfctSuVFRHST+JS3sFSMyVG/aTUVeL/QZJo3AA5Hk1uUXj4H1YElcxrc=, api error PermanentRedirect: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.

is that the call you'd expect to work?

edit:

[~/work/mcap/go/cli/mcap] (main) $ aws s3 ls s3://foxglove-wyatt-test-bucket/gps.mcap
2023-08-29 18:07:28    1454223 gps.mcap

@RileyEv
Copy link
Author

RileyEv commented Aug 30, 2023

aws

on AWS I see this - the object definitely does exist:

I think that means that something is not quite right with the authentication. I get the same message when trying to use your bucket which I expect I wouldn't have access to.

mcap/go/cli/mcap on  mcap-cli-s3-remote-file [?] via 🐹 v1.21.0 took 2s
➜ go run . info s3://foxglove-wyatt-test-bucket/gps.mcap
Failed to read file s3://foxglove-wyatt-test-bucket/gps.mcap: failed to build read seek closer: blob (key "gps.mcap") (code=NotFound): operation error S3: GetObject, https response error StatusCode: 301, RequestID: 8508VF35EXXJ6PXC, HostID: EqRD7aMLM5DFPJv77sDPoPIO9AtjRnCWUXj89dZb1NgcTtG1nNM+REKIHdhdGkbSRZJgyJxd5FQ=, api error PermanentRedirect: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.
exit status 1

How have you setup credentials? I've tested it with credentials from IAM Identity Center (SSO) and another auth method using aws-okta and both work for me.

mcap/go/cli/mcap on  mcap-cli-s3-remote-file [?] via 🐹 v1.21.0
➜ aws s3 ls s3://mcap-testing/
2023-08-30 21:03:04  512262937 nuscenes.mcap

mcap/go/cli/mcap on  mcap-cli-s3-remote-file [?] via 🐹 v1.21.0
➜ go run . info s3://mcap-testing/nuscenes.mcap
library: nuscenes2mcap
profile:
messages: 33886
duration: 19.192673s
start: 2018-07-24T04:28:47.604844+01:00 (1532402927.604844000)
end: 2018-07-24T04:29:06.797517+01:00 (1532402946.797517000)
compression:
	lz4: [638/638 chunks] [741.37 MiB/487.76 MiB (34.21%)] [25.41 MiB/sec]
channels:
  	(1) /imu                                      1898 msgs (98.89 Hz)     : IMU [jsonschema]
  	(2) /odom                                      937 msgs (48.82 Hz)     : Pose [jsonschema]
  	(3) /map                                         1 msgs (0.05 Hz)      : foxglove.Grid [protobuf]
  	(4) /semantic_map                                1 msgs (0.05 Hz)      : foxglove.SceneUpdate [protobuf]
  	(5) /tf                                       3002 msgs (156.41 Hz)    : foxglove.FrameTransform [protobuf]
  	(6) /drivable_area                              39 msgs (2.03 Hz)      : foxglove.Grid [protobuf]
  	(7) /RADAR_FRONT                               247 msgs (12.87 Hz)     : foxglove.PointCloud [protobuf]
  	(8) /RADAR_FRONT_LEFT                          262 msgs (13.65 Hz)     : foxglove.PointCloud [protobuf]
  	(9) /RADAR_FRONT_RIGHT                         258 msgs (13.44 Hz)     : foxglove.PointCloud [protobuf]
  	(10) /RADAR_BACK_LEFT                          253 msgs (13.18 Hz)     : foxglove.PointCloud [protobuf]
  	(11) /RADAR_BACK_RIGHT                         243 msgs (12.66 Hz)     : foxglove.PointCloud [protobuf]
  	(12) /LIDAR_TOP                                382 msgs (19.90 Hz)     : foxglove.PointCloud [protobuf]
  	(13) /CAM_FRONT/image_rect_compressed          224 msgs (11.67 Hz)     : foxglove.CompressedImage [protobuf]
  	(14) /CAM_FRONT/camera_info                    224 msgs (11.67 Hz)     : foxglove.CameraCalibration [protobuf]
  	(15) /CAM_FRONT/lidar                          224 msgs (11.67 Hz)     : foxglove.ImageAnnotations [protobuf]
  	(16) /CAM_FRONT/annotations                     39 msgs (2.03 Hz)      : foxglove.ImageAnnotations [protobuf]
  	(17) /CAM_FRONT_RIGHT/image_rect_compressed    217 msgs (11.31 Hz)     : foxglove.CompressedImage [protobuf]
  	(18) /CAM_FRONT_RIGHT/camera_info              217 msgs (11.31 Hz)     : foxglove.CameraCalibration [protobuf]
  	(19) /CAM_FRONT_RIGHT/lidar                    217 msgs (11.31 Hz)     : foxglove.ImageAnnotations [protobuf]
  	(20) /CAM_FRONT_RIGHT/annotations               39 msgs (2.03 Hz)      : foxglove.ImageAnnotations [protobuf]
  	(21) /CAM_BACK_RIGHT/image_rect_compressed     218 msgs (11.36 Hz)     : foxglove.CompressedImage [protobuf]
  	(22) /CAM_BACK_RIGHT/camera_info               218 msgs (11.36 Hz)     : foxglove.CameraCalibration [protobuf]
  	(23) /CAM_BACK_RIGHT/lidar                     218 msgs (11.36 Hz)     : foxglove.ImageAnnotations [protobuf]
  	(24) /CAM_BACK_RIGHT/annotations                39 msgs (2.03 Hz)      : foxglove.ImageAnnotations [protobuf]
  	(25) /CAM_BACK/image_rect_compressed           216 msgs (11.25 Hz)     : foxglove.CompressedImage [protobuf]
  	(26) /CAM_BACK/camera_info                     216 msgs (11.25 Hz)     : foxglove.CameraCalibration [protobuf]
  	(27) /CAM_BACK/lidar                           216 msgs (11.25 Hz)     : foxglove.ImageAnnotations [protobuf]
  	(28) /CAM_BACK/annotations                      39 msgs (2.03 Hz)      : foxglove.ImageAnnotations [protobuf]
  	(29) /CAM_BACK_LEFT/image_rect_compressed      219 msgs (11.41 Hz)     : foxglove.CompressedImage [protobuf]
  	(30) /CAM_BACK_LEFT/camera_info                219 msgs (11.41 Hz)     : foxglove.CameraCalibration [protobuf]
  	(31) /CAM_BACK_LEFT/lidar                      219 msgs (11.41 Hz)     : foxglove.ImageAnnotations [protobuf]
  	(32) /CAM_BACK_LEFT/annotations                 39 msgs (2.03 Hz)      : foxglove.ImageAnnotations [protobuf]
  	(33) /CAM_FRONT_LEFT/image_rect_compressed     224 msgs (11.67 Hz)     : foxglove.CompressedImage [protobuf]
  	(34) /CAM_FRONT_LEFT/camera_info               224 msgs (11.67 Hz)     : foxglove.CameraCalibration [protobuf]
  	(35) /CAM_FRONT_LEFT/lidar                     224 msgs (11.67 Hz)     : foxglove.ImageAnnotations [protobuf]
  	(36) /CAM_FRONT_LEFT/annotations                39 msgs (2.03 Hz)      : foxglove.ImageAnnotations [protobuf]
  	(37) /pose                                      39 msgs (2.03 Hz)      : foxglove.PoseInFrame [protobuf]
  	(38) /gps                                       39 msgs (2.03 Hz)      : foxglove.LocationFix [protobuf]
  	(39) /markers/annotations                       39 msgs (2.03 Hz)      : foxglove.SceneUpdate [protobuf]
  	(40) /markers/car                               39 msgs (2.03 Hz)      : foxglove.SceneUpdate [protobuf]
  	(41) /diagnostics                            22019 msgs (1147.26 Hz)   : diagnostic_msgs/DiagnosticArray [ros1msg]
attachments: 0
metadata: 1

azure

For Azure I think the default format expected by the library is a little weird. It requires that the storage account be set as an env var. It can also be passed as an option when building the url representing the blob. I think your usage shows that the need for an env var and unusual format is probably going to be confusing for users and it'd be much more helpful if you can paste in a blob store link like https://mcaptesting.blob.core.windows.net/mcap/nuscenes.mcap (maybe prefixed with azblob:// instead of https://?) and then its able to put it into the format that the library would like.

I did manage to get it working though, by setting the env vars.

mcap/go/cli/mcap on  mcap-cli-s3-remote-file [?] via 🐹 v1.21.0 took 2s
➜ az storage blob list --account-name mcaptesting --container-name mcap -o table
Name           Blob Type    Blob Tier    Length     Content Type              Last Modified              Snapshot
-------------  -----------  -----------  ---------  ------------------------  -------------------------  ----------
nuscenes.mcap  BlockBlob    Hot          512262937  application/octet-stream  2023-08-30T20:43:30+00:00

mcap/go/cli/mcap on  mcap-cli-s3-remote-file [?] via 🐹 v1.21.0
➜ AZURE_STORAGE_ACCOUNT=mcaptesting AZURE_STORAGE_KEY=… go run . info azblob://mcap/nuscenes.mcap
library: nuscenes2mcap
profile:
messages: 33886
duration: 19.192673s
start: 2018-07-24T04:28:47.604844+01:00 (1532402927.604844000)
end: 2018-07-24T04:29:06.797517+01:00 (1532402946.797517000)
compression:
	lz4: [638/638 chunks] [741.37 MiB/487.76 MiB (34.21%)] [25.41 MiB/sec]
channels:
  	(1) /imu                                      1898 msgs (98.89 Hz)     : IMU [jsonschema]
  	(2) /odom                                      937 msgs (48.82 Hz)     : Pose [jsonschema]
  	(3) /map                                         1 msgs (0.05 Hz)      : foxglove.Grid [protobuf]
  	(4) /semantic_map                                1 msgs (0.05 Hz)      : foxglove.SceneUpdate [protobuf]
  	(5) /tf                                       3002 msgs (156.41 Hz)    : foxglove.FrameTransform [protobuf]
  	(6) /drivable_area                              39 msgs (2.03 Hz)      : foxglove.Grid [protobuf]
  	(7) /RADAR_FRONT                               247 msgs (12.87 Hz)     : foxglove.PointCloud [protobuf]
  	(8) /RADAR_FRONT_LEFT                          262 msgs (13.65 Hz)     : foxglove.PointCloud [protobuf]
  	(9) /RADAR_FRONT_RIGHT                         258 msgs (13.44 Hz)     : foxglove.PointCloud [protobuf]
  	(10) /RADAR_BACK_LEFT                          253 msgs (13.18 Hz)     : foxglove.PointCloud [protobuf]
  	(11) /RADAR_BACK_RIGHT                         243 msgs (12.66 Hz)     : foxglove.PointCloud [protobuf]
  	(12) /LIDAR_TOP                                382 msgs (19.90 Hz)     : foxglove.PointCloud [protobuf]
  	(13) /CAM_FRONT/image_rect_compressed          224 msgs (11.67 Hz)     : foxglove.CompressedImage [protobuf]
  	(14) /CAM_FRONT/camera_info                    224 msgs (11.67 Hz)     : foxglove.CameraCalibration [protobuf]
  	(15) /CAM_FRONT/lidar                          224 msgs (11.67 Hz)     : foxglove.ImageAnnotations [protobuf]
  	(16) /CAM_FRONT/annotations                     39 msgs (2.03 Hz)      : foxglove.ImageAnnotations [protobuf]
  	(17) /CAM_FRONT_RIGHT/image_rect_compressed    217 msgs (11.31 Hz)     : foxglove.CompressedImage [protobuf]
  	(18) /CAM_FRONT_RIGHT/camera_info              217 msgs (11.31 Hz)     : foxglove.CameraCalibration [protobuf]
  	(19) /CAM_FRONT_RIGHT/lidar                    217 msgs (11.31 Hz)     : foxglove.ImageAnnotations [protobuf]
  	(20) /CAM_FRONT_RIGHT/annotations               39 msgs (2.03 Hz)      : foxglove.ImageAnnotations [protobuf]
  	(21) /CAM_BACK_RIGHT/image_rect_compressed     218 msgs (11.36 Hz)     : foxglove.CompressedImage [protobuf]
  	(22) /CAM_BACK_RIGHT/camera_info               218 msgs (11.36 Hz)     : foxglove.CameraCalibration [protobuf]
  	(23) /CAM_BACK_RIGHT/lidar                     218 msgs (11.36 Hz)     : foxglove.ImageAnnotations [protobuf]
  	(24) /CAM_BACK_RIGHT/annotations                39 msgs (2.03 Hz)      : foxglove.ImageAnnotations [protobuf]
  	(25) /CAM_BACK/image_rect_compressed           216 msgs (11.25 Hz)     : foxglove.CompressedImage [protobuf]
  	(26) /CAM_BACK/camera_info                     216 msgs (11.25 Hz)     : foxglove.CameraCalibration [protobuf]
  	(27) /CAM_BACK/lidar                           216 msgs (11.25 Hz)     : foxglove.ImageAnnotations [protobuf]
  	(28) /CAM_BACK/annotations                      39 msgs (2.03 Hz)      : foxglove.ImageAnnotations [protobuf]
  	(29) /CAM_BACK_LEFT/image_rect_compressed      219 msgs (11.41 Hz)     : foxglove.CompressedImage [protobuf]
  	(30) /CAM_BACK_LEFT/camera_info                219 msgs (11.41 Hz)     : foxglove.CameraCalibration [protobuf]
  	(31) /CAM_BACK_LEFT/lidar                      219 msgs (11.41 Hz)     : foxglove.ImageAnnotations [protobuf]
  	(32) /CAM_BACK_LEFT/annotations                 39 msgs (2.03 Hz)      : foxglove.ImageAnnotations [protobuf]
  	(33) /CAM_FRONT_LEFT/image_rect_compressed     224 msgs (11.67 Hz)     : foxglove.CompressedImage [protobuf]
  	(34) /CAM_FRONT_LEFT/camera_info               224 msgs (11.67 Hz)     : foxglove.CameraCalibration [protobuf]
  	(35) /CAM_FRONT_LEFT/lidar                     224 msgs (11.67 Hz)     : foxglove.ImageAnnotations [protobuf]
  	(36) /CAM_FRONT_LEFT/annotations                39 msgs (2.03 Hz)      : foxglove.ImageAnnotations [protobuf]
  	(37) /pose                                      39 msgs (2.03 Hz)      : foxglove.PoseInFrame [protobuf]
  	(38) /gps                                       39 msgs (2.03 Hz)      : foxglove.LocationFix [protobuf]
  	(39) /markers/annotations                       39 msgs (2.03 Hz)      : foxglove.SceneUpdate [protobuf]
  	(40) /markers/car                               39 msgs (2.03 Hz)      : foxglove.SceneUpdate [protobuf]
  	(41) /diagnostics                            22019 msgs (1147.26 Hz)   : diagnostic_msgs/DiagnosticArray [ros1msg]
attachments: 0
metadata: 1

@defunctzombie
Copy link
Contributor

I'm a bit meh on the CLI tool working directly on remote files but I see we already have GCS support. Is there anything else you need in this PR for the AWS support? I don't see why we need to block this on adding Azure support - someone else can come along and add that later.

@defunctzombie
Copy link
Contributor

Is this PR ready for review? It is still marked as draft. If you have lost interest can we close it?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@RileyEv @wkalt @defunctzombie