Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

File based naming resolver #1342

Closed
wants to merge 45 commits into from
Closed

Conversation

beiwei30
Copy link
Member

@beiwei30 beiwei30 commented Nov 29, 2021

Description

Provide a file based naming resolver to replace mDNS so that dapr can address each other either on the same host or via shared file system, which would be more stable and friendly for development.

The default directory to keep naming informations is: $HOME/.dapr/naming, and it can be customized in Dapr's config.yaml:

apiVersion: dapr.io/v1alpha1
kind: Configuration
metadata:
  name: appconfig
spec:
  nameResolution:
    component: "file"
    configuration:
      dir: "/var/tmp/dapr-naming"

Pro and con of this solution:

  • Pro: Since the name resolving is file based, it means that it can not only work in slim mode, but also in standalone mode, since the directory to keep the naming informations can be anywhere as long as it's accessible to Dapr's instances.
  • Con: Since there has no lifecycle mechanism for components, this solution doesn't have chance to remove (unregister) its naming info from the file when Dapr instance goes away. It may not a big deal in development stage, but in the worst case user needs to know where the directory locates and delete the files manually.

Issue reference

This PR will close: dapr/dapr#3256

Checklist

Please make sure you've completed the relevant tasks for this PR, out of the following list:

  • Code compiles correctly
  • Created/updated tests
  • Extended the documentation / Created issue in the https://github.com/dapr/docs/ repo: dapr/docs#[issue number]

@beiwei30 beiwei30 requested review from a team as code owners November 29, 2021 10:26
@yaron2
Copy link
Member

yaron2 commented Nov 29, 2021

There's also this PR: #1340.

Both close the same issue, however I do not see a problem accepting both. Thoughts?

@beiwei30
Copy link
Member Author

beiwei30 commented Dec 1, 2021

will continue to update dapr/dapr and dapr/docs once this solution is accepted.

beiwei30 and others added 26 commits December 14, 2021 15:25
Signed-off-by: Ian Luo <ian.luo@gmail.com>
Signed-off-by: Ian Luo <ian.luo@gmail.com>
Signed-off-by: Ian Luo <ian.luo@gmail.com>
Signed-off-by: Ian Luo <ian.luo@gmail.com>
…lemetry system and logic servers. (dapr#1155)

* feature/pubsub: add delay queue params for pulsar

* feature/pubsub: add delay queue params for pulsar

* feature/pubsub: add delay queue params for pulsar

* feature: add middleware/routerchecker to reduce dirty data report opentelemetry

* feature: add middleware/routerchecker to reduce dirty data report opentelemetry

* Update pulsar.go

* feature: add middleware/routerchecker to reduce dirty data report opentelemetry

* feature: add middleware/routerchecker to reduce dirty data report opentelemetry

Co-authored-by: Long Dai <long.dai@intel.com>
Co-authored-by: Simon Leet <31784195+CodeMonkeyLeet@users.noreply.github.com>
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
Signed-off-by: Ian Luo <ian.luo@gmail.com>
Signed-off-by: Ian Luo <ian.luo@gmail.com>
Signed-off-by: Ian Luo <ian.luo@gmail.com>
add retries for pubsub subscribe
fix race conditions

Signed-off-by: Ian Luo <ian.luo@gmail.com>
…apr#1290)

* Authentication for new Azure SDK

* Updated keyvault to use new Azure SDK

* 🙈

* Update authentication/azure/auth.go

* Update authentication/azure/auth.go

* Update authentication/azure/auth.go

* Reintroduce changes from PR 1132 without updating ASB

* Lint 💄

* Marking contexts as TODO as they'll need a timeout
As per conversation with @berndverst

* Update certification tests with no auth libraries

Co-authored-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
Signed-off-by: Ian Luo <ian.luo@gmail.com>
- Remove cloudstate state store component
- Update state/Readme.md

Fixes dapr#1297

Co-authored-by: Long Dai <long.dai@intel.com>
Signed-off-by: Ian Luo <ian.luo@gmail.com>
Co-authored-by: Long Dai <long.dai@intel.com>
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
Signed-off-by: Ian Luo <ian.luo@gmail.com>
With the addition of the certification tests, changes in the go.mod
definition for component-contrib will also need to be propagated to the
go.mod for each of the tests under tests/certification.

- Add a `make modtidy-all` action to support updating all go.mod files.
- Update developing-component.md to include use of `make modtidy-all`.

Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
Co-authored-by: Dapr Bot <56698301+dapr-bot@users.noreply.github.com>
Signed-off-by: Ian Luo <ian.luo@gmail.com>
Signed-off-by: Ian Luo <ian.luo@gmail.com>
Upgrade vmware-go-kcl dependency to the latest v1.5.0.
update mod tidy

Co-authored-by: Dapr Bot <56698301+dapr-bot@users.noreply.github.com>
There is an internal race condition in MQTT client Publish method when
configured with autoreconnect (default) where the call can fail if the
connection is broken right after the outgoing event is queued but before
the dead connection is blocked.

This change mitigates that on the Dapr side with a limited retry loop
3 times at fixed 200ms interval as a patch before the Dapr resiliency
feature is introduced that exposes more generalized retry controls to
the dev.

Co-authored-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
Co-authored-by: Long Dai <long.dai@intel.com>
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
Co-authored-by: Dapr Bot <56698301+dapr-bot@users.noreply.github.com>
Signed-off-by: Ian Luo <ian.luo@gmail.com>
Signed-off-by: Ian Luo <ian.luo@gmail.com>
* Move Service Bus Pubsub/Binding to common auth

Both the pubsub and input/output binding for Azure Service Bus were
connecting via a connection string. This is still supported but will
now fallback to using AAD from the common auth library. This is also
the recommended auth pattern going forward.

* Move AMPQ specific auth and fix linter issues

* Make conn string and namespace mutually exclusive

* Move resourceName to a constant

* Update auth_amqp.go

* Update auth.go

Co-authored-by: Long Dai <long.dai@intel.com>
Co-authored-by: Simon Leet <31784195+CodeMonkeyLeet@users.noreply.github.com>
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
Co-authored-by: Dapr Bot <56698301+dapr-bot@users.noreply.github.com>
Signed-off-by: Ian Luo <ian.luo@gmail.com>
…r#1283)

* feat(pulsar): add tenant and namesapce. fix topic parsing.

* add test

* fix review add topic format unit test

* fix review: add persistent to metadata, fix nits

* fix lint

Co-authored-by: Long Dai <long.dai@intel.com>
Co-authored-by: Mukundan Sundararajan <musundar@microsoft.com>
Co-authored-by: Simon Leet <31784195+CodeMonkeyLeet@users.noreply.github.com>
Co-authored-by: Dapr Bot <56698301+dapr-bot@users.noreply.github.com>
Signed-off-by: Ian Luo <ian.luo@gmail.com>
Co-authored-by: xujie <xujie1@shein.com>
Signed-off-by: Ian Luo <ian.luo@gmail.com>
…pr#1273)

Co-authored-by: Long Dai <long.dai@intel.com>
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
Co-authored-by: Dapr Bot <56698301+dapr-bot@users.noreply.github.com>
Signed-off-by: Ian Luo <ian.luo@gmail.com>
* support tracestate can be propagated in PubSub

* add trace state

Co-authored-by: Long Dai <long.dai@intel.com>
Co-authored-by: Ian Luo <ian.luo@gmail.com>
Co-authored-by: Simon Leet <31784195+CodeMonkeyLeet@users.noreply.github.com>
Co-authored-by: Dapr Bot <56698301+dapr-bot@users.noreply.github.com>
Signed-off-by: Ian Luo <ian.luo@gmail.com>
Signed-off-by: Ian Luo <ian.luo@gmail.com>
* bugfix for sns topic deletion upon termination

* removed upstream github workflow files

* Update snssqs.go

* dapr bot schedule

* read and append queue attributes

* unnecessary escaping in json tag

* unexporting structs

Signed-off-by: Ian Luo <ian.luo@gmail.com>
Signed-off-by: Ian Luo <ian.luo@gmail.com>
LaurenceLiZhixin and others added 18 commits December 14, 2021 15:27
* fix: add config subscription gr done

* unexpected comments

* fix comment

Co-authored-by: Dapr Bot <56698301+dapr-bot@users.noreply.github.com>
Signed-off-by: Ian Luo <ian.luo@gmail.com>
* Adds retry on CosmosDB Init in case of TooManyRequests error

* Use backoff v4

* missed some permanent errors

* clean up go.mod

* fix error type casting

* Add constant for HTTP 429

Signed-off-by: Ian Luo <ian.luo@gmail.com>
Co-authored-by: Dapr Bot <56698301+dapr-bot@users.noreply.github.com>
Signed-off-by: Ian Luo <ian.luo@gmail.com>
Signed-off-by: Ian Luo <ian.luo@gmail.com>
Signed-off-by: Ian Luo <ian.luo@gmail.com>
* bugfix for sns topic deletion upon termination

* removed upstream github workflow files

* wrapping errors

* fifo and naming sanitization

* fifo and name santization tests

* component carrying id

* Update snssqs.go

removed unneeded import

* Update snssqs.go

* add missing github actions file

* linting

* fix go.mod

* schedule yml file

* const max lenght, bugfix in ack

* dapr bot schedule

Co-authored-by: Dapr Bot <56698301+dapr-bot@users.noreply.github.com>
Signed-off-by: Ian Luo <ian.luo@gmail.com>
Signed-off-by: Ian Luo <ian.luo@gmail.com>
* Add certification tests for ASB Queue binding

This commit adds certification tests for the Azure Service Bus Queue
Input/Output binding. This utilizes the new certification framework
and performs a series of tests as described in the test plan readme.

The general purpose of these tests is to serve as an integration test
and as such requires an actual Azure Service Bus connection and a
Dapr sidecar.

dapr#957

* Add to github workflow and fix formatting

* Fix dependencies

Co-authored-by: Dapr Bot <56698301+dapr-bot@users.noreply.github.com>
Signed-off-by: Ian Luo <ian.luo@gmail.com>
Co-authored-by: Dapr Bot <56698301+dapr-bot@users.noreply.github.com>
Signed-off-by: Ian Luo <ian.luo@gmail.com>
Co-authored-by: Dapr Bot <56698301+dapr-bot@users.noreply.github.com>
Signed-off-by: Ian Luo <ian.luo@gmail.com>
* bugfix for sns topic deletion upon termination

* removed upstream github workflow files

* Update snssqs.go

* dapr bot schedule

* read and append queue attributes

* unnecessary escaping in json tag

* unexporting structs

* bugfix in policy

* bugfix in policy. merged from master

* fifo suffix as const

Signed-off-by: Ian Luo <ian.luo@gmail.com>
Co-authored-by: Dapr Bot <56698301+dapr-bot@users.noreply.github.com>
Signed-off-by: Ian Luo <ian.luo@gmail.com>
* Fix test infrastructure setup script

Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>

* use more unique ACR resource name

Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>

Co-authored-by: Looong Dai <long.dai@intel.com>
Signed-off-by: Ian Luo <ian.luo@gmail.com>
Signed-off-by: Ian Luo <ian.luo@gmail.com>
Signed-off-by: Amit Mor <amit.mor@hotmail.com>
Signed-off-by: Ian Luo <ian.luo@gmail.com>
* Adds CosmosDB Binding authentication tests

Signed-off-by: GitHub <noreply@github.com>
Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>

* Enable Cosmos DB Binding Conformance test

Signed-off-by: GitHub <noreply@github.com>
Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>

* Initial cosmosdb binding certification plan WIP

Signed-off-by: GitHub <noreply@github.com>
Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>

* Go mod tidy

Signed-off-by: GitHub <noreply@github.com>
Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>

* Update library and go mod tidy

Signed-off-by: GitHub <noreply@github.com>
Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>

* make modtidy-all

Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>

* CosmosDB Binding test plan details

Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
Signed-off-by: Ian Luo <ian.luo@gmail.com>
Signed-off-by: Ian Luo <ian.luo@gmail.com>
@codecov
Copy link

codecov bot commented Dec 14, 2021

Codecov Report

Merging #1342 (8c15cd8) into master (09fb60c) will decrease coverage by 0.00%.
The diff coverage is 19.84%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #1342      +/-   ##
==========================================
- Coverage   35.02%   35.01%   -0.01%     
==========================================
  Files         148      150       +2     
  Lines       12825    13004     +179     
==========================================
+ Hits         4492     4554      +62     
- Misses       7853     7950      +97     
- Partials      480      500      +20     
Impacted Files Coverage Δ
authentication/azure/auth_amqp.go 0.00% <0.00%> (ø)
bindings/aws/kinesis/kinesis.go 2.61% <0.00%> (ø)
bindings/azure/eventgrid/eventgrid.go 3.84% <0.00%> (ø)
bindings/mqtt/mqtt.go 25.00% <0.00%> (-1.42%) ⬇️
configuration/redis/redis.go 37.22% <0.00%> (-1.12%) ⬇️
pubsub/aws/snssqs/policy.go 0.00% <0.00%> (ø)
pubsub/rocketmq/rocketmq.go 0.00% <0.00%> (ø)
state/azure/cosmosdb/cosmosdb.go 13.91% <0.00%> (-1.35%) ⬇️
state/query/query.go 23.91% <ø> (ø)
authentication/azure/auth.go 40.35% <1.92%> (-15.75%) ⬇️
... and 11 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9dbdaee...8c15cd8. Read the comment docs.

@beiwei30
Copy link
Member Author

pls. use #1380 for further review.

@beiwei30 beiwei30 closed this Dec 14, 2021
@beiwei30 beiwei30 deleted the file-based-naming branch December 14, 2021 07:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Proposal] Provide alternative service resolution mechanism besides mdns