Drop anemone and use Spidr for Repo discovery

[sjha4]

To Do:

• Support basic auth
• Support proxy

What are the changes introduced in this pull request?

Considerations taken when implementing this change?

What are the testing steps for this pull request?

bundle install
Go to Content > Product > Repo discovery
Run repo discovery.

[sjha4]

@evgeni : Thoughts on spidr gem as replacement for anemone?

[evgeni]

Doesn't look crazy? Only dep is nokogiri, which we have anyway, tested on modern rubies. Why not.

[sjha4]

I am seeing significant performance difference with what's on the PR vs the existing workflow..Looking at ways to speed this up..Will push updates when I get the performance sorted.

Update: Should be good to go with latest commit.
Able to see chunked output in repo discovery page and the task finishes in about the same time as earlier.

[ekohl]

Implementation wise I think you should separate the crawler and Docker search into separate classes. Perhaps even the file crawl as well. Right now it's confusing.

[ekohl]

I think this heads in the right direction.

Review wise I wonder if it makes sense to split it up: 1 to create the 3 classes and a follow up that replaces anemone with spidr.

[sjha4]

Review wise I wonder if it makes sense to split it up: 1 to create the 3 classes and a follow up that replaces anemone with spidr.

The anemone -> spidr change is localized to yum_discovery only right now as far a changes go. My 2 cents is it's a small enough change to be in one PR?

[ekohl]

Review wise I always prefer to first have a refactor and then an actual code change. Right now if I read the commits then it's the other way around. That makes this whole change harder to follow, but I don't have commit permissions here so I'll let other reviewers weigh in on that aspect.

[ekohl]

This will be nil if upstream_credentials_and_search is overridden by a caller. If you want it to default to * you can use @search = upstream_credentials_and_search.fetch(:search, '*').

[ekohl]

I know it was already this way, but isn't this equivalent?

Suggested change

	@upstream_username = upstream_credentials_and_search[:upstream_username].empty? ? nil : upstream_credentials_and_search[:upstream_username]
	@upstream_password = upstream_credentials_and_search[:upstream_password].empty? ? nil : upstream_credentials_and_search[:upstream_password]
	@upstream_username = upstream_credentials_and_search[:upstream_username].presence
	@upstream_password = upstream_credentials_and_search[:upstream_password].presence

[ekohl]

I think this should live in the yum class