Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Race condition when registering ATC tables #8323

Open
Micah-Kolide opened this issue Apr 30, 2024 · 3 comments · May be fixed by #8324
Open

Race condition when registering ATC tables #8323

Micah-Kolide opened this issue Apr 30, 2024 · 3 comments · May be fixed by #8324
Milestone

Comments

@Micah-Kolide
Copy link
Contributor

Micah-Kolide commented Apr 30, 2024

It seems like #8233 has introduced a race condition when attempting to register both ATC tables and an extension at the same time.

The issue the PR was fixing was the first ATC table would attempt to generate twice, and therefore was causing an error message. The fix here was moving the Registry::call for attaching the ATC tables plugin to before the point where the table plugins are added to the registry.

The reason this seemed to work is because we drop the error: return Status::failure("Cannot call registry item: " + item_name); returns to:

      // Call is multiplexing plugins (usually for multiple loggers).
      for (const auto& item : osquery::split(item_name, ",")) {
        get().registry(registry_name)->call(item, request, response);
      }
      // All multiplexed items are called without regard for statuses.
      return Status(0);

Yes the Registry::call fails but the Registry::add doesn't, so attachTableInternal picks it up and generates the tables.

However, when an extension comes into play the extension registration may or may not run before the ATC registration. If it does run first, then it pushes out the ATC registration, which leaves nothing to attach the ATC tables since the Registry::call failed and they register after attachTableInternal has ran.

ATC registers first and is a success:

I0430 15:38:01.131906 -248333632 init.cpp:413] osquery initialized [version=5.12.1-3-g28c5df5ee-dirty]
...
I0430 15:38:01.132248 -248333632 auto_constructed_tables.cpp:99] Removing stale ATC entries
I0430 15:38:01.132252 1837101056 interface.cpp:299] Extension manager service starting: /tmp/osquery.socket
ATC registry call - fails and doesn't return the error.
I0430 15:38:01.134325 -248333632 auto_constructed_tables.cpp:233] ATC table: atc_test Registered
ATC registry add - success.
I0430 15:38:01.134421 -248333632 loader.cpp:45] No experiments selected
attachTableInternal
...
I0430 15:38:01.156646 1838247936 interface.cpp:137] Registering extension (example, 7818, version=0.0.1, sdk=5.12.1-3-g28c5df5ee-dirty)
I0430 15:38:01.156832 1838247936 registry_factory.cpp:116] Extension 7818 registered table plugin example
osquery> SELECT COUNT(*) FROM atc_test;
COUNT(*) = 29

Extension registers first which causes the ATC tables to not get picked up by attachTableInternal:

I0430 15:31:39.871309 -248333632 init.cpp:413] osquery initialized [version=5.12.1-3-g28c5df5ee-dirty]
...
I0430 15:31:39.871884 -248333632 auto_constructed_tables.cpp:99] Removing stale ATC entries
I0430 15:31:39.871989 1841197056 interface.cpp:299] Extension manager service starting: /tmp/osquery.socket
I0430 15:31:39.876617 1842343936 interface.cpp:137] Registering extension (example, 10072, version=0.0.1, sdk=5.12.1-3-g28c5df5ee-dirty)
attachTableInternal
...
ATC registry call - fails and doesn't return the error.
I0430 15:31:39.881490 -248333632 auto_constructed_tables.cpp:233] ATC table: atc_test Registered
ATC registry add - success.
I0430 15:31:39.881618 -248333632 loader.cpp:45] No experiments selected
...
I0430 15:31:39.895435 1842343936 registry_factory.cpp:116] Extension 10072 registered table plugin example
osquery> SELECT COUNT(*) FROM atc_test;
Error: no such table: atc_test
osquery version: 5.12.1-3-g28c5df5ee-dirty
extension: read_only_table_extension.ext
extension socket: /tmp/osquery.socket
ATC: {
	"auto_table_construction": {
		"atc_test": {
			"query": "SELECT service FROM access",
			"path": "/System/Volumes/Data/Library/Application Support/com.apple.TCC/TCC.db",
			"columns": ["service"]
		}
        }
}
@Micah-Kolide Micah-Kolide changed the title Race condition when registering both ATC tables and extension tables Race condition when registering ATC tables Apr 30, 2024
@directionless directionless added this to the 5.13 milestone May 1, 2024
@bgirardeau-figma
Copy link
Contributor

I did some digging as well and think this is on the right track, with a caveat that the error from trying to attach the table in the ATC plugin too early is dropped here (ie in the ATC plugin code). The multiplexing code in the registry factory that drops errors only happens if there are multiple active plugins separated by a comma, but the call to the SQL plugin specifically just requests "sql".

In any case, the Table plugin registry needs to be aware of a table before the "attach" request is made to the SQL plugin, meaning #8233 should be reverted in favor of an alternate fix for the duplicate ATC table registration issue.

Besides the ATC plugin, there are two other paths that add tables to the registry and SQL connection:

  • Built in tables:
    • Added to table registry at initialization time with a macro (template for this is here)
    • Added to SQL connection the first time it's initialized (at latest during the first query)
  • Extensions that provide tables
    • Added to table registry as "external" table and SQL connection in the same code path:
      • registerExtension calls RegistryFactory's addBroadcast to tell all the registries about the extension
      • RegistryFactory calls each registry's addExternal function
      • RegistryInterface first registers the route, then calls the implementation's addExternal function, then finishes registering the extension in table registry if successful in this code block
        • TablePlugin's addExternal function calls the SQL plugin to attach the extension table to the SQL connection
        • SQL plugin calls back to table registry for information on the extension table here
        • This returns an empty response here because the table's route has already been registered, but it doesn't register in the list of tables yet, so there is no duplicate issue like ATC

Because ATC tables can be added and removed dynamically (including in the race with table extensions described here), they need to call SQL attach explicitly -- only the built-in tables can safely rely on database initialization to attach the tables. There are a couple paths on how to fix both this bug and the duplicate ATC table registrations:

  • Have a partial pre-registration state for ATC tables similar to extensions
  • Initialize the database connection before trying to register or attach anything (either in ATC code or somewhere else in osquery's init code that we know runs before ATC will)

I think the second is a much simpler fix without much downside. I'll put up a quick PR as an example, not sure how much time I'll have to actually test it

@directionless
Copy link
Member

directionless commented May 1, 2024

@bgirardeau-figma -- @Micah-Kolide might have a fix in flight as well.

@bgirardeau-figma
Copy link
Contributor

OK I decided to go with the pending state idea instead, as it seems more correct/less hacky. I think forcing initialization in the ATC table would also mostly work, but in theory there could be (admittedly very unlikely) races where the database gets reinitialized in the middle, causing a duplicate registration again.

Here's a PR: #8324

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants