Eager loading of sibling relation causes fatal error

[karola-gr]

Description

Eager loading of sibling relation causes fatal error.

Fatal error: Cannot access field before it is initialized or fetched

The data in the database seems to be correctly created. The relations exist and are filled. However, I cannot access them in a query. The problem only occurs with some sibling relations, but only when querying a sibling that is held by an optional parent, or with a sibling that may be empty. Not with all of them.

Details

Case 1:

Sibling relation of optional parent

Example EntityOne with sibling relation to EntityTwo

final class EntityOne: Model, Content {
    static let schema = "entity_one"

    @Siblings(
        through: EntityOneAttachedEntityTwo.self,
        from: \.$entityOne,
        to: \.$entityTwo
    ) var   attachedEntityTwo:  [EntityTwo]
    
    init() {}

EntityThree with optional parent relation to EntityOne

final class EntityThree: Model, Content {
    static let schema = "entity_three"
    @ID                                                               var id: UUID?
    ...Fields..
    @OptionalParent(key: "entity_one_id")    var entityOne: EntityOne?

    init() {}
}

--> Query

let entity = try await EntityThree
           .query(on: req.db)
	  .with(\.$entityThree) { entityThree  in
		  entityThree
			  .with(\.$field) // Fields, Parents, etc. work fine
    			  .with(\.$attachedEntityTwo) // Causes fatal error in all()
	  }
            .all()
    }

BUT: entity.entityThree.load(on: db) works fine

Case 2:
Same behaviour as Case 1, but without optional parent relation

Example EntityOne with sibling relation to EntityTwo

 final class EntityOne: Model, Content {
 
    ...
    
    @Siblings(
        through: EntityOneAllowedEntityTwo.self,
        from: \.$entityOne,
        to: \.$entityTwo
    ) var allowedEntityTwo: [EntityTwo]
    
  ....

}

RelationTable (same for case 1)

final class EntityOneAllowedEntityTwo: Model {
    ...    
    @ID(key: .id)                           		                       var id: UUID?
    @Parent(key: "entity_one_id")             	               var entityOne: EntityOne
    @Parent(key: "entity_two_id")                               var entityTwo: EntityTwo
    
    init() {}
    
    init(id: UUID? = nil,
         entityOne: EntityOne,
         entityTwo: EntityTwo
    ) throws {
        self.id = id
        self.$entityOne = try entityOne.requireID()
        self.$entityTwo = try entityTwo.requireID()
    }
}

Eager loading in query failes, but loading afterwards works fine.

let entityOne = try await EntityOne
			.query(on: db)
                        .with(\.$field) // Everything works fine
//                      .with(\.$allowedEntityTwo) // Causes fatal error 
			.first()
			.unwrap(or: Abort(.notFound))
			.get()

But: try await entityOne.$allowedEntityTwo.loadIfNeeded(on: db) also works fine.

Migration for relation table

struct CreateEntityOneAllowedEntityTwoPivot: Migration {
    func prepare(on database: Database) -> EventLoopFuture<Void> {
        database.schema(EntityOneAllowedEntityTwo.schema)
            .id()
            .field("entity_one_id",
				.uuid,
				.references(EntityOne.schema, "id", onDelete: .cascade),
				.required)
            .field("entity_two_id",
				.uuid,
				.references(EntityTwo.schema, "id", onDelete: .cascade),
				.required)
            .unique(on: "entity_one_id", "entit_two_id")
            .create()
    }
    func revert(on database: Database) -> EventLoopFuture<Void> {
        database.schema(EntityOneAllowedEntityTwo.schema)
            .delete()
    }
}

Expected behavior

Sibling relations should be eager loaded / accessed in query

Environment

Vapor 4.77
MacOS: 13.5

[i-fire1773]

maybe need use .with(.$field.$pivots)

[karola-gr]

We tested it, but unfortunately the error persists.

[0xTim]

What does your code look like that actually triggers the error (where you're accessing the properties)

[karola-gr]

The error comes from the query, as I said, when loading the sibling relation.

The code for this is basically this:

func getAllHandler(
_ req: Request
) async throws -> [EntityOneReturnDto] {
return try await EntityOne
.query(on: req.db)
.with(.$allowedEntityTwo) <- Sibling relation (.$pivots) leads to the same issue
...Fields....
.all()
}

The entity and the migration look as previously described..
I attach the stack trace as a screenshot. Maybe that helps.

[0xTim]

In the code snippet you're returning a DTO - are you sure it's not happening there?

[karola-gr]

yes, I'm sure. The error is thrown before the dto is created and then of course when I remove the dto completely from the code. I think it fails in the call to the all() function in the query. At least that's the last point I can reasonably track with the debugger.

[karola-gr]

Do you have any idea where else this could be coming from, or can I provide any other information to look for the problem?

Sorry to ask again...but we are kind of stuck here

[0xTim]

@karola-gr I've been unable to reproduce this. Do you have a reproducible project you can share?

[Kishimotovn]

@karola-gr does it give fatalError if you don't select specific fields from the table?

[karola-gr]

@Kishimotovn No, this only happens with the specific sibling relations, not with other fields. as soon as I exclude the siblings from the query and reload them directly afterwards, it works without problems. Maybe there is some problem with the combination with the optionals that something is assumed to be nil... But yes...reload the relation is also our current workaround.

@0xTim , it is difficult to recreate the scenario in an minimal example project. I think the only way would be to look at our project remotely in a call or something.

[karola-gr]

Hi :) I have set up a small gitlab project in which I have rebuilt the part of the original project that fails. Maybe this will help. @0xTim ...I have invited you to the repo.

[karola-gr]

Hello,
I wanted to follow up on the repository invitation I sent to reproduce the problem we discussed previously. Have you had an opportunity to review it? Are there any updates or feedback on the issue?
I appreciate your time and assistance. Thank you.

[0xTim]

@karola-gr I've just tried running the tests from the GH repo you uploaded and I don't get any crashes. It retrieves items from the DB and returns them and doesn't fail like you are seeing. I get

Test Suite 'Selected tests' started at 2023-11-07 14:06:07.143.
Test Suite 'AppTests.xctest' started at 2023-11-07 14:06:07.143.
Test Suite 'Get' started at 2023-11-07 14:06:07.143.
Test Case '-[AppTests.Get testGetExercise]' started.
[GET /api/exercises, POST /api/exercises, GET /api/exercise_bases, POST /api/exercise_bases, POST /api/exercise_standards, POST /api/exercise_variation_categories]
2023-11-07T14:06:07+0000 info codes.vapor.request : request-id=43FC3D0D-E803-4010-AC7B-81F4CF64282B [Vapor] GET /api/exercises
DB8CDEA1-89BE-468E-B482-02B51704A406 Bench Press 
CED1F8E8-9DD3-4FCA-B160-17DD540AD133 Bench Press 
/Users/timc/Developer/playgrounds/vaportestenv/Tests/AppTests/Get.swift:36: error: -[AppTests.Get testGetExercise] : XCTAssertEqual failed: ("2") is not equal to ("0")
Test Case '-[AppTests.Get testGetExercise]' failed (15.481 seconds).
Test Suite 'Get' failed at 2023-11-07 14:06:22.625.
	 Executed 1 test, with 1 failure (0 unexpected) in 15.481 (15.481) seconds
Test Suite 'AppTests.xctest' failed at 2023-11-07 14:06:22.625.
	 Executed 1 test, with 1 failure (0 unexpected) in 15.481 (15.482) seconds
Test Suite 'Selected tests' failed at 2023-11-07 14:06:22.625.
	 Executed 1 test, with 1 failure (0 unexpected) in 15.481 (15.482) seconds
Program ended with exit code: 1

Are you running the latest packages etc?

[karola-gr]

Yes, I updated all the packages again, but still got this:

Test Suite 'Selected tests' started at 2023-11-07 18:32:08.480.
Test Suite 'AppTests.xctest' started at 2023-11-07 18:32:08.481.
Test Suite 'Get' started at 2023-11-07 18:32:08.481.
Test Case '-[AppTests.Get testGetExercise]' started.
[GET /api/exercises, POST /api/exercises, GET /api/exercise_bases, POST /api/exercise_bases, POST /api/exercise_standards, POST /api/exercise_variation_categories]
2023-11-07T18:32:08+0100 info codes.vapor.request : request-id=474AE927-1FD9-4FEC-B7AC-A65C52322BCD [Vapor] GET /api/exercises
FluentKit/Field.swift:23: Fatal error: Cannot access field before it is initialized or fetched: exercise_base_id
2023-11-07 18:32:13.697927+0100 xctest[62542:10452040] FluentKit/Field.swift:23: Fatal error: Cannot access field before it is initialized or fetched: exercise_base_id

Well, this confuses me even more now :D

[0xTim]

The best thing would be do a clean repo clone and follow the steps. If that fails then you must have a code difference. You can see the line in the stack trace that causes the issue

[karola-gr]

So I tried a few things again today. I used the postgres and the redis. I deleted derived data, did a clean build, recloned the repo in exactly the same state, reloaded all packages...
(M2, Xcode 15.0.1, macOS14)

I always did:
TestExerciseVariationCategory
TestExerciseBase
TestExerciseStandard
Test Exercise
so uploads are in the right order, then TestGetExercise and the error occurs.

In a few cases, which were not reproducible for me, it worked. The first time was when I switched to linux..then I tried to build the original project on linux and that did not work. The test repo didn't work after that either...it went on like that all day. It feels really random... :(

The stack trace doesn't really get me anywhere either, to be honest. It breaks between the .all() and the dto

[0xTim]

Oh I think I know what's going on. The error message is actually subtly different and it's nothing to do with sibling relations.

The error is about specific fields and not relations - I'm guessing you have a model field that has a null column in the database that should be @OptionalField and not @Field - that would explain the behaviour you're seeing

[karola-gr]

i created the data in such a way that there are two entries that are null in one column each (exerciseBaseId in the first entry andexerciseStandardId in the other). However, these two columns are declared as optionalParents. Even if I replace all null entries with data, I still get the error.

Reloading these fields also works without any problems. But it does not work directly in the query.

I am trying to access its siblings in the query via an optional parent. Maybe it searches for null in the sibling relation when the optional parent is empty and crashes at this point?

[karola-gr]

I removed the optional parent again and only looked at the sibling relation. Here I get a nil when it crashes, but there shouldn't actually be a nil. It is required and I don't see any entry in the database that has a nil in this column.
So yes, your previous answer is probably a problem, but how can this happen? Especially since this case does not occur when reloading after the query

[karola-gr]

I have double checked the data on the database as well as the db dump and it seems all correct. Also a nil value in a required column should already lead to a problem when saving to the database or lazy load them, right?
If I check the ids in
private struct SiblingsEagerLoader<From, To, Through>: EagerLoader
these are also correct and not nil. I don't get the point where the problem occurs :(

[rlziii]

@karola-gr, very likely unrelated, but .unique(on: "entity_one_id", "entit_two_id") has entit_two_id misspelled.

[theoks]

We have encountered the exact same problem. Whenever we try to eager load sibling relationships using .with and then .all it always triggers the Cannot access field before it is initialized or fetched fatalError.

[karola-gr]

Oh, that's good to read that we're not the only ones with this problem. Do you have any idea what the problem could be?

[theoks]

@karola-gr sorry for the delayed reply. Unfortunately no. I cannot eager load any sibling relationship. It doesn't fail sporadically, it fails consistently, and I can't find a workaround.

[karola-gr]

Thanks for the answer anyway. Are there any updates on this? We have now restructured our database so that we have removed all optional parents, but this does not change the error. As @theoks says, it fails consistently when preloading siblings of an entity.

The only thing that works for us is to load the siblings separately after fetching the entity.

[gwynne]

Is it possible to show actual code - or at least code which compiles - which exhibits this issue? The pseudocode in the original post is both incomplete and invalid in ways that make it difficult to see where things might be going wrong.

[karola-gr]

CodeExtract.zip

Unfortunately, I cannot provide the actual code. I tried to give an extract here:
An exercise has a parent exerciseDefinition and this in turn has a sibling, which contains attachesExerciseVariations.
If I query an exercise in the controller and want to fetch its definition with the siblings, I get the described error. If I do not preload, but then reload the attached exercise variations, it works.

[gwynne]

In the attached code I do not see the definition of ExerciseDefinitionAttachedExerciseVariation (the pivot model itself) - its migration is present, but not the model - and that model is the most relevant code for figuring out this issue.

[karola-gr]

oh sorry. This is the corresponding model

ExerciseDefinitionAttachedExerciseVariation.swift.zip

[gwynne]

I have so far been unable to reproduce the issue with the code you provided. I added the following additional code to put it together into a buildable project:

struct CreateBaseTables: AsyncMigration {
    func prepare(on database: any Database) async throws {
        try await database.schema(ExerciseDefinition.schema).id().create()
        try await database.schema(ExerciseVariation.schema).id().create()
    }

    func revert(on database: any Database) async throws {
        try await database.schema(ExerciseVariation.schema).delete()
        try await database.schema(ExerciseDefinition.schema).delete()
    }
}

struct AddTestData: AsyncMigration {
    func prepare(on database: any Database) async throws {
        // create a definition
        let def = ExerciseDefinition()
        try await def.create(on: database)

        // create a couple of variations
        let var1 = ExerciseVariation(), var2 = ExerciseVariation()
        try await [var1, var2].create(on: database)

        // attach the variations to the definition through pivots
        try await def.$attachedExerciseVariations.attach([var1, var2], on: database)

        // create an exercise using the definition
        let exc = Exercise(exerciseDefinitionID: try def.requireID())
        try await exc.create(on: database)
    }
    
    func revert(on database: any Database) async throws {}
}

I also had to comment out the authentication logic, the additional eager loading, and the DTO in the controller (as those types are not present in the provided code), and I converted the logic to be fully Concurrency-aware, with this result:

struct ExerciseController: RouteCollection {
    func boot(routes: any RoutesBuilder) throws {
        let routes = routes.grouped("api", .constant(Exercise.schema))
        routes.get(":ExerciseID", use: self.getHandler)
    }

    @Sendable func getHandler(_ req: Request) async throws -> Exercise {
        guard let exerciseIDString = req.parameters.get("ExerciseID"),
              let exerciseID = UUID(uuidString: exerciseIDString)
        else {
            throw Abort(.badRequest, reason: "Missing parameters in request")
        }
        guard let exercise = try await Exercise
            .query(on: req.db)
            .filter(\.$id == exerciseID)
            .with(\.$exerciseDefinition, { $0.with(\.$attachedExerciseVariations) })
            .first()
        else {
            throw Abort(.notFound)
        }
        return exercise
    }
}

I added these and the other code you provided to a default Vapor template project, and was unable to reproduce the crash.

As an aside, I ended up with some minor style and best practices notes with respect to the provided code, hopefully they are helpful in general despite being unrelated to this issue:

• Use the .constant() case of PathComponent (as shown in the controller above) instead of PathComponent.init(stringLiteral:), which is intended for use only by the compiler.
• It is strongly recommended to convert Migrations to AsyncMigrations when practical to do so.
• In async code, using guard (also as shown above) is much, much more efficient than using .unwrap(or:) and .get(); it saves several thread hops and allocations.