How to identify/represent federated containers of things?

[balessan]

At StartinBlox, we are working on distributed applications which need to use data coming from different pods and containers.

For example, we are working on the Happy Dev application which aims to provide distributed project management and instant messaging capabilities, to enforce the cooperation between our network of local companies.

We found no information about the proper way to represent the list of pods/sources constituting a federated universe as a LD resource.

Our technical proposal would be to have a specific type of resource that is a link to a container being a source of data. For now, we are using our own custom @type: sib:source (vocabulary not defined yet) for that.

The client application can then detect these sources and fetch the resources contained by the associated containers.

Any feedback welcome, would love to discuss that with you,

[elf-pavlik]

to have a specific type of resource that is a link to a container being a source of data

I don't think I understand that. Could you possibly provide some specific example with data snippet using mentioned sib:source ?

[balessan]

@elf-pavlik : what we are currently using is similar to the following:

{
  "@id": "",
  "@type": "ldp:Container",
  "ldp:contains": [
    {
      "@id": "https://api.test-paris.happy-dev.fr/sources/projects/1/",
      "container": "https://api.test-nantes.happy-dev.fr/projects/",
      "federation": "projects",
      "@type": "sib:source"
    },
    {
      "@id": "https://api.test-paris.happy-dev.fr/sources/projects/2/",
      "container": "https://api.test-paris.happy-dev.fr/projects/",
      "federation": "projects",
      "@type": "sib:source"
    }
  ],
  "@context": "https://cdn.happy-dev.fr/owl/hdcontext.jsonld"
}

We use a specific endpoint in order to provide the list of known containers of projects, each container of projects being fetched individually when we need to retrieve the federated list of projects. Is that clear enough ?

[RubenVerborgh]

We found no information about the proper way to represent the list of pods/sources constituting a federated universe as a LD resource.

Could you give a concrete example of something you want to achieve?

Because typically “pods/sources” are not first-class citizens in data land, in the sense that Linked Data documents contain links to folders and documents, but not to pods.

[TallTed]

It seems there may be a conceptual blur here between software-space (which may need to care about multiple pods in order to collect all relevant data) and user-space (which should not typically present pods per se).

In other words, from the user's perspective, it should be project- and/or person-based ... but in order for the software to present a unified project view to the user, with all tasks/people in that presentation, the software may need to federate information which has been stored on multiple pods (e.g., each person's tasks might be written to their pod, and not to the project's host-pod) -- so the presenting software must discover those pods and people somehow.

[jeff-zucker]

Isn't @TallTed's case handled by storing a list of webIDs of people on the project with federation consisting of looping through the webIDs, retrieving their profiles, finding their solid:account and visiting the projectname/tasks/ container relative to the account?

[balessan]

@RubenVerborgh : that is a truly important assertion.
@TallTed : You're exactly pointing out our issue here.

I drafted a use case, user oriented. I would define it as follows:

• Alice is a web developer living in Paris
• She is working as a freelancer with Happy Dev Paris (first organization)
• She is managing her tasks on the project Y through the Happy Dev Paris web application at https://paris.happy-dev.fr
• Her personal data pod is located at https://alice.com/ with her WebID installed at https://alice.com/#me
• She is working on project Y with Bob, who is a UX designer living in Nantes
• Bob is managing his tasks on the project Y through the Happy Dev Nantes web application located at https://nantes.happy-dev.fr
• His personal data pod is located at https://nantes.happy-dev.fr/bob/ with his WebID installed at https://nantes.happy-dev.fr/bob/#me

In this use case I am making the assumption that the Project Y is created on the https://paris.happy-dev.fr/ application at https://paris.happy-dev.fr/projects/y/#this

How can this project, after its creation on the https://paris.happy-dev.fr web application, be made available on https://nantes.happy-dev.fr through project lists ?

How can Alice, when creating this project, first describe that Bob is also a member of this project if there is no kind of federation between https://paris.happy-dev.fr/ and https://nantes.happy-dev.fr/ ?

Important to note that Alice WebID won't refer to Bob WebID as this is their first interaction.

Is What @jeff-zucker says that we should actually switch from a project oriented view from a user oriented view where belonging to a project is technically creating a project-y/ container to the user data pod (or chosen location) and being given the capabilities to edit information, create tasks and so on ?

A really more naive use case would be:

• Happy Dev is made of several independent organizations. Each one of them stores a list of their members (WebIDs) in a /members/ container
• Alice, being a member of Happy Dev Paris, wants to have a list of all the members of all the organizations being part of Happy Dev, and being able to search through it, contact someone, etc...

Is there an example of solid-app implementing the use case of being part of an organization, including generation of the needed containers somewhere ?

Lots of questions there, thanks in advance for your answers,

PS : this issue should probably have been created on solid/solid as it is really more general than the spec. Let me know if there is a better place to discuss this topic.

[RubenVerborgh]

My take

• Her personal data pod is located at alice.com

• His personal data pod is located at nantes.happy-dev.fr/bob

More specifically:

• Alice's projects will be located at https://alice.com/projects/happy-dev/
• Bob's projects will be at https://nantes.happy-dev.fr/bob/

So note how we are not identifying pods, but rather a folder of projects.

How can this project, after its creation on the paris.happy-dev.fr web application, be made available on nantes.happy-dev.fr through project lists ?

Two broad options:

• nantes.happy-dev.fr can link to individual projects at https://alice.com/projects/happy-dev/1 and https://nantes.happy-dev.fr/bob/project2/
  • The list of projects is updated by a PATCH append, or through Linked Data Notifications.
• nantes.happy-dev.fr can link to project folders https://alice.com/projects/happy-dev/ and https://nantes.happy-dev.fr/bob/; clients extract the projects from there. This is a kind of symlink (to a folder, not a pod).

Note that these are questions of shapes: is a project folder a flat folder, or can it link to other folders?

How can Alice, when creating this project, first describe that Bob is also a member of this project if there is no kind of federation between paris.happy-dev.fr and nantes.happy-dev.fr ?

Of what project? I'd need a more specific description to answer, but my general answer would be that Alice just states that Bob is a member as an RDF triple in the project.

Important to note that Alice WebID won't refer to Bob WebID as this is their first interaction.

Can you perhaps sketch a more precise timeline then? Not sure I get the order of steps.

• Happy Dev is made of several independent organizations. Each one of them stores a list of their members (WebIDs) in a /members/ container

That is possible today, right?

• Alice, being a member of Happy Dev Paris, wants to have a list of all the members of all the organizations being part of Happy Dev, and being able to search through it, contact someone, etc...

That would also be possible, given the document you describe right above?

---

In summary: I don't see the necessity to identify pods or a federation; quite the contrary, all of the above cases can seemingly be solved by linking domain resources (project, member, …) together.

[sylvainlb]

Actually there's no need to identify pods, that was a misunderstanding. It could be needed if we wanted to use some kind of discovery, but that's a different subject.

The need here is to be able to list all the projects of all the pods that are referenced by Happy Dev. There needs to be a way for Happy Dev to make a list of all the projects containers that contains projects referenced by Happy Dev.

The implementation we thought of was a container on https://ldp.happy-dev.fr/projects/ that contains a list of source of projects, which are themselves containers containing projects.

Now the question is: what is the best organization for that? Should sub containers be identified with a special @type? Should be in a project container alongside other resources like projects, or should they reside in a dedicated container?

We have several use cases among our clients where a trusted server hosts a federated lists of all "allowed" resources of third party entities. We think that whatever implementation we choose, we want to make sure it's seen as the best practice to be sure other editors would do it the same way.

[RubenVerborgh]

Now the question is: what is the best organization for that? Should sub containers be identified with a special @type? Should be in a project container alongside other resources like projects, or should they reside in a dedicated container?

I think there is no general answer to this.

The crucial thing is that everything is linked up, so that a client can find its way from one place to another. That always needs to happen, regardless of structure.

How data is structured would be determined by shapes and footprints. An early draft is available here: https://www.w3.org/DesignIssues/Footprints.html; I expect to release a blog post with more details on such mechanisms by the end of the week.