Full documentation of this API

[piwo1984]

Our goal is to build up a context driven IoT platform for energy management.
For this I'm trying to understand the NGSIv2 API for a couple of days now. But this is very difficult. Every example I can find (including the step by step guide) lacks some important points.
The main problem (for now) is that I have to register serveral devices, which are able to execute commands. Until now I haven't found any description/example how to do this.
What I've understand so far is:

• the devices will act as context-provider
• for this I have to implement kinds of iot-agents (transform data from/to context-broker and device)
• these iot-agents have to provide the REST endpoints /v2/op/update and /v2/op/query
• on startup they have to use the /v2/registration endpoint of the context-broker to self register
• ....

... and there isn't any description of the registration payload. I don't know how to notify the context-broker about: "Hey CB. I'm device .... I'm able to execute these commands: ...."

Can you please update your documentation/examples.

[fgalan]

Can we share which documentation references (links) have you been using so far, please?

[piwo1984]

Sure:

• https://swagger.lab.fiware.org/?url=https://raw.githubusercontent.com/Fiware/specifications/master/OpenAPI/ngsiv2/ngsiv2-openapi.json#/Registrations/Create%20Registration
• https://fiware-orion.readthedocs.io/en/master/user/context_providers/index.html
• https://fiware-orion.readthedocs.io/en/master/user/ngsiv2_implementation_notes/index.html
• https://fiware-tutorials.readthedocs.io/en/latest/context-providers/index.html
• https://fiware-tutorials.readthedocs.io/en/latest/iot-sensors/index.html
• https://github.com/telefonicaid/iotagent-node-lib

[fgalan]

So, as far as I understand, your goal is to build a system that acts as an IOT Agent. In that case, I'd recommend you to use the IOTA Lib. Note that library implements the IOTA-CB interface (northbound) so you will get "for free" all that stuf about registrations and /v2/op/update and /v2/op/query endpoints implementation.

There is a document on How to build and IOTA using the library in the library repository itself and probably more tutorials and documentation about it (not sure... I'm CCing @jason-fox in the case he knows more references on this).

[jason-fox]

A simple custom IoT Agent with working commands is described in this tutorial: https://github.com/FIWARE/tutorials.Custom-IoT-Agent/tree/master

There is an up-and-coming webinar on customizing IoT Agents scheduled for the beginning of March 2021: https://www.fiware.org/community/fiware-webinars/

[piwo1984]

Thanks for this hints.
Yes, in our project we have to rebuild a custom IoT Agent. For several reasons we can not use the IoT library you are providing.
And for this I was searching for a concrete documentation about the communication between context-broker and iot-agent (and vice versa)

[jason-fox]

For NGSI-v2 you will use the /v2/registrations endpoint as described here: https://github.com/FIWARE/tutorials.Context-Providers/tree/master#seven-request

The IoT Agent is then listening on /v2/op/update an example of processing this can be found here: https://github.com/telefonicaid/iotagent-node-lib/blob/master/lib/services/northBound/contextServer-NGSI-v2.js#L55

For NGSI-LD using Orion-LD an actuation example using registration is described here: https://github.com/FIWARE/tutorials.LD-Subscriptions-Registrations/tree/master#forwarded-update

The final definition of NGSI-LD actuations within the ETSI specification is still being discussed.

[piwo1984]

I still need further help. Here is my simple example:
There is a shallow entity

{
    _id: {
        id: 'urn:ngsi-ld:GeoHeatStorage:001',
        type: 'GeoHeatStorage',
        servicePath: '/'
    },
    attrNames: [
        'heatStorageCapacity'
    ],
    attrs: {
        heatStorageCapacity: {
            type: 'Number',
            creDate: 1611568405,
            modDate: 1611568405,
            value: 0,
            md: {
                unitCode: {
                    type: 'Text',
                    value: 'E14'
                }
            },
            mdNames: [
                'unitCode'
            ]
        }
    },
    creDate: 1611568405,
    modDate: 1611568405,
    lastCorrelator: '2b146c10-5ef3-11eb-8013-0242ac120003'
}

For regsitering a context-provider/iot-agent I used this:
POST /v2/registrations

{
  "description": "Current level of heat storage",
  "dataProvided": {
    "entities": [
      {
        "id": "urn:ngsi-ld:GeoHeatStorage:001",
        "type": "GeoHeatStorage"
      }
    ],
    "attrs": [
      "currentHeatStorageLevel"
    ]
  },
  "provider": {
      "http": {
          "url": "http://iot-agent:1880"
       },
            "supportedForwardingMode": "all"
  },
        "status": "active"
}

The resulting document within the registrations collection is:

{
    _id: ObjectId('600ec9586bcbafe46366999d'),
    description: 'Current level of heat storage',
    expiration: NumberLong(9223372036854775807),
    servicePath: '/',
    contextRegistration: [
        {
            entities: [
                {
                    id: 'urn:ngsi-ld:GeoHeatStorage:001',
                    type: 'GeoHeatStorage'
                }
            ],
            attrs: [
                {
                    name: 'currentHeatStorageLevel',
                    type: ''
                }
            ],
            providingApplication: 'http://iot-agent:1880'
        }
    ],
    status: 'active',
    format: 'normalized'
}

Know I query for this entity GET /v2/entities/urn:ngsi-ld:GeoHeatStorage:001 and get back what is within the entities collection. In the Logs I can see that the context-broker tries to access something with the correlator id (2b146c10-5ef3-11eb-8013-0242ac120003). The response is 404 .
Of course there is an endpoint /op/query in the iot-agent. And now I stuck. I don't know how to debug what's going on. Is there any possibility to get the request the cb tries to execute?
Can you please provide a simple example (or extend the documentation) how the cb forwards the request to the context-provider/iot-agent?
But maybe there is also another issue. The registration document hasn't the same structure as yours (https://fiware-tutorials.readthedocs.io/en/latest/context-providers/index.html#context-provider-registration-actions)

[fgalan]

Which CB version are you using? How are you starting it (i.e. ps ax | grep contextBroker)?

[piwo1984]

I'm using the docker image fiware/orion:2.4.0 and started it with /usr/bin/contextBroker -fg -multiservice -ngsiv1Autocast -dbhost context-broker-db

[fgalan]

Could you upgrade to fiware/orion:2.5.2 and start Orion adding -logLevel INFO to the CLI? Orion 2.5.0 introduces a re-work of the log subsystem that could be very useful to debug your forwarding scenario.

Once you do that, please repeat your test and provide the trace associated to the GET /v2/entities/urn:ngsi-ld:GeoHeatStorage:001 operation, please.

[piwo1984]

The log entry for the forwarded POST is:

time=2021-01-26T11:53:58.346Z | lvl=INFO | corr=2c6fb206-5fcd-11eb-8763-0242ac120004; cbfwd=1 | trans=1611660827-537-00000000080 | from=172.18.0.1 | srv=<none> | subsrv=<none> | comp=Orion | op=logTracing.cpp[212]:logInfoFwdRequest | msg=Request forwarded (regId: 600ec9586bcbafe46366999d): POST http://iot-agent:1880/op/query, request payload (114 bytes): {"entities":[{"id":"urn:ngsi-ld:GeoHeatStorage:001","type":"GeoHeatStorage"}],"attrs":["currentHeatStorageLevel"]}, response payload (149 bytes): <!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Error</title>
</head>
<body>
<pre>Cannot POST //op/query</pre>
</body>
</html>
, response code: 404

This response is from the IoT Agent (based on Node-Red). In the log you can see that the response ist from another url (//op/query). Do you have any explanation where this additional slash comes from? There isn't any in the registration document. I also tried (with success) a curl to the target url from the bash of the running docker container.

[piwo1984]

Workaround:
Extend the url of the context-provider http://iot-agent:1880 => http://iot-agent:1880/v2

So I think the context-broker isn't able to query a context-provider on it's root url. You have to add a path.

[fgalan]

The URL used by Orion to forward requests are:

• Queries: <url as used in context registration> + "/op/query"
• Update: <url as used in context registration> + "/op/update"

Is this different from what you have observed in your tests?

[piwo1984]

As you can see in the example above I POSTed a registration with the root url (http://iot-agent:8080) for my iot agent. The agent is listening on http://iot-agent:1880/op/query and http://iot-agent:1880/op/query for POST request. I tested this.
But the context-broker do not just simply add "/op/query" and "/op/update" to the provided url. It adds also a leading "/" between url and "/op/query" or "/op/update". It tries to make requests to http://iot-agent:1880//op/query.
If I extend the url of the registration to http://iot-agent:1880/v2 the context-broker generates the correct url (http://iot-agent:1880/v2/op/query & http://iot-agent:1880/v2/op/update)

[fgalan]

What about if you use http://iot-agent:8080/ (with final slash) instead of http://iot-agent:8080 in your registration? Do CB adds double slash in that case?

(Just to fully consider all the possible cases of a potential bug in the CB)

[piwo1984]

Same result with or without trailing slash: response is 404 (Cannot POST //op/query)

[fgalan]

I have created an issue in Orion Context Broker repository with the specific issue: telefonicaid/fiware-orion#3768

Apart from that, anything else pending? Or can this issue (#13) be closed?

[piwo1984]

Please extend the documentation https://swagger.lab.fiware.org/?url=https://raw.githubusercontent.com/Fiware/specifications/master/OpenAPI/ngsiv2/ngsiv2-openapi.json#/Registrations/Create%20Registration to show all possible property payload.

[fgalan]

Please extend the documentation https://swagger.lab.fiware.org/?url=https://raw.githubusercontent.com/Fiware/specifications/master/OpenAPI/ngsiv2/ngsiv2-openapi.json#/Registrations/Create%20Registration to show all possible property payload.

That swagger document is out of my scope, but maybe @jason-fox could move this to the people in charge of that document.