-
Notifications
You must be signed in to change notification settings - Fork 502
Dbusinterface
Prior to V2.0, NFS Ganesha used both SNMP and a simple TCP socket interface to export statistics. There was no control interface other than a few UNIX legacy signals that would trigger events like a re-reading of the configuration file. Although UNIX signals remain for some backwards compatibility, both of these management and statistics interfaces have been replaced by a DBUS server.
DBUS is a system local IPC mechanism for system management and peer-to-peer application communication. This page documents the NFS Ganesha DBUS service. For the details and documentation for DBUS itself, see the DBUS Project site.
- NOTE
- The d-feet tool is a very useful GUI based DBUS messaging debugger/tool. Using it in conjunction with the documentation below makes all this much more understandable. It is available as a package on most Linux distributions.
There is no configuration required on the NFS Ganesha server for it to connect to as use the DBus. The system libraries for DBus know where the DBus server is located. However, the DBus server must be made aware of NFS Ganesha. This section describes how to configure the DBus server to recognize and accept connections from NFS Ganesha.
The configuration file is located in src/scripts/ganeshactl/org.ganesha.nfsd.conf. It is an XML file that defines the org.ganesha.nfsd bus name and the permissions for accessing or connecting to it. The current permissions only allow programs or scripts run as the superuser to access NFS Ganesha.
The following instructions for configuring DBus to support NFS Ganesha assume that the system is Fedora or RHEL based. Other Linux distros and FreeBSD may be different. Consult your distribution's filesystem and documentation for the details of file and directory locations.
- Copy the file
src/scripts/ganeshactl/org.ganesha.nfsd.confto the directory/etc/dbus-1/system.d/. The DBus server ('messagebus') needs to be restarted to be able to load this file. - Start NFS Ganasha. Note that the connection to the DBus server is only made once at server initialization time so if you forgot to copy the file first you will have to restart NFS Ganesha.
- Check the server log, normally at
/var/log/nfs-ganesha.log. There should be a log message in the middle of the initialization messages indicating that the DBUS thread has started. If the message otherwise indicates that the thread terminated, this is probably related to the DBus server rejecting its connection attempt. Follow the advice in the error message. - As superuser, use either one of the supplied scripts in
src/scripts/ganeshactlor one of the standard DBus client tools tto verify that everything is connected. If you use d-feet, described below, you should see at least one/org/ganesha/nfsd services depending on what was included in the server's build. Recheck the server log and the system's message log if it is not present. d-feet is the best tool for this because it displays all DBus activity and updates itself in realtime whereas the supplied scripts only access NFS Ganesha services. - Each script has at least one command line option to display/report a status. As superuser, use one of the scripts to display status.
- If you have gotten this far, the DBus interface is up and running.
DBUS IPC is a bi-directional exchange of messages between the client and an interface on a particular object path in the service.
The DBUS specification defines a set of reserved common interfaces that clients can expect from a service. Two of these reserved interfaces are used in NFS Ganesha:
- org.freedesktop.DBus.Introspectable
- This interface has one method Introspect which returns an xml data string that describes all of the other interfaces and their methods for the particular object path. Every object path in NFS Ganesha's server provides this interface.
- org.freedesktop.DBus.Properties
- This interface is for setting and retrieving key/value pairs of properties. NFS Ganesha currently does not supply this interface yet. When it does, it will be used for setting configuration parameters in the running server such as log component levels.
| Name | Type | Details |
|---|---|---|
| status | boolean | true if method/command succeeded, false otherwise |
| error | string | OK if status == true, a people readable error message otherwise |
| time | struct |
The timestamp of the data in struct timespec format. (optional)
|
| ... | stuct or array | data payload (optional) |
Only the status and error fields are mandatory. The client is expected to use status for error checking and error for user display in GUI things like status bars.
The time reply is in nanosecond resolution. See below for how this is used for statistics calculations. It is optional but always present for statistics reporting.
- NOTE
- The introspect does not show a name for fields below the first level. It is a limitation in the DBUS spec. Client programs would be able to decode the "shape" of the row from the introspect and from the DBUS message parsing but DBUS provides no further help. The client application developer is forced to consult here, and as the final authority, the server code itself.
NFS Ganesha owns the org.ganesha.nfsd bus name. Any client application or system control service connects to this bus name to access the server.
The functions within the DBUS service are implemented as separate object paths dedicated to a specific subsystem of the server. Version 2.0 supports the following paths:
- /org/ganesha/nfsd/ClientMgr
- This is used for managing and reporting statistics for clients.
- /org/ganesha/nfsd/ExportMgr
- This is used for managing and reporting statistics for file system exports.
- /org/ganesha/nfsd/admin
- This is used to administer the server itself.
I/O statistics are reported below by client and by export use the same structures in their reply. There are two structures, one for I/O through the server and a second for NFSv4.1 layouts.
The following structures report I/O statistics. This applies to all protocols. In the NFSv4.1 case, these transfers are for reads and writes performed by the medadata server itself. The first table describes the reply for each statistics report separated out as read and write counts. The second table describes the statistics details for each type.
| Name | Type | Details |
|---|---|---|
| status | boolean | true if there are statistics, false if there is no activity |
| error | string | "OK" or "No activity..." |
| time | struct | Timestamp in nanoseconds when sampled (now) |
| read | struct | I/O struct described below |
| write | struct | I/O struct described below |
This is the I/O structure.
| Name | Type | Details |
|---|---|---|
| requested | uint64_t | bytes requested |
| transferred | uint64_t | actually tranferred |
| total | uint64_t | Total number of operations |
| errors | uint64_t | Number of operations that reported errors |
| latency | uint64_t | cumulative time consumed by operation in nanoseconds |
| queue wait | uint64_t | cumulative time spent in the rpc wait queue |
NFSv4.1 (pNFS) delegates I/O to transactions directly between the client and the data server or OSD. The following structure is used to report those statistics.
| Name | Type | Details |
|---|---|---|
| status | boolean | true if there are statistics, false if there is no activity |
| error | string | "OK" or "No activity..." |
| time | struct | Timestamp in nanoseconds when sampled (now) |
| getdevinfo | struct | get device info operations |
| layout_get | struct | layout get operations |
| layout_commit | struct | layout commit operations |
| layout_return | struct | layout return operations |
| layout_recall | struct | layout recalls by the server |
This is the layout statistics sub-structure for each of the layout operations above.
| Name | Type | Details |
|---|---|---|
| total | uint64_t | Total layout operations |
| errors | uint64_t | cumulative number of layout errors |
| delays | uint64_t | cumulative delay time in nanoseconds |
The statistics are accumulated and reported in a way that places a minimum of overhead on the server because each of these counters is in the main code path of every operation of the server. All counters and timers are cumulative from the start time of the server and use the atomic operations so that any individual counter is thread safe. However, this does not guarantee that all the counters in a set are guaranteed to be atomic relative to each other. This is not really an issue however because a single increment or added time one way or another is a minor error when calculating rates.
All the counters are cumulative from the start of the server. All the counters are also unsigned 64 bit integers, including the count of nanoseconds. Internal times are an unsigned 64 bit count of nanoseconds since server start which should be more than enough given that 65 bits of nanosecond precision will not be needed for at least 584 years. Every DBUS reply also contains a timestamp of nanosecond resolution. Rates are calculated by subtracting and old count from the new count and dividing the result by the interval caculated as the difference between the old timestamp and the new one followed by the usual normalization math to calculate ops/bytes per (second, minute, day, century).
The last time value is significantly different from the time value in statistics reply. It is just a timestamp recorded by the server whenever a particular counter set is updated. Its primary use is to simply record the last time when the server executed an operation of that type for either a particular client on a particular export.
The client manager is responsible for the client list in the server. Clients are referenced by IP address. IPv4 and IPv6 addresses are managed separately even in the case where a client may be accessing the server by both protocols. NFS Ganesha really has no way of distinguishing the two anyway and system administrators would probably want to know the nature of traffic in each protocol version. However, it would be an exercise left to the administrator to match up the addresses.
The client manager has an introspectable interface that is generated for all its interfaces and methods.
This interface is used to manage clients. It currently has three methods.
- AddClient
- This method takes an IP address as a formatted string and adds it to the client manager. The address can either be an IPv4 formatted address or an IPv6 formatted address. Each address is treated individually so that clients that use both protocols are treated separately. The reply has two fields, the status and error fields described above.
- RemoveClient
- This method takes an IP address as a formatted string and removed that entry from the client manager. Its reply also has the status and error fields. One side effect of this method is that all storage for statistics for this client are freed. Should the entry be re-created, all the counters will start at 0 (zero).
- ShowClients
- This method returns an array of structures which describes that statistics state for each client known to the server. The reply is of the following form:
| Name | Type | Details |
|---|---|---|
| time | struct |
The timestamp of the data in struct timespec format. (optional)
|
| clients | array | Array of client flags. See the clients table below |
The clents table is an array of structs where each row describes the statistics state of a client.
| Name | Type | Details |
|---|---|---|
| client | string | The IP address of the client as it is known by the client manager |
| nfsv3 | boolean | true if there has been any NFSv3 activity |
| mnt | boolean | true if there has been any MNTv3 activity |
| nlm4 | boolean | true if there has been any NLMv4 activity |
| rquota | boolean | true if there has been any RQUOTA activity |
| nfsv40 | boolean | true if there has been any NFSv4.0 activity |
| nfsv41 | boolean | true if there has been any NFSv4.1 activity |
| nfsv42 | boolean | true if there has been any NFSv4.2 activity |
| 9P | boolean | true if there has been any 9P activity |
| last time | struct |
The timestamp of the last data update in struct timespec format.
|
There are two timestamps in this reply, see below for the meanings of time and last time.
There are currently four methods defined. Statistics for the 9P protocol have not been defined yet.
Each method takes a single string ipaddr argument. This is the ipaddr field returned in the array returned by the ShowClients method above.
Each statistics method returns its own relevant data
This method reports I/O statistics for the NFSv3 protocol per client. Total operations counts are accumulated but only I/O statistics are reported at this time. See the I/O operations structure above for details.
This method reports I/O statistics for the NFSv4.0 protocol per client. Its reply format and semantics is identical to the NFSv3 reply.
This method reports I/O statistics for the NFSv4.1 protocol per client. Although its format is identical to the other protocols, its semantics are significantly different. These counts are for I/O done by the meta-data server not a rollup of statistics for dataservers or OSDs. Those statistics must be obtained directly from the the dataserver and summed for full server throughput calculations.
This method reports pNFS specific operations per client. These counters represent meta-data server operations and as such should equal the sum of the operations reported by the data servers for that client.
The export manager is responsible for the exports list in the server. The exports list is generated from the export blocks defined in the configuration file.
The export manager has an introspectable interface that is generated for all its interfaces and methods.
This interface is used to manage exports. There are currently four methods defined in this interface. All of them use the DBus status handling.
- AddExport
- This method takes two arguments. The first argument is the path name in the server's local filesystem for the configuration file to be processed. The second argument is a string that is a search expression that describes the block to be processed.
- RemoveExport
- This method takes a single integer argument that identifies the export by its ID parameter. A future version of this method may take an expression similar to the AddExport method.
- UpdateExport
- Similar to AddExport, this method takes two arguments. The first argument is the path name in the server's local filesystem for the configuration file to be processed. The second argument is a string that is a search expression that describes the block to be processed.
- NOTE:
- This UpdateExport feature was implemented in V2.4
- DisplayExport
- This method takes a single integer argument that identifies the export by its ID parameter. It returns four values described in the table below.
- ShowExports
- This method takes no input arguments. It returns an array of exports as described below.
A search expression is used to identify an export in the configuration file. This is the first use of these configuration file search expressions but this can be applied to any block in a parsed configuration. The expression uses the following form. White space is ignored. Only the tokens are important.
BlockName ( ParameterName = Value )
The BlockName is any valid block name recognized by the configuration processor, in this case EXPORT. The ParameterName can be any parameter defined within the block. It is best to choose one that would uniquely identify the block. In the export case, any one of the returned values of a DisplayExport method would work. The Value is either a "*" for a wildcard (match anything) or a value that is compared to the corresponding value set in the block.
This is the simplest expression. This can be extended in two ways. The first is a comma separated set of parameters.
ParameterOne = Value1 , ParameterTwo = Value2 , ...
Each of these expressions is taken from left to right and the , is equivalent to a logical AND. In other words, if ParameterOne equals Value1 and ParameterTwo equals Value2, then the whole list is true which means that the block is selected. If any are not equal, starting with the first, then the block is not selected.
The second case applies to sub-blocks. In this case, we are selecting a sub-block in the configuration and the match parameters for both must select. This case looks like,
BlockName ( some-set-of-parameters ) . SubBlockName ( subblock-paremeters ) . ...
If the parameters of BlockName match, the search continues with its sub-block SubBlockName. This, like the parameter set, can proceed to an arbitrary depth of sub-blocks.
- NOTE:
- This selection feature is currently, as of V2.1, only applicable to export blocks but this lookup function and the expression processing can be applied to any configuration block.
The following table describes the result returned by this method.
| Name | Type | Details |
|---|---|---|
| id | integer | The export ID |
| fullpath | string | The path to the root point in the backend filesystem to be exported |
| pseudopath | string | The path in the pseudo filesystem where the fullpath is to be "mounted". i.e. the junction |
| tag | string | An alternate export name to be used by NFSv3 mount requests. It is similar to pseudopath except that it is a token symbol, not a path. |
The following two tables describe the results returned by this method. There is one array row for each export active in the system.
| Name | Type | Details |
|---|---|---|
| time | struct |
The timestamp of the data in struct timespec format. (optional)
|
| exports | array | Array of export flags. See the exports table below |
- NOTE
- The interface does not have a status or error field. It will always return time but it can return an empty exports array which indicates that there are no exports defined in the server.
| Name | Type | Details |
|---|---|---|
| export id | int32 | The export id defined in the configuration file |
| path | string | the exported path root |
| nfsv3 | boolean | true if there has been any NFSv3 activity |
| mnt | boolean | true if there has been any MNTv3 activity |
| nlm4 | boolean | true if there has been any NLMv4 activity |
| rquota | boolean | true if there has been any RQUOTA activity |
| nfsv40 | boolean | true if there has been any NFSv4.0 activity |
| nfsv41 | boolean | true if there has been any NFSv4.1 activity |
| nfsv42 | boolean | true if there has been any NFSv4.2 activity |
| 9P | boolean | true if there has been any 9P activity |
| last time | struct |
The timestamp of the last data update in struct timespec format.
|
There are two timestamps here. The time field for the message is the current "now" time of the reply. The last time is the the timestamp of the last statistics update. If, for example, the last time there was any activity on this particular export was yesterday, time would be one day later than last time.
The booleans are used to select relevant statistics. Each boolean is associated with a method described below. If the boolean is false, its associated statistics method will return an "Export does not have any <protocol></protocol> activity" error. The client developer should first call the ShowExports method and then use the results to only call statistics methods that actually have something to report.
There are currently four methods defined. Statistics for the 9P protocol have not been defined yet.
Each method takes a single 32 bit integer export id argument. This is the export id field returned in the array returned by the ShowExports method above.
Each statistics method returns its own relevant data
This method reports I/O statistics for the NFSv3 protocol per export. Total operations counts are accumulated but only I/O statistics are reported at this time. See the I/O operations structure above for details.
This method reports I/O statistics for the NFSv4.0 protocol per export. Its reply format and semantics is identical to the NFSv3 reply.
This method reports I/O statistics for the NFSv4.1 protocol per export. Although its format is identical to the other protocols, its semantics are significantly different. These counts are for I/O done by the meta-data server not a rollup of statistics for dataservers or OSDs. Those statistics must be obtained directly from the the dataserver and summed for full server throughput calculations.
This method reports pNFS specific operations per export. These counters represent meta-data server operations and as such, the sum of the counters for all pNFS exports should equal the sum of the operations reported by the data servers for those exports.
This is the object path for server administration. There is an introspection interface that is generated for all methods and methods. Some interfaces have properties which are also introspectable.
This interface is used to manage overall server operation. All these methods return the usual status and error reply.
- grace
- This method takes a single IP address as its argument. Calling this method (re)starts the GRACE period in the server for state and locks for the named IP address. This is used to manage server clusters.
- reload
- This method (currently) takes no arguments. This is equivalent to sending the server a SIGHUP signal.
- shutdown
- This method (currently) takes no arguments. This is an orderly shutdown that releases all resources including backend fileserver state.
This interface controls the server's logger. This is the interface that the administrator's tools would use to change logging levels to aid the diagnosis of an event. The default level for most components is NIV_EVENT. The administrator can use COMPONENT_ALL to quiet all logging by setting it to NIV_FATAL. This will disable the reporting of all messages of less severity. The administrator can then set individual components of interest all the way to NIV_FULL_DEBUG to get every message for those components.
There are no methods currently defined. The properties have the Get, Set, and GetAll methods defined.
The logging subsystem reports events and errors to a log destination. The volume of log messages is managed by defining logging components for most of the subsystems in the server. This allows the administrator to focus log reporting on specific parts of the server. The log messages in each component have a logging severity value assigned to them to indicate the significance of the message. The following table of properties is used to control the components:
| Name | Description |
|---|---|
| COMPONENT_ALL | This is used to set the level for all components |
| COMPONENT_LOG | The logging subsystem itself is this component. |
| COMPONENT_LOG_EMERG | Emergency messages within the logging system |
| COMPONENT_MEMALLOC | Memory allocator errors, (unused) |
| COMPONENT_MEMLEAKS | Detected memory leaks |
| COMPONENT_FSAL | FSAL internal messages |
| COMPONENT_NFSPROTO | NFS (mainly NFSv3) protocol events and errors |
| COMPONENT_NFS_V4 | NFSv4 specific events and errors |
| COMPONENT_NFS_V4_PSEUDO | Pseudo filesystem (for NFSv4) events and errors |
| COMPONENT_FILEHANDLE | File handle encode/decode events and errors |
| COMPONENT_DISPATCH | NFS RPC and protocol operations decode and dispatch |
| COMPONENT_CACHE_INODE | Meta-data cache events and errors |
| COMPONENT_CACHE_INODE_GC | Meta-data cache garbage collect events and errors (unused) |
| COMPONENT_CACHE_INODE_LRU | Meta-data cache LRU processing events and errors |
| COMPONENT_HASHTABLE | Hashtable function events and debugging |
| COMPONENT_HASHTABLE_CACHE | Hashtable cache events |
| COMPONENT_LRU | (unused) |
| COMPONENT_DUPREQ | Duplicate NFS request processing events |
| COMPONENT_RPCSEC_GSS | GSS security events |
| COMPONENT_INIT | Server initialization and early startup events and errors |
| COMPONENT_MAIN | Administration functions and main server events |
| COMPONENT_IDMAPPER | Client ID mapping function errors and events |
| COMPONENT_NFS_READDIR | Readdir processing errors and events |
| COMPONENT_NFS_V4_LOCK | NFSv4 file locking, state, and delegation events |
| COMPONENT_NFS_V4_XATTR | (unused) |
| COMPONENT_NFS_V4_REFERRAL | (unused) |
| COMPONENT_MEMCORRUPT | (unused) |
| COMPONENT_CONFIG | Configuration file processing errors |
| COMPONENT_CLIENTID | NFSv4 client id handling errors and events |
| COMPONENT_STDOUT | (unused) |
| COMPONENT_SESSIONS | NFSv4.1 session id handling errors and events |
| COMPONENT_PNFS | pNFS errors and events |
| COMPONENT_RPC_CACHE | (unused) |
| COMPONENT_RW_LOCK | Read/write lock debugging |
| COMPONENT_NLM | NLM locking protocol events and errors |
| COMPONENT_RPC | RPC dispatch and processing errors and events |
| COMPONENT_NFS_CB | NFSv4 callback events and errors |
| COMPONENT_THREAD | General thread create and managment errors and events |
| COMPONENT_NFS_V4_ACL | NFSv4 ACL processing and access checking events |
| COMPONENT_STATE | General state abstraction layer errors and events |
| COMPONENT_9P | 9P protocol processing events and errors |
| COMPONENT_9P_DISPATCH | 9P protcol operations dispatch events and errors |
| COMPONENT_FSAL_UP | FSAL upcall to core server events and errors |
| COMPONENT_DBUS | DBus server and message processing events |
Each component has a level argument for the Set method and returns a level for the Get and GetAll methods. Either the name or the short name can be used as an argument to the Set. The level name is always returned by a Get or GetAll. The following table describes these levels and their use. The entries are in severity order, namely, if the level is NIV_EVENT any message with a level from NIV_FATAL through NIV_EVENT will be logged but any NIV_INFO through NIV_FULL_DEBUG will not be recorded.
| Level Name | Short Name | Description |
|---|---|---|
| NIV_NULL | NULL | Null level, no action (used internally to logger) |
| NIV_FATAL | FATAL | Very bad thing happened, server will exit |
| NIV_MAJ | MAJ | Major error. Server may not continue depending on error |
| NIV_CRIT | CRIT | Critical error. |
| NIV_WARN | WARN | Warning message, not fatal |
| NIV_EVENT | EVENT | Reporting a significant non-error event (default) |
| NIV_INFO | INFO | General information of less importance than an EVENT |
| NIV_DEBUG | DBG | Debug message for diagnosing server software problems |
| NIV_MID_DEBUG | M_DBG | More detailed debug message |
| NIV_FULL_DEBUG | F_DBG | Very verbose debug message |
A fatal error exits the server in the log subsystem. Major and critical errors can also cause the server to exit some time after the error occurs, usually after further error reporting and cleanup.