Improving API Reference format

[hildeweerts]

I think our current API Reference could benefit from restructuring. Some ideas:

Structure API Reference in tables

The current format puts the documentation of all functions/classes of a module on a single page. As the pages are getting quite long and often contain different types of functionality (e.g., "base" metrics versus fairness metrics in the metrics module), it can be difficult to get a good overview of the available functionality. This also makes it harder than necessary to find the documentation of a particular function/class.

Instead, I'd suggest to list the functions/classes in tables with a short description of each item. We can put the actual documentation on a separate page for each function/class. We can put all tables on a single page, similar to scikit-learn or on a separate page for each module, similar to pandas. My personal preference would go to the latter.

Ordering Attributes/Methods

The attributes and methods of a class are currently ordered alphabetically.

I'd suggest to group them separately instead, similar to e.g., this or this

Make References Consistent

The usage of references seems inconsistent. In fetch_boston() the References docstring is used, in fetch_adult() the reference (including the full url and explicit 'Source:') is in the description, in ThresholdOptimizer a reference is linked as a clickable name (year) in the Notes, and all functions/classes in the reductions module link the reference as a clickable name (year) in the description.

I'd suggest to follow the docstring guide and put implementation details/background theory in the Notes and proper references in the References (i.e. not just links, as they might break).

See Also

I always like the See Also sections of classes/functions, as they can lead you to different parts of the package you may not have considered yourself, especially if you're new to the field. This is probably something that's not as relevant with the current size of the package, but might be nice to consider as Fairlearn grows.

---

Do other people have thoughts on this? Tagging @riedgar-ms @MiroDudik @romanlutz

[romanlutz]

Couldn't agree more! I haven't looked into how scikit-learn achieves this yet, but I'm hoping that it's compatible with our docs generation setup.

Would you like to try this out @hildeweerts ? If not I'll mark it as "help-wanted".

[riedgar-ms]

Thanks for laying out your concerns @hildeweerts ! As we discussed in the metrics meeting, there's certainly plenty of space for improvements in the current docs.

[hildeweerts]

I'd be happy to take a stab, but if you know somebody (e.g., a prospective contributor?) who'd like to try out one of these that's totally fine with me as well!

[romanlutz]

I'm not aware of anyone who's looking for tasks right now. If you are interested in trying please do :-) The doc generation is a little quirky if you haven't used sphinx before, so please do reach out if you have questions. Both @riedgar-ms and I have worked with it extensively.

[riedgar-ms]

Something else I'll throw in for the references - it would be nice to use BibTeX .bib files. I did find some support in Sphinx, but.... it is a bit patchy compared to the real thing.

[hildeweerts]

Split issue into smaller components in #719 and #720