Educational Exercises and Activities

[AdamSobieski]

Introduction

Hello. I am pleased to share some brainstorming towards advancing the state of the art with respect to educational exercises and activities, e.g., homework, quiz, and exam items and sequences of such items.

Both items and sequences of items could be like EPUB resources, containing their necessary multimedia resources (e.g., hypertext, fonts, stylesheets, scripts, metadata, images, animations, 3D models, audio clips) in OCF-based archives.

As considered, HTML- and EPUB-based digital textbooks could include more featureful items and sequences of items using <iframe> elements.

<iframe width="600" height="400" src="123.items" />

Motivating Use Cases

Motivating use cases include enabling educational resources (e.g., websites, digital books, and digital textbooks), with exercises and activities in them, which could:

1. not require any local or remote service to provide readers with adaptive or personalized item sequences.
2. optionally utilize one or more local or remote services.
3. require access to one or more local or remote services to function.
4. operate while offline, storing educational data, while expecting to connect to the Web at a later point.

Important advanced topics considered include:

1. accessing or utilizing local or remote services configured elsewhere, e.g., on operating systems or in Web browsers.
2. interprocess and interapplication communication, e.g., with intelligent tutoring systems.

Sequences of Items and JavaScript

Sequences of items could be static or they could be dynamic, varying per client-side or server-side program logic. While server-side item-scheduling approaches can retrieve both individual items and sequences of items from itembanks, considered here are scenarios where client-side logic is desired to schedule items for learners.

Resembling how EPUB's <spine> elements are used to describe sequences of pages in EPUB resources, inside of each OCF-based archive for an item sequence, there could be an index.js resource which utilizes a global variable to present, or to serve up, a sequence of HTML-based content resources.

Note: the following JavaScript examples use decorators for providing metadata on functions, e.g., @metadata, which are not yet a standard language feature. Below, the @metadata decorators refer to metadata resources in self-contained OCF-based archives.

The following example shows a simple sequence comprised of one item.

@metadata('meta1.n3')
function main()
{
    env.next('item1.html');
}

The following example shows a simple sequence comprised of four items.

@metadata('meta2.n3')
function main()
{
    env.next('item1.html');
    env.next('item2.html');
    env.next('item3.html');
    env.next('item4.html');
}

The following example shows a more complex, featureful, and dynamic sequence of items.

@metadata('meta4.n3')
function subroutine()
{
    if(env.ask('query2.sparql'))
    {
        env.next('item3.html');
    }
    else
    {
        env.next('item4.html');
    }
}

@metadata('meta3.n3')
function main()
{
    env.next('item1.html');

    for (item of env.query('query1.sparql'))
    {
        env.next(item);
    }

    env.next('item2.html');

    subroutine();
}

The following example shows using return values from the next() function which would block while awaiting the learner's completion of the item. Note that the object returned by the next() function would be implementation-dependent, being that argument passed to parent.env.completed() by an HTML-based item.

@metadata('meta5.n3')
function main()
{
    let result = env.next('item1.html');

    if(result.isCorrect)
    {
        env.xapi.send(...);
    }
    else
    {
        env.xapi.send(...);      
    }
}

The following example shows a more intricate sketch indicating that item-authoring teams could make use of remote services for determining whether learners' responses to items were correct. This would mean that the correct answers would not need to be encoded in and stored in item packages. Note that item packages which depend upon remote services would not be self-contained.

var remote;
var uuid;

@metadata('meta6.n3')
function main()
{
    remote = env.service.connect('service1.config');
    uuid = ...;

    if(remote.isConnected)
    {
        fun('item1.html');
        fun('item2.html');
        fun('item3.html');
        fun('item4.html');
    }
    else
    {
        ...
    }
}

@metadata('meta7.n3')
function fun(x)
{
    if(remote.isConnected)
    {
        let result = env.next(x);

        let response = remote.isCorrect(uuid, x, result);

        if (response.status != 'error')
        {
            if (response.payload == 'true')
            {
                env.xapi.send(...);
            }
            else
            {
                env.xapi.send(...);      
            }
        }
        else
        {
            ...
        }
    }
    else
    {
        ...
    }
}

Global Scripting Variables

As the window and document global variables are important for HTML-based JavaScript functionalities and features, global variables for educational exercises and activities, here env, would be important.

Observer Pattern

As envisioned, the global variable, here env, would support an observer pattern for serving up sequences of HTML-based items to learners, providing functions including next() and error().

Querying Itembanks

As envisioned, the global variable, here env, would support querying a remote itembank using SPARQL to retrieve sequences of HTML-based items, per query() and, possibly, would support a Boolean ask().

Experience API (xAPI)

As envisioned, the global variable for items and item sequences, here env, would support generating and streaming xAPI events.

See also: xAPI.js.

Accessing Parent Scripting Scopes from HTML-based Item Content

Per HTML's <iframe> scripting scopes and parent global objects, it should be possible to access the global objects of a parent item or item-sequence scope from inside of those HTML-based items served up by it.

Something resembling the following would be called when an HTML-based item was completed, e.g. indicating that the parent scope's sequence of items should advance. An argument passed to the completed() function could be that value returned by the invoked next() function.

parent.env.completed(...);

Something resembling the following could be used for items' user-interface events to generate and stream xAPI events.

parent.env.xapi(...);

Advanced Topics

Accessing or Utilizing Configured Local or Remote Services

One can envision learners being able to easily configure local and remote services with their operating systems or Web browsers, or once per digital book or textbook, and for all of the features – and those items in them – to subsequently just work.

Automated Planning and Scheduling

Advanced topics include providing support for automated planning and scheduling components to use data available on or from clients to sort, sequence, and schedule items for learners.

Intelligent Tutoring Systems

Advanced topics include features for items and sequences of items to bidirectionally synchronize with and interoperate with AI components, e.g., intelligent tutoring systems, attached to digital textbooks.

Interprocess Communication

I am wondering about how to enable more features and capabilities from the item API side. For instance, via items' APIs, intelligent tutoring systems would be able to perform tasks including, but not limited to:

1. obtaining natural-language descriptions of items, their contents, their objectives, and more, from items' metadata
2. pointing to and highlighting contents in items (e.g., in interactive 3D visualizations)
3. advancing or rewinding some items to described, rendered keyframes.

I am also wondering about how applications (e.g., Web browsers), resources loaded in them, and/or items displayed in these resources might be able to signal to interoperating tutoring systems that, instead of homework, quizzes or exams were underway.

Conclusion

Thank you. Per the WICG proposal process,

1. Submit a proposal outlining your idea.
2. Get feedback and improve your proposal.
3. Find collaborators and create a GitHub repository.
4. Work on your proposal and seek consensus from the community.
5. Advocate for adoption of your proposal to the W3C or the WHATWG for standardization.

I am looking forward to discussing and improving this preliminary proposal with your feedback and to finding interested collaborators to create fuller documents with which to spur innovation and to seek consensus from the community and stakeholders.

[marcoscaceres]

Hi Adam,

Thank you for your proposal. I wanted to point out that everything you're suggesting can already be implemented using JavaScript without requiring changes to HTML standards or additional browser support.

Accessing Global Objects in iframes

• Same-Origin Frames: Direct access to the parent’s global objects using window.parent.
• Cross-Origin Frames: Communication is possible using the postMessage API, adhering to the same-origin policy for security.

Custom Formats and SPARQL

• Custom data formats like .n3 and querying languages such as SPARQL can be handled entirely in JavaScript. This allows for parsing, querying, and manipulating data without the need for standardization or native browser support.

Fetching Network Resources

• The Fetch API provides a powerful and flexible way to make network requests and handle responses, supporting a wide range of use cases without requiring any additional browser features.

Security Considerations

• The same-origin policy is crucial for protecting sensitive data, and existing APIs like postMessage ensure secure cross-origin communication.

By leveraging JavaScript, you can achieve the functionality you're looking for without necessitating changes to existing HTML standards.

If there are specific scenarios where these capabilities seem insufficient, please share more details so we can explore them further.

[AdamSobieski]

It does appear that that initial set of features can be implemented via a set of JavaScript libraries.

I have since updated the proposal with a few new ideas: (1) obtaining settings and configurations from operating systems or Web browsers (e.g., education-related service endpoints, e.g., learners' schools' servers), and (2) education-related bidirectional interprocess communication scenarios, e.g., with intelligent tutoring systems.

The current conceptual model utilizes nested frames:

• Educational resource
  • Item collection and scheduling logic
    • HTML5-based item

Adding considered MIME types:

• Educational resource (application/xhtml+xml, possibly from inside of an EPUB archive)
  • Item collection and scheduling logic (application/????+zip)
    • HTML5-based item (application/xhtml+xml from inside of the item collection)

Any thoughts on the conceptual model?

As considered, collections of homework, quiz, and exam items could be gathered together into zip archives along with those multimedia resources utilized by them (e.g., hypertext, fonts, stylesheets, scripts, metadata, images, animations, 3D models, audio clips). Should this be a way to go, a feature for Web browsers would be to recognize packages' MIME type to unzip and load up their contents for display.

Another detail would be to ensure that nested items could be maximized to fully utilize content display areas, and/or that fullscreen was possible, for those HTML5-based items in nested frames. It may be the case that these features are already possible with some scripting logic.

I would like a bit more time to brainstorm to your question about Web browser feature ideas. Presently, ideas include: (1) obtaining learners' operating system or Web browser settings with respect to education-related service endpoints (e.g., education-related service endpoints, e.g., learners' schools' servers), and (2) education-related bidirectional interprocess communication scenarios, e.g., with intelligent tutoring system applications.

[marcoscaceres]

Any thoughts on the conceptual model?

Right, but what I was getting to was: what does the Web Platform not give you (as a primitive) to meet your requirements?

Conceptually, this can't be domain specific. The web generally only deals with generalized user cases, not, say "education use cases"... those may be covered generally, however.

(1) obtaining learners' operating system or Web browser settings with respect to education-related service endpoints (e.g., education-related service endpoints, e.g., learners' schools' servers), and

That would need to be weighed against user privacy. There is little reason to trust such institutions from a user's perspective - or for those institutions to trust themselves with such privacy sensitive data.

(2) education-related bidirectional interprocess communication scenarios, e.g., with intelligent tutoring system applications.

As with the first question: what can't you do over fetch, web sockets, or WebRTC or whatever?

[AdamSobieski]

That would need to be weighed against user privacy. There is little reason to trust such institutions from a user's perspective - or for those institutions to trust themselves with such privacy sensitive data.

Traditionally, to turn in homework assignments or to hand in completed quizzes or exams, learners have provided their teachers and schools with some educational data. Beyond completed sets of items, more modern, granular forms of educational data include, but are not limited to: timing data (how long did an item or each part of an item take a learner), items' user-interface event logs, and dialogue transcripts or event logs from intelligent tutoring systems.

Educational data, e.g., xAPI data, can be stored in learning record stores. Educational data can be processed and analyzed per educational data mining techniques.

According to Wikipedia, applications of educational data mining include: (1) the analysis and visualization of data, (2) providing feedback for supporting instructors, (3) recommendations for students, (4) predicting student performance, (5) student modeling, (6) detecting undesirable student behaviors, (7) grouping students, (8) social network analysis, (9) developing concept maps, (10) constructing courseware, and (11) planning and scheduling.

With respect to points 3, 4, and 5, there are open learner modeling and analytics to consider. In these approaches, learners can access, view, and be benefitted by their learner models, using this information to be able to better select and prioritize their practice activities.

On these topics, there are preschool, kindergarten, elementary school, middle school, secondary or high school, trade and vocational school, university, and recreational and lifelong learning scenarios to consider. The topics also span sectors. Beyond academia, there are also industry (e.g., business training), public sector (e.g., government personnel training), and military domains to consider.

Brainstorming to your point: there could be user permissions when learners first initialize their educational resources (e.g., websites, digital books, digital textbooks) and when they connect these to any remote services, including servers at their schools?

As with the first question: what can't you do over fetch, web sockets, or WebRTC or whatever?

I have also thought about WebRTC on these topics.

The following recent video shows multimodal language models seeing displayed items and learners performing on these items while simultaneously engaging in dialogue and answering questions: https://www.youtube.com/watch?v=IvXZCocyU_M .

Based on that video (which shows two desktop windows), I am thinking about client-side interoperability, e.g., interprocess communication, between Web browsers displaying educational resources (e.g., websites, digital books, digital textbooks) and intelligent tutoring systems to enable new features and capabilities.

[AdamSobieski]

what does the Web Platform not give you (as a primitive) to meet your requirements?

One thing that I'm hoping to discuss is enabling content authors and developers to be able to provide data and metadata for items within nested frames in Web browsers to external connected applications, e.g., intelligent tutoring systems, on clients.

Here are some more thoughts with respect to bidirectional interprocess and interapplication communication between Web browsers and other software applications.

Using the following variables:

var item_description = 'http://www.example.com/2024/#item-description';
var item_instructions = 'http://www.example.com/2024/#item-learner-instructions';
var item_objectives = 'http://www.example.com/2024/#item-educational-objectives';
var item_hints = 'http://www.example.com/2024/#item-hints';

and with something like:

window.exportData(item_description, 'text/plain', 'en', 'data:text/plain;base64,SGVsbG8sIFdvcmxkIQ==');
window.exportData(item_instructions, 'text/plain', 'en', 'data:text/plain;base64,SGVsbG8sIFdvcmxkIQ==');
window.exportData(item_objectives, 'text/plain', 'en', 'data:text/plain;base64,SGVsbG8sIFdvcmxkIQ==');
window.exportData(item_hints, 'text/json', 'en', 'data:text/json;base64,SGVsbG8sIFdvcmxkIQ==');

and/or:

window.exportData(item_description, 'text/plain', 'en', my_js_callback_1);
window.exportData(item_instructions, 'text/plain', 'en', my_js_callback_2);
window.exportData(item_objectives, 'text/plain', 'en', my_js_callback_3);
window.exportData(item_hints, 'text/json', 'en', my_js_callback_4);

external processes, e.g., intelligent tutoring systems, would be able to connect and detect available exported data and functions (per semantic identifiers and other content-negotiation data) and could choose to retrieve or invoke these.

From the perspective of external software applications, implementation particulars for obtaining exported data from Web browsers' tabs would depend upon the operating system. There would also be a matter of enabling external software applications to detect changes in exported data or functions of interest to them, e.g., when items were completed by learners and new items were presented to them.

With respect to ensuring that multiple exported data could be synchronized, e.g., that all of the available exported data refer to the same item, something like the following could be considered:

window.exportOpen();
window.exportData(item_description, 'text/plain', 'en', 'data:text/plain;base64,SGVsbG8sIFdvcmxkIQ==');
window.exportData(item_instructions, 'text/plain', 'en', 'data:text/plain;base64,SGVsbG8sIFdvcmxkIQ==');
window.exportData(item_objectives, 'text/plain', 'en', 'data:text/plain;base64,SGVsbG8sIFdvcmxkIQ==');
window.exportData(item_hints, 'text/json', 'en', 'data:text/json;base64,SGVsbG8sIFdvcmxkIQ==');
window.exportClose();

and/or:

window.exportOpen();
window.exportData(item_description, 'text/plain', 'en', my_js_callback_1);
window.exportData(item_instructions, 'text/plain', 'en', my_js_callback_2);
window.exportData(item_objectives, 'text/plain', 'en', my_js_callback_3);
window.exportData(item_hints, 'text/json', 'en', my_js_callback_4);
window.exportClose();

Below, the sketches are refactored to show possibilities:

window.interprocess.open();
window.interprocess.setExport(item_description, 'text/plain', 'en', 'data:text/plain;base64,SGVsbG8sIFdvcmxkIQ==');
window.interprocess.setExport(item_instructions, 'text/plain', 'en', 'data:text/plain;base64,SGVsbG8sIFdvcmxkIQ==');
window.interprocess.setExport(item_objectives, 'text/plain', 'en', 'data:text/plain;base64,SGVsbG8sIFdvcmxkIQ==');
window.interprocess.setExport(item_hints, 'text/json', 'en', 'data:text/json;base64,SGVsbG8sIFdvcmxkIQ==');
window.interprocess.close();

and/or:

window.interprocess.open();
window.interprocess.setExport(item_description, 'text/plain', 'en', my_js_callback_1);
window.interprocess.setExport(item_instructions, 'text/plain', 'en', my_js_callback_2);
window.interprocess.setExport(item_objectives, 'text/plain', 'en', my_js_callback_3);
window.interprocess.setExport(item_hints, 'text/json', 'en', my_js_callback_4);
window.interprocess.close();

There are a variety of related technologies (e.g., fetch, cross-document messaging, web sockets, WebRTC, etc.) and interprocess and interapplication communication topics have been explored previously (e.g., Web Intents).

I have some additional ideas about JS API with respect to the other direction of communication, how external processes or applications might provide data properties and functions to scripts running in Web browsers.

Any thoughts on a JS API for unidirectional or bidirectional interprocess and interapplication communication?

[AdamSobieski]

A different approach for enabling interprocess and interapplication communication is presented below, one more directly inspired by the Web Platform.

In an operating-system-dependent and -mediated manner, other processes (e.g., intelligent tutoring systems) could connect to Web browser processes and provide them with interfaces with which to enable message passing.

With respect to browser-side JavaScript, something like the following would enable Web developers to get those processes connected to Web browser and then to get those window objects within the processes.

var w = window.interprocess.getProcess(/* ? */).getWindow(/* ? */);

w.postMessage(...);

As shown in the following example, some sort of app:// URL scheme could be of use for representing the origins of connected processes.

window.interprocess.addEventListener(
    "message",
    (event) => {
        if (event.origin !== "app://vendor/application/major.minor.build.revision/instanceNumber") return;
        // ...
    },
    false,
);

The following example shows how one could utilize a new Process interface to access connected processes' metadata. Connected processes' metadata could be provided by operating systems, using applications' digitally-signed manifests, or by other means.

window.interprocess.addEventListener(
    "message",
    (event) => {
        var process = window.interprocess.getProcess(event.origin);
        if(typeof process !== "undefined" && process !== null) {
            if(process.about...) {
                // ...
            }
        }
    },
    false,
);

Any thoughts on these more Web-Platform-inspired ideas for bidirectional interprocess and interapplication communication?

[marcoscaceres]

Right, browsers would never allow that because the web app could directly attack the process.

I'd encourage you to take a look at how Web Authn, Payment Request API, or the new Digital Credentials API work... those shows specialized, secure, and privacy preserving approaches to talking to native applications while minimizing the risk of attacks from message passing.

[AdamSobieski]

Ok, I will take a look at how Web Authn, Payment Request API, and new Digital Credentials API work towards enabling specialized, secure, and privacy-preserving bidirectional communication between intelligent tutoring systems and Web browsers presenting learners with educational exercises and activities.

In these regards, thus far considered:

1. Web browsers could relay information about homework items to intelligent tutoring systems, e.g., items' descriptions, instructions, educational objectives, and hints.
2. Intelligent tutoring systems could select and highlight elements within homework items' accompanying multimedia resources.
   1. In a dialogue about a physics homework item, for example, as an intelligent tutoring system said "gravitational force", a downward-pointing arrow in that item's illustration could be selected and visually highlighted, and, as the intelligent tutoring system said "normal force", an upward-pointing arrow in that illustration could be selected and visually highlighted.
   2. Virtual cameras could be moved through 3D-graphics visualizations to showcase specific elements in them, including those selected and visually highlighted elements, and animations could be navigated to frames depicting elements.
3. Applications could stream data, e.g.,xAPI data, to any local or remote educational services, e.g., LRS's, pre-configured and selected by end-users.