The next evolution of Cycle.js -- Cycle.js Neo

[jvanbruegge]

Hi all, this is the branch where all the future development of Cycle.js will happen. It is a big rewrite of the core and all drivers, but there are no changes to the overall philosophy of Cycle and the user facing changes will be minimal. On the other hand, the new design cleanly separates drivers out even further, so they are really just interpreting streams of commands and return streams of data. This means that more code can be used directly in tests and less code that still needs mocking.

The design goals

As already discussed in #760, we figured out that splitting drivers even further into a driver part that only works on streams and a wrapper part that creates a nice API (like the DOMSource for example) would be beneficial. The idea was to handle every effect separated from each other:

This has one severe limitation though: Unlike today, it is not possible any more to inspect and modify streams that your app emits in a normal main wrapper. I for example frequently use this functionality to add a header with a token to all outgoing requests where the token in question gets requested at app startup and then stored in the state.

On the other hand, question and answer side effects like HTTP is a bit awkward at the moment. FRP (being based on stream) is a really nice fit for user interfaces, so the DOM driver for example, but it gets tedious with stuff like HTTP.

Take this simple component for example, it is quite hard to read because instead of having a linear flow of information, we start with the response, wondering where the request came from, have to manually correlate response with request by using category and basically the whole right hand side of res$ is boilerplate`.

function main(sources) {
    const res$ = sources.HTTP.select("manIhateNamingThings").flatten();

    return {
        DOM: res$.map(view).startWith("loading"),
        HTTP: xs.of({
            url: "/wherever",
            method: "GET",
            category: "manIhateNamingThings"
        })
    };
}

With the new design, we wanted to turn this component into something similar to this:

function main(sources) {
    return {
        DOM: sources.HTTP.get("/whatever")
            .map(view)
            .startWith("loading")
    };
}

And while it is totally clear what it does and is nice to read, we have the first problem again: Now that we don't have any request that comes out of the sink, how can I intercept those requests in user code and modify them to my liking?

To find a solution for this, @staltz and me had a two hour long video call where with the help of a whiteboard and throwing ideas to each other we finally found a solution that solves the problem in an elegant way and allows not only to intercept HTTP requests, but any action to any driver! This means you can for example log the attachment of event listeners even though there won't be actual event listeners on the DOM because of how Cycle simulates event bubbling on its own (if this interests you, you can read my article about it).

The final design

The final design is a bit more complex from the general flow of information, but @cycle/run takes care of connecting all the different components together. With Cycle.js Neo run will connect three different core components together instead of just two as currently. At the moment, run just connects all the sinks of your application with the input of the drivers and gives whatever the drivers return back to the application as sources. This means if the driver returns a stream, the source the user sees will be a stream and if the driver returns an object like the DOMSource, the user will see that.

Cycle.js Neo splits its drivers into two seperate pieces, the driver that just operates on an input and output stream and the API that takes that low level stream and offers a high level interface to the user that in turn emits these low level events. So to come back to the example from earlier, sources.HTTP.get will send a request to the driver, filter out the response that matches the request and returs this response stream for the user to consume directly.

So the only thing missing now is being able to intercept and modify those low level streams from user code. For this purpose and different than the design in #760, we have added a new part. The master wrapper (name most likely subject to change). We can think of the API parts of the drivers as fascades, they present a nice interface to the user, but on the other side they expect a low level stream as input and return a low level stream as output. This is done on each channel (ie HTTP, DOM, whatever) seperately. Seen together with your application, after all APIs are applied, we basically have a new main function but instead of expecting e.g. a DOMSource as input this new master main just expects the low level streams and also only returns those. The master wrappers then can wrap this new master main. This means that the master wrappers have access to all the events and commands from all drivers and can do whatever they want with them.

This is also where @cycle/state will be implemented. In #760, state was supposed to be an API without driver, but this would make having access to the state in the master wrappers impossible. But we did not want to make this a special case for state only, so we defined specific purposes to each of the parts:

• A driver is responsible for interpreting a stream of commands and returning a stream of data by doing side effects. A driver may be read-only (ie it ignores the sink) or write-only (it does not return a source), but it always has to do side effects. Pure effects like state or i18n should be implemented as a master wrapper.
• An API is an object that provides a high level interface to the user of a driver. The API itself is completely pure as it only interprets the calls to its methods by returning low level commands to the driver.
• The master main is the main function that is already wrapped by all its APIs. It is still a pure fuction, but only accepts streams and only returns streams.
• The master wrapper wraps the master main and can do arbitrary modifications to the streams. @cycle/run applies master wrappers in order, as inner wrappers can access potential new pure APIs that where created by the master wrapper (@cycle/state will provide a StateSource for example). The fully wrapped master main is then hooked up to the drivers. A master wrapper is a function that takes a master main and returns a new master main

A fully wrapped and connected main function may look like this then:

To connect everything together, @cycle/run does roughly these steps:

1. forall channels: Create sinksProxies
2. forall channels: call driver.consumeSink(sinkProxy), collect subscriptions
3 forall channels: call driver.produceSource(), collect masterSources
4. forall-in-order master wrappers: wrap this
  4.1 forall channels: call makeApi(masterSources[channel]), collect [api, apiSink]
  4.2 sinks = main({ ...masterSources, ...apis }) // Pass through sources that master wrappers created
  4.3 masterSinks = merge sinks apiSinks
5. Connect sinksProxies with masterSinks

PAQ (potentially asked questions)

How does multiple stream library support work with this new version?

The standard @cycle/xx packages like @cycle/http will provide a driver that uses Callbags as streaming library because of its small size and it being only based on callbacks (ie no common core you need to always ship, it's just some callbacks). It will also provide a HttpAPI that returns Callbags from all methods and also returns Callbags to the driver. You can use this API directly if you want to write your application with Callbags as streaming library.

For those that prefer rxjs, most or xstream, there will be packages like @cycle-rxjs/dom or @cycle-xstream/dom that just wrap those Callbag APIs and provide an rxjs, most or xstream interface. We have not fully decided if we want to have those packages as official cycle packages or better have them community maintained. But those packages will be very simple as they just convert from callbag to another stream lib. This also solves the Typescript issues we have currently for people that use e.g. rxjs, because @cycle/run won't implicitly convert or adapt the streams any more. Everything is explicit.

Is it now easier to run a Cycle.js app in a web worker?

cc @aronallen

Yes, because @cycle/run will expose a method that just connects main with its APIs returning the master main. This master main can then be wrapped with the master wrappers as needed. On the outmost layer you can have a master wrapper that takes the low level streams, serializes them and uses postMessage to send them over to the main thread. On the main thread, you basically only have a master wrapper that takes all the low level streams and deserializes them.

When will this be done?

We don't know. We have started the efford already. I've created @cycle/callbags, that will be the basis of the new implementation. I've also started working on a first version of the HTTP driver to try things out. To further downsize a typical Cycle.js app, I've created minireq, a request library that will be the base of the new HTTP driver

We have opened this PR so that everyone can see that we are actively working on it and to allow others to give their feedback to the new design. We will open more, smaller PRs that target this branch where all the drivers and other components will be implemented.

How can we help?

There are several things that you can do. First and foremost, give feedback to our ideas. We think the current design is pretty great, but we might have missed something! So please feel free to comment here. The second thing is supporting us through our Open Collective. Until now, we have not taken any money regarding the rewrite (the two hour design meeting excluded), but justifying spending a lot of time on Open Source work is a lot easier with financial support. We both love what we do, but we both have to live off something :)

[mightyiam]

I appreciate your work!

This change formalizes an existing common usage — wrapping main. One instance is @cycle/state. Another example is logging HTTP requests. Is this correct, please?

[jvanbruegge]

Yes, but it also changes main wrappers as they don't wrap main directly any more as is the case today. first main is wrapped with the APIs to interpret the XXXSource calls into a plain stream that your wrapper can then inspect

[teohhanhui]

I've created minireq, a request library that will be the base of the new HTTP driver

I really hope there will be an official fetch driver. That's been a deal breaker in the past, and will continue to be so if not addressed...

[jvanbruegge]

@teohanhui why is this a dealbreaker? Why does the implementation matter?

[teohhanhui]

Because fetch is a modern standard and we want to use it? And there are polyfills that allow it to be used in NodeJS too. I don't think cyclejs should be opinionated on this front, but rather it should be agnostic.

[jvanbruegge]

fetch is still missing some key features like reporting upload/download progress. How the driver is implemented now allows you to change the function that does the requests, so with a small wrapper around fetch this would work. The the default will be XMLHTTPRequest though

[jvanbruegge]

I've create a kanban board to track the progress here: https://github.com/cyclejs/cyclejs/projects/5

[jvanbruegge]

And another step is done: Isolation!

[wclr]

@jvanbruegge how is it going with neo?

I've got a question.

A driver is responsible for interpreting a stream of commands and returning a stream of data by doing side effects.

An API is an object that provides a high-level interface to the user of a driver. The API itself is completely pure as it only interprets the calls to its methods by returning low-level commands to the driver.

Who is responsible for handling/creating subscriptions and how they will be handled by the driver? If it is the API who create the subscription, so it is probably not so pure?

[jvanbruegge]

@whitecolor I am currently working on @cycle/state, currently only missing the collection stuff, see #959 for that.

The driver interface forces a driver to return a subscription, ie the driver has to subscribe (for a writeable driver). The subscription itself is then managed by @cycle/run.
Copied from https://github.com/cyclejs/cyclejs/blob/neocycle/run/src/types.ts#L31:

export interface Driver<Source, Sink> {
  provideSource?(): Producer<Source>;
  consumeSink?(sink: Producer<Sink>): Dispose;
  cleanup?(): void;
}

[jvanbruegge]

And a new milestone is done: @cycle/state including the collection stuff is here 🎉

[FbN]

Great work @jvanbruegge. After reading the post on DEV I'm impatient tro see it ready for testing.
One idea that crossed my mind several times was to replace Cycle Functions input sources with an high order stream of sources. If I can make a parallel to hardware you can think to a Cycle Function as a device with fixed input cable and fix output cable. An high order stream of sources in input/output can be seen as the introduction o a single USB protocol/cable in replacement of many ones. One advantage is that a Cycle Function can dinamically add drivers that can be added and remove during application lifecycle. This 'bus' could even be used in a standardized way for cross component messaging. You could write stream filters to select other component output stream and react/respond to their message.

[jvanbruegge]

I don't really see what Cycle would gain from the ability to dynamically add drivers. And even if there is a use case it will be very minor. The way cycle Neo structures messages (everything is serializable and wrappers have full access to the raw messages), it would be possible to build something like that yourself. A higher order stream would make it impossible to use drivers over APIs like postMessage.