I've started a grand review of the systems architecture of all the projects I've done for the learning sciences grants starting back in 2014. They're all Javascript projects, built as we were still figuring out how the ecosystem worked. Today, we are much more opinionated about what we like about Javascript and what works for our particular type of application development.
The new grant is to extend Net.Create, which we coded back in 2018 using D3js (v4) and React (v16) from pieces of other projects as a proof of concept. It's all held together by a control system to manage a network of web browser clients to update a shared network graph in realtime. The original work was done with a limited budget so the underlying code is "prototype quality"...it works pretty well for what it is, but we need to do a lot of retooling to make it secure and deployable outside of local area networks. We also need to add real account support instead of our generated "user tokens" for use on LANs, with permissions and templatable content and classroom lists and everything.
We'd like to take everything we've learned from doing these app into modules that will snap together more easily in the future based on the peculiar needs of our distributed realtime graphics systems. Today I started by reviewing the old Net.Create 2018 codebase to remember how we slapped it together and seeing what was under the hood with the idea of making a list of universal modules to develop for the new version. We don't want to rewrite it completely, but with a good univeral module outline perhaps the new features will be more reusable.
Our particular focus is on "distributed multiuser realtime simulation using embodied input" in real classrooms using Chromebooks and mobile devices with web browsers. Our interesting abstractions are:
Automatic Network Connection - When a device comes online, it automatically joins without the user having to figure out how to connect beyond the provided URL, and the system tracks users as they come on and offline.
Addressless Network Messages - We have a lot of data flying around, encoded as named messages that are subscribed to by different devices using a few different signaling approaches. It's sort of like MQTT but we support asynchronous in-order transactions between arbitrary clients to make it easier to write that kind of tricky code.
Multiple Device I/O Controllers - Our realtime sims are "embodied", meaning that the students get to run around a space with various tracking mechanisms inputing their positions into the simulation being displayed from the main computer running the simulation. Other students can use tablets to control other elements in the simulation. The system can handle a variable number of such controllers and remap them to unique IDs in the simulation as they leave and join so we can use rules-based assignment of a class of controller instead of manually assigning each controller to an entity in the simulation. This also allows us to have a crude form of distributed rendering using display lists, with a clear separation between modeled data and their visual representations.
Lifecycle Management - We don't let the UI control the simulation, but use our own master module to provide synch across different phases of the simulation loop. Multiple modules can subscribe to a particular phase and there is a distinct order of operations to ensure that the simulation is stable, and each phase completes only when all modules have finished to enforce order-of-operation.
There are three challenges that I'd like to address too:
A real problem has been communicating the nuances of time-based simulation and asynchronous event-driven programming. We need better abstractions documented so new devs in our system can learn these concepts and feel empowered instead of weighed-down by requirements that just seem like dogma.
A big challenge has been building consensus on what the difference is between a message, event, signal, callback, and handler and how to structure them logically so everyone can see what they are supposed to do.
A more esoteric challenge has been about managing dataflow between modules and making the distinction between pure data, derived data, working state, component state, and application state. Organizing these logically and naming them appropriately is a huge challenge.
Then of course there's all the usual stuff that we have to make, like account authentication, secure communications, database operations, multi-server management, classroom adminstration, logging, rendering tech, annotation, user-generated asset management, and so on.
It's going to be an interesting challenge see how to achieve the big challenge goals by surgically updating our 2018 prototype with robust systems we can reuse!