Fork this repo, fill in your markdown and <html> for the 15 slides (max 20 slides), record your presentation and save it as recorded-talk.m4a (or change the code to reflect the new name.) A blank version of this github is here
TensorflowJS with TfMicro in the web browser using WebSerial
Arduino Nano33BleSense Demo here
TensorflowJS encredible web machine learning using vision and sound.
TfLite / TfMicro / RTLite: ML training of low cost Edge Devices
TinyMLjs proof of concept
Google can combine TensorflowJS with Tensorflow Micro
Proof of concept Overview TinyMLjs
Script
Let's explore the exciting potential of combining TensorFlowJS with TfMicro using WebSerial.
We're familiar with TensorFlowJS for vision and sound models, but now let's push further.
This proof of concept uses TinyMLjs to train sensor-based ML models for actuators on low-cost microcontrollers—all within the browser!
2
Why TensorflowJS?
Client-side
Secure and Private
Easier for Education and Makers
Community of Web Developers
Total Control
Version Control
Install Arduino IDE Library rocksettaTinyML a verison of TfMicro
Script
Why choose TensorflowJS for this approach? First, it's client-side, which means no data needs to leave your device—ensuring both privacy and security.
It’s also perfect for education and maker communities because it's simple to use. Plus, you have full control over the process, with robust version management and support from a massive community of web developers.
3
Why TfMicro
Millions of microcontrollers
ML at the Edge
Solves bandwidth
Low Energy
Low Power
Personal Solutions
Low Cost $
Right to Repair
Connecting MCU to Browser with webSerial
Script
And why TfMicro? The ability to run machine learning on millions of microcontrollers brings ML to the edge.
It solves common bandwidth problems by processing data locally, with minimal energy consumption.
These microcontrollers offer low-power, affordable solutions—making this approach accessible for personal projects as well as commercial applications.
Our "Right to Repair" allows us easy ways to reuse, repair and restore these products.
4
TinyMLjs Steps (1 of 3)
Explaining the videos on the right side of the page.
Compile-upload to micrcontroller, check serial LED on/off
Script
Let’s talk about how to use TinyMLjs with TfMicro. Step one: install the 'RocksettaTinyML' library, compile the sensor code, and upload it to your microcontroller using WebSerial. Once that’s done, generate your sensor data, train the model right in your browser, test it, and then export the final model. Simple and efficient!
5
TinyMLjs Steps (2 of 3)
Install or online load Python
TensorflowJS_converter from TFJS to Keras
From Keras to TFLITE
xxd from TfLite to a c-header file
I use an iPython notebook that bundles the outputs into a .zip folder
Connect the Microcontroller to the webpage using webSerial
Script
In step two, you’ll need Python installed or loaded online. Then, use TensorflowJS_Converter to convert your trained TFJS model into Keras format, then to TFlite and finally using xxd convert
the tFlite file into a c-header file which is an array of the binary characters needed to be run in the microcontroller. I use an iPython notebook that bundles all of the output into a zipped folder
6
TinyMLjs Steps (3 of 3)
RocksettaTinyML library examples Arduino Code
Replace c-header file library
Compile, upload and test in the serial monitor
Can also test back in the browser
Create motion x, y, z data
Script
Using the RocksettaTinyML arduino Library and finding the ML model for your micrcontroller. The code has two files an .ino file and a c-header file. The c-header file needs to be replaced with the converted file you just made and then the entire sketch
needs to be compiled and uploaded to the micrcontroller and then tested in the serial monitor. Note: This testing can also be done in the browser.
7
Why Do we need the Web Developers Community and Google? (1 of 4)
General
WebSerial barely works on Android
TensorflowJS_Converter is written in Python
Flashing code to MCU from the Browser only for the ESP boards
No community written code for the multitude of mcu's
Advanced: code can't yet be compiled in the browser client-side?
Train the model in the browser
Script
Now, why do we need support from the Web Developers, Maker Community and Google? First, WebSerial on Android needs some improvements, along with a Javascript lite version Python TensorflowJS_Converter. Flashing code directly from the browser will make it easier to support a wide range of microcontrollers. Imagine if we could even compile code right in the browser. These innovations are only possible with strong community and corporate support
8
Why Do we need the community and Google? (2 of 4)
WebSerial and TensorflowJS_Converter
WebSerial/WebUSB needs to be stable on Android so all of this can be done from your cell phone
A lite version of TensorflowJS_Converter written in Javascript is needede to make the c-header file
After training the model test it in the browser
Script
For WebSerial and TensorflowJS_Converter to reach their full potential, they need to be stable on Android. Additionally, a lightweight version of TensorflowJS_Converter written in Javascript would make generating C-header files much easier for developers.
That ability needs to be able to be done on multiple mcu's
The community can write the C++ code needed for the multitude of different mcu's sensors and actuators
View and export model from TensorflowJS
Script
The ESP tool already allows us to flash code directly from the browser for the ESP32 microcontroller board, but we need to expand that capability to other MCUs. The Community of Web Developers and Makers is essential for developing C++ code that will support a wide range of microcontrollers, sensors, and actuators and checking that these sketches work within the web browser environment.
10
Why Do we need the community and Google? (4 of 4)
Advanced: compiling microcontroller code from the browser!
It’s good to remember Atwood’s Law: any application that can be written in JavaScript will eventually be written in JavaScript.
Do we even need a microcontroller IDE? Can the code be written from the browser client-side or an image be saved and flashed directly to the microcontroller? These are questions that may become more importent and easier to achieve over the
next decade.
Load Python and use the TensorflowJS_Converter
Script
Finally, we reach the ultimate goal: compiling microcontroller code directly in the browser. Following Atwood’s Law, any application that can be written in Javascript eventually will be written in Javascript.
Do we even need a dedicated IDE anymore? Could all this be done from within the browser itself? Those are advanced questions that may take a while to answer.
11
Google Can Democritize ML on Microcontrollers in the Browser
Google can stabilize webSerial on Android
Google can make a lite TensorflowJS_converter in Javascript
Google can improve browser code flashing
Google can brainstorm clientside compiling
Python generates the tflite file then a c-header file all zipped
Script
Google has the power to truly democratize machine learning at the edge. Google can stabilize webSerial on Android. Google can make a lite TensorflowJS_converter in Javascript. Google can improve browser code flashing. Google can brainstorm clientside compiling.
Imagine millions of web developers having the ability to adapt machine learning models for physical world applications!
12
Who is Jeremy Ellis
Technology teacher for ~35 years
In the 1990's was making 3 layer 8 node neural networks
Manually adjusted the weights. Nothing really worked
RocksettaTinyML machine learning code to be compiled
Script
Jeremy Ellis has been a technology teacher for over 30 years. In the 1990's he was making 3 layer, 8 node, neural networks and manually adjusting the weights. Nothing really worked.
In 2015 Tensorflow made everything easier. By 2019 he was an expert at TensorflowJS. Then he switched to physical mcu's. By 2024 he had become an expert at ML models on microcontrollers.
13
TinyMLjs Proof of Concept
TinyMLjs with RocksettaTinyML proves that the browser can train machine learning models and those models can be converted and installed on microcontrollers.
There are issues that could streamline this process, but solving thosee issues will take a wider community involvement beyond what I can do.
Replace the c-header file compile and test in the serial monitor
Script
TinyMLjs has shown that browser-based machine learning works, and it’s capable of training models that can then be converted and deployed onto microcontrollers.
TensorFlowJS can be used to train a multitude of low-power, low-cost microcontrollers, giving power, privacy, freedom, and stability to everyone in this fast-changing world of microcontroller cloud costs and unstable software.
Let’s be a part of using the greatest combination of Google frameworks: TensorFlowJS with TensorFlowMicro.
14
TFJS with rtLite the new tfLite/Micro
Let's start making rtLite the new tf-micro and TensorflowJS work together in the browser.
It can be done, it just needs a community and some interest from Google.
The ML serial output can be tested in the browser
Script
In conclusion, let’s work together to make Tf-Micro and TensorflowJS function seamlessly in the browser.
With the strength of the community, interest by Google, the future of machine learning at the edge is bright and full of possibilities making it accessible and practical for a wide range of people and applications.
While there are challenges, the benefits of local execution, data security, Javascript simpicity and offline capability make it a compelling endeavor.
15
Credits
It is exciting to be part of the TensorflowJS community that has so much potential for the future of ML on Edge Devices.
Thank you, by Jeremy Ellis
Script
Thank you all for being here today. It’s an exciting time to be part of the TensorflowJS community, and I’m thrilled to be on this journey with you. Let’s keep pushing the limits of what’s possible!