TinyMLjs - WebBLE Enabled

This website makes machine learning models from Web Bluetooth connected micro-controller sensors using TensorflowJS. That model can be saved and converted to a tFlite model, then to a C-header model.h file using Tensorflow command line or Python converters. I simplify the conversion using a Gitpod. Finally the model.h file is combined with C/C++ code using an Arduino IDE ready library called RocksettaTinyML and compiled to the device for testing.

Presently, for complex vision or sound data it is easier to use EdgeImpulse.com as it will achieve the needed model compression that we have not yet achieved.

Connect via BLE

Send 'a' (LED Cmd) Send 'b' (LED Cmd)
Clear and send 'start' Send 'stop' Clear Sensor Data Send firstline Send 'start' only Nano33-fancy1 Nano33-fancy2
Cell Phone: Motion

Clean, Trim or Fill Label: Keep Auto Next Auto Next and Save
Save CSV CSV FileName:

Show Graph Classify Data Show All Data
...

Machine Learning models often need very specific data.
Clean, Trim or Fill All Count CSV: Count Senses: Count Total:
Number of Samples/count: Number of Senses/sample:

Click here to see the working HTML code.
Convert Data to Tensor
Enter number of epochs: , Learning rate:
Just Fit - Retrain View Model

Export Model

1. Try the following steps:
2. Convert the exported model to Arduino ready c-header model.h file as well as a model.tflite file using one of the following methods:
   

3. 1:	Method	Description
   1:	tensorflowjs-to-arduino-for-tinymljs	Best to install the modules needed to client-side do the conversions yourself. This github shows what you need to do. The installation might be different on your computer. I always find this hard to do at school.
   2:	iPython Notebook TFJS to TFlite	This web based Google Colab iPython notebook (Needs a google login) loads the necessary code then you click on an upload button to load your tensorflowjs exported "model.json" and "model.weights.bin" files and it zips and downloads the tflite and c-header files.
   3:	Use a Gitpod: https://github.com/hpssjellis/Gitpod-auto-tensorflowJS-to-arduino or direct load the gitpod: Gitpod Here	A Gitpod browser docker like program that auto loads the necessaray python files and then runs a bash program to do the command line conversions. All code is easy to view, if you have a github account it loads fast.

   
   
   
4. Use https://netron.app/ to check and visualize your downloaded model.tflite
   
5. Once you have made a model.h file then install this Arduino Library RocksettaTinyML based on EloquentArduino to load the code onto your Arduino IDE.
   

...

---

Show File Uploading

Upload from a raw CSV file or an Arduino style microcontroller using Web Bluetooth (Android Pixel phones also work) or your cell phone motion sensor. Keep the raw sensor data then Machine Learning train a tensorflowJS model for export or for live classification all on this single vanilla Javascript webpage!

Show: Hide: Load .csv files:

Following is the list of actual labels used in the same order as uploaded (comma-separated) Note: expecting files to be named: "name-lable.csv" or "name-lable (1).csv" etc.

CSV Lables (careful):

---

Senses Labels (In the order collected):

Select model file (.json):
Select weights file (.bin):
Select labels file (.txt):
Enter Labels (comma seperated): Re-set labels
Upload Model
...

---

The following sketch should be uploaded to your Arduino or ESP32 board. Make sure to delete any conflicting `ArduinoBLE` library if you are using ESP32 with `BLEDevice.h`.

Copy Sketch

The "fancy" Arduino sketch for the Nano33BleSense is on the github here https://github.com/hpssjellis/tinyMLjs/blob/main/public/acceleration/a00-accell-nano33-fancy.txt

/* * WebBLE for testing javascript connection with an Arduino/ESP32 * * Based on user provided code. * * NOTE: For ESP32, you need to use the ESP32 BLE Library (BLEDevice.h, BLEServer.h etc.) * and ensure ArduinoBLE is NOT installed or it will conflict. * For other Arduino boards (e.g., Nano 33 BLE, Nicla Vision), use ArduinoBLE.h. * * Service UUID and Characteristic UUID must match the JavaScript code. * MY_SERVICE_UUID: "19b10000-e8f2-537e-4f6c-d104768a1214" * MY_CHARACTERISTIC_UUID: "19b10000-e8f2-537e-4f6c-d104768a1214" */ #if defined(ESP32) #include <BLEDevice.h> #include <BLEServer.h> #include <BLEUtils.h> #include <BLE2902.h> #else #include <ArduinoBLE.h> #endif // Define your Service and Characteristic UUIDs - MUST MATCH JAVASCRIPT! #define MY_SERVICE_UUID "19b10000-e8f2-537e-4f6c-d104768a1214" #define MY_CHARACTERISTIC_UUID "19b10000-e8f2-537e-4f6c-d104768a1214" unsigned long myLastSendTime = 0; // For throttling sensor data sending // Define sensor parameters for data collection #define FREQUENCY_HZ 36 // how many samples per second #define COLLECTION_SECONDS 1 // how many seconds to collect samples #define INTERVAL_MS (1000 / (FREQUENCY_HZ + 1)) // need for the timer #define CONVERT_G_TO_MS2 9.80665f // accleration conversion int myMaxData = FREQUENCY_HZ * COLLECTION_SECONDS; int myCount = 0; unsigned long myStart; bool mySendData = false; // Control flag for sending sensor data #if defined(ESP32) BLECharacteristic *myCharacteristic; bool myDeviceConnected = false; // Callback for when the characteristic is written to (Web -> Arduino) class MyCallbacks : public BLECharacteristicCallbacks { void onWrite(BLECharacteristic* pChar) override { String value = pChar->getValue(); if (value.length() > 0) { Serial.print("Received over BLE: "); Serial.println(value); // Command handling if (value == "start") { Serial.println("Command: start - Starting sensor data."); mySendData = true; // Start sending data myStart = millis(); // Reset timer myCount = 0; // Reset sample count } else if (value == "stop") { Serial.println("Command: stop - Stopping sensor data."); mySendData = false; // Stop sending data Serial.println("Stopping at count: " + String(myCount)); } else if (value == "a") { Serial.println("Command: a - Toggling LED."); digitalWrite(LED_BUILTIN, !digitalRead(LED_BUILTIN)); // Toggle LED } else if (value == "b") { Serial.println("Command: b - Toggling LED."); digitalWrite(LED_BUILTIN, !digitalRead(LED_BUILTIN)); // Toggle LED (same as 'a' for this simple example) } else if (value == "firstline"){ Serial.println("accX,accY,accZ"); // Send CSV header } else { Serial.println("Command: unknown"); // For TinyML 'fancy' commands, you'd add more 'if' statements here } } } }; // Callback for BLE server connection/disconnection class MyServerCallbacks : public BLEServerCallbacks { void onConnect(BLEServer* pServer) { myDeviceConnected = true; Serial.println("Central connected"); } void onDisconnect(BLEServer* pServer) { myDeviceConnected = false; Serial.println("Central disconnected"); pServer->getAdvertising()->start(); // Restart advertising } }; #else // For Arduino boards like Nano 33 BLE, Nicla Vision #include <Arduino_LSM9DS1.h> // Example IMU library, replace if your board has different sensors BLEService myService(MY_SERVICE_UUID); // Characteristic for read, write, and notify. Max length 32 bytes for basic BLE. BLECharacteristic myCharacteristic(MY_CHARACTERISTIC_UUID, BLERead | BLEWrite | BLENotify, 32); #endif void setup() { Serial.begin(115200); pinMode(LED_BUILTIN, OUTPUT); digitalWrite(LED_BUILTIN, HIGH); // Onboard LED, HIGH = off (common for some boards) // Initialize IMU if using a sensor (e.g., Nano 33 BLE Sense, Nicla Vision) #if !defined(ESP32) // Only for Arduino-specific boards with IMU. ESP32 would have its own sensor setup. if (!IMU.begin()) { Serial.println("Failed to initialize IMU!"); while (1); // Halt if IMU fails } #endif randomSeed(analogRead(A0)); // Seed for random numbers (if used) Serial.println("Serial ready. Setting up BLE..."); #if defined(ESP32) BLEDevice::init("ESP32-TinyML-BLE"); // Name for your ESP32 device BLEServer *pServer = BLEDevice::createServer(); pServer->setCallbacks(new MyServerCallbacks()); BLEService *pService = pServer->createService(MY_SERVICE_UUID); myCharacteristic = pService->createCharacteristic( MY_CHARACTERISTIC_UUID, BLECharacteristic::PROPERTY_READ | BLECharacteristic::PROPERTY_WRITE | BLECharacteristic::PROPERTY_NOTIFY ); myCharacteristic->addDescriptor(new BLE2902()); // For client characteristic configuration myCharacteristic->setValue("ready"); // Initial value myCharacteristic->setCallbacks(new MyCallbacks()); pService->start(); pServer->getAdvertising()->addServiceUUID(BLEUUID(MY_SERVICE_UUID)); // Advertise your service UUID pServer->getAdvertising()->start(); Serial.println("BLE ready on ESP32"); #else // Arduino BLE (e.g., Nano 33 BLE) if (!BLE.begin()) { Serial.println("BLE failed to start"); while (1); // Halt if BLE fails } BLE.setLocalName("Arduino-TinyML-BLE"); // Name for your Arduino device BLE.setAdvertisedService(myService); myService.addCharacteristic(myCharacteristic); BLE.addService(myService); myCharacteristic.writeValue("ready"); // Initial value BLE.advertise(); // Start advertising Serial.println("BLE ready on Arduino BLE"); #endif } void loop() { // Read and send sensor data (e.g., IMU data) via BLE #if defined(ESP32) if (myDeviceConnected && mySendData && (millis() - myStart >= INTERVAL_MS)) { myStart = millis(); // Reset timer myCount += 1; float x, y, z; // Example: Read accelerometer data (replace with your sensor code) // For ESP32, you'd likely use an external sensor library (e.g., for MPU6050, LSM9DS1, etc.) // For demonstration, let's just send dummy data if no sensor is integrated. // In a real application, replace this with actual sensor reads. x = random(-500, 500) / 100.0f; // Dummy data y = random(-500, 500) / 100.0f; z = random(-500, 500) / 100.0f; String sensorData = String(x) + "," + String(y) + "," + String(z); myCharacteristic->setValue(sensorData.c_str()); myCharacteristic->notify(); Serial.println("Sent BLE: " + sensorData); if (myCount >= myMaxData) { mySendData = false; // Stop sending after max samples Serial.println("Max samples reached. Stopping data sending."); } } #else // Arduino BLE (e.g., Nano 33 BLE) BLEDevice central = BLE.central(); // Check for connected central if (central) { Serial.print("Connected to: "); Serial.println(central.address()); while (central.connected()) { // Handle incoming data (Web -> Arduino) if (myCharacteristic.written()) { // Read incoming value into a buffer first, then convert to String // This provides explicit control over the buffer and might resolve // subtle issues with direct String conversion on some ArduinoBLE versions. int len = myCharacteristic.valueLength(); uint8_t buffer[32]; // Max characteristic length is 32 bytes for this example myCharacteristic.readValue(buffer, len); // Ensure null termination if the incoming data is not exactly 'len' and contains nulls. // For String conversion, it's safer to ensure it's null-terminated if it's treated as a C-string. // However, String(char*, length) constructor is better if the data might not be null-terminated by sender. // If the sender always null-terminates, String((char*)buffer) is fine. // Given the Web-side uses TextEncoder().encode(), it's likely not null-terminated. String incoming = ""; // Initialize an empty string for (int i = 0; i < len; i++) { incoming += (char)buffer[i]; // Append char by char } Serial.print("Web says: "); Serial.println(incoming); // Command handling (same as ESP32 version) if (incoming == "start") { Serial.println("Command: start - Starting sensor data."); mySendData = true; myStart = millis(); myCount = 0; } else if (incoming == "stop") { Serial.println("Command: stop - Stopping sensor data."); mySendData = false; Serial.println("Stopping at count: " + String(myCount)); } else if (incoming == "a") { Serial.println("Command: a - Toggling LED."); digitalWrite(LED_BUILTIN, !digitalRead(LED_BUILTIN)); } else if (incoming == "b") { Serial.println("Command: b - Toggling LED."); digitalWrite(LED_BUILTIN, !digitalRead(LED_BUILTIN)); } else if (incoming == "firstline"){ myCharacteristic.writeValue("accX,accY,accZ"); // Send CSV header back via BLE Serial.println("Sent BLE: accX,accY,accZ"); } else { Serial.println("Command: unknown"); } } // Send sensor data (Arduino -> Web) if (mySendData && (millis() - myStart >= INTERVAL_MS)) { myStart = millis(); myCount += 1; float x, y, z; IMU.readAcceleration(x, y, z); // Read actual IMU data x *= CONVERT_G_TO_MS2; y *= CONVERT_G_TO_MS2; z *= CONVERT_G_TO_MS2; String sensorData = String(x, 2) + "," + String(y, 2) + "," + String(z, 2); // Format to 2 decimal places myCharacteristic.writeValue(sensorData.c_str()); // Send sensor data Serial.println("Sent BLE: " + sensorData); if (myCount >= myMaxData) { mySendData = false; Serial.println("Max samples reached. Stopping data sending."); myCharacteristic.writeValue("stop_data"); // Optionally send a signal that data collection stopped } } } Serial.println("Disconnected."); } #endif }

Use at your own risk!
By Jeremy Ellis @rocksetta
Github at https://github.com/hpssjellis/tinyMLjs/tree/main
Demo's Index at https://hpssjellis.github.io/tinyMLjs/public/index.html
This page should be at https://hpssjellis.github.io/tinyMLjs/public/acceleration/a00-best-acceleration.html

A couple of gotchas (as of Jun 19, 2024):


1. File names have to be in the format "name-lable.csv" or "name-lable (1).csv" or "name-lable (2).csv" etc. Unfortunately Android and iPhone don't auto make the numbering for you.
   
2. Android and Apple device have an opposite orientation, so I have made negative all the android motion data so when your phone is on a table z = -9.8 m/s^2 etc. When looking veritacally at your phone y = -9.8 m/s^2. The auto detect of this only works if an Android phone is in mobile format not "desktop site"
   
3. Real data has lots of rough data, machine learning models do not like missing data. If your results show "NaN" either your training data or classification data has errors. Note: If the loss is not changing your trained data probably has errors. The "clean,trim,fill" buttons might help.
   
4. Presently a CSV label upload bug happens sometimes. Easy to fix by entering the correct labels in the correct order. I will try to fix the issue when I figure out what is causing it.
   
5. **HTTPS Required for Web BLE:** Web Bluetooth, like Web Serial, only works on secure contexts (HTTPS). If testing locally, you will need to serve your page over HTTPS.
   
6. **BLE UUIDs:** The provided Arduino sketch and JavaScript use specific Service and Characteristic UUIDs (`19b10000-e8f2-537e-4f6c-d104768a1214`). Ensure your BLE device firmware uses these exact UUIDs for communication to work.