Sensing temperature using MQTT

I recently built a simple temperature sensor using a NodeMCU (ESP8266) and a DHT22. This reads the temperature and humidity roughly every 30 seconds and pushes the data to a MQTT server so I can handle the data elsewhere.

The NodeMCU wakes up every 30 seconds and connects to my local WiFi network. Once connected it takes a sensor reading and publishes it to my MQTT server. It then delays for a few seconds to ensure the message is sent, then goes into deep sleep.

The deep sleep is a function I wasn’t aware of on the ESP8266 before. It just a case of wiring up D0 to RST and calling ESP.deepsleep() in the code. When the device wakes up, it’s like its been turned on for the first time.

The code and wiring is available on my GitHub https://github.com/robertprice/NodeMCU_MQTT_Temperature.

Reading the data from MQTT

As the purpose of the sensor was to just send the data to a MQTT server in a simple format, I needed to write something to listen to the published messages the device was sending.

I was able to do this using a simple NodeRED program to listen to incoming MQTT messages, and send them to debug console, and to a CSV file for later processing.

I added a timestamp and built the CSV line using a mustache template. This is then appended to a CSV file I put in my /tmp directory.

The data looks like this

2020-05-07T13:09:17.635Z,29.70
2020-05-07T13:09:54.586Z,29.80
2020-05-07T13:10:31.508Z,29.80
2020-05-07T13:11:06.936Z,29.80
2020-05-07T13:11:42.409Z,29.80

Using the CSV data

To test the data is being logged correctly, I wrote a simple R script parse the data and plot a diagram from it. As the CSV had no header columns, I had to rename the first column names as read_csv used the first line values by default.

library(tidyverse)
library(readr)
library(ggplot2)

temperatures <- read_csv("temperatures.csv", col_types = cols(`2020-05-07T13:09:17.635Z` = col_datetime()))
temperatures <- rename(temperatures, c("2020-05-07T13:09:17.635Z"="datetime", "29.70"="temperature"))
shedtemperatures <- temperatures %>% select(datetime, temperature) %>% filter(datetime >= as.Date("2020-05-09") & datetime < as.Date("2020-05-10"))
ggplot(data=shedtemperatures, aes(x=datetime, y=temperature, group=1)) geom_line() + ggtitle("Temperature in the Dev Shed")

This gave me the following plot…

The maximum temperature in the shed I placed the sensor in was 33.1C.

Changing an RGB LED from a Mendix app

The Nordic Thingy:52 is a compact multi-sensor prototyping platform. It exposes a variety of configurable sensors and actuators over Bluetooth.

Modern versions of the Google Chrome browser support the Web Bluetooth API. This means we can write JavaScript that runs in a web browser to talk to Bluetooth devices. We can use this to get Chrome to talk to a Nordic Thingy:52.

The guys at Timeseries created a quick primer on How to get started with IoT using Mendix. Willem Van Zantvoort also gave a webinar that included using a Nordic Thingy:52. This demo only showed how to read data, not write.

Timeseries have released the Bluetooth (BLE) Connector module to the Mendix App Store. This allows developers to connect and write to Bluetooth devices from a Mendix app. These are exposed as JavaScript actions.

If we want to change the colour of the Nordic Thingy:52’s LED, how do we do this from a Mendix app?

We can use the DeviceConnect JavaScript Action in a Nanoflow to connect to a Nordic Thingy:52. The Bluetooth (BLE) Connector module doesn’t allow data to be written to a device. We need this functionality to tell the Nordic Thingy:52 what colour we want to the LED to show. So we will need to write our own JavaScript Action to do this.

Create the SetLED JavaScript Action

JavaScript Actions in Mendix are very easy to write. We create a JavaScript Action called SetLED. This needs to take 3 String parameters. A “primaryServiceUUID”, a “characteristicUUID”, and “mycolour”. The “primaryServiceUUID” describes what type of service we want to talk to. The “characteristicUUID” is the specific characteristic, in this case the LED. Finally, “mycolour” is the colour we want to show described as a CSS style rgb string (e.g. “rgb(10, 255, 10)”).

A Mendix JavaScript Action works by using a JavaScript promise to return data to a Nanoflow. Mendix has created a small scaffolded bit of JavaScript for us to change.

The first thing we need to do is to see if Bluetooth is supported. We do this by trying to access the window.gattServer. If it’s not available, we need to reject the Promise.

var gattServer = window.gattServer;
  if(gattServer){
    // Bluetooth available
  } else {
    return Promise.reject("No gatt server found");
  }

Now we need to try to access the LED characteristic on the device. We do this by creating a new Promise within which we get the primary service, then the LED characteristic. At this point we should be able to talk to the device and issue an instruction for the LED.

return new Promise((resolve, reject) => {
    gattServer.getPrimaryService(primaryServiceUUID)
    .then( (s) => {
        return s.getCharacteristic( characteristicUUID );
    })
    .then( (c) => {
        characteristic = c;
        return c;
    })
    .then(characteristic => {
        // do something with the LED
        resolve(true);
    });
});

The data is sent as binary using the writeValue method of the characteristic. We need to convert our RGB string into binary values before we send this. We can extract the individual values for R, G, and B using a regular expression. We can then insert these into a JavaScript Uint8ClampedArray. The Uint8ClampedArray prevents values under 0 and over 255. This helps us avoid buffer overflows which could cause bugs.

const match = mycolour.match(/rgba?\((\d{1,3}), ?(\d{1,3}), ?(\d{1,3})\)?(?:, ?(\d(?:\.\d?))\))?/);
let buffer = new Uint8ClampedArray(4);
buffer.set([1, match[1], match[2], match[3]], 0);

characteristic.writeValue(buffer);

Our final action should look like this…

 // This file was generated by Mendix Studio Pro.
//
// WARNING: Only the following code will be retained when actions are regenerated:
// - the code between BEGIN USER CODE and END USER CODE
// Other code you write will be lost the next time you deploy the project.

/**
 * @param {string} primaryServiceUUID
 * @param {string} characteristicUUID
 * @param {string} mycolour
 * @returns {boolean}
 */
function SetLED(primaryServiceUUID, characteristicUUID, mycolour) {
 // BEGIN USER CODE
 var gattServer = window.gattServer;
 if(gattServer){
 return new Promise((resolve, reject) => {
            gattServer.getPrimaryService(primaryServiceUUID)
            .then( (s) => {
 return s.getCharacteristic( characteristicUUID );
            })
            .then( (c) => {
                characteristic = c;
 return c;
            })
            .then(characteristic => {
 const match = mycolour.match(/rgba?\((\d{1,3}), ?(\d{1,3}), ?(\d{1,3})\)?(?:, ?(\d(?:\.\d?))\))?/);
 let buffer = new Uint8ClampedArray(4);
                buffer.set([1, match[1], match[2], match[3]], 0);

                characteristic.writeValue(buffer);
                resolve(true);
            });
        });
    } else {
 return Promise.reject("No gatt server found");
    }
 // END USER CODE
}

Now we have our JavaScript action, we can wire it into a Mendix app.

Using the SetLED JavaScript Action

We need to add a non persistable entity to hold our RGB string. We’ll call this “LED”, and add one String attribute “RGBString”.

Add a Data view to the page, and set its data source to the a new Nanoflow called CreateLED. All the Nanoflow needs to do is create an instance of our LED entity.

Mendix have provided an excelled RGB colour picker in the App Store called Color Picker. We can drop this into our Data view setting the Color Attribute field to the RGB string in our new LED entity. Set the Color Format to RGB. In the Appearance Tab set the Picker type to Hue. Now when we use the colour picker, it will save the selected value to our LED entity. That’s great, but we want to actually change the colour on Nordic Thingy:52. To do this, we need to go to Events and On change call a new Nanoflow we’ll call ChangeLED.

ChangeLED needs to first connect to the Nordic Thingy:52. We do this using the DeviceConnect JavaScript action from the Bluetooth (BLE) Connector. This takes a Primary service UUID of ” ef680100-9b35-4933-9b10-52ffa9740042″. It also takes an Optional service UUID of ” ef680300-9b35-4933-9b10-52ffa9740042″. I found these values in the example JavaScript code on Nordic’s GitHub account. When this is called it will trigger a prompt to connect to a Bluetooth device.

Once we are connected, we call our SetLED JavaScript action. The Primary service UUID is ” ef680300-9b35-4933-9b10-52ffa9740042″. This was the value we used as the optional UUID when we connected. For the Characteristic UUID we use the value “ef680301-9b35-4933-9b10-52ffa9740042” for the LED. For Mycolour we use the value in the RGBString attribute in our LED entity.

We can now run the App.

Running the App

The first time the ChangeLED Nanoflow is run it will prompt to connect to a Bluetooth device. This is where we look for our Nordic Thingy:52. Future requests are not prompted for while in the same session.

If we’ve connected to our device the LED should now be changing colour to match the value in the colour picker.

Thingy LED change from a Mendix App

Conclusion

Hopefully this should be enough information to get you connecting to a Bluetooth device and sending data.

If you aren’t using Mendix at the moment it’s free to get started, you just need to signup for a Mendix account.

Using Bluetooth on Windows 10 running in Parallels on a Mac

I was trying to use a Bluetooth device on Windows 10 running in Parallels on a Macbook Pro, but without luck.

Parallels can share its Bluetooth connectivity, but it’s not very reliable.

The solution is to get a Bluetooth USB dongle, and being a Macbook Pro, I also had to get a USB C to USB A adaptor.

I found the Plugable USB-BT4LE USB Bluetooth 4.0 Low Energy Micro Adapter with a Ailun Type C Adapter worked.

Before you plug in the dongle, you need to go into Parallels and stop sharing built in Bluetooth device to your Windows 10 instance.

Now plug the dongle in. You will need to assign the USB device to Windows 10 in Parallels when prompted.

Windows 10 will complain that the device doesn’t work. To fix this download the Windows 10 drivers. Once installed you need to restart Windows 10, but after you do Windows 10 will have a working Bluetooth connection.

Parallels has a habit of sharing the built in Bluetooth service when you restart, so make sure this is disabled if you are having problems.

Logging from a Mendix Java action

How to add logging to a Mendix Java action.

It is a common requirement to be able to log data from an application. Mendix has a good logging system built in, but how do we access this from a custom Java action?

The Mendix development team have already thought of this, and have provided a method called getLogger in the Core.

To use the logger from a Java action, we first need to add a couple of imports.

import com.mendix.core.Core;
import com.mendix.logging.ILogNode;

Now we need to provide access to the logger. We can do this by providing a static variable which we’ll call LOG. We use the getLogger method we mentioned earlier, passing in the name of the log node name we want to use. In this case, we’ll use “RobTest” as the log node name.

// BEGIN EXTRA CODE
public static ILogNode LOG = Core.getLogger("RobTest");
// END EXTRA CODE

Now, this is in place, we can access LOG from elsewhere in our Java action.

Let’s say our action wants to log when it starts and also wants to list all the microflows available within the App. We can change the log levels, so the start notification is at “info” level, and the microflows are at “debug” level. We can do this using the following.

public java.lang.Boolean executeAction() throws Exception
{
    // BEGIN USER CODE
    LOG.info("Running executeAction()");

    for (String mf : Core.getMicroflowNames()) {
        LOG.debug("Microflow: " + mf);
    }
    return true;
    // END USER CODE
}

When we execute the Java action, we see the following on the Mendix console.

Using an HC-06 Bluetooth adapter with a ESP8266 NodeMCU.

I’ve been looking at getting a ESP8266 NodeMCU to talk over Bluetooth using an HC-06.

The example will provide a simple way of echoing data from Bluetooth to a serial port.

The HC-06 exposes a serial port that the NodeMCU can use. In this example, I’m going to echo data between a the HC-06 and the serial port connected up to my Mac. As the NodeMCU only supports one serial port in hardware, we need to use Software Serial as well. This provides us a second serial port.

Wiring between the HC-06 and the NodeMCU is as follows

GND to GND
VCC to 3V
TXD to D4
RXD to D3

The code is a variation on the example code provided by Software Serial. We need to set the baud rate to 9600, and specify pins D4 and D3 for RX and TX.

#include <SoftwareSerial.h>

#define BAUD_RATE 9600

SoftwareSerial swSer(D4, D3);

void setup() {
  Serial.begin(BAUD_RATE);
  swSer.begin(BAUD_RATE);

  Serial.println("\nSoftware serial test started");

  for (char ch = ' '; ch <= 'z'; ch++) {
    swSer.write(ch);
  }
  swSer.println("");

}

void loop() {
  while (swSer.available() > 0) {
    Serial.write(swSer.read());
    yield();
  }
  while (Serial.available() > 0) {
    swSer.write(Serial.read());
    yield();
  }

}

To test this on a Mac, plug it into the serial port, and set the Arduino Serial Monitor to 9600 baud. The message “Software serial test started” should appear.

Go to Bluetooth and add the device. Mine showed up as HC-06. The default pairing code is 1234. Once paired, open a Terminal window and look for the device in /dev/. My device name is /dev/tty.HC-06-SPPDev . Connect to the device by typing

screen /dev/tty.HC-06-SPPDev

The message “ !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz” should appear.

What you type in one window is now echoed to the other window.

An improved SDS011 connector cable for Airrohr devices

At the Clean Air Eastbourne project we build Airrohr Luftdaten devices.

These use the SDS011 sensor. The official instructions suggest plugging dupont cables directly into the sensor, however we have found these tend to become loose and can be unreliable.

Our solution is to build our own cables.

We buy 7pin 2.54m 6S1P JST-XH cables from AliExpress. These come with wires already attached. We take out the red, black, and yellow wire by pressing down on the connector and gently pulling the wire out the back. This leaves us with the 4 wires we need in the correct slots.

Next we attach female dupont connectors with double housing to the other ends of the wires. One double connector goes on the blue / green pair, and another on the white / orange pair of wires. I used a PA-09 crimping tool from Amazon.co.uk (affiliate link) to make the attachments.

When plugging into the NodeMCU the connections are
White – VU
Orange – Ground
Green – D2
Blue – D1

Debugging a Mendix widget

I had to debug a custom Mendix widget that was failing. All that was shown on screen in red was “Could not create widget BHCCDropZone.widget.BHCCDropZone”.

We’d recently upgraded to Mendix 7.18.1, so something looked like it happened during the upgrade.

The first step was to make sure the appropriate access rights to the entity the widget uses were right. In this case they were.

Next was to up the log levels from INFO to DEBUG in the Mendix Modeller to see if that gave me any clues. I could see the widget’s constructor and postCreate being run as I had logger.debug calls in those functions being run. However, one of my debug calls was not being reached, so I had narrowed down the location of problem.

After this I added JavaScript breakpoints in the failing method using Chrome’s developer tools. The widget is only loaded the first time the page is reached. When I reached this page I singled stepped through, and found an exception being thrown by the JavaScript but being caught by Mendix. My error was…

"TypeError: Converting circular structure to JSON
    at JSON.stringify ()
    at http://localhost:8080/mxclientsystem/mxui/mxui.js?636758982080189581:74:164930
    at Array.map ()
    at t.log (http://localhost:8080/mxclientsystem/mxui/mxui.js?636758982080189581:74:164840)
    at http://localhost:8080/mxclientsystem/mxui/mxui.js?636758982080189581:74:196250
    at http://localhost:8080/mxclientsystem/mxui/mxui.js?636758982080189581:74:166406
    at Array.forEach ()
    at t.log (http://localhost:8080/mxclientsystem/mxui/mxui.js?636758982080189581:74:166377)
    at t.debug (http://localhost:8080/mxclientsystem/mxui/mxui.js?636758982080189581:74:165718)
    at Object.postCreate (http://localhost:8080/widgets/BHCCDropZone/widget/BHCCDropZone.js?636758982080189581:80:11)"

I could now see I had a circular data structure.

Working backwards through the stack trace, I could see the problem was in my widget’s postCreate function, and it was a debug function causing the problem. This was the faulty code causing the widget to fail.

My postCreate function looked like this…

postCreate: function () {
  console.log("BHCCDropZone session", mx.session);
  logger.debug(this.id + ".postCreate");
  this.initBHCCDropZone();
  logger.debug("this", this);
},

The faulty line was this…

logger.debug("this", this);

Trying to send the contents of this to the debug logs was causing problems as it was a circular data structure. Removing this line of JavaScript fixed the problem, and the widget started to work again as expected.

I hope this helps others in future when debugging “Could not create widget” errors in Mendix.

Using a http proxy from a Mendix Java action

As part of some work I have been undertaking to integrate the UK Government Notifications service into Mendix, I needed to be able to make API calls from behind a firewall using a proxy in a Java action.

Due to the lower level Java actions in Mendix run at, proxy settings are not automatically applied, and must be added manually. I wanted to explain how to get the proxy settings from Mendix, and use them a Java action.

I’ve previously explained how to add proxy settings to Mendix, so I assume this step has been completed.

In a Java action, we need to get these from the HttpConfiguration singleton.

import com.mendix.http.HttpConfiguration;
import com.mendix.http.IHttpConfiguration;
import com.mendix.http.IProxyConfiguration;

IHttpConfiguration httpconf = com.mendix.http.HttpConfiguration.getInstance();
IProxyConfiguration proxyconf = httpconf.getProxyConfiguration().orElse(null);

We can now check if we have a proxy configuration set, if we don’t proxyconf will be null.

The username and password for the proxy can be retrieved using the getUser() and getPassword() methods.

String username = proxyconf.getUser().orElse(null);
String password = proxyconf.getPassword().orElse(null);

If they are present we can build a Java Authenticator object and set it as the default authenticator.

import java.net.Authenticator;
import java.net.PasswordAuthentication;

if (username != null && password != null) {
    Authenticator authenticator = new Authenticator() {
        public PasswordAuthentication getPasswordAuthentication() {
           return (new PasswordAuthentication(username, password.toCharArray()));
        }
    };

    Authenticator.setDefault(authenticator);
}

Next we need to create the Proxy object. We need to get the host and port of our proxy server from Mendix using the getHost() and getPort() methods.

import java.net.InetSocketAddress;
import java.net.Proxy;

InetSocketAddress proxyLocation = new InetSocketAddress(proxyconf.getHost(), proxyconf.getPort());
Proxy proxy = new Proxy(Proxy.Type.HTTP, proxyLocation);

The proxy can be used for Java network actions.

An example of using this would be the UK Government Notifications client. It has a second optional paramater in it’s constructor for a Proxy.

client = new NotificationClient('APIKey', proxy);

Using a proxy server from the Mendix Modeller

There are times when building online services you find yourself behind a firewall and need to use a proxy. Sometimes these are transparent, but other times you need to add settings by hand.

In a Mendix app, an example may be when you need to consume a REST service from outside you home network.

To configure proxy settings in Mendix, you need to go to our Project’s “Settings”. Open “Configurations”, select your working configuration, and click “Edit”. Select the “Custom” tab and add the following “Names” and “Values”.

http.proxyHost The name your proxy
http.proxyPort The port your proxy is running off of.

If my proxy was running on proxy.robertprice.co.uk:8080, my settings would be

http.proxyHost proxy.robertprice.co.uk
http.proxyPort 8080

Sometimes the proxy will also need a username and password. You can set these using http.proxyUser and http.proxyPassword. For example

http.proxyUser RobertPrice
http.proxyPassword SecretPassword

You should now be able to access external services through the proxy from Mendix.

Example proxy settings for the Mendix Modeller

More information on using a proxy in Mendix is available at Using a proxy to call a REST service.

Extracting text in Mendix using RegexReplaceAll

It’s a fairly common requirement to be able to extract text from a larger string.

Using Mendix, the easiest way I’ve found to do this is using the RegexReplaceAll Java action from the Community Commons module.

We use a regular expression extract the text, then return this selected text in the action.

For example, take the following string returned from the Nexmo SMS module.

--------- part [ 1 ] ------------Status [ 9 ] ...SUBMISSION FAILED!Message-Id [ null ] ...Error-Text [ Quota Exceeded - rejected ] ...Message-Price [ 0.03330000 ] ...Remaining-Balance [ 0.03200000 ]

If we want to extract the Error-Text we can use the following regular expression.

^.*Error-Text \[ (.*?) \].*$

Here we’re saying look for the text between the square brackets after the string Error-Text. We use round brackets to say we want to remember this matched text. We can then use the regular expression match position to return the matched text – in this case $1.

If we run this over our string we get the following

Quota Exceeded - rejected

To use this in a Mendix microflow, assume we have our status message in a String $StatusMessage that we pass into the Java action. This is our Haystack.

Next, we use the regular expression as a String for our Needle regex.

Finally, we say we want $1 as a String to be our Replacement.

We return the String as $Details.

This is what the microflow and action should look like.

View Robert Price’s Mendix Profile.