Using an HC-06 Bluetooth adapter with a ESP8266 NodeMCU.

I’ve been looking at getting a ESP8266 NodeMCU to talk over Bluetooth using an HC-06.

The example will provide a simple way of echoing data from Bluetooth to a serial port.

The HC-06 exposes a serial port that the NodeMCU can use. In this example, I’m going to echo data between a the HC-06 and the serial port connected up to my Mac. As the NodeMCU only supports one serial port in hardware, we need to use Software Serial as well. This provides us a second serial port.

Wiring between the HC-06 and the NodeMCU is as follows

GND to GND
VCC to 3V
TXD to D4
RXD to D3

The code is a variation on the example code provided by Software Serial. We need to set the baud rate to 9600, and specify pins D4 and D3 for RX and TX.

#include <SoftwareSerial.h>

#define BAUD_RATE 9600

SoftwareSerial swSer(D4, D3);

void setup() {
  Serial.begin(BAUD_RATE);
  swSer.begin(BAUD_RATE);

  Serial.println("\nSoftware serial test started");

  for (char ch = ' '; ch <= 'z'; ch++) {
    swSer.write(ch);
  }
  swSer.println("");

}

void loop() {
  while (swSer.available() > 0) {
    Serial.write(swSer.read());
    yield();
  }
  while (Serial.available() > 0) {
    swSer.write(Serial.read());
    yield();
  }

}

To test this on a Mac, plug it into the serial port, and set the Arduino Serial Monitor to 9600 baud. The message “Software serial test started” should appear.

Go to Bluetooth and add the device. Mine showed up as HC-06. The default pairing code is 1234. Once paired, open a Terminal window and look for the device in /dev/. My device name is /dev/tty.HC-06-SPPDev . Connect to the device by typing

screen /dev/tty.HC-06-SPPDev

The message “ !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz” should appear.

What you type in one window is now echoed to the other window.

Debugging a Mendix widget

I had to debug a custom Mendix widget that was failing. All that was shown on screen in red was “Could not create widget BHCCDropZone.widget.BHCCDropZone”.

We’d recently upgraded to Mendix 7.18.1, so something looked like it happened during the upgrade.

The first step was to make sure the appropriate access rights to the entity the widget uses were right. In this case they were.

Next was to up the log levels from INFO to DEBUG in the Mendix Modeller to see if that gave me any clues. I could see the widget’s constructor and postCreate being run as I had logger.debug calls in those functions being run. However, one of my debug calls was not being reached, so I had narrowed down the location of problem.

After this I added JavaScript breakpoints in the failing method using Chrome’s developer tools. The widget is only loaded the first time the page is reached. When I reached this page I singled stepped through, and found an exception being thrown by the JavaScript but being caught by Mendix. My error was…

"TypeError: Converting circular structure to JSON
    at JSON.stringify ()
    at http://localhost:8080/mxclientsystem/mxui/mxui.js?636758982080189581:74:164930
    at Array.map ()
    at t.log (http://localhost:8080/mxclientsystem/mxui/mxui.js?636758982080189581:74:164840)
    at http://localhost:8080/mxclientsystem/mxui/mxui.js?636758982080189581:74:196250
    at http://localhost:8080/mxclientsystem/mxui/mxui.js?636758982080189581:74:166406
    at Array.forEach ()
    at t.log (http://localhost:8080/mxclientsystem/mxui/mxui.js?636758982080189581:74:166377)
    at t.debug (http://localhost:8080/mxclientsystem/mxui/mxui.js?636758982080189581:74:165718)
    at Object.postCreate (http://localhost:8080/widgets/BHCCDropZone/widget/BHCCDropZone.js?636758982080189581:80:11)"

I could now see I had a circular data structure.

Working backwards through the stack trace, I could see the problem was in my widget’s postCreate function, and it was a debug function causing the problem. This was the faulty code causing the widget to fail.

My postCreate function looked like this…

postCreate: function () {
  console.log("BHCCDropZone session", mx.session);
  logger.debug(this.id + ".postCreate");
  this.initBHCCDropZone();
  logger.debug("this", this);
},

The faulty line was this…

logger.debug("this", this);

Trying to send the contents of this to the debug logs was causing problems as it was a circular data structure. Removing this line of JavaScript fixed the problem, and the widget started to work again as expected.

I hope this helps others in future when debugging “Could not create widget” errors in Mendix.

Using a http proxy from a Mendix Java action

As part of some work I have been undertaking to integrate the UK Government Notifications service into Mendix, I needed to be able to make API calls from behind a firewall using a proxy in a Java action.

Due to the lower level Java actions in Mendix run at, proxy settings are not automatically applied, and must be added manually. I wanted to explain how to get the proxy settings from Mendix, and use them a Java action.

I’ve previously explained how to add proxy settings to Mendix, so I assume this step has been completed.

In a Java action, we need to get these from the HttpConfiguration singleton.

import com.mendix.http.HttpConfiguration;
import com.mendix.http.IHttpConfiguration;
import com.mendix.http.IProxyConfiguration;

IHttpConfiguration httpconf = com.mendix.http.HttpConfiguration.getInstance();
IProxyConfiguration proxyconf = httpconf.getProxyConfiguration().orElse(null);

We can now check if we have a proxy configuration set, if we don’t proxyconf will be null.

The username and password for the proxy can be retrieved using the getUser() and getPassword() methods.

String username = proxyconf.getUser().orElse(null);
String password = proxyconf.getPassword().orElse(null);

If they are present we can build a Java Authenticator object and set it as the default authenticator.

import java.net.Authenticator;
import java.net.PasswordAuthentication;

if (username != null && password != null) {
    Authenticator authenticator = new Authenticator() {
        public PasswordAuthentication getPasswordAuthentication() {
           return (new PasswordAuthentication(username, password.toCharArray()));
        }
    };

    Authenticator.setDefault(authenticator);
}

Next we need to create the Proxy object. We need to get the host and port of our proxy server from Mendix using the getHost() and getPort() methods.

import java.net.InetSocketAddress;
import java.net.Proxy;

InetSocketAddress proxyLocation = new InetSocketAddress(proxyconf.getHost(), proxyconf.getPort());
Proxy proxy = new Proxy(Proxy.Type.HTTP, proxyLocation);

The proxy can be used for Java network actions.

An example of using this would be the UK Government Notifications client. It has a second optional paramater in it’s constructor for a Proxy.

client = new NotificationClient('APIKey', proxy);

Using a proxy server from the Mendix Modeller

There are times when building online services you find yourself behind a firewall and need to use a proxy. Sometimes these are transparent, but other times you need to add settings by hand.

In a Mendix app, an example may be when you need to consume a REST service from outside you home network.

To configure proxy settings in Mendix, you need to go to our Project’s “Settings”. Open “Configurations”, select your working configuration, and click “Edit”. Select the “Custom” tab and add the following “Names” and “Values”.

http.proxyHost The name your proxy
http.proxyPort The port your proxy is running off of.

If my proxy was running on proxy.robertprice.co.uk:8080, my settings would be

http.proxyHost proxy.robertprice.co.uk
http.proxyPort 8080

Sometimes the proxy will also need a username and password. You can set these using http.proxyUser and http.proxyPassword. For example

http.proxyUser RobertPrice
http.proxyPassword SecretPassword

You should now be able to access external services through the proxy from Mendix.

Example proxy settings for the Mendix Modeller

More information on using a proxy in Mendix is available at Using a proxy to call a REST service.

Extracting text in Mendix using RegexReplaceAll

It’s a fairly common requirement to be able to extract text from a larger string.

Using Mendix, the easiest way I’ve found to do this is using the RegexReplaceAll Java action from the Community Commons module.

We use a regular expression extract the text, then return this selected text in the action.

For example, take the following string returned from the Nexmo SMS module.

--------- part [ 1 ] ------------Status [ 9 ] ...SUBMISSION FAILED!Message-Id [ null ] ...Error-Text [ Quota Exceeded - rejected ] ...Message-Price [ 0.03330000 ] ...Remaining-Balance [ 0.03200000 ]

If we want to extract the Error-Text we can use the following regular expression.

^.*Error-Text \[ (.*?) \].*$

Here we’re saying look for the text between the square brackets after the string Error-Text. We use round brackets to say we want to remember this matched text. We can then use the regular expression match position to return the matched text – in this case $1.

If we run this over our string we get the following

Quota Exceeded - rejected

To use this in a Mendix microflow, assume we have our status message in a String $StatusMessage that we pass into the Java action. This is our Haystack.

Next, we use the regular expression as a String for our Needle regex.

Finally, we say we want $1 as a String to be our Replacement.

We return the String as $Details.

This is what the microflow and action should look like.

View Robert Price’s Mendix Profile.

Building a simple air quality indicator

I demonstrated Using a bicolour LED with an NodeMCU in an earlier blog.

We now extend that code and instead of having a single LED, we have two. One LED will indicate the current levels of PM2.5, and the other LED will indicate the current levels of PM10. To do this we need to connect to the internet and download these levels from the Clean Air Eastbourne API. The API returns a single float as a string with the current value in µg/m3.

The wiring is very similar to the previous blog post, except we include an extra LED connected to D3 and D4 on the NodeMCU.

The code

The NodeMCU has support for WiFi built in thanks to it’s ESP8266 chip, so it’s very easy to go online in our in setup()loop() function we poll for values every 30 seconds. We turn these values into floats and compare our snapshot against the WHO annual guidelines limits, and UK annual legal limits. If we’re below WHO and UK limits, we show green. If we’re above WHO limits, but below UK limits, we show amber. If we’re above both WHO and UK limits, we show red.

#include <ESP8266WiFi.h>
#include <ESP8266HTTPClient.h>

// Wifi connection details
const char* ssid = "Wifi Network";
const char* password = "Wifi Password";

// Poll URLs
const char* PM10_URL = "http://api.eastbourneair.com/readings/906088-pm10";
const char* PM25_URL = "http://api.eastbourneair.com/readings/906088-pm25";

// How often to poll the website for updates (in microseconds)
const int POLL_DELAY = 30000;

// Limits for particulates from the WHO and UK.
// These are annual limits.
const float WHO_PM10 = 20.0;
const float WHO_PM25 = 10.0;
const float UK_PM10 = 40.0;
const float UK_PM25 = 25.0;

// The TriColour LEDs are wired to the following pins
int redPinPM10 = D2, greenPinPM10 = D1;
int redPinPM25 = D3, greenPinPM25 = D4;

void setup () {
  pinMode(redPinPM10, OUTPUT);
  pinMode(redPinPM25, OUTPUT);
  pinMode(greenPinPM10, OUTPUT);
  pinMode(greenPinPM25, OUTPUT);

  Serial.begin(115200);
  WiFi.begin(ssid, password);

  while (WiFi.status() != WL_CONNECTED) {
    delay(1000);
    Serial.println("Connecting to Wifi...");
  }

}

void loop() {

  if (WiFi.status() == WL_CONNECTED) { //Check WiFi connection status

    HTTPClient http;  //Declare an object of class HTTPClient

    Serial.println("Getting PM10 data...");
    http.begin(PM10_URL);  //Specify request destination
    int httpCode = http.GET();                                        //Send the request
    if (httpCode > 0) { //Check the returning code

      String payload = http.getString();   //Get the request response payload
      Serial.println(payload);             //Print the response payload

      int reading = payload.toFloat();

      digitalWrite(redPinPM10, LOW);
      digitalWrite(greenPinPM10, LOW);
      if (reading >= WHO_PM10 && reading < UK_PM10) {  // turn the LED if over WHO limits, but under UK
        Serial.println("Over WHO guidelines ");
        digitalWrite(redPinPM10, HIGH);
        digitalWrite(greenPinPM10, HIGH);
      } else if (reading >= UK_PM10) {                 // turn the LED on if over UK limits
        Serial.println("Over UK limits");
        digitalWrite(redPinPM10, HIGH);
      } else {
        Serial.println("under limit");
        digitalWrite(greenPinPM10, HIGH);
      }
    } else {
      Serial.println("Unable to get data from API");
    }
    http.end();   //Close connection


    Serial.println("Getting PM2.5 data...");
    http.begin(PM25_URL);  //Specify request destination
    httpCode = http.GET();                                            //Send the request
    if (httpCode > 0) { //Check the returning code

      String payload = http.getString();   //Get the request response payload
      Serial.println(payload);             //Print the response payload

      int reading = payload.toFloat();

      digitalWrite(redPinPM25, LOW);
      digitalWrite(greenPinPM25, LOW);
      if (reading >= WHO_PM25 && reading < UK_PM25) {  // turn the LED if over WHO limits, but under UK
        Serial.println("Over WHO guidelines ");
        digitalWrite(redPinPM25, HIGH);
        digitalWrite(greenPinPM25, HIGH);
      } else if (reading >= UK_PM25) {                 // turn the LED on if over UK limts
        Serial.println("Over UK limits");
        digitalWrite(redPinPM25, HIGH);
      } else {
        Serial.println("under limit");
        digitalWrite(greenPinPM25, HIGH);
      }
    } else {
      Serial.println("Unable to get data from API");
    }
    http.end();   //Close connection

  } else {
    Serial.println("Not connected to Wifi");
  }

  delay(POLL_DELAY);    // wait before we poll again.
}

Running a SDS011 particulate sensor on a Mac using PHP

The Novafit SDS011 particulate sensor is an a very affordable sensor for detecting particulate pollution. It is capable of detecting both PM2.5 and PM10 with a relative error margin of +/- 10µg/m3.

It can output data via it’s serial port. The one I bought came with a serial to USB adaptor, allowing it to be plugged into my Mac. It did need a driver, and I used ch340g-ch34g-ch34x-mac-os-x-driver.

Data is sent at 9600 baud, with 8 data bits, no parity bit, and 1 stop bit. 10 bytes are sent at a time.

Byte Name Content
0 Message Header AA
1 Commander No C0
2 DATA 1 PM2.5 Low byte
3 DATA 2 PM2.5 High byte
4 DATA 3 PM10 Low byte
5 DATA 4 PM10 High byte
6 DATA 5 ID byte 1
7 DATA 6 ID byte 2
8 Check-sum DATA 1+DATA 2+..+DATA 6
9 Message tail AB

PM2.5 (μg /m3) = ((PM2.5 High byte *256) + PM2.5 low byte)/10
PM10 (μg /m3) = ((PM10 high byte*256) + PM10 low byte)/10

We can read this using PHP. The following PHP script can be run on the command line and outputs the PM2.5 and PM10 levels every 2 seconds to a terminal window.

<?php
exec('stty -f /dev/cu.wchusbserial1420 9600 raw');
while (true) {
    $handle = @fopen( '/dev/cu.wchusbserial1420', 'r' ); # Open device for Read access
    if ($handle) {
        $binarydata = fread( $handle, 10 ); # Read data from device
        $data = unpack('H2header/H2commander/vpm25/vpm10/Sid/H2checksum/H2tail', $binarydata);
        echo sprintf("PM2.5: %dµg/m³\nPM10:  %dµg/m³\n", $data['pm25']/10, $data['pm10']/10);
        fclose ($handle); # Close device file
    } else {
        echo "Unable to connect to pollution sensor\n";
    }
    sleep(2);
}

The device is used by reading the 10 bytes it returns each message from the USB device. We can use PHP’s unpack function to break the binary data returned into an associative array for ease of use. We do have to do divide the PM2.5 and PM10 results by 10 to get their real values though. This can then be echo’d to the terminal.

NodeMCU WiFi setup

NodeMCU devices have WiFi built in, and it’s easy to configure using Lua.

One of the easiest examples of setting this up was on Limpkin’s blog. I took this and tweaked it slightly so it works with the more recent Lua wifi module.

station_cfg={}
station_cfg.ssid="Rob's iPhone"
station_cfg.pwd="secret password"
wifi.setmode(wifi.STATION)
wifi.sta.config(station_cfg)
wifi.sta.connect()
tmr.alarm(1, 1000, 1, function()
    if wifi.sta.getip() == nil then
        print("IP unavailable, Waiting...")
    else
        tmr.stop(1)
        print("ESP8266 mode is: " .. wifi.getmode())
        print("The module MAC address is: " .. wifi.ap.getmac())
        print("Config done, IP is "..wifi.sta.getip())
    end
end)

Upload this using luatool or similar as the init.lua file that runs when the NodeMCU powers up.

In this example, the script tries to connect to my iPhone hotspot. We use tmr.alarm to check once a second if we have an IP address. If we don’t, we let the user know. If we do, we stop the timer from continually executing, and print out some of our wifi configuration details.

It’s at this point we could start using the connection to talk to the internet if we needed to.

Using an i2c 128×64 OLED display with Lua on a NodeMCU

Small 128×64 OLED i2c displays can picked up for just over £3 on Amazon, and they are great for interfacing with a NodeMCU IoT device.

Wiring is simple…

  • NodeMCU 3.3v – OLED VCC
  • NodeMCU GND – OLED GND
  • NodeMCU D1 – OLED SDA
  • NodeMCU D2 – OLED SCL or SCK

Wiring an i2c OLED display to a NodeMCU

Your NodeMCU firmware must have been compiled with the i2c and u8g modules included. If you need to add these, see my previous post – NodeMCU Lua setup using a Mac. It also explains how to upload the example Lua code below.

The u8g module comes with some great examples. I’ve taken some of the relevant parts for a simple two page Lua script that prints out two alternating screens of text.

sda = 1 -- SDA Pin
scl = 2 -- SCL Pin

function init_display(sda,scl) --Set up the u8glib lib
     sla = 0x3C
     i2c.setup(0, sda, scl, i2c.SLOW)
     disp = u8g.ssd1306_128x64_i2c(sla)
     disp:setFont(u8g.font_6x10)
     disp:setFontRefHeightExtendedText()
     disp:setDefaultForegroundColor()
     disp:setFontPosTop()
end

function updateDisplay(func)
  -- Draws one page and schedules the next page, if there is one
  local function drawPages()
    func()
    if (disp:nextPage() == true) then
      node.task.post(drawPages)
    end
  end
  -- Restart the draw loop and start drawing pages
  disp:firstPage()
  node.task.post(drawPages)
end

function drawScreen1()
     disp:drawFrame(2,2,126,62)
     disp:drawFrame(5,5,121,57)
     disp:drawStr(5, 20, "  RobertPrice.co.uk")
     disp:drawStr(5, 35, "    @robertprice")
end

function drawScreen2()
     disp:drawFrame(2,2,126,62)
     disp:drawFrame(5,5,121,57)
     disp:drawStr(5, 20, "  NodeMCU OLED Test")
     disp:drawStr(5, 35, "   12th July 2017")
end

local drawDemo = { drawScreen1, drawScreen2 }

function demoLoop()
  -- Start the draw loop with one of the demo functions
  local f = table.remove(drawDemo,1)
  updateDisplay(f)
  table.insert(drawDemo,f)
end

init_display(sda,scl)
demoLoop()
tmr.alarm(4, 5000, 1, demoLoop)

This may look a bit complex for just displaying text, but it aims to be reusable. We use the tmr.alarm to avoid bogging the CPU down in a loop.

NodeMCU driving an i2c OLED display

I’ve tried this with two different i2c OLED displays, and they both work.

NodeMCU Lua setup using a Mac

I recently bought a NodeMCU. This is a small ESP8266 based card with built in WiFi, MicroUSB, and a Lua interpreter that can be used for developing IoT (Internet of Things) devices.

I had trouble getting it working initially, so I wanted to share how I fixed this on a Mac.

Install Mac drivers

If you’re not using a Mac, you can skip this part.

You plug the NodeMCU into the Mac via USB. The Mac won’t support this by default, and you need to install a driver. The driver you need to install is the Silcon Labs CP210x USB to UART Bridge.

Once installed, plug in the NodeMCU and check that the device /dev/tty.SLAB_USBtoUART exists.

To connect to the NodeMCU, you’ll need some tools. To test, download CoolTerm. In Options, for Port select SLAB_USBtoUART, and for Baudrate select 115200.

If you are lucky you’ll get a prompt, if not you may need to build and install some new firmware.

If you do get a prompt type…

print "Hello";

… and Lua should echo “Hello” back to you.

Build and install the NodeMCU firmware

If you need to build new firmware, there is a very useful online site called NodeMCU-Build.com that I used to build the firmware with the right modules I wanted for my project.

Once you’ve build the firmware, you’ll need to install the Python esptool.py to flash the firmware to the NodeMCU.

git clone https://github.com/themadinventor/esptool.git
cd esptool
sudo python ./setup.py install

Check the flash size of your NodeMCU.

To flash the firmware hold down “FLASH” and press “RST” on the NodeMCU, then use the following command (remembering to disconnect CoolTerm first if connected)…

esptool.py --port /dev/tty.SLAB_USBtoUART write_flash -fm dio 0x00000 ~/Downloads/nodemcu-master-12-modules-2017-07-07-21-24-03-float.bin

As esptool.py runs, you should see something like this…

esptool.py v2.0.1
Connecting........_
Detecting chip type... ESP8266
Chip is ESP8266
Uploading stub...
Running stub...
Stub running...
Configuring flash size...
Auto-detected Flash size: 4MB
Flash params set to 0x0240
Compressed 754992 bytes to 505060...
Wrote 754992 bytes (505060 compressed) at 0x00000000 in 44.5 seconds (effective 135.6 kbit/s)...
Hash of data verified.

Leaving...
Hard resetting...

Connect to it using CoolTerm, then press the “RST” button. You should see some messy characters, then something like following (depending on what you built into your firmware)…

NodeMCU custom build by frightanic.com
.branch: master
.commit: c8ac5cfb912ff206b03dd7c60ffbb2dafb83fe5e
.SSL: true
.modules: adc,bit,cron,crypto,encoder,file,gpio,http,i2c,net,node,ow,pcm,sjson,sntp,spi,struct,tmr,u8g,uart,websocket,wifi,tls
 build .built on: 2017-07-12 09:30
 powered by Lua 5.1.4 on SDK 2.1.0(116b762)
lua: cannot open init.lua
>

Upload Lua code to the NodeMCU

The best way to get Lua code onto the NodeMCU is to use the Python luatool.

git clone https://github.com/4refr0nt/luatool
cd luatool

To test we’ll upload a simple Lua script that will blink the NodeMCU’s onboard LED, save this as “blink.lua”.

LED_PIN = 0

gpio.mode(LED_PIN, gpio.OUTPUT)
value = true

tmr.alarm(0, 500, 1, function ()
    gpio.write(LED_PIN, value and gpio.HIGH or gpio.LOW)
    value = not value
end)

Upload it to the NodeMCU using the following command (making sure Coolterm is disconnected first)…

python luatool/luatool.py --port /dev/tty.SLAB_USBtoUART --src blink.lua --dest init.lua --dofile

You should see the following…

->file.open("init.lua", "w") -> ok
->file.close() -> ok
->file.remove("init.lua") -> ok
->file.open("init.lua", "w+") -> ok
->file.writeline([==[LED_PIN = 0]==]) -> ok
->file.writeline([==[]==]) -> ok
->file.writeline([==[gpio.mode(LED_PIN, gpio.OUTPUT)]==]) -> ok
->file.writeline([==[value = true]==]) -> ok
->file.writeline([==[]==]) -> ok
->file.writeline([==[tmr.alarm(0, 500, 1, function ()]==]) -> ok
->file.writeline([==[gpio.write(LED_PIN, value and gpio.HIGH or gpio.LOW)]==]) -> ok
->file.writeline([==[value = not value]==]) -> ok
->file.writeline([==[end)]==]) -> ok
->file.flush() -> ok
->file.close() -> ok
->dofile("init.lua") -> send without check
--->>> All done <<<--

The onboard LED on the NodeMCU should now be blinking.

The NodeMCU executes the init.lua script by default, so when we upload we tell luatool to upload our blink.lua script as init.lua.