NodeMCU WiFi setup

NodeMCU devices have WiFi built in, and it’s easy to configure using Lua.

One of the easiest examples of setting this up was on Limpkin’s blog. I took this and tweaked it slightly so it works with the more recent Lua wifi module.

station_cfg={}
station_cfg.ssid="Rob's iPhone"
station_cfg.pwd="secret password"
wifi.setmode(wifi.STATION)
wifi.sta.config(station_cfg)
wifi.sta.connect()
tmr.alarm(1, 1000, 1, function()
    if wifi.sta.getip() == nil then
        print("IP unavailable, Waiting...")
    else
        tmr.stop(1)
        print("ESP8266 mode is: " .. wifi.getmode())
        print("The module MAC address is: " .. wifi.ap.getmac())
        print("Config done, IP is "..wifi.sta.getip())
    end
end)

Upload this using luatool or similar as the init.lua file that runs when the NodeMCU powers up.

In this example, the script tries to connect to my iPhone hotspot. We use tmr.alarm to check once a second if we have an IP address. If we don’t, we let the user know. If we do, we stop the timer from continually executing, and print out some of our wifi configuration details.

It’s at this point we could start using the connection to talk to the internet if we needed to.

Using an i2c 128×64 OLED display with Lua on a NodeMCU

Small 128×64 OLED i2c displays can picked up for just over £3 on Amazon, and they are great for interfacing with a NodeMCU IoT device.

Wiring is simple…

  • NodeMCU 3.3v – OLED VCC
  • NodeMCU GND – OLED GND
  • NodeMCU D1 – OLED SDA
  • NodeMCU D2 – OLED SCL or SCK

Wiring an i2c OLED display to a NodeMCU

Your NodeMCU firmware must have been compiled with the i2c and u8g modules included. If you need to add these, see my previous post – NodeMCU Lua setup using a Mac. It also explains how to upload the example Lua code below.

The u8g module comes with some great examples. I’ve taken some of the relevant parts for a simple two page Lua script that prints out two alternating screens of text.

sda = 1 -- SDA Pin
scl = 2 -- SCL Pin

function init_display(sda,scl) --Set up the u8glib lib
     sla = 0x3C
     i2c.setup(0, sda, scl, i2c.SLOW)
     disp = u8g.ssd1306_128x64_i2c(sla)
     disp:setFont(u8g.font_6x10)
     disp:setFontRefHeightExtendedText()
     disp:setDefaultForegroundColor()
     disp:setFontPosTop()
end

function updateDisplay(func)
  -- Draws one page and schedules the next page, if there is one
  local function drawPages()
    func()
    if (disp:nextPage() == true) then
      node.task.post(drawPages)
    end
  end
  -- Restart the draw loop and start drawing pages
  disp:firstPage()
  node.task.post(drawPages)
end

function drawScreen1()
     disp:drawFrame(2,2,126,62)
     disp:drawFrame(5,5,121,57)
     disp:drawStr(5, 20, "  RobertPrice.co.uk")
     disp:drawStr(5, 35, "    @robertprice")
end

function drawScreen2()
     disp:drawFrame(2,2,126,62)
     disp:drawFrame(5,5,121,57)
     disp:drawStr(5, 20, "  NodeMCU OLED Test")
     disp:drawStr(5, 35, "   12th July 2017")
end

local drawDemo = { drawScreen1, drawScreen2 }

function demoLoop()
  -- Start the draw loop with one of the demo functions
  local f = table.remove(drawDemo,1)
  updateDisplay(f)
  table.insert(drawDemo,f)
end

init_display(sda,scl)
demoLoop()
tmr.alarm(4, 5000, 1, demoLoop)

This may look a bit complex for just displaying text, but it aims to be reusable. We use the tmr.alarm to avoid bogging the CPU down in a loop.

NodeMCU driving an i2c OLED display

I’ve tried this with two different i2c OLED displays, and they both work.

NodeMCU Lua setup using a Mac

I recently bought a NodeMCU. This is a small ESP8266 based card with built in WiFi, MicroUSB, and a Lua interpreter that can be used for developing IoT (Internet of Things) devices.

I had trouble getting it working initially, so I wanted to share how I fixed this on a Mac.

Install Mac drivers

If you’re not using a Mac, you can skip this part.

You plug the NodeMCU into the Mac via USB. The Mac won’t support this by default, and you need to install a driver. The driver you need to install is the Silcon Labs CP210x USB to UART Bridge.

Once installed, plug in the NodeMCU and check that the device /dev/tty.SLAB_USBtoUART exists.

To connect to the NodeMCU, you’ll need some tools. To test, download CoolTerm. In Options, for Port select SLAB_USBtoUART, and for Baudrate select 115200.

If you are lucky you’ll get a prompt, if not you may need to build and install some new firmware.

If you do get a prompt type…

print "Hello";

… and Lua should echo “Hello” back to you.

Build and install the NodeMCU firmware

If you need to build new firmware, there is a very useful online site called NodeMCU-Build.com that I used to build the firmware with the right modules I wanted for my project.

Once you’ve build the firmware, you’ll need to install the Python esptool.py to flash the firmware to the NodeMCU.

git clone https://github.com/themadinventor/esptool.git
cd esptool
sudo python ./setup.py install

Check the flash size of your NodeMCU.

To flash the firmware hold down “FLASH” and press “RST” on the NodeMCU, then use the following command (remembering to disconnect CoolTerm first if connected)…

esptool.py --port /dev/tty.SLAB_USBtoUART write_flash -fm dio 0x00000 ~/Downloads/nodemcu-master-12-modules-2017-07-07-21-24-03-float.bin

As esptool.py runs, you should see something like this…

esptool.py v2.0.1
Connecting........_
Detecting chip type... ESP8266
Chip is ESP8266
Uploading stub...
Running stub...
Stub running...
Configuring flash size...
Auto-detected Flash size: 4MB
Flash params set to 0x0240
Compressed 754992 bytes to 505060...
Wrote 754992 bytes (505060 compressed) at 0x00000000 in 44.5 seconds (effective 135.6 kbit/s)...
Hash of data verified.

Leaving...
Hard resetting...

Connect to it using CoolTerm, then press the “RST” button. You should see some messy characters, then something like following (depending on what you built into your firmware)…

NodeMCU custom build by frightanic.com
.branch: master
.commit: c8ac5cfb912ff206b03dd7c60ffbb2dafb83fe5e
.SSL: true
.modules: adc,bit,cron,crypto,encoder,file,gpio,http,i2c,net,node,ow,pcm,sjson,sntp,spi,struct,tmr,u8g,uart,websocket,wifi,tls
 build .built on: 2017-07-12 09:30
 powered by Lua 5.1.4 on SDK 2.1.0(116b762)
lua: cannot open init.lua
>

Upload Lua code to the NodeMCU

The best way to get Lua code onto the NodeMCU is to use the Python luatool.

git clone https://github.com/4refr0nt/luatool
cd luatool

To test we’ll upload a simple Lua script that will blink the NodeMCU’s onboard LED, save this as “blink.lua”.

LED_PIN = 0

gpio.mode(LED_PIN, gpio.OUTPUT)
value = true

tmr.alarm(0, 500, 1, function ()
    gpio.write(LED_PIN, value and gpio.HIGH or gpio.LOW)
    value = not value
end)

Upload it to the NodeMCU using the following command (making sure Coolterm is disconnected first)…

python luatool/luatool.py --port /dev/tty.SLAB_USBtoUART --src blink.lua --dest init.lua --dofile

You should see the following…

->file.open("init.lua", "w") -> ok
->file.close() -> ok
->file.remove("init.lua") -> ok
->file.open("init.lua", "w+") -> ok
->file.writeline([==[LED_PIN = 0]==]) -> ok
->file.writeline([==[]==]) -> ok
->file.writeline([==[gpio.mode(LED_PIN, gpio.OUTPUT)]==]) -> ok
->file.writeline([==[value = true]==]) -> ok
->file.writeline([==[]==]) -> ok
->file.writeline([==[tmr.alarm(0, 500, 1, function ()]==]) -> ok
->file.writeline([==[gpio.write(LED_PIN, value and gpio.HIGH or gpio.LOW)]==]) -> ok
->file.writeline([==[value = not value]==]) -> ok
->file.writeline([==[end)]==]) -> ok
->file.flush() -> ok
->file.close() -> ok
->dofile("init.lua") -> send without check
--->>> All done <<<--

The onboard LED on the NodeMCU should now be blinking.

The NodeMCU executes the init.lua script by default, so when we upload we tell luatool to upload our blink.lua script as init.lua.

Analysing Beat The Street data using R

Beat The Street is a game currently running in East Sussex with the aim of increasing the amount of walking and cycling people do. Players are given a free Beat The Street card that they tap on Beat Boxes to record their journeys. At least 2 different Beat Boxes must be tapped in the space of an hour for points to be recorded. Each 2nd box tapped within an hour of the last earns at least 10 points.

While there isn’t an officially documented API to the Beat The Street data, the website itself is built using JSON feeds we can use to extract data ourselves. There can be multiple schemes running at once. I’m playing in East Sussex which is scheme 47, we’ll need this ID shortly.

Firstly we’ll need to login as this sets a cookie that identifies us for subsequent calls.

library(httr)
loginurl <- "https://beatthestreet.me/api/Login?schemeId=47"
loginbody <- list(Username = "Your Username", Password = "Your Password", RememberMe = "false")
r <- POST(loginurl, body = loginbody, encode = "form", verbose())

We can use R's httr library to handle the login for us. We POST our details to the form. I'm using the verbose() option for debugging, but that's optional.

Now we are logged in, we should have a cookie set identifying us, so we can request a record of our journeys so far.

swipesurl <- "https://beatthestreet.me/api/UserCardSwipesWidget?schemeId=47"
r <- GET(swipesurl, accept_json())
cont <- content(r, as = "parsed", type="application/json")

The returned JSON data is an array of objects looking like this.

[
    {
        "SwipeDatetime":"2017-07-11T07:18:38",
        "Name":"E019",
        "Description":"E.Sussex",
        "Line1":
        "N/A",
        "Line2":"N/A",
        "Line3":"N/A",
        "Postcode":"ES1",
        "City":"E.Sussex",
        "Points":10
    },{
        "SwipeDatetime":"2017-07-11T07:14:40",
        "Name":"E024",
        "Description":"E.Sussex",
        "Line1":"N/A",
        "Line2":"N/A",
        "Line3":"N/A",
        "Postcode":"ES1",
        "City":"E.Sussex",
        "Points":0
    }
]

We can parse this by calling content().

The values are all strings apart from "Points" which is an integer. However, sometimes Beat The Street returns a null value, which we'll want to convert to an R NA. We can do this by applying a small utility function over the data to convert any nulls to NA.

nullToNA <- function(x) {
    x[sapply(x, is.null)] <- NA
    return(x)
}
cont <- lapply(cont, nullToNA)

Now we can convert our data to a data.frame, cast the "SwipeDatetime" as a DateTime, and add in an extra field called "Date" that's just the Date without the time.

library(plyr)
swipes <- ldply(cont, data.frame)
swipes$SwipeDatetime <- as.POSIXct(swipes$SwipeDatetime, format = "%Y-%m-%dT%H:%M:%S")
swipes$Date <- as.Date(swipes$SwipeDatetime)

Now we have our data in usable format, we can extract some useful information.

I can plot a graph of how many Beat Boxes I've tapped each day.

library(ggplot2)
ggplot(swipes, aes(Date)) + geom_bar(stat="count") + ggtitle("Beat Boxes Visited Per Day") + ylab("Visits") + xlab("Date") + theme(axis.text.x = element_text(angle = 90, hjust = 1))

Number of Beat Boxes visited by day

Or I can plot which Beat Boxes I visit most.

swipesFrame <- data.frame(sort(table(swipes$Name)))
ggplot(data=swipesFrame, aes(x=Var1, y=Freq)) + geom_bar(stat="identity") + ggtitle("Most Visited Beat Boxes") + ylab("Visits") + xlab("Beat Box") + theme(axis.text.x = element_text(angle = 90, hjust = 1))

Most visited Beat Boxes

I can even plot this onto a map assuming I have the latitude and longitudes of the Beat Boxes already available in a data.frame called "beatboxes", and the map in "map". The larger the frequency of visit, the larger the plotted point on the map.

library(ggmap)
swipeFrameLatLng <- merge(swipesFrame, beatboxes, by.x = "Var1", by.y="id")
ggmap(map) + geom_point(data = swipeFrameLatLng, aes(x = longitude, y = latitude, color="Red", size=Freq*10)) + ylab("") + xlab("") + guides(colour = "colorbar",size = "none")

Most visited Beat Boxes map

Using file_get_contents to PUT JSON data over HTTPS to a webservice

It’s possible to use PHP’s file_get_contents to PUT JSON data to a RESTful webservice if you don’t want to use cURL.

In this example I’m going to try to add or update a person using the NationBuilder API.

Firstly, we need the URL of the webservice, and the data you want to update as a JSON string.

$access_token = 'set access token here';
$nation = 'set nation name here';
$url = "https://$nation.nationbuilder.com/api/v1/people/push?access_token=$access_token";
// details we want to update
$data = [
    'person' => [
        'email_opt_in' => true,
        'do_not_contact' => false,
        'do_not_call' => false,
        'first_name' => 'Robert',
        'last_name' => 'Price',
        'email' => 'robert.price@email.adress',
        'home_address' => [
            'zip' => 'AB1 2CD'
        ],
        'bio' => 'Software developer - robertprice.co.uk'
    ]
];
$json_data = json_encode($data);

Now we need to setup a stream_context so file_get_contents knows how to handle the request.

Even though we’re PUTing data over HTTPS, we need to use the http context. We set method to PUTContent-type to application/json, Accept to application/json, Connection to close, and Content-length to the length of our JSON string. We’re PUTing this data over HTTP1.1 so we set the to 1.1. We’re sending the JSON in the body of the request, so we set this in the content.

As we are using HTTPS, we need to configure the ssl settings. In this case I’m not going to verify the peer or the peer name.

$context = stream_context_create([
    'http' => [
        'method' => 'PUT',
        'header' => "Content-type: application/json\r\n" .
                    "Accept: application/json\r\n" .
                    "Connection: close\r\n" .
                    "Content-length: " . strlen($json_data) . "\r\n",
        'protocol_version' => 1.1,
        'content' => $json_data
    ],
    'ssl' => [
        'verify_peer' => false,
        'verify_peer_name' => false
    ]
]);

Finally we PUT the data by calling file_get_contents. NationBuilder returns a JSON response to this call, so we can decode that and echo a confirmation message to the screen.

$rawdata = file_get_contents($url, false, $context);
if ($rawdata === false) {
    exit("Unable to update data at $url");
}
$data = json_decode($rawdata, true);
if (JSON_ERROR_NONE !== json_last_error()) {
    exit("Failed to parse json: " . json_last_error_msg());
}
echo "Updated details for {$data['person']['first_name']} {$data['person']['last_name']}\n";

The full code for this, and some other examples of querying NationBuilder using PHP, can be found in my NationBuilder People In Nation – GitHub repository.

You may also be interested in my previous article on POSTing JSON to Web Service.

Sparkline Creation Using PHP

Sparkline graphs have had a bit of press lately, so I’ve been playing about with generating them on my commutes to and from work.

If you’ve not heard of them before, a Sparkline is a small graph that is often used inline with text.

I’ve been using PHP, so I’ve created a simple class called Spark, with a single static method called line. This returns a simple string as an inline graph from an array of values that have been passed in.

Sparkline Examples

Here’s an example plotting a simple staircase type graph.

require_once('lib/Spark.php');

echo Spark::line(array(1, 2, 3, 4, 5, 6, 7, 8)) . "n";

> ▁▂▃▄▅▆▇█

Here’s an example plotting a cosine wave.

$cos = array();
for($i=0; $i <= 360; $i=$i+20) {
    $cos[] = cos($i * M_PI / 180);
}
echo Spark::line($cos) . "n";

> ██▇▆▅▄▃▂▁▁▁▂▃▄▅▆▇██

Summary

The code is available on GitHub as php-spark, and is free to use. It was inspired by a simple project for Go called Spark, which was inspired from elsewhere.

Fixing The National Rail Mobile Website For iPhones

I’ve been frustrated that the National Rail Enquiries mobile website doesn’t work properly on my Apple iPhone 4s, running iOS7, so I decided to take a look and see what is actually going on.

You can turn on debugging on your iPhone in the settings, so I turned this on, and loaded Safari on my Macbook Pro to take a look at what was actually happening.

The “Go” button on the mobile website acts as a submit button so I put a breakpoint on it to see what happened.

The event handler calls a function called setRecents, and it fails when it tries to save my recent stations to localStorage.

if (hasStorage) {
    localStorage.setItem("recentStations", recVal);
} else {
    setCookie("recentStations", recVal, 365);
}

The hasStorage global variable has a true value, allowing it to try to save to local storage, so I tried setting this to false and tried again. This time to website worked as it falls back to using cookies. So the problem is with localStorage on mobile Safari, I’m not sure why.

I then took a look to see how hasStorage was set. It was being set in the following function.

function checkStorage() {
    // Feature detect + local reference
    var storage,
    fail,
    uid;
    try {
        uid = new Date;
        (storage = window.localStorage).setItem(uid, uid);
        fail = storage.getItem(uid) != uid;
        storage.removeItem(uid);
        fail && (storage = false);
    } catch (e) {}
    if (storage) {
        hasStorage = true; //if localstorage available
    }
};

I set breakpoints through the function, reloaded and single stepped what was happening in the debugger.

The uid was correctly set to a Date object with the current date and time. The next line was where the problem occured. storage was assigned as a Storage object, but the setItem method failed. This meant that the storage variable was set, but an exception was thrown. The exception handler does nothing, so the line after was executed. This sees that storage is available, but not that it didn’t work, so sets hasStorage incorrectly to true.

The fix would be to modify the exception handler to set storage to false instead of doing nothing.

If any developers of the National Rail website are reading this, please add this line so I can use your website to plan my train journeys again using my iPhone!

The National Rail mobile website on an iPhone

All information correct at time of writing and publishing. 20th April 2014.

Returning A 410 HTTP Status Code In Apache

I had an SEO requirement to return a 410 HTTP status code for any URL’s with /advanced-search/ in them.

The following snippet can be added to your Apache config to achieve this.

# Return a 410 for any URL's with advanced-search in
RewriteCond %{REQUEST_URI} /advanced-search/
RewriteRule ^.*$ - [G]

Here we’re looking in the URI for the words /advanced-search/ and if found, the RewriteRule below is activated.

The RewriteRule takes the URL, keeps it intact and sets the [G] flag. The [G] flag forces the server to return a 410 Gone status as it’s response.

If Apache sees a [G] flag, it returns immediately and no further rewrite rules are evaluated.

A 410 HTTP status code indicates that a resource was once available, but isn’t any more.

Vagrant Failing To Launch

I use Vagrant based virtual machines to develop on. These are great as you can create custom environments at will without affecting your main machine.

Today I had the following error while trying to bring up my development environment.

wks:Vagrant robertprice$ vagrant up devvm
[ccfs] VM already created. Booting if it's not already running...
[ccfs] Clearing any previously set forwarded ports...
[ccfs] Forwarding ports...
[ccfs] -- 22 => 2222 (adapter 1)
[ccfs] -- 80 => 4567 (adapter 1)
[ccfs] Creating shared folders metadata...
[ccfs] Clearing any previously set network interfaces...
There was an error executing the following command with VBoxManage:

["hostonlyif", "create"]

For more information on the failure, enable detailed logging with
VAGRANT_LOG.

This turned out to be an error with VirtualBox, the virtualisation environment Vagrant uses. The solution is to force a restart of VirtualBox.

sudo /Library/StartupItems/VirtualBox/VirtualBox restart

I hope this helps others experiencing the same problem.

Data URI’s – Using and Generating

A recent project of mine needed an image embedding into some HTML via JavaScript. Rather than use a separate image, I decided to embed it directly using a data URI.

An image in a data URI is the MIME type of the image and it’s content encoded with base64 into a string. This is great as it cuts down HTTP requests but does cause the initial page weight to increase and be difficult to update as each change means the image needs re-encoding. Modern browsers support data URI’s very well, but older browsers such as IE 7 and below won’t like it.

Examples Using A Data URI Encoded Image

HTML Example

Here’s how I can embed the image of a red cross into an HTML <img> tag.

<img src="data:image/gif;base64,R0lGODlhFAAUAJEAAP/9/fYQEPytrflWViH5BAAAAAAALAAAAAAUABQAQAJKhI+pGe09lnhBnEETfodatVHNh1BR+ZzH9LAOCYrVYpiAfWWJOxrC/5MASbyZT4d6AUIBlUYGoR1FsAXUuTN5YhxAEYbrpKRkQwEAOw==" alt="red cross" width="20" height="20" />

CSS Example

Here’s how I can embed the image of a red cross into background of an HTML element using CSS.

body { 
  background: url(data:image/gif;base64,R0lGODlhFAAUAJEAAP/9/fYQEPytrflWViH5BAAAAAAALAAAAAAUABQAQAJKhI+pGe09lnhBnEETfodatVHNh1BR+ZzH9LAOCYrVYpiAfWWJOxrC/5MASbyZT4d6AUIBlUYGoR1FsAXUuTN5YhxAEYbrpKRkQwEAOw==) no-repeat left center;
}

JavaScript Example

Here’s how I can add an image element with the red cross in to an HTML page using JavaScript.

var imagestring = "data:image/gif;base64,R0lGODlhFAAUAJEAAP/9/fYQEPytrflWViH5BAAAAAAALAAAAAAUABQAQAJKhI+pGe09lnhBnEETfodatVHNh1BR+ZzH9LAOCYrVYpiAfWWJOxrC/5MASbyZT4d6AUIBlUYGoR1FsAXUuTN5YhxAEYbrpKRkQwEAOw==";
var image = new Image();
image.src = imagestring;
image.onload = function() {
  document.body.appendChild(image);  
}

Encoding An Image To A Data URI

It’s easy to create the encoded image string using PHP as it comes with a Base64 encoder as part of the language. Automatically detecting the MIME type of an image is a bit harder, but we can use finfo_file with comes as an extension to PHP 5.3 and above to do this.

So assuming the filename of the image is in the variable $filename we can use the following code to get the mimetype, read the image and encode it to a data URI string.

$finfo = finfo_open(FILEINFO_MIME_TYPE);
$mimetype = finfo_file($finfo, $filename);
finfo_close($finfo);

$contents = file_get_contents($filename);
echo "data:" . $mimetype . ";base64," . base64_encode($contents);

Conclusion

We’ve seen it’s easy to embed encoded images into code. I have wrapped the encoding routine into a command line PHP script and placed it on GitHub as php-base64-encode so it’s easy to quickly generate data URI’s.