Hardware Access
and Device APIs
with JavaScript and HTML5

Wes Bos
wesbos.com
@wesbos

September 27, 2012

Hey there, I'm Wes Bos

Independent Web UI Designer / Web Developer

I design and build the front end of applications and websites.

Huge JavaScript / CSS3 / HTML5 Fan.

Instructor at Ladies Learning Code and HackerYou

I hang out on Twitter @wesbos

Also on LinkedIn linkedin.com/in/wesbos

This Talk

About accessing hardware inside mobile browsers

Device APIs that we have today, or ones that are coming in the near future

No phonegap, no ruby motion, no webviews.

A Word on native vs HTML5

Obviously HTML5 isn't a solution to everything

Many stories to support both sides of the argument

This talk isn't about how HTML5 is going to replace native

It's about how we finally are getting the same APIs in the browser, and what we can use them for in both apps and mobile websites.

Hardware Access

Today we look at 5 different types of hardware access for the mobile web and then leave some time at the end for questions.

Geolocation has been around for quite some time now and is widely supported by most devices.

Permission is required by the user before you can access a devices location

The Codes

To Access it with javascript, it couldn't be simpler!

On navigator we have an object called geolocation with a few available methods.

For a one time position, we use the following code:

navigator.geolocation.getCurrentPosition(function(data){
    console.log('Oh hey, we got you some',data);
});

Watching position

Sometimes you will want to watch a device's position as they move around. Instead of setting up an interval, we let the device tell us whenever they have new geolocation information available.

navigator.geolocation.watchPosition(
    function(data){
        console.log('New Data!',data);
    },
    function(err){
        console.log('error man',err);
    }
);

Want to save your battery? You can also pass in an object as a third argument with one of the possible values being:

{enableHighAccuracy:false}

How does it work?

Each mobile browser implements geolocation in their own way, but there are three main ways:

There are many uses for geolocation, most of them obvious - see where in the world a device is.

One of the cooler implementation is geofencing

With geofencing we create virtual areas and boundaries that a device must be inside of to perform tasks.

This opens up the doors many applications that are based on a users current location

Remember this output from: navigator.geolocation.getCurrentPosition()


You'll see the speed is returned as null. On mobile devices, speed is returned as meters per second.

Cool!

I thought it would be fun to try and make a pure HTML5/CSS3 Speedometer

Start off with some basic HTML

<div class="meterInner metal">
    <span class="needle">
        <span class="needleInner"></span>
    </span>
    <ul class="ticks">
        <li></li><li></li><li></li><li></li><li></li><li></li><li></li><li></li><li></li><li></li><li></li><li></li><li></li><li></li><li></li><li></li><li></li><li></li><li></li>
    </ul>
    <div class="speedWrapper">
        <span class="speed">255</span>
        <span class="speedMeasure">kph</span>
    </div>
</div>

Use the geolocation API and the watchPosition() method to track the speed

navigator.geolocation.watchPosition(function(d){
    sp.writeSpeed(d);
});

And then calculating the speed and visually displaying it was a breeze too

sp.writeSpeed = function(d) {
    var currSpeed = d.coords.speed; // Meters per second
        currSpeed = currSpeed * 60 * 60; // meters per hour
        currSpeed = currSpeed / 1000; // k/hr
        // Check if we are converting to MPH
        if (sp.localStorage.measure === 'm') {
            currSpeed = currSpeed * 0.621371192;    
        }
        
        // needle
        var speedPercentage = currSpeed / sp.opts.maxSpeed,
            degrees = speedPercentage * 180;
        $('.meter .needle').css({'-webkit-transform' : 
        'rotate('+degrees+'deg) translateZ(0) translate3d(0,0,0)'});
}

But does it work while circling on an airplane?

Perhaps the biggest limitation of mobile websites in the past was not being able to access a users filesystem / camera roll from the browser.

What if we want the raw binaries of files right inside the browser?

Examples:

Enter File input

An old friend in the browser world, been around in Android for some time, but only recently supported in iOS 6.

There are no polyfills or shims, so its been kind of a showstopper

UNTIL:

<input type="file">

Clicking this brings up the Camera roll

So, what can we do with that?

Obviously we can upload the file to the server

BORING

But its 2012, we want to work with files in the browser!

FileReader

The HTML5 spec allows browsers to access a local filesystem or files that a user selects.

Once selected, the browser has the blob or the raw binary data.

This means we can upload images and manipulate them right inside the browser!

Crapstagram

A simple use case for this is building a crappy version of instagram but doing it entirely in the browser.

Demo Time

Let's step through how we did this

  1. Create a webpage with 1 file input and 1 canvas
  2. Use the file input to select a photo from the camera roll
  3. Use JS FileReader() to read the file
  4. Spit the image into the canvas
  5. Manipulate the canvas pixels so it looks like crappy filter
  6. Optional: Export canvas to JPG

The Accelerometer is a chip inside devices that gives us info on how the user is holding and moving their device.

It is helpful for determining what orientation someone is holding the phone, and has been used in a ton of games.

Accessing it with JavaScript is really easy and can be a ton of fun!

Accelerometer Events

To hook into the accelerometer, you simple bind to the following events:

deviceorientation MozOrientation devicemotion

There is some normalization that needs to be done across devices

webkitCompassHeading

If we take a look at the previously mentioned deviceorientation event, we will see something called webkitCompassHeading

This returns the number of degrees relative to north.

You can also get webkitCompassAccuracy with is the # of degrees it is accurate to.

Code

window.addEventListener('deviceorientation', function(e) {
    var rel = e.webkitCompassHeading,
        giveOrTake = e.webkitCompassAccuracy;
    console.log('You are',rel,'degrees relative of north, 
    give or take',giveOrTake);

});

Example

James Pearce wrote a really great example of how it works

http://tripleodeon.com/2011/10/taking-a-new-device-api-for-a-spin/ http://jamesgpearce.github.com/compios5/

Web Real Time Communications

There is a new technology on the block called WebRTC

Still heavily under development but we are able to create something called a PeerConnection which allows for a 1:1 channel between two peers.

We can stream audio, video and eventually data/files over this connection without relying on a server in the middle.

We would only need websockets/xmlHTTPRequest to do the initial handshake

The real benefit here is that we will be seeing HTML5 Chatroulette clones!

What this really means...

We are now able to get webcam access, 100% in HTML/JavaScript!

Thats right, no flash required.

You are able to stream your webcam right into an HTML5 Video element.

We use a method called getUserMedia();

Enter Canvas

Once the video is streaming into a HTML5 video element, you can then dump the frames into a canvas.

HTML5 Canvas gives you pixel level detail of an image, and we can play with those pixels!

I've done some fun stuff on the desktop


Start Stop Hipster Threshold Blur Green HTML5 Glasses

Web RTC on Mobile

So, thats cool on the desktop, but what about mobile?

WebRTC is very new, so not very many mobile or desktop browsers have implemented getUserMedia();

Lucky for us, Opera Mobile for android does support it! (and has for a while)

Story Time

In order to grasp the concept and use of this, I need to tell you about my #FirstWorldProblem

Hrm.

I don't want to get up and check every few hours,
like a sucker.

Solution!

Wes' Law:
Any problem will go away if you throw enough technology at it.

4 Requirements

Step #1


Pull camera video feed into a web page, dump into canvas, and take a frame every 3 seconds.

Step #2


Write some code to analyze and compare that frame to the previous one

Step #3

If the two frames are different enough, we have detected motion.

Motion Detection

Already at step 3, we have a motion detection / security camera

Not good enough, I need something to notify me and I want it to be instant!

We have two files:

client.html - running on the device with a camera

viewer.html - running on anything that wants to view detected frames

When motion is detected, send that image over a websocket to everyone with viewer.html open.

At this time, we can also use the node.js server to send emails, trigger SMS, push notifications and so on...

Final Step

Anyone on the other end (viewer.html), sees a stream of triggered photos.

This could obviously be stepped up with email/sms/growl

I waited...

and waited...

and waited...

and waited...

until suddenly!

#FirstWorldProblem SOLVED

Technology Review:

Device API Overview

Device APIs in the browser let you access the hardware directly like many native apps do

While they aren't your common browser APIs, they can definitely come in handy as well as open the door for some previously impossible use cases

Device support in some areas is great, and others have a long road ahead of them.

As browsers become faster and hardware get beefier, we are able to push the limits of what we can do in a website.

Thats it!

Questions?

@wesbos / LinkedIn

All code / slides will be up online soon if not already on github.

Thanks!