Testing & Comparing Bluetooth Beacons

Most apps that use location analytics leverage a device’s built-in GPS, WiFi or cellular network functionality to generate an approximation of where the user is in the world, and can usually be considered reliable when determining the user’s current city or suburb.

This works fine for automatically adding the suburb to a Facebook post or for rights management on streaming services like Netflix.

But what if an app needs to track a user’s location accurately to the nearest metre – or even a few centimetres?

This presents challenges for apps like Commbank Offers, a collaboration between Westfield and Commonwealth Bank, where shoppers are presented with coupons (such as discounted coffee) based on their current location inside the shopping centre.

"The problem with tracking devices using the traditional methods is that the location precision is drastically affected when indoors."

Beams from GPS and cellular tracking bounce off objects and walls in a room – sometimes knocking location data off by hundreds of metres.

Where serving content based on a precise location is required, this technology simply doesn’t cut it. That’s where low energy, battery operated Bluetooth beacons come in.

Still a relatively emerging technology, tiny bluetooth beacons allow apps to precisely track how far away a device is – with particular emphasis on operating indoors.

These small devices are placed throughout the area, and their signals can be picked up by any device with Bluetooth 4.0 capability.

Understanding how this technology works, its limitations and ways to best leverage beacons is key to pushing the boundaries of location-driven user experiences.

Conduct was recently approached by an art gallery to create an iPad application that served exhibit information based on the visitor’s location within the gallery.

While it became obvious to us early on that Bluetooth beacons would be the way to go, what wasn’t so clear was which beacon brand, configuration, or position in the room would be most reliable and why.

We decided to conduct some in-house research and testing of our own, and what we discovered along the way was rather surprising.

1. There is no “best” brand of beacon

With most electronic devices, quality is determined by looking at the price tag. Since an iPhone Pro is more expensive than the regular iPhone, most of us will correctly conclude that it’s a superior product before we even see the feature list.

This is not the case with bluetooth beacons. Small bundles of beacons (usually three) can range anywhere from $50 to $200+, but from our own tests, we found this made little difference to the performance over a small area like an art gallery.

Our first instinct was to test out a high end product from Square Metrics. As a comparison, we also tested kontakt.io beacons to see how the performance of a mid-range product stacked up against a high end product.

Square Metrics (high end) vs. Kontakt.io (mid range)

Once we got familiar with both products, we started to test out how reliable their signal strength was over a sustained period of time. The test was simple:

Setup two beacons along a flat wall so that each beacon represents an art gallery exhibit.
Stand in front of each “exhibit” for one minute with an iPad Mini to record signal strength.
Repeat this process five times for each product, and take the average values of the five test runs respectively.

Much to our surprise, the mid-range kontakt.io beacons were undeniably superior in performance over this small area when compared with the high end Beacon Inside beacons.

While Beacon Inside is a market leader in larger scale signal performance, kontakt.io was going to perform better in a space like the art gallery.

It’s clear that the actual effectiveness of specific products is going to vary from project to project.

The bottom line: There really is no best brand of beacon and we recommend testing a range of different brands before deciding on a product.

2. Device orientation affects beacon signals

Another really interesting thing we discovered was that device orientation can have quite an impact on the reliability of Bluetooth beacon signals.

In the case of our project, visitors to the art gallery were always going to be facing the exhibit, which got us thinking about the hardware inside the iPad Mini and where specifically the Bluetooth receptor was located.

We discovered the Bluetooth chip is located in the bottom right hand corner of the iPad Mini, and is the same for all other iPad models to date.

"If a user is holding the iPad in landscape, the chip is going to be covered by their hand as they hold it – thus reducing its signal reliability."

We ran a new test with the kontakt.io beacons to see if we could prove our concept:

Setup nine beacons equally around an open symmetrical room to better reflect the setup of the art gallery.
Stand in front of each exhibit for 30 seconds.
Repeat five times in landscape, and a further five times in portrait to compare the averages of the results.

As we suspected, results proved signal consistency depends on the orientation of the device.

We suggest you do a similar test for your devices, too. After seeing these test results, we decided to change the art gallery application from landscape to portrait mode.

3. Positioning really matters

The layout of Bluetooth beacons in the room is instrumental in setting reliable and precise zone boundaries for your application.

We discovered pretty quickly that a room dimension and layout can dramatically affect the performance and behaviour of beacon signals.

Since Bluetooth operates by sending low powered radio signals through the air, it is highly susceptible to bouncing off walls, ceilings and other objects in its path.

This meant that test results could vary quite significantly to performance in the actual art gallery – which is comprised of different types of walls and objects to our test environment.

Bluetooth also struggles to travel through water and other forms of moisture. Whilst this might not sound like a major problem, it is easy to forget that our own bodies are over 55 per cent water.

The number of people in the immediate area is clearly going to play a factor in how you setup your beacon zones. It’s important to place beacons in a position where the signals will have to travel through as few people as possible, or readings can become unreliable.

In the case of the art gallery, a few tweaks were required in terms of where we positioned the beacons for each exhibit based on the flow of people entered the room.

4. Use testing software

As you can see, testing was a huge part of our process, and it helped us overcome many challenges that we encountered during the project.

Unfortunately there wasn’t much software available to run our tests. Some brands such as Bluecats Beacons come with really functional and interactive applications, which has an iPhone app that lists all your active beacons and how far away they are to the nearest centimetre.

Whilst all this real-time data is great, it’s difficult to properly record and then analyse it visually in graphs and tables.

Since there wasn’t a generic iOS beacon testing tool that served our needs, we decided to create our own tool for measuring the signal strength of beacons over time. We used this tool to conduct all of the tests covered in this blog post. We’ll be excited for you to try it when we make it available for free on the app store.

"Beacon functionality is greatly limited when a device is in lock mode."

5. Understand the limitations

As the scale of your beacon project increases, both deploying and managing the hardware becomes increasingly difficult. Beacons are difficult to track since they have no serial numbers – so lost, fallen or misplaced beacons can be extremely challenging to replace.

Rolling out a fully fledged beacon system in a large space, like many museums and shopping centres have already tried, may require full-time staff working on site to keep things running smoothly.

The technology is currently far from perfect, and there’s definitely a higher chance of something going wrong as the number of beacons you have increases.

Another thing to be mindful of is how often your users’ devices are going to be locked.

With some projects, like the art gallery we worked with, you can safely assume that the in-house iPads visitors receive will have the screen active for their entire session.

This wouldn’t be the case for something like a shopping centre, where a much greater percentage of users’ phones are going to be locked and away in their pockets.

iBeacon technology is somewhat reliable at picking up zone changes while a device is locked (although delays are higher than normal), but metrics like distance and signal strength are not available.