An Augmented Reality Board Game System



Augmon introduces a groundbreaking way to play board games, with LEGO bricks under a camera and a projector that are connected to a personal computer.

Also, it demonstrates along with Escape, an asymmetric tabletop strategy game, designed for this system to show how the design intertwined with the technology.


  • Personal C++ Engine

    • Multiple render outputs: on table, on computer screen

    • Multiple window modes: border, borderless, borderless-fullscreen

    • Game specific data structure

  • OpenCV Library

    • Object detection and analysis

    • Color space conversion

  • Intel RealSense SDK

    • Access to color image and depth image

    • Color & depth image alignment


A LEGO unit is a 2x2 LEGO brick or plate. Moreover, a grid can fit precisely one LEGO unit. So with this definition, the gameplay area on the table can fit 42x23 LEGO units, 966 units in total.​


The idea behind the gridlization to know which grids have LEGO units and what color are those. Players have to place the LEGO unit mostly inside a grid (> 0.8) so that the system can know which grid has LEGO unit. If the LEGO unit is placed between 2 or more grids, the system cannot decide at which grid the LEGO unit is.


Camera Calibration

  • Why? 

    • The camera needs to know the size and origin of the board in order to calculate an object’s relative position on the board.

  • How? 

    • Render markers at the corner.

    • Calculate each marker’s position in camera space.​

    • System knows the boundary of the board, as well as the width and height of the board


Keystone Correction

However, there are other cases where the projector is not facing downwards straight or rotates a little bit; then we cannot calculate the grid center according to the width and the height of the board.

For example, when we need to find the center of the grid (31, 7), it is not simply counting 31 grids right, and seven grids up from the bottom left the grid. Instead, we calculate the fraction on each edge of the boundary and get 4 points on the edges. Then we generate two line segments, and their intersection point is the center of the grid.



  • Multi-threading

    • Processing camera data is much slower than updating and rendering the game

    • The game should be able to render at least 60 FPS while processing camera data

Camera Thread​


Board Data


Game Thread

  • Triple Buffer

    • Prevents either camera thread or game thread waiting for available data slot to write/read

    • Provide three pieces of board data instead of one.

    • The camera thread writes back and forth between the two back buffers and (at best) once every refresh the front buffer is swapped for the back buffer containing the most recent board data.

Game Content: Escape

In a game of Escape, one player plays as the Thief who has just broken into a museum in an attempt to steal treasures as many as possible. The other players play as Guards patrolling around the building and trying to find out and capture the thief.


Playtest - 1 Thief vs. 2 Guards 


System Overview