December, 2023

Nerfbot

A robot that searches for targets and blasts them down!

ros2
python
arduino
franka
yolo
realsense
computer vision
git

Overview

This system is capable of detecting colored bowling pins (targets) through an object detection algorithm, locating them in the world using information provided by the robot-mounted depth camera, and blasting them down using nerf guns upon command. Integration of subsystems was performed using ROS2.

Check this project's repository on

Subsystems

  • Franka: the robot arm is controlled using the Moveit2 ROS API.

  • Computer vision: images provided by an Intel Realsense D435i camera are used to detect pins and locate them in the world using depth information and robot position. The images are also used for april tag detection, which indicates the position of the Nerf blasters.

  • Voice input: python speech recognition libraries are used with an external microphone to tell the system which pin color to blast.

  • Nerf guns interfacing: the nerf gun triggers were hacked to be operated using an arduino and integrated to ROS2.

    • Program flow

  • System initialization: all nodes spinned up, blasters loaded with 6 rounds each. The system assumes that is the starting number of rounds on each blaster.

  • User input: the user tells the system through the microphone which pin color to shoot next.

  • Pin scanning: the robot points the camera to two predefined "scan positions", and on each one of them the system processes the images to detect and localize the pins.

  • Gun scanning: the system goes to "gun scan position", where it locates both blasters through their top-mounted april tags. After detecting both blasters, the system heuristically picks the left blaster (blaster #1) and blasts until there are no more pins of the commanded color, or the ammo is out.

  • Gun picking: the robot picks up the Nerf blaster using cartesian paths and blasts the commanded-color pins from right-to-left.

  • Blasting: when the robot is at each aim position, an arduino operates relays that make the corresponding gun blast.

  • Placing: after blasting, the robot places the gun at the pick position (with minor offsets) and wait for command. If there are pins missing and the blaster is out of rounds, the system switches blasters automatically.

  • Repeat: while there is ammo available, the user can keep indicating which colored pins to blast. The system only scans the pins once (upon the first command), so subsequent commands will just trigger the picking-aiming-firing-placing sequence.

    • Personal work

    This project was developed in a group as the final project for ME495: Embedded Systems in Robotics.
    Personal work includes:

  • Hacking of Nerf gun electronics to trigger them using Arduino

  • Development of ROS2 interface for Nerf gun blasting

  • Collaboration in the development of Moveit2 API wrapper to interface with the Franka arm.

  • Collaboration in the development of pin-locating code and Realsense interfacing.

  • Collaboration in integration of subsystems/components.