System initialization: the python software start running, establishing a connection with the robot and the camera. The robot is sent to the home position.

Calibration: to measure the relationship between the camera frame and the robot's end effector, the user places the pen on the center of the gripper, the robot retracts and the system measures the pen's position. This way, the system is able to know the offset between arm coordinates and camera coordinates. After calibration, the arm proceeds to pick the pen to confirm the calibration is successful.

User input: the system waits for the user to confirm the pen is ready to be picked. Upon confirmation, the pen's location is transferred to robot coordinates

Pick: one the pen is located, the robot turns its first axis to point towards the pen. As the Pincher is a 4 degree-of-freedom robot, this step is crucial to avoid colliding with the pen.

Rest: robot goes to its sleep position with the pen.

Place (or throw?): robot goes back to home position and releases the pen.

Repeat: this cycle can keep going. The calibration is only done once, only on the first run.

This is an introductory level challenge for students going into the MS in Robotics at Northwestern, so there is a lot of freedom as to how to approach the problem, and how to extend it. Many aspects are simplified, mind this is a task to be done in a few hours.

The robot and camera are supossed to be at a 90 degree offset. This simplifies the conversion between frames a lot.

Multiprocessing was used to separate the camera processing from the robot's communication, as the camera live feed was stuttery when everything was running on the same process.