Perception and Manipulation for Robotic Block Stacking
Abstract
This work presents a perception and manipulation system for constructing a block tower with a Franka Panda robot. The task required the robot to collect AprilTag-labeled blocks from both a stationary platform and a rotating turntable, then stack as many blocks as possible within five minutes. Distinct pipelines were developed for the static and dynamic cases because they imposed different sensing and timing constraints. Static blocks were localized with a wrist-mounted camera and transformed into the robot base frame, whereas moving blocks were acquired with a fixed wait-and-pick policy. A damped least-squares inverse-kinematics solver improved stability near a singular grasp configuration. In the final hardware evaluation, the system stacked four static blocks and three dynamic blocks and achieved first place in the class competition.
System Overview
The manipulation pipeline comprised block detection, pose transformation, inverse kinematics, grasp verification, and placement. Rather than computing a new motion plan for every transition, the system used a finite set of reusable arm configurations: a neutral configuration, a camera configuration above the static platform, block-specific grasp configurations, a stack-inspection configuration, a placement configuration, and a waiting configuration adjacent to the turntable. The static-block sequence consisted of scanning, grasping, inspecting the stack, placing the block, and returning to the scanning configuration. The dynamic-block sequence reused the inspection and placement stages but routed the arm through the turntable waiting configuration.
The use of repeatable configurations reduced online computation and constrained the physical robot to motions that could be tested systematically.
Author Contributions
The project was completed as a team effort. My primary contributions were the path-planning framework, the predefined robot configurations, and the coordinate transformations among the camera, end-effector, and robot base frames.
Static-Block Localization and Grasp Planning
For a detected static block, the AprilTag detector reported a pose in the camera coordinate frame. The system transformed this pose through the calibrated camera-to-end-effector transformation and the end-effector pose obtained from forward kinematics. The resulting composition produced the block pose in the robot base frame. All visible blocks were recorded during a single scan and subsequently visited in an order intended to reduce interference with neighboring blocks: the red-side routine processed blocks from left to right, whereas the blue-side routine processed them from right to left.
Gripper orientation required additional geometric processing because an AprilTag coordinate frame rotates with its associated block. Consequently, fixed assumptions about the tag axes in the world frame were invalid. The algorithm first identified the block axis most closely aligned with the robot’s vertical axis. It excluded this axis from the planar orientation calculation and used the remaining two axes to determine the gripper alignment. This procedure provided a block-relative grasp orientation without requiring a separate online optimization step for every detection.
Dynamic-Block Acquisition
Moving blocks were collected with a wait-and-pick strategy. The arm moved to a fixed configuration beside the rotating platform and repeatedly attempted to close the gripper as blocks passed through the grasp region. The gripper was tilted by 10 degrees to reduce the risk of contact with the turntable. This grasp configuration was close to a kinematic singularity, for which the conventional Jacobian pseudoinverse produced unstable joint updates. Damped least squares was therefore used to regularize the inverse-kinematics solution.
Grasp success was evaluated from the measured gripper opening. A very small final separation indicated that the fingers had closed without acquiring an object, whereas a larger separation indicated that a block remained between them. Force feedback was considered for this decision but was less reliable in the tested setup. After an unsuccessful attempt, the arm remained at the waiting configuration and repeated the grasp. This policy reduced acquisition speed but avoided reliance on noisy visual interception of a moving object.
Experimental Results
Before every placement, the wrist camera estimated the upper surface of the existing tower. The nominal release position was approximately 5 mm above that surface, limiting lateral disturbance of the stack. Static blocks aligned nearly exactly in simulation. On hardware, the vision pipeline produced approximately 5 mm of lateral misalignment, while errors in estimated height occasionally caused contact with the upper block or release from an excessive height. Changes in illumination during the competition appeared to increase these errors.
Despite these uncertainties, the robot stacked four static blocks and three dynamic blocks during the five-minute evaluation. The resulting seven-block tower was sufficient to win the class competition. In this run, the hybrid strategy—vision-based pose estimation for static objects and mechanically constrained acquisition for moving objects—completed the task despite the stated sensing and computation constraints.
Limitations and Discussion
The system relied on hard-coded platform geometry and did not explicitly represent perception uncertainty. Its performance was therefore sensitive to calibration error and illumination changes. The wait-and-pick routine also traded throughput for robustness and did not estimate the turntable state. Moreover, the reported outcome comes from one five-minute competition run; repeated trials would be required to estimate task-completion reliability. A more general implementation would localize the platforms through their AprilTags, quantify uncertainty in the block-pose estimates, and evaluate vision-based interception for moving blocks. These extensions would reduce dependence on fixed workspace geometry while permitting more direct comparison between reactive and model-based acquisition strategies.
Conclusion
The project integrated camera-based block localization, frame transformations, regularized inverse kinematics, grasp verification, and closed-loop stack inspection on a physical Franka Panda. Separating static and dynamic acquisition produced a system matched to the sensing characteristics of each subtask. The hardware evaluation shows that the selected motion primitives and regularized inverse kinematics were sufficient to complete the task once within the imposed time limit; repeated trials would be required to estimate reliability.