Computer visionFebruary 2025 - May 2025

Wavedrive — Gesture-Controlled Robot Car

Wave at a webcam to drive an ESP32 Wi-Fi robot

Hand Tracking
MediaPipe runs locally on any webcam
7 Gestures
TFLite classifier on landmark coordinates, 0.98 held-out accuracy
Wi-Fi Tele-op
ESP32 AP + TCP control loop

System Overview

Wavedrive turns webcam hand gestures into commands for an ESP32 robot car over local Wi-Fi.

MediaPipe tracks hand landmarks, a TensorFlow Lite classifier recognizes ten gestures, and the laptop sends the corresponding command over TCP. The ESP32 drives two DC motors and displays the active command on an OLED, without a cloud inference service.

How it actually works

A hand sign becomes a motor command

Follow a recorded hand through normalisation, the trained classifier running in your browser, and the ESP32 firmware that sets the motors and OLED.

Things to try

In 3 Classify, raise Blend amount until the prediction flips.
In 1 Track, shrink Hand size in frame, then check 2 Normalise.
In 5 Drive, pick Speed up and press Repeat.

Built from the project's own code and documents.from code: A constant or rule copied from the project's repositoryfrom report: A value from the project's report or presentationsimulated: Computed in your browser by running the project's logic

Gesture (the seven trained classes in keypoint_classifier_label.csv)

Each webcam frame is captured at 960 by 540, mirrored, and passed to MediaPipe Hands, which is limited to one hand. The 21 landmarks become integer pixel coordinates. The gold dot is the wrist (landmark 0) and the green dot is the index tip (landmark 8).

Hand size in frame220 px

Your input: sets the largest wrist-to-landmark distance

Hand position50%

Your input: slides the hand across the frame

Capture

960 x 540

Max hands

Detect conf.

0.7

Track conf.

0.5

from codeyour inputsimulated

Model output: 7 Forward → sends "forward\n" simulated: The real network weights run in your browser on the landmarks above

main.py: capture and trackView on GitHub

while True:
fps = cvFpsCalc.get()
# キー処理(ESC：終了) #################################################
key = cv.waitKey(10)
if key == 27: # ESC
break
number, mode = select_mode(key, mode)
# カメラキャプチャ #####################################################
ret, image = cap.read()
if not ret:
break
image = cv.flip(image, 1) # ミラー表示
debug_image = copy.deepcopy(image)
# 検出実施 #############################################################
image = cv.cvtColor(image, cv.COLOR_BGR2RGB)
image.flags.writeable = False
results = hands.process(image)
image.flags.writeable = True
# ####################################################################
if results.multi_hand_landmarks is not None:
for hand_landmarks, handedness in zip(results.multi_hand_landmarks,
results.multi_handedness):
# 外接矩形の計算
brect = calc_bounding_rect(debug_image, hand_landmarks)
# ランドマークの計算
landmark_list = calc_landmark_list(debug_image, hand_landmarks)
# 相対座標・正規化座標への変換
pre_processed_landmark_list = pre_process_landmark(
landmark_list)
pre_processed_point_history_list = pre_process_point_history(
debug_image, point_history)
# 学習データ保存
logging_csv(number, mode, pre_processed_landmark_list,
pre_processed_point_history_list)
# ハンドサイン分類
hand_sign_id = keypoint_classifier(pre_processed_landmark_list)

Hand-drawn gesture sheet: go forwards, go backwards, turn right, turn left, stop, spin around, speed up, slow down, and two undecided panels — The gesture design sheet in the repo. Go backwards and the two question-mark panels never made it into the label file or the firmware.gestures.jpg

Real: the labels, command strings, endpoint, PWM values and OLED icons are from main.py and esp32.ino; the hands are recorded rows from keypoint.csv; the probabilities come from the trained network weights (1,180 parameters from keypoint_classifier.keras), evaluated in your browser. Simulated: where the hand sits in the frame, and the blend between two recorded hands. The robot has no reverse: firmware drives one PWM pin per motor.

How it works

Turn hand movement into a robot command.

1
Track the hand
Capture webcam frames on the laptop and use MediaPipe to obtain hand-landmark coordinates for the classifier.
2
Classify the gesture
Run the TensorFlow Lite model on normalised landmark coordinates and map the predicted gesture to one of seven commands.
3
Send the command
Transmit the selected command through a TCP socket to the ESP32 access point at 192.168.4.1 on port 12345.
4
Drive and display
The ESP32 sets a PWM duty for each of its two motors and shows the matching gesture icon on the OLED.

Engineering features

Trained a TFLite gesture classifier over normalised MediaPipe hand landmarks for seven gestures, reaching 0.98 accuracy on 1,133 held-out samples.
Built ESP32 firmware that hosts a Wi-Fi access point and sets the two motors' PWM duty from TCP commands: forward, left, right, stop, spin, speed up and slow down.
Added a 128×64 OLED that shows an icon of the active gesture for at-a-glance feedback.
End-to-end pipeline runs in real time on a laptop webcam with no cloud dependencies.

Inside the build

The control logic, integration and design decisions, one note at a time.

Landmark classification on the laptop

The classifier is trained on MediaPipe landmark coordinates rather than direct image pixels. Python and OpenCV provide the webcam pipeline, with TensorFlow Lite classifying the resulting coordinates. Tracking and classification run locally, without a cloud inference dependency.

Robot-hosted wireless control and feedback

The ESP32 hosts the Wi-Fi access point and listens for TCP commands at a fixed local endpoint. Its firmware sets one PWM pin per motor, so the car drives forward, turns, spins and changes speed but cannot reverse, and shows a gesture icon for the current command on the OLED. This divides gesture interpretation and motor actuation between the laptop and the robot.

Evidence & limitations

What the project documents show, and where each claim comes from.

Source code available to explore

The project repository is linked below for the gesture pipeline and ESP32 firmware. The training notebook reports 0.98 accuracy on 1,133 held-out samples across the seven trained gestures. There is no report or demonstration video, so command latency and driving performance are not quantified.

Wavedrive source repository
Gesture-recognition pipeline and embedded robot firmware.

Next projectMaze Runner & Ball ShooterTwo robots for a first-semester university competition

Hiring for robotics or embedded work?

I'm looking for internships. The résumé has the short version.

Get in touch Résumé

Wavedrive — Gesture-Controlled Robot Car

Hand Tracking

7 Gestures

Wi-Fi Tele-op