Session 6: Real-time Human Body & Hand Pose Tracking

🤸 Lesson 6: Pose Tracking — A Robot That Copies Your Moves

🤖 AI that understands body movement

In the previous lesson, the AI detected faces. In this lesson, we help the computer recognise the pose of the whole body using Pose Estimation.

The system finds key points on the body, then uses their coordinates to work out whether you are standing, leaning or raising your arms.

🎯 Learning objectives

After this lesson, you will be able to:

• Explain what body keypoints are.
• Turn on pose detection in PictoBlox.
• Read the coordinates of the shoulders, elbows and wrists.
• Recognise a raised-arm gesture.
• Control a sprite with body movements.

1. What is Pose Estimation?

Pose Estimation is a technology that lets AI locate body parts in an image or video.

The PoseNet system divides the body into 17 keypoints:

Each keypoint usually has:

• An x coordinate: the horizontal position.
• A y coordinate: the vertical position.
• A confidence score: how sure the AI is about the detection.

2. Preparing the camera

For accurate pose detection:

1. 1Place the camera in a stable position.
2. 2Stand far enough away for the camera to see your upper body or whole body.
3. 3Choose a well-lit place.
4. 4Avoid having several people in the frame.
5. 5Clear some space around you before moving.

3. Adding the Pose Detection extension

In PictoBlox:

1. 1Click Extension.
2. 2Find and select Pose Detection or Human Body Detection.
3. 3Allow PictoBlox to use the webcam.
4. 4Check that the pose detection blocks have appeared.

Block names may differ slightly between PictoBlox versions.

4. Turning on and analysing the video

Create a basic program:

when green flag clicked
turn video on stage with 20% transparency

forever
    analyse pose from camera

The analysis block must be inside a forever loop so the AI keeps updating body positions as you move.

5. Reading keypoint coordinates

To recognise arm movements, we need these points:

Left shoulder
Right shoulder
Left wrist
Right wrist

Create the variables:

Left_Shoulder_Y
Right_Shoulder_Y
Left_Wrist_Y
Right_Wrist_Y

Update their values:

forever
    analyse pose from camera

    set [Left_Shoulder_Y] to
        [y position of left shoulder]

    set [Left_Wrist_Y] to
        [y position of left wrist]

When you raise your left arm, the wrist moves higher than the shoulder.

if <Left_Wrist_Y greater than Left_Shoulder_Y> then
    say "You're raising your left arm!"

6. Recognising each gesture

if <Left_Wrist_Y greater than Left_Shoulder_Y> then
    say "Left arm is up!"

if <Right_Wrist_Y greater than Right_Shoulder_Y> then
    say "Right arm is up!"

if
    <Left_Wrist_Y greater than Left_Shoulder_Y>
    and
    <Right_Wrist_Y greater than Right_Shoulder_Y>
then
    say "Both arms are up!"

7. Mini project: the copycat robot

Make the sprite change costume to match the user's pose.

when green flag clicked
turn video on stage with 20% transparency

forever
    analyse pose from camera

    set [Left_Shoulder_Y] to [y position of left shoulder]
    set [Right_Shoulder_Y] to [y position of right shoulder]
    set [Left_Wrist_Y] to [y position of left wrist]
    set [Right_Wrist_Y] to [y position of right wrist]

    if
        <Left_Wrist_Y greater than Left_Shoulder_Y>
        and
        <Right_Wrist_Y greater than Right_Shoulder_Y>
    then
        switch costume to [Both arms up]
        say "Awesome!" for 1 seconds
    else
        if <Left_Wrist_Y greater than Left_Shoulder_Y> then
            switch costume to [Left arm up]
        else
            if <Right_Wrist_Y greater than Right_Shoulder_Y> then
                switch costume to [Right arm up]
            else
                switch costume to [Standing]

The sprite changes its look to match your pose.

8. Controlling a robot with poses

You could agree on:

Example:

if <both arms up> then
    robot forward
else
    if <left arm up> then
        robot turn left
    else
        if <right arm up> then
            robot turn right
        else
            robot stop

9. If the AI is not accurate

Check:

• Are both shoulders and arms inside the frame?
• Is the room bright enough?
• Do your clothes stand out from the background?
• Are several people in the frame?
• Is the camera shaking?
• Is the pose analysis block inside a loop?

If the sprite keeps switching states, hold each pose steady for about 1 second before it acts.

🧩 Challenge

Upgrade the project with one idea:

1. 1Create an exercise game that scores each arm raise.
2. 2Steer a spaceship by leaning your body.
3. 3Play a sound when the player raises both arms.
4. 4Make the sprite jump when a wrist is higher than the head.
5. 5Add a Reps variable to count correct movements.

✅ Remember

• Pose Estimation helps AI recognise body poses.
• PoseNet uses 17 keypoints.
• Each keypoint has x and y coordinates and a confidence score.
• Comparing the wrist with the shoulder recognises a raised arm.
• Conditions turn poses into sprite or robot actions.
• The camera needs good light and a clear view of the body.

1. 1What is a keypoint?
2. 2How many keypoints does PoseNet detect?
3. 3Which points do you compare to recognise a raised arm?
4. 4Why should you test on the stage before controlling a real robot?