Skip to main content

Lesson 13.3: Testing, Validation, and Sim-to-Real Preview

Your simulation is built. Now validate it works. This lesson teaches you how to systematically test whether your implementation meets the specification you wrote in Lesson 13.1.

This is the moment of truth: does your design actually solve the problem you defined?

Validation Methodology​

Step 1: Prepare for Testing​

Before running your simulation:

Checklist:

  • ✅ All URDF/SDF files are in your workspace
  • ✅ ros_gz_bridge configuration (bridge.yaml) is ready
  • ✅ Behavior node (or launch file) is ready to start
  • ✅ You have your success criteria from Lesson 13.1
  • ✅ You have a way to measure results (timer, console output, log files)

Create a test log file (capstone_test_results.txt):

CAPSTONE TEST RESULTS
====================

Specification: [Your specification intent]
Test Date: [Today's date]
Tester: [Your name]

Success Criteria:
1. [Criterion 1] - Target: [value], Measurement: [actual]
2. [Criterion 2] - Target: [value], Measurement: [actual]
3. [Criterion 3] - Target: [value], Measurement: [actual]
4. [Criterion 4] - Target: [value], Measurement: [actual]
5. [Criterion 5] - Target: [value], Measurement: [actual]

Trial Results:
--------------
Trial 1: [PASS/FAIL] - Notes
Trial 2: [PASS/FAIL] - Notes
Trial 3: [PASS/FAIL] - Notes

Summary:
--------
Specification Met: [YES/NO]
Issues Encountered: [List any failures]

Step 2: Run Trial 1​

Terminal 1: Launch Gazebo

gz sim world.sdf

Terminal 2: Start ros_gz_bridge

ros2 run ros_gz_bridge parameter_bridge --ros-args -p config_file:=bridge.yaml

Terminal 3: Start behavior node

python3 behavior_node.py

Terminal 4: Monitor execution

# Watch robot state
ros2 topic echo /scan
ros2 topic echo /cmd_vel

Observation window: Let the simulation run until one of these happens:

  • Robot completes the task (SUCCESS)
  • Robot fails spectacularly (FAILURE)
  • Timeout (robot stuck or wandering—FAILURE)

Record observations:

  • Did robot start moving? YES/NO
  • Did robot encounter obstacles? YES/NO
  • Did robot reach goal? YES/NO
  • Time taken? [X seconds]
  • Any collisions? YES/NO
  • Any unexpected behaviors? [describe]

Step 3: Measure Success Criteria​

For each criterion in your specification, measure the actual value:

Example Criteria (delivery robot):

Criterion 1: Deliverability (reach all goals)

Target: Robot reaches goal location (within 0.5m)
Measurement: Distance from robot to goal at end of trial
Trial 1 result: 0.3m (PASS - within tolerance)

Criterion 2: Safety (no collisions)

Target: Zero collisions with obstacles
Measurement: Bumper sensor triggered count
Trial 1 result: 0 collisions (PASS - no bumper events)

Criterion 3: Efficiency (complete in time limit)

Target: Delivery complete in under 5 minutes
Measurement: Elapsed time from start to goal
Trial 1 result: 3m 42s (PASS - under limit)

Criterion 4: Reliability (succeed multiple times)

Target: 3 out of 3 trials successful
Measurement: Success count after 3 trials
Trial 1: PASS, Trial 2: [pending], Trial 3: [pending]

Criterion 5: ROS 2 Integration (topics working)

Target: All sensor/command topics publishing
Measurement: Topic count and message frequency
Trial 1 result: /scan 30Hz, /cmd_vel 10Hz, /imu 50Hz (PASS)

Step 4: Run Trials 2 and 3​

Repeat Steps 2-3 for two more runs. Multiple trials show reliability:

  • Does your design work consistently?
  • Or does it work sometimes and fail other times?

Why 3 trials?

  • Trial 1 might be a fluke (lucky obstacle avoidance)
  • Trial 2 confirms repeatability (wasn't a fluke)
  • Trial 3 confirms stability (works consistently)

Record Trial 2 and 3 results in your test log.

Step 5: Evaluate Results​

Create a summary:

VALIDATION SUMMARY
==================

Specification Criteria:
1. Reach goal: Trial 1 PASS, Trial 2 PASS, Trial 3 PASS → SUCCESS
2. Zero collisions: Trial 1 PASS, Trial 2 PASS, Trial 3 PASS → SUCCESS
3. Complete in time: Trial 1 PASS (3:42), Trial 2 PASS (4:15), Trial 3 PASS (3:58) → SUCCESS
4. Reliability: 3/3 trials passed → SUCCESS (100% success rate)
5. ROS 2 integration: All topics active, data flowing → SUCCESS

Overall: SPECIFICATION MET ✓

All success criteria achieved. Capstone validated.

Iterating on Failures​

If your first trial FAILS, don't panic. This is normal. Simulation is hard. Debug and improve.

Failure Case 1: Robot doesn't move​

Symptoms:

  • Robot spawns but doesn't move
  • Console shows no motion commands

Diagnosis:

  1. Check behavior node is running: ps aux | grep behavior
  2. Check topics are connected: ros2 topic list | grep cmd_vel
  3. Check data flows: ros2 topic echo /scan (should see LIDAR data)
  4. Check bridge is active: ros2 topic echo /cmd_vel (any messages?)

Fix:

  • Verify behavior node subscribers are working (add debug prints)
  • Confirm cmd_vel topic is being published
  • Check motor joint names in URDF match bridge configuration
  • Test manually: ros2 topic pub /cmd_vel geometry_msgs/msg/Twist '{"linear":{"x":0.5}}'

Failure Case 2: Robot collides with obstacles​

Symptoms:

  • Robot hits walls/obstacles
  • Bumper sensor triggers
  • Collision forces flip or push robot

Diagnosis:

  1. Is LIDAR detecting obstacles? ros2 topic echo /scan | head -20
  2. Are distance readings accurate? (compare visual position to LIDAR range)
  3. Is behavior logic responding? (check minimum distance threshold)

Fix:

  • Lower obstacle detection threshold (react sooner)
  • Increase minimum obstacle distance
  • Add safety margin to collision avoidance
  • Test behavior logic: ros2 run ros_gz_bridge parameter_bridge ... & then manually publish scans

Failure Case 3: Robot doesn't reach goal​

Symptoms:

  • Robot moves but doesn't navigate to goal
  • Robot wanders or goes in circles
  • Robot gets stuck

Diagnosis:

  1. Is odometry available? (check robot position estimation)
  2. Are motion commands appropriate? (check cmd_vel values)
  3. Is goal location reachable? (check world layout vs goal position)

Fix:

  • Implement path planning (instead of random motion)
  • Use odometry to track position and navigate systematically
  • Add waypoint navigation (go to waypoint 1, then 2, then goal)
  • Test in simpler world first (empty space, no obstacles)

Failure Case 4: Simulation is unstable (physics wild)​

Symptoms:

  • Robot shakes violently
  • Objects fall through ground
  • Physics behaves unrealistically

Diagnosis:

  1. Check URDF mass distribution (too light or top-heavy?)
  2. Check world physics settings (friction, gravity correct?)
  3. Check time step (too large = unstable simulation)

Fix:

  • Increase robot mass slightly
  • Lower friction coefficient in world (less slippery surfaces help stability)
  • Increase physics time step accuracy in SDF (default often works)
  • Test with simple object (sphere) in empty world—verify gravity works

Iteration Workflow​

Run Trial 1
    ↓
Did it PASS?
    ├─ YES → Run Trials 2-3, validate
    └─ NO → Diagnose failure
         ↓
         Identify issue (collision, doesn't move, unstable, etc.)
         ↓
         Fix URDF, SDF, or behavior code
         ↓
         Test fix (quick trial)
         ↓
         Ready to re-run full trials?
         ├─ YES → Run Trial 1 again
         └─ NO → Debug more

When to stop iterating: When all 3 trials pass all success criteria.

Sim-to-Real Gap: What's Different About Real Robots?​

Your simulation works perfectly. But real robots behave differently. Here's why, and what you'll learn in Module 3:

Gap 1: Sensor Noise​

In simulation: LIDAR distance is exact. Camera image is perfect. In reality: Sensors have noise. LIDAR might read 2.5m ± 0.1m. Camera images have compression artifacts.

What to expect: Real robot needs noise filtering. Your behavior logic might need thresholds to handle noisy data.

Module 3 preview: You'll learn sensor noise modeling in Isaac Sim, making simulation more realistic.

Gap 2: Actuator Dynamics​

In simulation: Motor commands execute instantly. "Go 0.5 m/s" and the robot goes 0.5 m/s immediately. In reality: Motors have acceleration limits. "Go 0.5 m/s" takes time to reach that speed.

What to expect: Real robot behavior is slower, smoother. Your timing assumptions might be wrong.

Module 3 preview: You'll model motor dynamics in Isaac Sim—acceleration limits, backlash, torque curves.

Gap 3: Environmental Variation​

In simulation: World is static and predictable. Obstacles stay exactly where you placed them. In reality: Environments change. Obstacles move, lighting changes, surfaces are variable.

What to expect: Real robot needs adaptation. Your hardcoded obstacle detection thresholds might fail in different lighting.

Module 3 preview: You'll learn adaptive control and sensor fusion in Isaac Sim.

Gap 4: Timing and Latency​

In simulation: ROS 2 messages arrive instantly. Brain-to-motor latency is zero. In reality: Network latency, processing delay, motor response time all add up (20-100ms typical).

What to expect: Real robot behavior lags behind commands. Timing-sensitive behaviors (fast obstacle avoidance) might fail.

Module 3 preview: You'll model communication latency in Isaac Sim, testing behavior robustness.

Gap 5: Physics Accuracy​

In simulation: Friction is exactly as configured. Collisions are perfectly elastic or plastic as specified. In reality: Physics varies—wet vs dry floor, worn vs new wheels, unexpected mass distribution.

What to expect: Real robot behavior might differ. Your floor friction assumption might be wrong. Wheelslip might occur.

Module 3 preview: You'll learn physics domain randomization in Isaac Sim—testing behavior across physics variations.

From Simulation to Real Hardware: The Path Forward​

You are here (Module 2 Capstone):

  • Simulation works perfectly
  • Behavior is tuned for your specific simulation

Module 3 (Isaac Sim):

  • Simulation includes realistic noise, dynamics, timing
  • You'll adapt behavior to handle sim-to-real gaps
  • Transfer learning: which behaviors transfer? Which don't?

Module 4 (Real Hardware):

  • Deploy simulation-trained behavior to real robot
  • Real-world validation: does it work?
  • Online learning: robot improves from real experience

The key insight: Simulation is training, reality is evaluation. Your capstone teaches you to build and validate in simulation. Module 3 teaches you to bridge the gap.

Capstone Completion Checklist​

Before declaring success:

  • All 5 success criteria measured and documented
  • 3 trials completed
  • 3/3 trials passed all criteria
  • Test log filled out completely
  • All failures diagnosed and fixed (or acceptable trade-offs explained)
  • Can explain why each criterion matters
  • Can identify 3 sim-to-real gaps relevant to your project
  • Can describe how Module 3 will address those gaps

If all checks pass, you've completed Module 2!

Module 2 Completion Celebration​

You've accomplished:

✓ Understood why simulation matters (Chapter 8) ✓ Built robot models in URDF (Chapter 9) ✓ Created simulation worlds (Chapter 10) ✓ Configured sensors (Chapter 11) ✓ Integrated ROS 2 with Gazebo (Chapter 12) ✓ Designed and implemented a complete capstone project (Chapter 13)

You now understand:

  • Digital twin concepts
  • Robot description formats (URDF/SDF)
  • Physics simulation fundamentals
  • Sensor simulation and configuration
  • ROS 2 integration with simulators
  • Specification-driven development
  • Sim-to-real transfer concepts

You're ready for:

  • Module 3: NVIDIA Isaac Sim (advanced simulation, AI training)
  • Module 4: Real robot deployment (transfer learning, real-world validation)

Try With AI​

Setup: Open ChatGPT or Claude and get guidance on validation and next steps.

Prompt Set:

Prompt 1 (Designing validation strategy):

I've completed my capstone simulation. Here's my specification:

[Paste your success criteria]

Create a detailed validation plan:
1. How should I measure each criterion?
2. What tools should I use (timers, console output, ROS 2 tools)?
3. How many trials should I run to confirm reliability?
4. What failure modes should I test for?

Prompt 2 (Debugging simulation failures):

My capstone simulation isn't working. Here's what's happening:

[Describe: robot doesn't move, collides, gets stuck, etc.]

My simulation has:
- [Describe robot specs]
- [Describe world setup]
- [Describe behavior logic]

Help me diagnose and fix this. What's likely causing the problem?

Prompt 3 (Understanding sim-to-real transfer):

My simulation works perfectly. I'm wondering: what happens when
this same behavior runs on a real robot?

My robot is: [description]
My behavior does: [description]

For each of these sim-to-real gaps, explain what might go wrong
and how I should design differently:
1. Sensor noise (my sensors are noisy in reality)
2. Actuator latency (motors don't respond instantly)
3. Environmental variation (real world isn't as controlled as simulation)
4. Physics changes (friction, masses vary in reality)

Prompt 4 (Next steps planning):

I've completed Module 2 capstone. What should I expect from Module 3 (Isaac Sim)?

Based on my capstone project [describe], which sim-to-real gaps will
Isaac Sim help me address?

How should I prepare my behavior code for Module 3?

Expected Outcomes:

  • AI helps design validation methodology
  • AI assists in debugging failures
  • AI explains sim-to-real transfer concepts
  • AI guides transition to Module 3

Safety Note: When testing robot behavior, especially with physical consequences:

  • Always have an emergency stop ready
  • Test in confined spaces first (not open floor)
  • Verify velocity limits (don't command dangerous speeds)
  • Monitor for unexpected behaviors