Chapter 3: Sensors, Actuators, and Physical Limits

Introduction

The physical capabilities of AI systems are fundamentally constrained by the laws of physics. Understanding these limitations is crucial for designing effective embodied AI systems that can operate reliably in the real world. This chapter explores the physical principles governing sensors and actuators, analyzes the constraints they impose on intelligent systems, and presents strategies for working within and around these limitations.

Physical constraints are not limitations to be eliminated, but design parameters that shape the possibilities and characteristics of intelligent systems. Working with physics rather than against it leads to more robust and efficient designs.

3.1 Sensing in the Physical World

3.1.1 Fundamentals of Sensing

Sensing is the process by which physical systems gather information about their environment and internal state. All sensing ultimately involves energy exchange between the system and its environment.

Diagram: Sensing Process Pipeline

[Physical Phenomenon] → [Transducer] → [Signal Processing] → [Feature Extraction] → [Perception]
        ↓                    ↓              ↓                  ↓              ↓
     Light, Sound        Energy          Amplification     Pattern        Understanding
     Force, Heat          Conversion      Filtering         Recognition    Decision
     Chemical, EM         Measurement     Digitization      Estimation     Action
     Magnetic            Detection       Calibration       Tracking       Planning

3.1.2 Physical Limitations of Sensors

Signal-to-Noise Ratio (SNR) The fundamental limit of any sensor is its ability to distinguish signal from noise:

$$SNR = \frac{P_{signal}}{P_{noise}} = \frac{\mu_{signal}^2}{\sigma_{noise}^2}$$

Where:

• $P_{signal}$ - Signal power
• $P_{noise}$ - Noise power
• $\mu_{signal}$ - Signal mean
• $\sigma_{noise}$ - Noise standard deviation

Nyquist-Shannon Sampling Theorem Digital sensors are limited by sampling rate and quantization:

$$f_{sampling} \geq 2 \cdot f_{max}$$

Where $f_{max}$ is the maximum frequency component of the signal.

Energy Constraints Sensing requires energy exchange with the environment:

$$E_{sensing} = P_{transducer} \cdot t_{measurement} + E_{processing}$$

3.1.3 Vision Systems

Camera Physics Cameras convert light into electrical signals through photoelectric effects:

$$I_{pixel} = \int_{\lambda} L(\lambda) \cdot R(\lambda) \cdot A \cdot \Delta t \, d\lambda$$

Where:

• $I_{pixel}$ - Pixel intensity
• $L(\lambda)$ - Spectral radiance
• $R(\lambda)$ - Sensor responsivity
• $A$ - Aperture area
• $\Delta t$ - Exposure time

Depth Sensing Active depth measurement methods:

Diagram: Depth Sensing Technologies

Structured Light
┌─────────────────────────────────┐
│ Projector ───→ Pattern         │
│     ↓                           │
│  Object surface                │
│     ↓                           │
│ Camera ────→ Deformation      │
│     ↓                           │
│  Depth calculation             │
└─────────────────────────────────┘

Time-of-Flight
┌─────────────────────────────────┐
│ Laser pulse ───→ Object        │
│     ↓                           │
│  Reflection                    │
│     ↓                           │
│ Detector ────→ Time delay      │
│     ↓                           │
│  Depth = c·t/2                 │
└─────────────────────────────────┘

Stereo Vision
┌─────────────────────────────────┐
│ Scene → Camera 1 → Image 1     │
│    ↘                           │
│      ↘                         │
│        ↘ Disparity             │
│      ↗                         │
│    ↗                           │
│ Scene ← Camera 2 ← Image 2     │
└─────────────────────────────────┘

Example: Stereo Vision Depth Calculation

import numpy as np
import cv2

class StereoVision:
    def __init__(self, focal_length, baseline):
        self.f = focal_length    # Camera focal length (pixels)
        self.B = baseline        # Distance between cameras (meters)

    def compute_depth(self, disparity):
        """
        Compute depth from disparity
        disparity: disparity in pixels
        """
        # Avoid division by zero
        disparity = np.maximum(disparity, 0.1)
        depth = (self.f * self.B) / disparity
        return depth

    def rectify_images(self, img_left, img_right):
        """Rectify stereo images for correspondence"""
        # Use OpenCV for stereo rectification
        # This is a simplified example
        stereo = cv2.StereoBM_create(numDisparities=16, blockSize=15)
        disparity = stereo.compute(img_left, img_right)
        return disparity

    def point_cloud(self, disparity, img_left):
        """Generate 3D point cloud from disparity"""
        depth = self.compute_depth(disparity)
        height, width = depth.shape

        # Create coordinate grids
        u, v = np.meshgrid(range(width), range(height))

        # Convert to 3D coordinates
        x = (u - width/2) * depth / self.f
        y = (v - height/2) * depth / self.f
        z = depth

        # Stack coordinates and colors
        points = np.stack([x.flatten(), y.flatten(), z.flatten()], axis=1)
        colors = img_left.reshape(-1, 3)

        return points, colors

# Example usage
stereo = StereoVision(focal_length=800, baseline=0.1)
# disparity = stereo.rectify_images(left_img, right_img)
# points, colors = stereo.point_cloud(disparity, left_img)

3.2 Actuation Physics

3.2.1 Fundamentals of Actuation

Actuation converts control signals into physical work through energy transformation. All actuators are limited by conservation laws and material properties.

Energy and Power Constraints

$$P_{output} \leq P_{input} \cdot \eta$$

Where $\eta$ is the efficiency of the actuator.

Torque-Speed Characteristics Most actuators exhibit inverse relationships between torque and speed:

$$P_{mechanical} = \tau \cdot \omega$$

Where:

• $\tau$ - Torque
• $\omega$ - Angular velocity

3.2.2 Electric Motors

DC Motor Physics DC motors convert electrical energy to mechanical rotation:

$$\tau = K_t \cdot I$$ $$V = K_e \cdot \omega + I \cdot R$$

Where:

• $K_t$ - Torque constant
• $K_e$ - Back-EMF constant
• $I$ - Current
• $R$ - Resistance

Motor Dynamics The dynamic equation of a DC motor:

$$J \frac{d\omega}{dt} + B\omega = \tau - \tau_{load}$$

Where:

• $J$ - Moment of inertia
• $B$ - Damping coefficient

Example: DC Motor Simulation

import numpy as np
import matplotlib.pyplot as plt

class DCMotor:
    def __init__(self):
        # Motor parameters
        self.R = 1.0        # Resistance (Ohms)
        self.L = 0.001      # Inductance (H)
        self.Kt = 0.1       # Torque constant
        self.Ke = 0.1       # Back-EMF constant
        self.J = 0.01       # Moment of inertia
        self.B = 0.001      # Damping coefficient

        # State variables
        self.i = 0.0        # Current
        self.omega = 0.0    # Angular velocity
        self.theta = 0.0    # Position

    def step(self, V, tau_load, dt):
        """Simulate one time step"""
        # Electrical dynamics: L * di/dt + R*i = V - Ke*omega
        di_dt = (V - self.R * self.i - self.Ke * self.omega) / self.L

        # Mechanical dynamics: J * domega/dt + B*omega = Kt*i - tau_load
        domega_dt = (self.Kt * self.i - self.B * self.omega - tau_load) / self.J

        # Update states using Euler integration
        self.i += di_dt * dt
        self.omega += domega_dt * dt
        self.theta += self.omega * dt

        return self.theta, self.omega, self.i

# Simulate motor response
motor = DCMotor()
dt = 0.001  # 1ms time step
V = 12.0    # 12V input
tau_load = 0.1  # Load torque

# Record trajectory
time_points = []
theta_history = []
omega_history = []
i_history = []

for t in range(1000):
    theta, omega, i = motor.step(V, tau_load, dt)

    if t % 10 == 0:  # Record every 10ms
        time_points.append(t * dt)
        theta_history.append(theta)
        omega_history.append(omega)
        i_history.append(i)

# Plot results
plt.figure(figsize=(12, 8))

plt.subplot(3, 1, 1)
plt.plot(time_points, theta_history)
plt.ylabel('Position (rad)')
plt.title('DC Motor Response')

plt.subplot(3, 1, 2)
plt.plot(time_points, omega_history)
plt.ylabel('Speed (rad/s)')

plt.subplot(3, 1, 3)
plt.plot(time_points, i_history)
plt.ylabel('Current (A)')
plt.xlabel('Time (s)')

plt.tight_layout()
plt.show()

3.2.3 Advanced Actuation

Shape Memory Alloys Materials that change shape with temperature:

$$\epsilon = \epsilon_0 + \alpha \Delta T + \sigma/E + \epsilon_{SMA}$$

Piezoelectric Actuators Materials that deform with electric field:

$$\epsilon = d_{33} \cdot E$$

Where $d_{33}$ is the piezoelectric coefficient and $E$ is the electric field.

Artificial Muscles Technologies mimicking biological muscle:

• Pneumatic artificial muscles (PAMs)
• Electroactive polymers (EAPs)
• Dielectric elastomers

3.3 Physical Constraints on Intelligence

3.3.1 Speed of Computation vs. Speed of Physics

Computational systems operate on timescales that may not match physical requirements:

Diagram: Timescale Mismatch

Electronics:        nanoseconds to microseconds
├── CPU cycles:     ~1 GHz = 1ns per cycle
├── Memory access:  ~100ns
└── Communication:  ~1-10μs

Mechanical Systems:  milliseconds to seconds
├── Actuator response: 10-100ms
├── Mechanical motion: 100ms-1s
└── Environmental changes: 1s-1min

The Gap: 3-6 orders of magnitude difference!

3.3.2 Energy Constraints

Power Density Limits Biological systems demonstrate high power density efficiency:

$$P_{density} = \frac{P}{m} \text{ (W/kg)}$$

Human muscle: ~400 W/kg Typical electric motor: ~100-200 W/kg

Energy Efficiency

$$\eta = \frac{P_{useful}}{P_{total}}$$

3.3.3 Bandwidth Limitations

Communication Bandwidth Internal and external communication face bandwidth limits:

$$C = B \log_2(1 + SNR)$$

Where $C$ is channel capacity, $B$ is bandwidth.

Information Processing Limits Von Neumann-Landauer limit for computation:

$$E_{min} = k_B T \ln 2 \approx 2.8 \times 10^{-21} \text{ J at room temperature}$$

3.4 Sensor Fusion and State Estimation

3.4.1 Kalman Filtering

Kalman filters optimally combine multiple sensor measurements:

Prediction Step

$$\hat{x}_{k|k-1} = F_k \hat{x}_{k-1|k-1} + B_k u_k$$ $$P_{k|k-1} = F_k P_{k-1|k-1} F_k^T + Q_k$$

Update Step

$$K_k = P_{k|k-1} H_k^T (H_k P_{k|k-1} H_k^T + R_k)^{-1}$$ $$\hat{x}_{k|k} = \hat{x}_{k|k-1} + K_k (z_k - H_k \hat{x}_{k|k-1})$$

Example: Kalman Filter Implementation

import numpy as np

class KalmanFilter:
    def __init__(self, dim_x, dim_z):
        self.dim_x = dim_x  # State dimension
        self.dim_z = dim_z  # Measurement dimension

        # State estimate
        self.x = np.zeros((dim_x, 1))

        # Covariance matrix
        self.P = np.eye(dim_x)

        # State transition matrix
        self.F = np.eye(dim_x)

        # Measurement matrix
        self.H = np.zeros((dim_z, dim_x))

        # Process noise covariance
        self.Q = np.eye(dim_x)

        # Measurement noise covariance
        self.R = np.eye(dim_z)

        # Control input matrix
        self.B = None

        # Kalman gain
        self.K = None

    def predict(self, u=None):
        """
        Predict next state
        u: control input (optional)
        """
        if u is not None and self.B is not None:
            self.x = self.F @ self.x + self.B @ u
        else:
            self.x = self.F @ self.x

        self.P = self.F @ self.P @ self.F.T + self.Q

        return self.x

    def update(self, z):
        """
        Update state with measurement z
        """
        # Innovation
        y = z - self.H @ self.x

        # Innovation covariance
        S = self.H @ self.P @ self.H.T + self.R

        # Kalman gain
        self.K = self.P @ self.H.T @ np.linalg.inv(S)

        # State update
        self.x = self.x + self.K @ y

        # Covariance update
        I = np.eye(self.dim_x)
        self.P = (I - self.K @ self.H) @ self.P

        return self.x

# Example: 2D position tracking
kf = KalmanFilter(dim_x=4, dim_z=2)  # [x, y, vx, vy] and measure [x, y]

# State transition (constant velocity model)
dt = 0.1
kf.F = np.array([[1, 0, dt, 0],
                 [0, 1, 0, dt],
                 [0, 0, 1,  0],
                 [0, 0, 0,  1]])

# Measurement matrix (only position measured)
kf.H = np.array([[1, 0, 0, 0],
                 [0, 1, 0, 0]])

# Process noise
kf.Q = 0.01 * np.eye(4)

# Measurement noise
kf.R = 0.1 * np.eye(2)

# Initialize state
kf.x = np.array([[0], [0], [1], [0.5]])
kf.P = 100 * np.eye(4)

3.4.2 Particle Filtering

For non-linear, non-Gaussian systems, particle filters provide flexible estimation:

Diagram: Particle Filter Process

1. Initialize: Spread particles
   ★ ★ ★ ★ ★ ★

2. Predict: Move particles
   → → → → → →

3. Weight: Based on measurements
   ● ● ● ● ● ●
   (size = weight)

4. Resample: Keep likely particles
   ★ ★ ★ ★ ★ ★

3.4.3 Multi-sensor Fusion

Centralized Fusion All sensor data processed at central location:

• Optimal estimation
• High bandwidth requirements
• Single point of failure

Distributed Fusion Local processing with global consensus:

• Robust to failures
• Lower bandwidth
• Suboptimal but scalable

Example: Autonomous Vehicle Sensor Fusion

An autonomous vehicle combines:

• LiDAR (10-20 Hz, 3D points, ~100m range)
• Radar (20-50 Hz, velocity, ~200m range)
• Cameras (30 Hz, 2D images, ~200m range)
• IMU (100+ Hz, orientation, local)
• GPS (10 Hz, global position, ~10m accuracy)

Fusion provides comprehensive environmental understanding despite individual sensor limitations.

3.5 Control Under Physical Constraints

3.5.1 Model Predictive Control (MPC)

MPC optimizes control inputs subject to physical constraints:

$$\min_{u_{0:k}} \sum_{i=0}^{k} (x_i^T Q x_i + u_i^T R u_i)$$

Subject to:

$$x_{i+1} = Ax_i + Bu_i$$ $$u_{min} \leq u_i \leq u_{max}$$ $$x_{min} \leq x_i \leq x_{max}$$

3.5.2 Robust Control

Control systems must handle uncertainties and disturbances:

H-infinity Control Minimizes worst-case error:

$$\|G(s)\|_\infty < \gamma$$

Sliding Mode Control Robust to parameter variations and disturbances.

3.5.3 Adaptive Control

Systems that adapt to changing dynamics:

Diagram: Adaptive Control Architecture

[Reference] → [Controller] → [Actuator] → [Plant] → [Output]
     ↑            ↑                            ↑           |
     |            |                            |           ↓
     └─[Adaptation]←─────[Performance]←─────[Sensors]←───┘

3.6 Emerging Technologies

3.6.1 Quantum Sensors

Quantum effects enable ultra-sensitive measurements:

• Atomic interferometers for gravity sensing
• SQUIDs for magnetic field detection
• NV centers in diamond for magnetic imaging

3.6.2 Neuromorphic Sensing

Brain-inspired sensor architectures:

• Event cameras (change detection)
• Silicon retinas (sparse coding)
• Tactile sensors (spiking output)

3.6.3 Soft Robotics

Compliant actuators and sensors:

• Dielectric elastomer actuators
• Hydrogel sensors
• Fiber optic strain sensors

3.7 Design Implications

3.7.1 Working with Constraints

Exploiting Physics

• Use passive dynamics for efficiency
• Leverage mechanical computation
• Exploit environmental interactions

Managing Uncertainty

• Robust sensing strategies
• Redundant sensor modalities
• Graceful degradation

Energy Efficiency

• Match sensing frequency to task requirements
• Use intermittent sensing
• Co-design sensing and computation

The most successful embodied AI systems work with physics rather than against it, using natural dynamics and constraints to simplify control and improve efficiency.

3.7.2 Design Trade-offs

Speed vs. Accuracy Faster sensing often reduces accuracy:

• Integration time vs. noise reduction
• Bandwidth vs. resolution
• Computation time vs. precision

Energy vs. Performance Higher performance requires more energy:

• Sensor resolution vs. power consumption
• Actuator force vs. energy efficiency
• Computation complexity vs. power

Cost vs. Capability Advanced sensing and actuation increase costs:

• Sensor precision vs. price
• Actuator performance vs. cost
• System reliability vs. expense

Summary

Physical constraints fundamentally shape the capabilities and design of embodied AI systems. Understanding these constraints is essential for creating intelligent systems that can operate effectively in the real world.

Key takeaways:

• All sensing and actuation is limited by physical laws
• Energy, power, and bandwidth constraints shape system design
• Sensor fusion can overcome individual sensor limitations
• Control strategies must account for physical constraints
• Working with physics rather than against it leads to efficient designs

Exercises

Exercise 3.1: Sensor Analysis

Choose a specific sensor (e.g., camera, LiDAR, IMU) and analyze:

• Physical principles of operation
• Limitations and noise characteristics
• Power consumption and bandwidth requirements
• Typical applications and failure modes

Exercise 3.2: Actuator Selection

For a specific robotic task (e.g., arm manipulation, mobile locomotion):

• Identify required force/torque and speed characteristics
• Select appropriate actuator technology
• Analyze energy efficiency and control complexity
• Discuss alternative approaches

Exercise 3.3: Kalman Filter Implementation

Implement a Kalman filter for a specific estimation problem:

• Define system dynamics and measurement model
• Implement prediction and update steps
• Test with simulated or real data
• Analyze performance and robustness

Exercise 3.4: Constraint Optimization

Design a control system that explicitly handles physical constraints:

• Identify relevant constraints (actuator limits, safety boundaries)
• Implement constraint handling (MPC, barrier functions)
• Test with simulation
• Analyze trade-offs and performance

Exercise 3.5: Sensor Fusion Design

Design a multi-sensor fusion system for a specific application:

• Select complementary sensors
• Design fusion architecture
• Implement and test the system
• Evaluate performance improvements over individual sensors

Glossary Terms

• Signal-to-Noise Ratio (SNR): Ratio of signal power to noise power in a measurement
• Nyquist-Shannon Theorem: Minimum sampling rate required to capture signal information
• Transducer: Device that converts energy from one form to another
• Shape Memory Alloy (SMA): Material that returns to predetermined shape when heated
• Piezoelectric Effect: Electric charge generation in response to mechanical stress
• Kalman Filter: Optimal recursive data processing algorithm for linear systems
• Model Predictive Control (MPC): Control method optimizing future behavior subject to constraints
• Quantum Sensor: Sensor exploiting quantum mechanical effects for enhanced sensitivity
• Event Camera: Vision sensor that outputs changes rather than full frames
• Soft Robotics: Field of robotics using compliant materials and structures