Tactile Sensing and Proprioception: What Robots Need to Feel
Why tactile sensing and robot proprioception matter more than vision for contact-rich manipulation, and how a shared somatosensory event schema could help.
Most robots are built to see. Cameras, depth sensors, and increasingly large vision models do the heavy lifting, and the rest of the body is treated as an actuator to be commanded. Yet the moments that decide whether a manipulation task succeeds (the instant of contact, the first millimeter of slip, the give of a compliant part) are the moments vision handles worst.
Biology made the opposite bet. Touch and proprioception are primary senses, processed quickly near the periphery and fused centrally with a running prediction of what the body expects to feel. If embodied AI is going to handle contact-rich manipulation reliably, it needs to take that design seriously.
Why vision alone fails at contact-rich manipulation
Vision answers “where is the object?” well. It answers “what is happening between my fingers and the object right now?” poorly, for four structural reasons.
- Occlusion at the moment of grasp. When a gripper closes on an object, the gripper itself hides the contact region. The camera loses the most important view exactly when it matters.
- Slip is fast and small. Incipient slip begins as micro-displacements and shear changes at the contact surface. By the time slip is visible from a camera, the object is often already moving.
- Compliance is invisible. A foam part, a cable, a bag of produce, and a rigid casting can look similar and behave completely differently under load. Stiffness and damping have to be felt.
- Force regulation needs force. Inserting a connector, wiping a surface, or holding a fragile object depends on regulating contact force. Inferring force from pixels is indirect, slow, and sensitive to calibration.
The practical consequence is familiar to anyone who has deployed manipulation systems: policies that look strong in free space degrade at contact, and failures cluster around grasp, insertion, and handover.
The robot touch sensor landscape
Tactile and proprioceptive sensing is not one technology. At a qualitative level, the options fall into a few families, each with different tradeoffs.
| Sensor family | What it measures | Typical strengths | Typical limits |
|---|---|---|---|
| Force/torque sensors (wrist or joint) | Net forces and moments at a mounting point | Mature, calibrated, good for force control | No spatial detail about where contact occurs |
| Capacitive and resistive skins | Pressure distribution over a surface | Can cover large areas, thin form factors | Variable resolution, drift, wiring complexity |
| Vision-based tactile sensors (GelSight-style optical) | Deformation of an elastomer imaged by an internal camera | High spatial resolution, shear and texture cues | Bulk, bandwidth, limited coverage per sensor |
| Joint encoders and IMUs | Joint position, velocity, body acceleration and orientation | Core proprioception, available on most robots | Says little about the contact itself |
Proprioception deserves emphasis. Robot proprioception, the sense of the body’s own configuration and motion, is what lets a system notice that a joint is deflecting under unexpected load or that a limb did not arrive where it was commanded. Most robots already have these signals. They are often logged and ignored.
The tactile data problem
The deeper obstacle is not hardware. It is representation.
Every tactile sensor produces its own format: an optical sensor emits image frames, a capacitive skin emits a grid of taxel values at some sampling rate, a force/torque sensor emits a six-axis vector. Each is tied to its own geometry and mounting. A dataset collected on one gripper with one sensor rarely transfers to another gripper with a different sensor, let alone to a different robot morphology.
This is part of why tactile learning lags vision. Vision benefits from a shared representation (images) and enormous shared datasets. Touch has neither. Each lab and each product line effectively starts over.
The case for an embodiment-independent somatosensory event schema
One way through is to stop exchanging raw sensor streams and start exchanging events. A somatosensory event schema would describe physical interaction in terms that do not depend on the specific sensor or body:
- where on the body contact occurred, expressed in a body-relative frame
- when it began, changed, or ended
- estimated normal and shear force, with uncertainty
- event type, such as contact onset, slip onset, release, or impact
- the proprioceptive state of the relevant limb at that moment
- provenance: which sensor produced the event and how it was encoded
With a representation like this, a slip event from an optical fingertip sensor and a slip event from a capacitive palm skin become comparable. Tactile datasets become shareable across morphologies, and models trained on one embodiment have a plausible path to transfer.
Peripheral encoding and central prediction
The biological analogy is useful here because it describes an architecture, not just a sensor.
Peripheral nerves do not forward raw pressure maps to the brain. Receptors respond to change, and early processing extracts events such as onset, vibration, and slip close to where they happen. The central nervous system then compares incoming signals to what it predicted the body would feel, and attends mainly to the difference.
A robotic version of this looks like two layers:
- Event-based local encoding at the periphery. Sensor-adjacent processing converts raw tactile and proprioceptive streams into discrete events. This keeps latency low for reflex-like responses (tightening a grip on slip onset) and cuts bandwidth to the central controller.
- Central predictive control. A central model maintains an expectation of contact given the current plan and body state. Surprises, such as contact where none was expected or force exceeding prediction, drive correction, replanning, or a request for human attention.
This split also makes the system easier to reason about. Reflexes are local and bounded. Deliberation is central and can be slower.
Observability and simulation for contact
Contact is notoriously hard to simulate and hard to debug. Physics engines approximate friction and deformation, and the gap between simulated and real contact is a common source of sim-to-real failure.
Event-level representations help on both fronts. They give engineers something inspectable: a timeline of contact onsets, slips, and force surprises that can be replayed and compared across runs. They also give simulation a clearer target. Instead of matching raw sensor output exactly, a simulator can be judged on whether it produces the right events at the right times.
A practical checklist for teams working on contact-rich manipulation:
- Log proprioceptive signals (joint positions, torques, IMU) alongside vision for every trial, even before you use them.
- Record grasp, insertion, and release outcomes with timestamps so failures can be aligned with sensor data.
- Define a small set of contact events you care about (onset, slip, release, unexpected contact) and detect them consistently.
- Express contact locations in a body-relative frame, not sensor pixel coordinates.
- Compare simulated and real runs at the event level, not only at the trajectory level.
- Treat unexpected contact as an observable fault, not just noise.
How QuantumWorks approaches embodied sensing
The Embodied Predictive Cortex is QuantumWorks work on tactile and proprioceptive intelligence built on the ideas above: distributed peripheral sensing, locally encoded contact events, proprioception, and central predictive control, unified by an embodiment-independent somatosensory event schema. It sits within our broader autonomy and robotics area alongside MissionGuard, which applies a related observability principle at the mission level: make autonomous behavior legible enough to check. If you are working on manipulation where contact is the hard part, we would like to compare notes.