A closed-loop neurobotic system for fine touch sensing

Figure 1(A) shows the complete closed-loop architecture, which included (i) an artificial touch sensor providing an array of analogue responses to mechanical tactile indentations; (ii) a network of primary neurons responsible for transducing the analogue output of the touch sensor into spiking, mechanoreceptor-like, activity; (iii) a network of second-order neurons that, similarly to cuneate neurons of the brainstem, processed primary afferent signals to facilitate downstream tactile discrimination and sensorimotor control; (iv) a probabilistic classification system for tactile input recognition; (v) a high-level controller that shaped active sensing based on optimality classification principles; (vi) a low-level controller subserving cerebellar-like fine movement adaptation; and (vii) a robotic arm-hand setup. Implementation-wise, all these components communicated through user datagram protocol (UDP) channels and were integrated in a framework whose system-clock timestep was 4 ms.

Zoom In Zoom Out Reset image size

The tactile stimulation protocol involved a set of 26 different probes reproducing a scaled version (1 : 1.7) of Braille characters (see figure S1 of the supplementary material (available from stacks.iop.org/JNE/10/046019/mmedia)). The artificial fingertip scanned a series of Braille lines actively, which emulated human fingertip deformations exerted by Braille dots. The feed-forward processing mediated by first- and second-order neural networks aimed at allowing rapid and reliable Braille discrimination to be carried out online, under optimized high-level velocity control strategies. We performed a series of quantitative analyses to assess information coding and neurotransmission at both primary and secondary processing stages, to measure performance in terms of fine touch classification, and to characterize adaptive sensorimotor control.

The artificial skin prototype⁷ (Cannata et al 2008, Bologna et al 2010) consisted of 24 capacitive square sensors disposed according to a rectangular grid layout (figure 1(B)). A 2.5 mm thick neoprene layer covered the array to protect the sensors during mechanical indentation and to passively modulate the exerted pressure. Each sensor had a dimension of 3 mm and the inter-sensor distance was 1 mm, for a total sensitive surface of approximately 18 × 23 mm (figure 1(C)). Given the spatial rescaling applied to the system and Braille characters (see section 2.1 of methods), the density of sensors on the artificial skin reached 17 sensor cm⁻². A series of preliminary experiments allowed the response properties of the artificial touch sensor to be characterized in the presence of both static and dynamic stimulation protocols (see supplementary materials, section S2 for more details (available from stacks.iop.org/JNE/10/046019/mmedia)). The analogue response of each sensor had an intensity proportional to the indentation level, with a high signal-to-noise ratio (50 dB), and a Gaussian-shaped profile with a peak amplitude of 200 ± 3.3 (mean ± std) femtoFarads (fF) and a width of 2.5 ± 0.044 mm. The acquisition frequency of each capacitive pad was 20 Hz.

The dataset of analogue responses collected through these experiments served as a basis for developing a simulated version of the artificial touch sensor. A family of 24 noisy Gaussian kernels sampled the input space based on the spatial location distribution of the real array. Each kernel modelled the response profile of an individual sensor, with white noise added to the amplitude and width of the response, 2.5 fF and 0.1 mm respectively. Also, a Gaussian noise on the position of each simulated stimulus (std = 0.1 mm) accounted for possible positional errors during real stimulation conditions.

The analogue readouts A_i(t), with i ∈ [1, 24], provided by the artificial touch sensor formed the excitatory inputs to a network of spiking neurons, based on a one-to-one connection scheme (figure 2(A)). This network of first-order neurons acted as a population of cutaneous fingertip mechanoreceptors, which converted analogue skin deformations following mechanical indentations onto spiking spatiotemporal patterns.

Zoom In Zoom Out Reset image size

We implemented the dynamics of each first-order neuron's membrane potential V_i(t) according to a leaky integrate-and-fire model (Lapicque 1907):

$$\begin{equation} C\cdot \frac{\,{\rm d}V_i(t)}{\,{\rm d}t} = -g\cdot (V_i(t) - V^{\prime } ) - k \cdot A_i(t) \end{equation} \tag{ 1 }$$

with C = 0.5 nF denoting the membrane capacitance, g = 25 nS the passive conductance (which gives a membrane time constant equal to τ = C/g = 20 ms), and V' = −70 mV the resting membrane potential. The term k · A_i(t) denotes the depolarizing input current, whose intensity was computed by multiplying the analogue response of the afferent touch sensor A_i(t) by a gain factor of k = −390 pA/fF, determined by comparing the output spike trains of simulated first-order neurons against recorded mechanoreceptor responses (Johansson and Birznieks 2004).

Each neuron emitted an action potential (spike) whenever its membrane potential V_i(t) reached a threshold V_th(t). Immediately after a spike event, the neuron was hyperpolarized to V_i(t) = V_reset = −100 mV, and the dynamics of its membrane potential were frozen during a refractory period Δt_ref = 2 ms. We also modelled the spiking adaptation phenomenon by means of a 'threshold fatigue' (Chacron et al 2003), which consisted of increasing the threshold V_th by 50 mV following each neuron discharge, making it harder for the neuron to spike again (i.e. bounding its response firing rate). In the absence of spikes, V_th decreased exponentially back to its resting value V'_th:

$$\begin{equation} \frac{\,{\rm d}V_{\mathrm{th}}(t)}{\,{\rm d}t} = -\frac{V_{\mathrm{th}}(t)-V_{\mathrm{th}}^{\prime }}{\tau _{\mathrm{th}}} \end{equation} \tag{ 2 }$$

with τ_th = 100 ms, and V'_th = −50 mV.

Second-order processing along the implemented somatosensory pathway occurred through a downstream network of spiking neurons (figure 2(A)), modelling a population of brainstem cuneate cells. The latter constitute the first relay mediating peripheral-to-central transmission of haptic signals (Johansson and Flanagan 2009). The neural model used in this study is derived from a previous work based on neurophysiological data from cat cuneate neurons (Bengtsson et al 2013).

The excitatory projections from first- to second-order neurons followed a scheme so as to generate the receptive fields shown in figure 2(B). Each cuneate neuron received non-plastic afferents from either one or several adjacent mechanoreceptors, with on average 1.9 ± 0.6 (mean ± std) connections per cuneate neuron. This sparse connectivity is coherent with physiological observations in which only a few primary afferents (less than ten), out of the hundreds of existing connections (Jones 2000), functionally contribute to individual cuneate neuron inputs (Bengtsson et al 2013). The dimension and shape of the receptive fields and the synaptic weight distribution of the mechanoreceptor-to-cuneate projections allowed topographical information to be maintained at the level of the second-order output space. The adopted connectivity layout enabled cuneate neurons collecting signals from larger receptive fields to account for both single primary neuron activation and coincidence detection of multiple co-activations, thus enriching their coding dynamics.

We described the activity of each cuneate neuron according to the spike-response model (Gerstner and Kistler 2002), by incorporating a noise model (i.e. escape noise) that followed a stochastic process, thereby providing a linear probabilistic neuronal model (Brasselet et al 2009). Whenever multiple afferent spikes excited the neuron within a short time window, they induced a compound membrane potential depolarization equal to:

$$\begin{equation} V(t) =V^{\prime }+\sum _{i,j} w_{\mathrm{i}} \cdot \Delta V \big(t-\hat{t}_{\mathrm{i}}^{\mathrm{\,j}}\big) \end{equation} \tag{ 3 }$$

$$\begin{equation} \Delta V (t) \propto \sqrt{t} \cdot \exp (-t/\tau ) \end{equation} \tag{ 4 }$$

with V' = −70 mV denoting the resting potential, i the presynaptic neurons, and j indexing the spikes emitted by a presynaptic neuron i at times $\hat{t}_{\mathrm{i}}^{\mathrm{\,j}}$. The synaptic weight w_i of the projection from the presynaptic unit i was taken so as to guarantee a reliable transmission of primary afferent signals and to avoid saturation of second-order neuron activity. We chose w_i = 0.04 and w_i = 0.028 for mechanoreceptors belonging to cuneate units receptive fields populated by one and two/three units respectively (figure 2(B)). The function ΔV(t) represented a unitary excitatory postsynaptic potential (EPSP), with τ = 2 ms indicating the decay time constant of the EPSP profile.

At each time step, the spiking probability p(t) of the neuron depended on the following functions:

$$\begin{equation} p(t) = 1-\exp (-f(t) \cdot R(t) ) \end{equation} \tag{ 5 }$$

$$\begin{equation} f(t) = r_{\mathrm{0}} \cdot \log \bigg ( 1+\exp \bigg (\frac{V(t)-V_{\mathrm{th}}}{V_{\mathrm{f}}} \bigg) \bigg) \end{equation} \tag{ 6 }$$

$$\begin{equation} R(t) = \frac{(t-\hat{t}-\tau _{\mathrm{abs}})^2}{\tau _{\mathrm{rel}}^2+(t-\hat{t}-\tau _{\mathrm{abs}})^2}\; {H}(t-\hat{t}-\tau _{\mathrm{abs}}) \end{equation} \tag{ 7 }$$

with f(t) denoting the instantaneous firing rate, determined by the constant r₀ = 11 Hz, the threshold potential V_th = −65 mV, and a gain factor V_f = 0.1 mV. The function R(t) determined the refractoriness property of the neuron, with ${\hat{t}}$ indicating the time of the last spike emitted, τ_abs = 3 ms the time constant of the absolute refractory period, τ_rel = 9 ms the time constant of relative refractory period, and H the Heaviside function.

We employed the Event-Driven neural simulator based on LookUp Tables (EDLUT)⁸ simulation environment (Ros et al 2006) to implement the second-order processing stage mediated by the network of model cuneate neurons. EDLUT is designed for efficient simulation of complex neural network models and takes advantage of both time-driven and event-drive procedures so as to guarantee fast computation and updates of neural state variables (Garrido et al 2011). Thanks to its properties, the EDLUT environment provided our system with the computational efficiency required for neurobotic applications.

In order to perform online tactile discrimination during Braille reading tasks, we trained a naïve Bayesian classifier (NBC) via multinomial distributions (McCallum and Nigam 1998). The NBC belongs to the family of probabilistic classifiers relying on Bayes' rule to compute the posterior probability of sample classes (Duda et al 2001). Relying on the hypothesis of independence between features, the NBC provides fast classification and is suited for applications requiring either a frequent estimation of class likelihoods or strict execution-time constraints (e.g. closed-loop systems, real-time robotic applications).

The spatiotemporal patterns provided by second-order cuneate neurons worked as class features, whose posterior probabilities could be incrementally estimated as spikes flew in the NBC—see supplementary materials, section S3 for details (available from stacks.iop.org/JNE/10/046019/mmedia), and Truccolo et al (2008) for a previous application of the NBC to recorded neural activity. The NBC training procedure involved a sample base of 100 trials (scans) for each Braille character. A gradually increasing temporal window allowed the spatiotemporal readouts from the cuneate network to be mapped onto a spike count data vector in a cumulative manner (see figure S3A of the supplementary material (available from stacks.iop.org/JNE/10/046019/mmedia)). Using these data vectors, the NBC computed the posterior probability of a letter being read to correspond to a known character class every 4 ms timestep. For online classification, we averaged the resultant probability distribution (across all Braille characters) over a 40 ms history, and as soon as the mean peak of the probability distribution exceeded the 90% threshold we considered the NBC to have classified the currently scanned Braille character.

The shape of the posterior probability distribution estimated by the NBC across all Braille letters evolved over time as the artificial fingertip scanned a given character. The ongoing degree of peakedness of this probability distribution provided a simple and effective basis to optimize the scanning speed control. A decision-making module (algorithmically implemented) monitored the evolution of the excess Kurtosis index k(t) of the NBC's output probability distribution on a 4 ms timestep basis. The fingertip scanning speed v(t) was then modulated according to:

$$\begin{equation} \dot{v}(t) = \frac{k(t) - k(t-1)}{C \cdot v(t)} \end{equation} \tag{ 8 }$$

with $\dot{v}(t)$ denoting the scanning acceleration (cm s⁻²), and C a constant factor tuned to 600 cm² s⁻². An increase of the k(t) function indicated a narrowing of the distribution, reflecting a decrease in the uncertainty of the probabilistic classification of the character being scanned. The high-level controller consequently increased the scanning speed proportionally to the gradient value. By contrast, a decrease of k(t) indicated a broadening of the estimate distribution, which induced a deceleration of the fingertip.

In order to compensate for movement execution errors, the closed-loop architecture incorporated a low-level controller (figure 1(A)) responsible for online sensorimotor adaptation. The low-level controller consisted of a spiking neural network mimicking the role of the cerebellum in fine movement control and coordination (Ito 1974, Thach et al 1992, Miall et al 2001, Attwell et al 2002).

We modelled the main information processing stages of the cerebellar microcomplex (see figure S4 of the supplementary material (available from stacks.iop.org/JNE/10/046019/mmedia)), and included long-term plasticity mechanisms to adapt its input–output dynamics through training. In agreement with Marr–Albus–Ito theory (Marr 1969, Albus 1971, Ito and Kano 1982), we assumed that the cerebellum can learn internal representations of sensorimotor interactions in multiple microcomplexes (Wolpert et al 1998). We implemented four cerebellar microcomplexes and trained them to predict the outcomes of motor commands prior to their actual execution, i.e. the microcomplexes acted as forward predictor models (Ito 1984, Miall et al 1993, Wolpert and Miall 1996). These predictions allowed the motor commands to be finely adjusted online, by avoiding otherwise inaccurate movement execution (e.g. local drifts in trajectories). See supplementary materials, section S4 (available from stacks.iop.org/JNE/10/046019/mmedia) for a more detailed description of the cerebellar model, which we developed in a previous work (Passot et al 2012).

We simulated a 2-dof robotic arm with noisy dynamics, taken from Carrillo et al (2008). The arm model included two joints (shoulder, elbow), with the arm end-point carrying the simulated touch sensor described in section 2.2. The four cerebellar forward models learned to predict the future angular position and velocity of each of the two joints. During movement control, the predicted joint states were algorithmically mapped onto the arm end-point position (in Cartesian coordinates). Then, a trajectory generator (Carrillo et al 2008) compared the desired and predicted position of the arm end-point and updated the motor commands consequently.

We also tested the performance of the overall Braille reading system in a real-world scenario. Section 3.5 of the results presents a preliminary experiment in which a robotic arm-hand platform, carrying the real artificial touch sensor, scanned and classified a Braille line (figure 8).

To quantify the information content of first and second-order neural responses we applied the metrical information theory (Brasselet et al 2011b). Unlike Shannon mutual information (Shannon 1948), this measure takes into account the metrical properties of the spike train space (Victor and Purpura 1996, Schreiber et al 2003, van Rossum 2001), which proved to be suited to decode the responses of human mechanoreceptors obtained via microneurography recordings (Brasselet et al 2011b).

The metrical mutual information I*(R; S), with R and S denoting the response and stimulus space, respectively, is defined as follows:

$$\begin{eqnarray} \fl I^*(R;S) &=& H^*(R)-H^*(R|S) \nonumber\\ &=& \sum _{s, r} p(r,s) \log _2 \bigg( \frac{\sum _{r^{\prime }} p(r^{\prime }|s) \phi (r,r^{\prime })}{\sum _{r^{\prime }} p(r^{\prime }) \phi (r,r^{\prime })} \bigg) \end{eqnarray} \tag{ 9 }$$

where

$$\begin{eqnarray} &&\fl H^*(R) = -\sum _{r} p(r) \log _2 \bigg( \sum _{r^{\prime }} p(r^{\prime }) \phi (r,r^{\prime }) \bigg) \end{eqnarray} \tag{ 10 }$$

$$\begin{eqnarray} \fl H^*(R|S) &=& \sum _{s} p(s) H(R|s)\nonumber\\ &=&-\sum _{s, r} p(r,s) \log _2 \bigg( \sum _{r^{\prime }} p(r^{\prime }|s) \phi (r,r^{\prime }) \bigg) \end{eqnarray} \tag{ 11 }$$

$$\begin{eqnarray} &&\fl \phi (r,r^{\prime }) = {H} { (}D_{c}-D(r,r^{\prime }) { )} \end{eqnarray} \tag{ 12 }$$

where H*(R) and H*(R|S) are the marginal and conditional metrical entropies, respectively; p(r), with r ∈ R, is the response marginal probability; p(s), with s ∈ S, the stimulus a priori probability; p(r|s) and p(r, s) the conditional and joint probabilities, respectively. The function ϕ(r, r') measures the similarity between two responses, and it is defined as the Heaviside function of the Victor–Purpura distance D(r, r') between two spike trains r, r' ∈ R (Victor and Purpura 1996). The term D_c is the cutoff parameter, and is called the critical distance. For D(r, r') < D_c, responses r, r' are considered as identical (i.e. ϕ(r, r') = 1), otherwise they are considered as different. If D_c = 0 we recover the Shannon entropy from equation (10).

Optimal information transmission occurs when the metrical mutual information I*(R; S) is maximized and (at the same time) the conditional entropy H*(R|S) is minimized. Under this optimality condition, perfect input discrimination is reached (Brasselet et al 2011b).

Several complexity measures have been established in the field of psychophysics to describe visual (Alexander and Carey 1968, Chipman 1977, Yodogawa 1982) and vibro-tactile stimuli (Horner 1991), attempting to link an objective measure to perceived complexity (Aksentijevic and Gibson 2012). To the best of our knowledge, their extension to the study of the Braille alphabet has concentrated solely around the number of dots as a complexity estimator (Nolan and Kederis 1969, Newman et al 1984). In order to study the correlation between Braille letter complexity and movement policy (see section 3.3 of the results), we considered a linear combination of two measures, namely the number of dots Dot (Nolan and Kederis 1969) and the number of subsymmetries Sym (Alexander and Carey 1968):

$$\begin{equation} C =\alpha \cdot Dot + (1-\alpha ) \cdot Sym \end{equation} \tag{ 13 }$$

where

$$\begin{equation} Sym = \sum ^{X}_{x=1} \sum ^{Y}_{y=1} \left(\sum ^{X-x}_{i=1} \sum ^{Y-y}_{j=1} i \cdot j \cdot NSym_{(x,y)}(i,j) \right) \end{equation} \tag{ 14 }$$

with α = 0.94. The Sym function counted the number of symmetries of each rectangular sub-grid of the Braille character matrix and added them in a sum weighted by the area of the sub-grid; with X and Y referring to the size of the first and second cell dimensions respectively (i.e. 2 × 3 in the case of Braille cells), and NSym_{(x, y)}(i, j) being the number of symmetries observed in the rectangle of size i × j at position (x, y). The symmetries considered consisted of vertical, horizontal, and central symmetries for all sub-grid dimensions, as well as both diagonal symmetries when i = j.

We first compared simulated and real human mechanoreceptor responses to moving Braille probes passively sensed by a steady fingertip (Bologna et al 2011). To do so, we reproduced in simulation the experimental protocol used by (Phillips et al 1990) to characterize primary afferent representations of dotted spatial patterns. Braille characters were dynamically swiped at 60 mm s⁻¹ over the receptive field of a recorded mechanoreceptor while the fingertip was kept still. Once the stimulus exited the receptive field, its position was shifted by 0.2 mm along the radial-to-ulnar axis, and the process repeated. This procedure allowed the so-called spatial event plot (SEP) (Phillips et al 1988) to be constructed from the recorded mechanoreceptor spikes (figure 3(A)). Simulated SEPs, which illustrated the spatiotemporal characteristics of a first-order response to a moving Braille stimulus, were qualitatively similar to those of slow adaptive type I (SA-I) human mechanoreceptors. Receptive field sizes of recorded and simulated afferents were also comparable at, respectively, 4.8 ± 1.2 mm² (mean ± std) and 4.7 ± 1.5 mm² after correcting for the scale difference. However, due to mechanical constraints, the modelled receptive fields had little overlap and consequently covered the artificial fingertip with a density of 17 units mm⁻², one fourth of the 70 units mm⁻² reached by their biological counterparts (Phillips et al 1990). Nonetheless, they showed a topological stimulus-response mapping, an important property for the encoding of fine spatial discontinuities (Johansson and Flanagan 2009).

Zoom In Zoom Out Reset image size

Figure 3(B) shows an example of discharge pattern recorded from a simulated first-order neuron in response to a single dot stimulus moving at 30 mm s⁻¹. Responses are shown as a raster plot of spike times (top) and the corresponding poststimulus time histogram (PSTH) (centre) during 150 stimulation trials. The analogue output signal from the indented touch sensor serves as depolarizing current to the model neurons (bottom). The spike timing reliability of the firing pattern was high at the onset of the protraction stimulation phase and decreased with time, as observed for human SA-I mechanoreceptors (Johansson and Flanagan 2009) and more generally in central neurons (Mainen and Sejnowski 1995).

We then compared the first-spike jitter distribution of simulated mechanoreceptors against experimental SA-I data (figure 3(C), top). Despite a time lag of about 2 ms, there was no statistical difference between the two distribution shapes (Kolmogorov–Smirnov, p > 0.076). Indeed, when accounting for the 2 ms time lag, we did not observe any statistical difference between the two distribution medians (Mann–Whitney U, p > 0.11). A comparison between the interspike interval (ISI) distributions of simulated and real mechanoreceptors (figure 3(C), bottom) showed that the ISI variability of model neurons was smaller than in recorded SA-I afferents, although the medians of the two distributions were statistically equivalent (Mann–Whitney U test, p > 0.16). The difference in ISI variability may reflect the viscoelastic properties and more complex dynamics of human skin compared to the artificial finger.

We studied neurotransmission reliability at the first- and second-order stages of the simulated somatosensory pathway. The stimulation protocol consisted in scanning the entire 26 character Braille alphabet (see figure S1 of the supplementary material (available from stacks.iop.org/JNE/10/046019/mmedia)) at a constant speed of 30 mm s⁻¹ during 200 trials. We performed a metrical information analysis of simulated mechanoreceptor and cuneate responses. Figure 4(A) illustrates the evolution of the information content of first-order and second-order neural activities over time (i.e. as spikes flew in during Braille scanning). At the mechanoreceptor level (figure 4(A), left), a complete discrimination of all Braille characters occurred within 350 ms of stimulus onset, i.e. when the optimality condition was verified. Indeed, according to metrical information theory (Brasselet et al 2011b), after 350 ms the clusters of simulated mechanoreceptor responses to all Braille characters became completely separated from each other, allowing perfect context separation to be achieved. Expectedly, the information curve exhibited a plateau after about 100 ms and lasting approximately 75 ms. This time interval corresponded to the stimulation phase during which the fingertip was only contacting the first column of Braille dots, whereas the second column did not yet stimulate any touch sensor. The information value at the plateau was about half of the total information transmitted. We observed a similar information content profile at the level of simulated cuneate responses (figure 4(A), right), for which perfect input discrimination occurred as rapidly as in first-order neurons. Yet, the redundancy of second-order responses was slightly larger, as indicated by the non-zero conditional metrical entropy after 350 ms.

Zoom In Zoom Out Reset image size

Even if after 350 ms both mechanoreceptor and cuneate responses contained enough information to discriminate all Braille letters theoretically, we wanted to investigate to what extent the implemented probabilistic decoder would benefit, in terms of classification, from the processing carried out at the cuneate level. We extended the above stimulation protocol by varying the fingertip speed within the range [10, 80] mm s⁻¹, with a step of 10 mm s⁻¹. For each speed value, the fingertip scanned all 26 Braille characters for 100 trials. The performance of the NBC was comparable when decoding first and second-order responses at movement speeds lower than 30 mm s⁻¹ (figure 4(B), top). By contrast, processing at the cuneate level led to a statistically significant increase of 12% in classification performance at 30 mm s⁻¹ (Mann–Whitney U, p < 0.01). The improvement in performance was even larger for higher scanning speeds, showing that cuneate processing enhanced the generalization and robustness of the classification process in the presence of speed modulation. We also quantified the nonclassification rate, i.e. how often, on average, the system was unable to classify a Braille letter during a unique scanning (figure 4(B), bottom). This measure is related to the mean number of reversal movements during Braille reading, which consist of backtracking the fingertip to rescan a unrecognized letter. In accordance with the above results, for scanning speeds higher than 30 mm s⁻¹, the nonclassification rate was significantly larger when decoding first-order responses than cuneate activity.

To further address this issue, we focused on some Braille dot arrangements that were likely to evoke similar mechanoreceptor responses due to the homogeneous structure of the artificial touch sensor. For instance, symmetrical letters (e.g. 'e' and 'i', 'd' and 'f', 'h' and 'j' and 'r' and 'w') would activate the same subset of sensors, increasing the probability of cross-character interference during scanning. To illustrate this point we performed a principal component analysis of both first order and cuneate responses obtained at a scanning speed of 30 mm s⁻¹. The example of figure 4(C) shows that the separation between the clusters of first-order responses to letters 'e' and 'i' decreased over time, whereas it remained rather constant at the level of cuneate responses (left and right diagram, respectively). The reduced interference between symmetrical letters at the second-order level partially explains the corresponding better classification performance at higher speeds (figure 4(B)). In the context of the full Braille alphabet, an early discrimination was prevented by interference from letters with identical first dot columns (e.g. 'a', 'c' and 'd' for letter 'e'). In the case of first-order responses, subsequent interference by symmetrical characters left only a small time window available for the classification to occur. For high scanning speeds, this window shortened and the classification performance decreased consequently. By contrast, cuneate processing avoided this effect.

We then tested the performance of the entire closed-loop system, including high- and low-level movement control, in a Braille reading task. The artificial fingertip scanned multiple Braille lines containing 8 letters each, for a total of 200 trials (i.e. repetitions) per letter. As scanning proceeded, feed-forward processing through first and second-order networks provided the probabilistic classifier with continuously evolving spike responses. The NBC estimated the online posterior probability distribution. The latter eventually converged to a narrow single peaked distribution, allowing classification of one Braille letter to be achieved. Figure 5(A) shows the time course of two examples of posterior probability distributions corresponding to the scanning of the letter 'r' (top) and 'e' (bottom). In the first example, early cuneate activity did not allow the probabilistic classifier to distinguish between 'r' and other Braille characters with a similar dot arrangement (i.e. 'l', 'p', 'q', 'v'). Nevertheless, as movement progressed, the probability distribution started to peak, indicating a decrease of uncertainty, until a correct classification became possible.

Zoom In Zoom Out Reset image size

As shown by the confusion matrix in figure 5(B) and by the distribution in figure 5(C), the overall online classification performance was characterized by a recognition rate of 95 ± 1.5% (mean ± s.e.m), a nonclassification rate (i.e. reversal movement rate) of 1 ± 0.4%, and a false positive rate of 4 ± 1.3%.

Adaptive speed modulation was a function of the evolution of the posterior probability distribution through time, which determined the active sensing policy. The example of figure 6(A) displays the time course of the posterior probability distribution (top) and of the corresponding finger acceleration profile (bottom) while scanning a line with three letters ('d', 'y', and 'n'). Over the entire Braille reading task, the mean number of finger accelerations (both positive and negative) per letter was equal to 11.8, which is of the same order of magnitude as that observed in human Braille reading experiments (Hughes 2011, Hughes et al 2011). Importantly, the movement policy (and consequently the number of accelerations) varied significantly depending on the letter being scanned (figure 6(B), top; Kruskal–Wallis ANOVA, p < 0.01). In addition, we observed a significant correlation between the distribution of Braille character complexity (defined according to equation (13)) and the distribution of the number of accelerations per letter (figure 6(B), bottom; Spearman's ρ = 0.66, p < 10⁻³).

Zoom In Zoom Out Reset image size

During Braille reading, the low-level cerebellar controller adjusted the trajectory of the fingertip online. The training of the cerebellar network involved ten sessions. Each session consisted of ten trials during which the fingertip (i.e. the end-point of the 2-dof arm carrying the artificial touch sensor) had to scan a Braille line containing eight evenly spaced characters, at a constant command speed of 30 mm s⁻¹. After training, the output of the cerebellar network provided a good estimator of the noisy arm dynamics, allowing motor commands to be tuned online—and, consequently, the movement accuracy to be improved. The mean position error, computed as the discrepancy between desired and actual fingertip trajectory, decreased significantly through cerebellar training, from 0.81 ± 0.04 mm (mean ± s.e.m) to 0.54 ± 0.04 mm (Mann–Whitney U, p < 0.01).

Figure 7(A) shows a sample of fingertip trajectory (top), the corresponding PSTH (centre) of the cerebellar output (i.e. the activity of simulated neurons in the deep cerebellar nuclei (DCN)), and two examples of DCN spikegrams. Figure 7(B) shows a fingertip trajectory, averaged over ten trials, with and without cerebellar-dependent adaptation (blue and red curve, respectively). The cerebellar online adjustment proved effective at reducing fingertip oscillation amplitudes. We quantified the percentage of times that the fingertip trajectory exceeded a ±1 mm bounded region (dashed lines in figure 7(B)). The resulting error, averaged over 10 trials, decreased significantly through cerebellar training (figure 7(C); Mann–Whitney U, p < 10⁻³). About 82% of the positions belonging to a finger trajectory fell, on average, within the ±1 mm boundary by the end of training.

Zoom In Zoom Out Reset image size

We then studied the influence of position inaccuracy on the probabilistic classification performance through the following protocol. A Braille line was swiped at 30 mm s⁻¹ over the immobile artificial fingertip. After 80 repetitions of each letter, the Braille line was shifted along the distal-to-proximal axis by 0.5 mm, and the process repeated. This shift corresponded to a fingertip position change along the Y axis in the active sensing scenario. Figure 7(D) shows the recognition rate (mean ± s.e.m.) as a function of the Y position. The mean classification performance remained high within the ±1 mm range, whereas it decreased sharply beyond these boundaries. The best recognition rate of 99% occurred at Y positions of 0 and −0.5 mm (Mann–Whitney U, p > 0.4). Finally, figure 7(E) displays the classification performance distribution across letters depending on the Y position. The recognition rate of individual letters tended to follow an all-or-none pattern, with few intermediate values.

In order to test the performance of the system in a real-world scenario, we ran a preliminary experiment in which a robotic arm-hand platform had to solve a Braille reading task (figure 8(A)). We fixed the artificial touch sensor (Cannata et al 2008) on the index digit of the DLR-HIT hand II (Liu et al 2008) (figure 8(B)). Then, we mounted the hand on the DLR light-weight robot III (LWR) (Albu-Schäffer et al 2007a), in order to rub the fingertip over a Braille line at a constant speed of 30 mm s⁻¹ (figure 8(C)). Both the LWR robot and the DLR-HIT hand were operated in impedance control mode (Albu-Schäffer et al 2007b), which ensured stability even in physical contact situations as required in this task. The calculated adjustments to the reading velocity as well as the corrections to the movement trajectory were relayed to the robotic system via an UDP interface. Yet, compared to the simulated arm, noise in the pitch and roll dynamics of the real arm-hand platform remained marginally compensated, which increased the probability of inhomogeneous and discontinuous contacts between the fingertip and Braille characters.

Zoom In Zoom Out Reset image size

We selected a subset of representative Braille characters and recorded 150 trials per character. Again, first and second order neural networks mediated feed-forward processing of tactile signals prior to their use for probabilistic classification. We trained the NBC through the same offline procedure used for simulated data, which aimed at estimating the posterior probability distribution through increasing temporal windows that sampled cuneate responses (see supplementary materials, section S3 for details (available from stacks.iop.org/JNE/10/046019/mmedia)). We randomly selected 100 trials for training and we employed the remaining 50 trials for probing Braille character recognition. The mean classification rate, averaged over seven characters, was 89 ± 5.3% (mean ± s.e.m.; figure 8(D)), with a false positive rate of 11 ± 5.3%. This preliminary robotic validation gave rise to a real-time and online Braille reading demonstrator in the framework of the European project 'Sensopac', no. IST-028056-IP (see supplementary material videos (available from stacks.iop.org/JNE/10/046019/mmedia)).