Cortical model with reinforcement learning drives realistic virtual arm (Dura-Bernal et al 2015)

 Download zip file   Auto-launch 
Help downloading and running models
Accession:183014
We developed a 3-layer sensorimotor cortical network of consisting of 704 spiking model-neurons, including excitatory, fast-spiking and low-threshold spiking interneurons. Neurons were interconnected with AMPA/NMDA, and GABAA synapses. We trained our model using spike-timing-dependent reinforcement learning to control a virtual musculoskeletal human arm, with realistic anatomical and biomechanical properties, to reach a target. Virtual arm position was used to simultaneously control a robot arm via a network interface.
Reference:
1 . Dura-Bernal S, Zhou X, Neymotin SA, Przekwas A, Francis JT, Lytton WW (2015) Cortical Spiking Network Interfaced with Virtual Musculoskeletal Arm and Robotic Arm. Front Neurorobot 9:13 [PubMed]
2 . Dura-Bernal S, Li K, Neymotin SA, Francis JT, Principe JC, Lytton WW (2016) Restoring behavior via inverse neurocontroller in a lesioned cortical spiking model driving a virtual arm. Front. Neurosci. Neuroprosthetics 10:28
Model Information (Click on a link to find other models with that property)
Model Type: Realistic Network;
Brain Region(s)/Organism:
Cell Type(s): Neocortex M1 pyramidal pyramidal tract L5B cell; Neocortex M1 pyramidal intratelencephalic L2-5 cell; Neocortex M1 interneuron basket PV cell; Neocortex fast spiking (FS) interneuron; Neostriatum fast spiking interneuron; Neocortex spiking regular (RS) neuron; Neocortex spiking low threshold (LTS) neuron;
Channel(s):
Gap Junctions:
Receptor(s): GabaA; AMPA; NMDA;
Gene(s):
Transmitter(s): Gaba; Glutamate;
Simulation Environment: NEURON; Python (web link to model);
Model Concept(s): Synaptic Plasticity; Learning; Reinforcement Learning; STDP; Reward-modulated STDP; Sensory processing; Motor control;
Implementer(s): Neymotin, Sam [samn at neurosim.downstate.edu]; Dura, Salvador [ salvadordura at gmail.com];
Search NeuronDB for information about:  Neocortex M1 pyramidal intratelencephalic L2-5 cell; Neocortex M1 pyramidal pyramidal tract L5B cell; Neocortex M1 interneuron basket PV cell; GabaA; AMPA; NMDA; Gaba; Glutamate;
/
arm2dms_modeldb
mod
drspk.mod *
infot.mod *
intf6.mod *
intfsw.mod *
misc.mod *
mySTDP.mod
nsloc.mod
nstim.mod *
place.mod *
stats.mod *
updown.mod *
vecst.mod *
parameters.multi
                            
: $Id: updown.mod,v 1.16 2009/02/16 22:56:52 billl Exp $

NEURON {
  SUFFIX nothing
  : BVBASE is bit vector base number (typically 0 or -1)
  GLOBAL UPDOWN_INSTALLED, SHM_UPDOWN, NOV_UPDOWN, DEBUG_UPDOWN
}

PARAMETER {
  UPDOWN_INSTALLED=0
  DEBUG_UPDOWN=0
  SHM_UPDOWN=4   : used in updown() for measuring sharpness
  NOV_UPDOWN=1   : used in updown() to eliminate overlap of spikes
  CREEP_UPDOWN=0 : used in updown() to allow left/right "creep" to local minima
}

VERBATIM
#include <stdlib.h>
#include <math.h>
#include <limits.h> // contains LONG_MAX 
#include <sys/time.h> 
extern double* hoc_pgetarg();
extern double hoc_call_func(Symbol*, int narg);
extern FILE* hoc_obj_file_arg(int narg);
extern Object** hoc_objgetarg();
extern void vector_resize();
extern int vector_instance_px();
extern void* vector_arg();
extern double* vector_vec();
extern double hoc_epsilon;
extern double chkarg();
extern void set_seed();
extern int ivoc_list_count(Object*);
extern Object* ivoc_list_item(Object*, int);
extern int hoc_is_double_arg(int narg);
extern char* hoc_object_name(Object*);
char ** hoc_pgargstr();
int list_vector_px();
int list_vector_px2();
int list_vector_px3();
double *list_vector_resize();
int ismono1();
static void hxe() { hoc_execerror("",0); }
static void hxf(void *ptr) { free(ptr); hoc_execerror("",0); }
ENDVERBATIM

:* src.updown(thresh,dlist,nqslist)
:  dest.updown(src)  -- default thresh=0; returns indices
: look for multiple threshold crossings to define peaks
: creates multiple parallel vectors for an NQS db
: counts peaks pointing upward -- should be all pos
: see eg decnqs.hoc:fudup() for usage
VERBATIM

  //** declarations
#define UDSL 500
#define UDNQ 11
// nq=new NQS("LOC","PEAK","WIDTH","BASE","HEIGHT","START","SLICES","SHARP","INDEX","FILE","NESTED")
#define LOC     nq[0] // loc of peak of spike
#define PEAK  	nq[1] // value at peak (absolute height)
#define WIDTH  	nq[2] // rt flank - lt flanks (? isn't it rt flank - LOC ?)
#define BASE  	nq[3] // height at base
#define HEIGHT  nq[4] // peak - base
#define START  	nq[5] // left flank of spike?
#define SLICES  nq[6] // how many slices found this spike
#define SHARP  	nq[7] // 2nd deriv at peak
#define INDEX  	nq[8] // consecutive numbering of spikes
//        	nq[9] // will use to fill in trace's file name at hoc level
#define NESTED  nq[10] // how many bumps are nested within this one
  //** procedure updown()
static double updown (void* vv) {
  int i, k, m, n, nqsz, nsrc, jj[UDSL], f[UDSL], lc, dsz[UDSL], nqmax, thsz, lc2, done, dbn;
  double *src, *tvec, *th, *dest[UDSL], *nq[UDNQ], *tmp, *dbx, lt, thdist;
  Object *ob, *ob2;
  void *vvd[UDSL], *vvth, *vnq[UDNQ];
  //** read in vectors and verify sizes, etc
  nsrc = vector_instance_px(vv, &src); // trace to analyze
  thsz = vector_arg_px(1, &th);        // vector of thresholds to check
  ob =  *hoc_objgetarg(2);             // storage for values for each threshold
  ob2 = *hoc_objgetarg(3);             // list of NQS vectors for returning values
  tmp = (double *)ecalloc(nsrc, sizeof(double));  // tmp is size of trace
  lc =  ivoc_list_count(ob);
  lc2 = ivoc_list_count(ob2);
  if (lc>UDSL) {printf("updown ERRF mismatch: max slice list:%d %d\n",UDSL,lc); hxf(tmp);}
  if (lc2!=UDNQ){printf("updown ERRB mismatch: NQS sz is %d (%d in list)\n",UDNQ,lc2);hxf(tmp);}
  if (nsrc<lc) {printf("updown ERRC mismatch: %d %d\n",lc,nsrc); hxf(tmp);} // ??
  if (lc!=thsz) {printf("updown ERRA mismatch: %d %d\n",lc,thsz); hxf(tmp);}
  if (!ismono1(th,thsz,-1)) {printf("updown ERRD: not mono dec %g %d\n",th[0],thsz); hxf(tmp);}
  // thdist=(th[thsz-2]-th[thsz-1])/2; // NOT BEING USED: the smallest spike we will accept
  for (k=0;k <lc;k++)  dsz[k] =list_vector_px3(ob , k, &dest[k], &vvd[k]);
  for (k=0;k<lc2;k++) {
    i=list_vector_px3(ob2, k, &nq[k],   &vnq[k]);
    if (k==0) nqmax=i; else if (i!=nqmax) { // all NQ vecs same size
      printf("updown ERRE mismatch: %d %d %d\n",k,i,nqmax); hxf(tmp); }
  }
  //** store crossing points and midpoints in dest[k]
  // dest vectors dest[k] will store crossing points and midpoints at each th[k] slice location
  // as triplets: up/max/down
  for (k=0; k<lc; k++) {   // iterate thru thresholds
    jj[k]=f[k]=0; // jj[k] is ind into dest[k]; f[k] is flag for threshold  crossings
    for (i=0;i<nsrc && src[i]>th[k];i++) {} // start somewhere below this thresh th[k]
    for (; i<nsrc; i++) { // iterate through trace
      if (src[i]>th[k]) { 
        if (f[k]==0) { // ? passing thresh 
          if (jj[k]>=dsz[k]){printf("(%d,%d,%d) :: ",k,jj[k],dsz[k]);
            hoc_execerror("Dest vec too small in updown ", 0); }
          dest[k][jj[k]++] = (i-1) + (th[k]-src[i-1])/(src[i]-src[i-1]); // interpolate
          f[k]=1; 
          tmp[k]=-1e9; dest[k][jj[k]]=-1.; // flag in tmp says that a thresh found here
        }
        if (f[k]==1 && src[i]>tmp[k]) { // use tmp[] even more temporarily
          tmp[k]=src[i]; // pick out max
          dest[k][jj[k]] = (double)i; // location of this peak
        }
      } else {          // below thresh 
        if (f[k]==1) {  // just passed going down 
          jj[k]++;      // triplet will be indices of cross-up/peak/cross-down
          dest[k][jj[k]++] = (i-1) + (src[i-1]-th[k])/(src[i-1]-src[i]);
          f[k]=0; 
        }
      }
    }
  }
  //** truncate dest vectors to multiples of 3:
  for (k=0;k<lc;k++) vector_resize(vvd[k],(int)(floor((double)jj[k]/3.)*3.));
  for (i=0; i<nsrc; i++) tmp[i]=0.; // clear temp space
  //** go through all the slices to find identical peaks and save widths and locations
  // tmp[] uses triplets centered around a location corresponding to a max loc in the
  // original vector; the widest flanks for each are then on either side of this loc
  for (k=0;k<lc;k++) { // need to go from top to bottom to widen flanks
    for (i=1;i<jj[k];i+=3) { // through centers (peaks)
      m=(int)dest[k][i]; // hash: place center at location
      if (tmp[m-2]<0 || tmp[m-1]<0 || tmp[m+1]<0 || tmp[m+2]<0) continue; // ignore; too crowded
      tmp[m]--;  // count how many slices have found this peak (use negative)
      tmp[m-1]=dest[k][i-1]; tmp[m+1]=dest[k][i+1]; // flanks
    }
  }
  //** 1st (of 2) loops through tmp[] -- pick up flanks
  // step through tmp[] looking for negatives which indicate the slice count and pick up 
  // flanks from these
  // nq=new NQS("LOC","PEAK","WIDTH","BASE","HEIGHT","START","SLICES","SHARP","INDEX","FILE")
  for (i=0,k=0; i<nsrc; i++) if (tmp[i]<0.) { // tmp holds neg of count of slices
    if (k>=nqmax) { printf("updown ERRG OOR in NQ db: %d %d\n",k,nqmax); hxf(tmp); }
    LOC[k]=(double)i;  // approx location of the peak of the spike
    WIDTH[k]=tmp[i+1]; // location of right side -- temp storage
    START[k]=tmp[i-1]; // start of spike (left side)
    SLICES[k]=-tmp[i];  // # of slices
    k++;
  }
  nqsz=k;   // k ends up as size of NQS db
  if (DEBUG_UPDOWN && ifarg(4)) { dbn=vector_arg_px(4, &dbx); // DEBUG -- save tmp vector
    if (dbn<nsrc) printf("updown ERRH: Insufficient room in debug vec (%d<%d)\n",dbn,nsrc); 
    else for (i=0;i<nsrc;i++) dbx[i]=tmp[i]; 
  }
  //** adjust flanks to handle nested bumps
  // 3 ways to handle spike nested in a spike or elongated base:
  // NB always using same slice for both L and R flanks; NOV_UPDOWN flag: (no-overlap)
  //   0. nested spike(s) share flanks determined by shared base
  //   1. nested spike(s) have individual bases, 1st and last use flanks from base
  //   2. nested spike(s) have individual bases, base flanks listed separately w/out peak
  // here use 
  // search nq vecs to compare flanks to neighboring centers
  // if flanks overlap the centers on LT or RT side,
  // correct them by going back to original slice loc info (in dest[])
  //*** look at left side -- is this flank to left of center of another bump?
  if (NOV_UPDOWN) for (i=0;i<nqsz;i++) { // iterate through NQS db
    if ((i-1)>0 && START[i] < LOC[i-1]) { // flank is to left of prior center
      if (DEBUG_UPDOWN) printf("LT problem %d %g %g<%g\n",i,LOC[i],START[i],LOC[i-1]);
      for (m=lc-1,done=0;m>=0 && !done;m--) { // m:go from bottom (widest) to top
        for (n=1;n<jj[m] && !done;n+=3) {     // n:through centers
          // pick out lowest slice with this peak LOC whose flank is to RT of prior peak
          if (floor(dest[m][n])==LOC[i] && dest[m][n-1]>LOC[i-1]) {
            // ??[i]=START[i]; // temp storage for L end of this overlap
            // replace both left and right flanks at this level -- #1 above
            START[i]=dest[m][n-1]; WIDTH[i]=dest[m][n+1]; done=1; 
          }
        }
      }
    }
    //*** now look at RT side
    if ((i+1)<nqsz && WIDTH[i]>LOC[i+1]) {
      if (DEBUG_UPDOWN) printf("RT problem %d %g %g>%g\n",i,LOC[i],WIDTH[i],LOC[i+1]);
      for (m=lc-1,done=0;m>=0 && !done;m--) { // m: go from bottom to top
        for (n=1;n<jj[m] && !done;n+=3) {     // n: through centers
          // pick out lowest slice with this peak LOC whose flank is to LT of next peak
          if (floor(dest[m][n])==LOC[i] && dest[m][n+1]<LOC[i+1]) {
            // ??[i]=WIDTH[i]; // end of overlap
            START[i]=dest[m][n-1]; WIDTH[i]=dest[m][n+1]; done=1;
          }
        }
      }        
    }
  }

  //make sure left and right sides of bump occur at local minima
  //shouldn't creeping be before NOV_UPDOWN=1 overlap check???
  //creeping can result only in equal borders btwn two bumps
  //on one side, so it should be ok here...
  if(CREEP_UPDOWN) for(i=0,k=0;i<nsrc;i++) if(tmp[i]<0.){

    //move left side to local minima
    int idx = (int)START[k];
    while(idx >= 1 && src[idx] >= src[idx-1]) idx--;
    START[k] = idx;

    //move right side to local minima
    idx = (int)WIDTH[k];
    while(idx < nsrc-1 && src[idx] >= src[idx+1]) idx++;
    WIDTH[k] = idx;

    k++;
  }

  //** 2nd loop through tmp[] used to fill in the rest of NQS
  // needed to split into 2 loops so that could check for overlaps and correct those
  // before filling in the rest of nq
  for (i=0,k=0; i<nsrc; i++) if (tmp[i]<0.) { // tmp holds neg of count of slices
    // calculate a base voltage lt as interpolated value on left side
    lt=src[(int)floor(START[k])]+(START[k]-floor(START[k]))*\
      (src[(int)floor(START[k]+1.)]-src[(int)floor(START[k])]);
    BASE[k]=lt;         // base voltage
    PEAK[k]=src[i];     // peak voltage
    WIDTH[k] = WIDTH[k] - START[k]; // width = RT_flank-LT_flank
    HEIGHT[k]=PEAK[k]-BASE[k]; // redund measure -- can eliminate
    // measure of sharpness diff of 1st derivs btwn peak and SHM_UPDOWN dist from peak
    // to get 2nd deriv would be normalized by 2*SHM_UPDOWN*tstep
    // ??could take an ave. or max first deriv for certain distance on either side
    SHARP[k]=(src[i]-src[i-(int)SHM_UPDOWN])-(src[i+(int)SHM_UPDOWN]-src[i]);
    INDEX[k]=(double)k;
    k++;
  }
  
  int iNumBumps = k;

  //count # of other bumps nested within each bump
  if(!NOV_UPDOWN){
    for(i=0; i<iNumBumps; i++){
      NESTED[i] = 0;
      int j = 0;
      for(;j<iNumBumps;j++){
        if(i!=j && LOC[j] >= START[i] && LOC[j] <= START[i]+WIDTH[i]){
          NESTED[i]+=1.0;
        }
      }
    }
  } else for(i=0;i<iNumBumps;i++) NESTED[i]=0.0;

  //** finish up
  for (i=0;i<lc2;i++) vector_resize(vnq[i], nqsz);
  if (k!=nqsz) { printf("updown ERRI INT ERR: %d %d\n",k,nqsz); hxf(tmp); }
  free(tmp);
  return jj[0];
}


ENDVERBATIM

:* PROCEDURE install_updown()
PROCEDURE install_updown () {
  if (UPDOWN_INSTALLED==1) {
    printf("$Id: updown.mod,v 1.16 2009/02/16 22:56:52 billl Exp $\n")
  } else {
  UPDOWN_INSTALLED=1
  VERBATIM {
  install_vector_method("updown", updown);
  }
  ENDVERBATIM
  }
}

Dura-Bernal S, Zhou X, Neymotin SA, Przekwas A, Francis JT, Lytton WW (2015) Cortical Spiking Network Interfaced with Virtual Musculoskeletal Arm and Robotic Arm. Front Neurorobot 9:13[PubMed]

References and models cited by this paper

References and models that cite this paper

Almassy N, Edelman GM, Sporns O (1998) Behavioral constraints in the development of neuronal properties: a cortical model embedded in a real-world device. Cereb Cortex 8:346-61 [PubMed]

Alstermark B, Isa T (2012) Circuits for skilled reaching and grasping. Annu Rev Neurosci 35:559-78 [Journal] [PubMed]

Barrett Tech (2012) WAM Training Documentation

Bergenheim M, Ribot-Ciscar E, Roll JP (2000) Proprioceptive population coding of two-dimensional limb movements in humans: I. Muscle spindle feedback during spatially oriented movements. Exp Brain Res 134:301-10 [PubMed]

Berger DJ, d'Avella A (2014) Effective force control by muscle synergies. Front Comput Neurosci 8:46 [Journal] [PubMed]

Carmena JM (2013) Advances in neuroprosthetic learning and control. PLoS Biol 11:e1001561 [Journal] [PubMed]

Carnevale NT, Hines ML (2006) The NEURON Book

Carrillo RR, Ros E, Boucheny C, Coenen OJ (2009) A real-time spiking cerebellum model for learning robot control. Biosystems 94:18-27 [PubMed]

Chadderdon GL, Neymotin SA, Kerr CC, Lytton WW (2012) Reinforcement learning of targeted movement in a spiking neuronal model of motor cortex. PLoS One 7:e47251-57 [PubMed]

Demandt E, Mehring C, Vogt K, Schulze-Bonhage A, Aertsen A, Ball T (2012) Reaching movement onset- and end-related characteristics of EEG spectral power modulations. Front Neurosci 6:65 [Journal] [PubMed]

DeWolf T, Eliasmith C (2011) The neural optimal control hierarchy for motor control. J Neural Eng 8:065009 [Journal] [PubMed]

Dura-Bernal lS,Chadderdon GL,Neymotin SA,Francis JT,Lytton WW (2014) Towards a real-time interface between a biomimetic model of sensorimotor cortex and a robotic arm Pattern Recognition Lett. 36:204-212

Dura-Bernal S,Prins N,Neymotin S,Prasad A,Sanchez J,Francis J,et al (1887) Evaluating hebbian reinforcement learning bmi using an in silico brain model and a virtual musculoskeletal arm Neural Control of Movement

Edelman GM (1987) Neural Darwinism: The Theory of Neuronal Group Selection

Fagg AH, Hatsopoulos NG, de Lafuente V, Moxon KA, Nemati S, Rebesco JM, Romo R, Solla SA, Rei (2007) Biomimetic brain machine interfaces for the control of movement. J Neurosci 27:11842-6 [Journal] [PubMed]

Featherstone R,Orin D (2000) Robot dynamics: Equations and algorithms In ICRA (International Conference Robotics and Automation) :826-834

Flint RD, Lindberg EW, Jordan LR, Miller LE, Slutzky MW (2012) Accurate decoding of reaching movements from field potentials in the absence of spikes. J Neural Eng 9:046006 [Journal] [PubMed]

Francis JT (2009) The neural representation of kinematics and dynamics in multiple brain regions: the use of force field reaching paradigms in the primate and rat Mechanosensitivity of the Nervous System, Mechanosensitivity in Cells and Tissues, Kamkim A:Kiseleva I, ed. pp.215

Hatsopoulos N, Joshi J, O'Leary JG (2004) Decoding continuous and discrete motor behaviors using motor and premotor cortical ensembles. J Neurophysiol 92:1165-74 [Journal] [PubMed]

Hines ML, Carnevale NT (2001) NEURON: a tool for neuroscientists. Neuroscientist 7:123-35 [Journal] [PubMed]

   Spatial gridding and temporal accuracy in NEURON (Hines and Carnevale 2001) [Model]

Hogan N, Sternad D (2009) Sensitivity of smoothness measures to movement duration, amplitude, and arrests. J Mot Behav 41:529-34 [Journal] [PubMed]

Holzbaur KR,Murray WM,Delp SL (2005) A model of the upper extremity for simulating musculoskeletal surgery and analyzing neuromuscular control Ann. Biomed. Eng. 33:829-840

Izhikevich EM (2007) Solving the Distal Reward Problem through Linkage of STDP and Dopamine Signaling. Cereb Cortex 17(10):2443-2452 [Journal] [PubMed]

   Linking STDP and Dopamine action to solve the distal reward problem (Izhikevich 2007) [Model]

Kerr CC, Neymotin SA, Chadderdon GL, Fietkiewicz CT, Francis JT, Lytton WW (2012) Electrostimulation as a prosthesis for repair of information flow in a computer model of neocortex IEEE Transactions on Neural Systems & Rehabilitation Engineering 20(2):153-60 [Journal] [PubMed]

   Prosthetic electrostimulation for information flow repair in a neocortical simulation (Kerr 2012) [Model]

Lee G,Matsunaga A,Dura-Bernal S,Zhang W,Lytton W,Francis J,Et AL (2014) Towards real-time communication between in vivo neurophysiological data sources and simulator-based brain biomimetic models. J Comput Surg

Li K,Dura-Bernal S,Francis J,Lytton W,Principe J (2015) Repairing lesions via kernel adaptive inverse control in a biomimetic model of sensorimotor cortex Neural Engineering (NER), 2015 7th International IEEE/EMBS Conference. (Montpellier)

Luque NR, Garrido JA, Carrillo RR, Coenen OJ, Ros E (2011) Cerebellar input configuration toward object model abstraction in manipulation tasks. IEEE Trans Neural Netw 22:1321-8 [PubMed]

Lytton WW, Neymotin SA, Hines ML (2008) The virtual slice setup. J Neurosci Methods 171:309-15 [Journal] [PubMed]

   The virtual slice setup (Lytton et al. 2008) [Model]

Lytton WW, Omurtag A (2007) Tonic-clonic transitions in computer simulation. J Clin Neurophysiol 24:175-81 [PubMed]

   Tonic-clonic transitions in a seizure simulation (Lytton and Omurtag 2007) [Model]

Lytton WW, Omurtag A, Neymotin SA, Hines ML (2008) Just in time connectivity for large spiking networks Neural Comput 20(11):2745-56 [Journal] [PubMed]

   JitCon: Just in time connectivity for large spiking networks (Lytton et al. 2008) [Model]

Lytton WW, Stewart M (2005) A rule-based firing model for neural networks Int J Bioelectromagn 7:47-50

Lytton WW, Stewart M (2006) Rule-based firing for network simulations. Neurocomputing 69:1160-1164

Mahmoudi B, Pohlmeyer EA, Prins NW, Geng S, Sanchez JC (2013) Towards autonomous neuroprosthetic control using Hebbian reinforcement learning. J Neural Eng 10:066005 [Journal] [PubMed]

Marsh BT, Tarigoppula VS, Chen C, Francis JT (2015) Toward an autonomous brain machine interface: integrating sensorimotor reward modulation and reinforcement learning. J Neurosci 35:7374-87 [Journal] [PubMed]

Neymotin SA, Chadderdon GL, Kerr CC, Francis JT, Lytton WW (2013) Reinforcement learning of 2-joint virtual arm reaching in a computer model of sensorimotor cortex Neural Computation 25(12):3263-93 [Journal] [PubMed]

   Sensorimotor cortex reinforcement learning of 2-joint virtual arm reaching (Neymotin et al. 2013) [Model]

Neymotin SA, Lee H, Park E, Fenton AA, Lytton WW (2011) Emergence of physiological oscillation frequencies in a computer model of neocortex. Front Comput Neurosci 5:19-75 [Journal] [PubMed]

   Emergence of physiological oscillation frequencies in neocortex simulations (Neymotin et al. 2011) [Model]

Prins NW, Sanchez JC, Prasad A (2014) A confidence metric for using neurobiological feedback in actor-critic reinforcement learning based brain-machine interfaces. Front Neurosci 8:111 [Journal] [PubMed]

Roll JP, Albert F, Ribot-Ciscar E, Bergenheim M (2004) "Proprioceptive signature" of cursive writing in humans: a multi-population coding. Exp Brain Res 157:359-68 [Journal] [PubMed]

Sanchez J, Lytton W, Carmena J, Principe J, Fortes J, Barbour R, Francis J (2012) Dynamically repairing and replacing neural networks: using hybrid computational and biological tools. IEEE Pulse 3:57-9 [Journal] [PubMed]

Sanchez J, Tarigoppula A, Choi J, Marsh B, Chhatbar P (2011) Control of a center-out reaching task using a reinforcement learning brain-machine interface Neural Engineering (NER), 2011 5th International IEEE-EMBS Conference on. IEEE :525-528

Sartori M, Gizzi L, Lloyd DG, Farina D (2013) A musculoskeletal model of human locomotion driven by a low dimensional set of impulsive excitation primitives. Front Comput Neurosci 7:79 [Journal] [PubMed]

Schutte LM,Rodgers MM,Zajac F,Glaser RM (1993) Improving the efficacy of electrical stimulation-induced leg cycle ergometry: an analysis based on a dynamic musculoskeletal model Rehabil. Eng. IEEE Trans. 1:109-125

Shadmehr R, Mussa-Ivaldi FA (1994) Adaptive representation of dynamics during learning of a motor task. J Neurosci 14:3208-24 [PubMed]

Song W, Kerr CC, Lytton WW, Francis JT (2013) Cortical plasticity induced by spike-triggered microstimulation in primate somatosensory cortex. PLoS One 8:e57453-18 [PubMed]

Sussillo D, Churchland MM, Kaufman MT, Shenoy KV (2015) A neural network that finds a naturalistic solution for the production of muscle activity. Nat Neurosci 18:1025-33 [Journal] [PubMed]

Teulings HL, Contreras-Vidal JL, Stelmach GE, Adler CH (1997) Parkinsonism reduces coordination of fingers, wrist, and arm in fine motor control. Exp Neurol 146:159-70 [Journal] [PubMed]

Thelen DG, Anderson FC, Delp SL (2003) Generating dynamic simulations of movement using computed muscle control. J Biomech 36:321-8 [PubMed]

Wolpert DM, Diedrichsen J, Flanagan JR (2011) Principles of sensorimotor learning. Nat Rev Neurosci 12:739-51 [Journal] [PubMed]

Zajac FE (1989) Muscle and tendon: properties, models, scaling, and application to biomechanics and motor control. Crit Rev Biomed Eng 17:359-411 [PubMed]

Dura-Bernal S, Li K, Neymotin SA, Francis JT, Principe JC, Lytton WW (2016) Restoring behavior via inverse neurocontroller in a lesioned cortical spiking model driving a virtual arm. Front. Neurosci. Neuroprosthetics 10:28 [Journal]

   Cortical model with reinforcement learning drives realistic virtual arm (Dura-Bernal et al 2015) [Model]

Dura-Bernal S, Neymotin SA, Kerr CC, Sivagnanam S, Majumdar A, Francis JT, Lytton WW (2017) Evolutionary algorithm optimization of biological learning parameters in a biomimetic neuroprosthesis. IBM Journal of Research and Development (Computational Neuroscience special issue) 61(2/3):6:1-6:14 [Journal]

   Motor system model with reinforcement learning drives virtual arm (Dura-Bernal et al 2017) [Model]

(51 refs)

Dura-Bernal S, Li K, Neymotin SA, Francis JT, Principe JC, Lytton WW (2016) Restoring behavior via inverse neurocontroller in a lesioned cortical spiking model driving a virtual arm. Front. Neurosci. Neuroprosthetics 10:28

References and models cited by this paper

References and models that cite this paper

Alstermark B, Isa T (2012) Circuits for skilled reaching and grasping. Annu Rev Neurosci 35:559-78 [Journal] [PubMed]

Arle JE, Shils JL (2008) Motor cortex stimulation for pain and movement disorders. Neurotherapeutics 5:37-49 [Journal] [PubMed]

Bensmaia SJ, Miller LE (2014) Restoring sensorimotor function through intracortical interfaces: progress and looming challenges. Nat Rev Neurosci 15:313-25 [Journal] [PubMed]

Calvin WH (1988) Neural Darwinism. The Theory of Neuronal Group Selection. Gerald M. Edelman. Basic Books, New York, 1987. xxii, 371 pp., illus. $29.95. Science 240:1802 [Journal] [PubMed]

Carandini M (2012) From circuits to behavior: a bridge too far? Nat Neurosci 15:507-9 [PubMed]

Chadderdon GL, Mohan A, Suter BA, Neymotin SA, Kerr CC, Francis JT, Shepherd GM, Lytton WW (2014) Motor cortex microcircuit simulation based on brain activity mapping. Neural Comput 26:1239-62 [Journal] [PubMed]

   Motor cortex microcircuit simulation based on brain activity mapping (Chadderdon et al. 2014) [Model]

Chen B, Zhao S, Zhu P, Príncipe JC (2012) Quantized kernel least mean square algorithm. IEEE Trans Neural Netw Learn Syst 23:22-32 [Journal] [PubMed]

Ching S, Ritt JT (2013) Control strategies for underactuated neural ensembles driven by optogenetic stimulation. Front Neural Circuits 7:54 [Journal] [PubMed]

Choi JS,DiStasio MM,Brockmeier AJ,Francis JT (2012) An electric field model for prediction of somatosensory (s1) cortical field potentials induced by ventral posterior lateral (vpl) thalamic microstimulation Neural Systems and Rehabilitation Engineering, IEEE Transactions on 20:161-169

Clark KL, Armstrong KM, Moore T (2011) Probing neural circuitry and function with electrical microstimulation. Proc Biol Sci 278:1121-30 [Journal] [PubMed]

Douglas RJ, Martin KA (2012) Behavioral architecture of the cortical sheet. Curr Biol 22:R1033-8 [Journal] [PubMed]

Dura-Bernal S, Chadderdon GL, Neymotin SA, Francis JT, Lytton WW (2014) Towards a real-time interface between a biomimetic model of sensorimotor cortex and a robotic arm. Pattern Recognit Lett 36:204-212 [Journal] [PubMed]

Dura-Bernal S, Zhou X, Neymotin SA, Przekwas A, Francis JT, Lytton WW (2015) Cortical Spiking Network Interfaced with Virtual Musculoskeletal Arm and Robotic Arm. Front Neurorobot 9:13 [Journal] [PubMed]

   Cortical model with reinforcement learning drives realistic virtual arm (Dura-Bernal et al 2015) [Model]

Dura-Bernal S,Kerr C,Neymotin S,Suter B,Shepherd G,Francis J,Lytton W (2015) Large-scale m1 microcircuit model with plastic input connections from biological pmd neurons used for prosthetic arm control 24th Annual Computational Neuroscience Meeting (CNS15) BMC Neuroscience

Featherstone R,Orin D (2000) Robot dynamics: Equations and algorithms In ICRA (International Conference Robotics and Automation) :826-834

Grahn PJ, Mallory GW, Berry BM, Hachmann JT, Lobel DA, Lujan JL (2014) Restoration of motor function following spinal cord injury via optimal control of intraspinal microstimulation: toward a next generation closed-loop neural prosthesis. Front Neurosci 8:296 [Journal] [PubMed]

Grammont F, Riehle A (1999) Precise spike synchronization in monkey motor cortex involved in preparation for movement. Exp Brain Res 128:118-22 [PubMed]

Gupta RK, Przekwas A (2013) Mathematical Models of Blast-Induced TBI: Current Status, Challenges, and Prospects. Front Neurol 4:59 [Journal] [PubMed]

Hampson RE, Gerhardt GA, Marmarelis V, Song D, Opris I, Santos L, Berger TW, Deadwyler SA (2012) Facilitation and restoration of cognitive function in primate prefrontal cortex by a neuroprosthesis that utilizes minicolumn-specific neural firing. J Neural Eng 9:056012 [Journal] [PubMed]

Hampson RE, Song D, Opris I, Santos LM, Shin DC, Gerhardt GA, Marmarelis VZ, Berger TW, Deadw (2013) Facilitation of memory encoding in primate hippocampus by a neuroprosthesis that promotes task-specific neural firing. J Neural Eng 10:066013 [Journal] [PubMed]

Harris KD, Shepherd GM (2015) The neocortical circuit: themes and variations. Nat Neurosci 18:170-81 [Journal] [PubMed]

Hartmann CJ, Chaturvedi A, Lujan JL (2015) Quantitative analysis of axonal fiber activation evoked by deep brain stimulation via activation density heat maps. Front Neurosci 9:28 [Journal] [PubMed]

Hatsopoulos N, Joshi J, O'Leary JG (2004) Decoding continuous and discrete motor behaviors using motor and premotor cortical ensembles. J Neurophysiol 92:1165-74 [Journal] [PubMed]

Hiscott R (2014) Darpa: On the hunt for neuroprosthetics to enhance memory Neurology Today

Holzbaur KR,Murray WM,Delp SL (2005) A model of the upper extremity for simulating musculoskeletal surgery and analyzing neuromuscular control Ann. Biomed. Eng. 33:829-840

Hwang EJ, Shadmehr R (2005) Internal models of limb dynamics and the encoding of limb state. J Neural Eng 2:S266-78 [PubMed]

Jackson A, Mavoori J, Fetz EE (2006) Long-term motor cortex plasticity induced by an electronic neural implant. Nature 444:56-60 [Journal] [PubMed]

Jefferson SC, Clayton ER, Donlan NA, Kozlowski DA, Jones TA, Adkins DL (2015) Cortical Stimulation Concurrent With Skilled Motor Training Improves Forelimb Function and Enhances Motor Cortical Reorganization Following Controlled Cortical Impact. Neurorehabil Neural Repair [Journal] [PubMed]

Kasthuri N, Hayworth KJ, Berger DR, Schalek RL, Conchello JA, Knowles-Barley S, Lee D, Vázque (2015) Saturated Reconstruction of a Volume of Neocortex. Cell 162:648-61 [Journal] [PubMed]

Kerr CC, Neymotin SA, Chadderdon GL, Fietkiewicz CT, Francis JT, Lytton WW (2012) Electrostimulation as a prosthesis for repair of information flow in a computer model of neocortex IEEE Transactions on Neural Systems & Rehabilitation Engineering 20(2):153-60 [Journal] [PubMed]

   Prosthetic electrostimulation for information flow repair in a neocortical simulation (Kerr 2012) [Model]

Kerr CC,O`Shea DJ,Goo W,Dura-Bernal S,Francis JT,Diester I,Kalanithi P,Deisseroth K,Shenoy KV (2014) Network-level effects of optogenetic stimulation in a computer model of macaque primary motor cortex BMC Neuroscience 15:p107

Klaes C, Shi Y, Kellis S, Minxha J, Revechkis B, Andersen RA (2014) A cognitive neuroprosthetic that uses cortical stimulation for somatosensory feedback. J Neural Eng 11:056024 [Journal] [PubMed]

Kleim JA, Bruneau R, VandenBerg P, MacDonald E, Mulrooney R, Pocock D (2003) Motor cortex stimulation enhances motor recovery and reduces peri-infarct dysfunction following ischemic insult. Neurol Res 25:789-93 [Journal] [PubMed]

Kocaturk M, Gulcur HO, Canbeyli R (2015) Toward Building Hybrid Biological/in silico Neural Networks for Motor Neuroprosthetic Control. Front Neurorobot 9:8 [Journal] [PubMed]

Koch C, Buice MA (2015) A Biological Imitation Game. Cell 163:277-80 [Journal] [PubMed]

Koralek AC, Jin X, Long JD, Costa RM, Carmena JM (2012) Corticostriatal plasticity is necessary for learning intentional neuroprosthetic skills. Nature 483:331-5 [Journal] [PubMed]

Kreuz T, Chicharro D, Greschner M, Andrzejak RG (2011) Time-resolved and time-scale adaptive measures of spike train synchrony. J Neurosci Methods 195:92-106 [PubMed]

Kreuz T, Mulansky M, Bozanic N (2015) SPIKY: a graphical user interface for monitoring spike train synchrony. J Neurophysiol 113:3432-45 [Journal] [PubMed]

Lee G, Matsunaga A, Dura-Bernal S, Zhang W, Lytton WW, Francis JT, Fortes JA (2014) Towards real-time communication between in vivo neurophysiological data sources and simulator-based brain biomimetic models. J Comput Surg 3:1-23 [Journal] [PubMed]

Li K,Dura-Bernal S,Francis J,Lytton W,Principe J (2015) Repairing lesions via kernel adaptive inverse control in a biomimetic model of sensorimotor cortex Neural Engineering (NER), 2015 7th International IEEE/EMBS Conference. (Montpellier)

Li L, Park IM, Brockmeier A, Chen B, Seth S, Francis JT, Sanchez JC, Príncipe JC (2013) Adaptive inverse control of neural spatiotemporal spike patterns with a reproducing kernel Hilbert space (RKHS) framework. IEEE Trans Neural Syst Rehabil Eng 21:532-43 [Journal] [PubMed]

Ling G (2013) Newsmaker interview: Geoffrey Ling. DARPA aims to rebuild brains. Interview by Emily Underwood. Science 342:1029-30 [Journal] [PubMed]

Liu W,Pokharel P,Principe JC (2008) The kernel least mean square algorithm 56:543-554

Liu W,Principe JC,Haykin S (2010) Kernel Adaptive Filtering: A Comprehensive Introduction

Loeb GE, Tsianos GA (2015) Major remaining gaps in models of sensorimotor systems. Front Comput Neurosci 9:70 [Journal] [PubMed]

Lytton W,Stark J,Yamasaki D,Sober S (1999) Computer models of stroke recovery: Implications for neurorehabilitation. The Neuroscientist 5:100-111

Lytton WW, Neymotin SA, Hines ML (2008) The virtual slice setup. J Neurosci Methods 171:309-15 [Journal] [PubMed]

   The virtual slice setup (Lytton et al. 2008) [Model]

Lytton WW, Omurtag A, Neymotin SA, Hines ML (2008) Just in time connectivity for large spiking networks Neural Comput 20(11):2745-56 [Journal] [PubMed]

   JitCon: Just in time connectivity for large spiking networks (Lytton et al. 2008) [Model]

Lytton WW, Stewart M (2006) Rule-based firing for network simulations. Neurocomputing 69:1160-1164

Mante V, Sussillo D, Shenoy KV, Newsome WT (2013) Context-dependent computation by recurrent dynamics in prefrontal cortex. Nature 503:78-84 [Journal] [PubMed]

Marcus G, Marblestone A, Dean T (2014) Neuroscience. The atoms of neural computation. Science 346:551-2 [PubMed]

Markram H, Muller E, Ramaswamy S, Reimann MW, Abdellah M, Sanchez CA, Ailamaki A, Alonso-Nanclares L, Antille N, Arsever S, Kahou GA, Berger TK, Bilgili A, Buncic N, Chalimourda A, Chindemi G, Courcol JD, Delalondre F, Delattre V, Druckmann S, Dumusc R, Dynes J, Eilemann S, Gal E, Gevaert ME, Ghobril JP, Gidon A, Graham JW, Gupta A, Haenel V, Hay E, Heinis T, Hernando JB, Hines M, Kanari L, Keller D, Kenyon J, Khazen G, Kim Y, King JG, Kisvarday Z, Kumbhar P, Lasserre S, Le Be JV, Magalhães BR, Merchan-Perez A, Meystre J, Morrice BR, Muller J, Muñoz-Cespedes A, et al. (2015) Reconstruction and Simulation of Neocortical Microcircuitry. Cell 163:456-92 [Journal] [PubMed]

   [5 reconstructed morphologies on NeuroMorpho.Org]

McIntyre CC, Mori S, Sherman DL, Thakor NV, Vitek JL (2004) Electric field and stimulating influence generated by deep brain stimulation of the subthalamic nucleus. Clin Neurophysiol 115:589-95 [PubMed]

Miranda RA, Casebeer WD, Hein AM, Judy JW, Krotkov EP, Laabs TL, Manzo JE, Pankratz KG, Pratt (2015) DARPA-funded efforts in the development of novel brain-computer interface technologies. J Neurosci Methods 244:52-67 [Journal] [PubMed]

Nelson JT, Tepe V () Neuromodulation research and application in the U.S. Department of Defense. Brain Stimul 8:247-52 [Journal] [PubMed]

Neymotin SA, Chadderdon GL, Kerr CC, Francis JT, Lytton WW (2013) Reinforcement learning of 2-joint virtual arm reaching in a computer model of sensorimotor cortex Neural Computation 25(12):3263-93 [Journal] [PubMed]

   Sensorimotor cortex reinforcement learning of 2-joint virtual arm reaching (Neymotin et al. 2013) [Model]

Neymotin SA, Lee H, Park E, Fenton AA, Lytton WW (2011) Emergence of physiological oscillation frequencies in a computer model of neocortex. Front Comput Neurosci 5:19-75 [Journal] [PubMed]

   Emergence of physiological oscillation frequencies in neocortex simulations (Neymotin et al. 2011) [Model]

Neymotin SA,Fenton AA,Lytton WW (2015) Tracking recurrence of correlation structure in neuronal ensembles

Nirenberg S, Pandarinath C (2012) Retinal prosthetic strategy with the capacity to restore normal vision. Proc Natl Acad Sci U S A 109:15012-7 [Journal] [PubMed]

Nishimura Y, Perlmutter SI, Fetz EE (2013) Restoration of upper limb movement via artificial corticospinal and musculospinal connections in a monkey with spinal cord injury. Front Neural Circuits 7:57 [Journal] [PubMed]

O'Doherty JE, Lebedev MA, Ifft PJ, Zhuang KZ, Shokur S, Bleuler H, Nicolelis MA (2011) Active tactile exploration using a brain-machine-brain interface. Nature 479:228-31 [Journal] [PubMed]

Overduin SA, d'Avella A, Carmena JM, Bizzi E (2012) Microstimulation activates a handful of muscle synergies. Neuron 76:1071-7 [Journal] [PubMed]

Overstreet CK, Klein JD, Helms Tillery SI (2013) Computational modeling of direct neuronal recruitment during intracortical microstimulation in somatosensory cortex J Neural Eng. 10(6):066016 [Journal]

   Direct recruitment of S1 pyramidal cells and interneurons via ICMS (Overstreet et al., 2013) [Model]

Paiva AR, Park I, Príncipe JC (2009) A reproducing kernel Hilbert space framework for spike train signal processing. Neural Comput 21:424-49 [Journal] [PubMed]

Palop JJ, Mucke L (2010) Amyloid-beta-induced neuronal dysfunction in Alzheimer's disease: from synapses toward neural networks. Nat Neurosci 13:812-8 [PubMed]

Potjans TC, Diesmann M (2014) The cell-type specific cortical microcircuit: relating structure and activity in a full-scale spiking network model. Cereb Cortex 24:785-806 [Journal] [PubMed]

Ramanathan D, Conner JM, Tuszynski MH (2006) A form of motor cortical plasticity that correlates with recovery of function after brain injury. Proc Natl Acad Sci U S A 103:11370-5 [Journal] [PubMed]

Rickgauer JP, Deisseroth K, Tank DW (2014) Simultaneous cellular-resolution optical perturbation and imaging of place cell firing fields. Nat Neurosci 17:1816-24 [Journal] [PubMed]

Riehle A, Grun S, Diesmann M, Aertsen A (1997) Spike synchronization and rate modulation differentially involved in motor cortical function. Science 278:1950-3 [PubMed]

Rowan MS, Neymotin SA, Lytton WW (2014) Electrostimulation to reduce synaptic scaling driven progression of Alzheimer's disease. Front Comput Neurosci 8:39 [Journal] [PubMed]

   Electrostimulation to reduce synaptic scaling driven progression of Alzheimers (Rowan et al. 2014) [Model]

Rubino D, Robbins KA, Hatsopoulos NG (2006) Propagating waves mediate information transfer in the motor cortex. Nat Neurosci 9:1549-57 [Journal] [PubMed]

Sanchez J, Lytton W, Carmena J, Principe J, Fortes J, Barbour R, Francis J (2012) Dynamically repairing and replacing neural networks: using hybrid computational and biological tools. IEEE Pulse 3:57-9 [Journal] [PubMed]

Sanchez J, Tarigoppula A, Choi J, Marsh B, Chhatbar P (2011) Control of a center-out reaching task using a reinforcement learning brain-machine interface Neural Engineering (NER), 2011 5th International IEEE-EMBS Conference on. IEEE :525-528

Scholkopf B, Smola AJ (2001) Learning with kernels: Support vector machines, regularization, optimization, and beyond

Scholkopf B,Herbrich R,Smola AJ (2001) A generalized representer theorem Proc. 14th Annual Conf. on Comput. Learn. Theory 2111:416-426

Schutte LM,Rodgers MM,Zajac F,Glaser RM (1993) Improving the efficacy of electrical stimulation-induced leg cycle ergometry: an analysis based on a dynamic musculoskeletal model Rehabil. Eng. IEEE Trans. 1:109-125

Song W, Kerr CC, Lytton WW, Francis JT (2013) Cortical plasticity induced by spike-triggered microstimulation in primate somatosensory cortex. PLoS One 8:e57453-18 [PubMed]

Spuler M,Nagel S,Rosenstiel W () A Spiking Neuronal Model Learning a Motor Control Task by Reinforcement Learning and Structural Synaptic Plasticity

Stanley GB (2013) Reading and writing the neural code. Nat Neurosci 16:259-63 [Journal] [PubMed]

Sussillo D, Churchland MM, Kaufman MT, Shenoy KV (2015) A neural network that finds a naturalistic solution for the production of muscle activity. Nat Neurosci 18:1025-33 [Journal] [PubMed]

Suter BA, Migliore M, Shepherd GM (2013) Intrinsic electrophysiology of mouse corticospinal neurons: a class-specific triad of spike-related properties. Cereb Cortex 23:1965-77 [PubMed]

Suter BA, Yamawaki N, Borges K, Li X, Kiritani T, Hooks BM, Shepherd GM (2014) Neurophotonics applications to motor cortex research. Neurophotonics [Journal] [PubMed]

Tessadori J, Bisio M, Martinoia S, Chiappalone M (2012) Modular neuronal assemblies embodied in a closed-loop environment: toward future integration of brains and machines. Front Neural Circuits 6:99 [Journal] [PubMed]

Thelen DG, Anderson FC, Delp SL (2003) Generating dynamic simulations of movement using computed muscle control. J Biomech 36:321-8 [PubMed]

Van Acker GM, Amundsen SL, Messamore WG, Zhang HY, Luchies CW, Kovac A, Cheney PD (2013) Effective intracortical microstimulation parameters applied to primary motor cortex for evoking forelimb movements to stable spatial end points. J Neurophysiol 110:1180-9 [Journal] [PubMed]

Warden MR, Cardin JA, Deisseroth K (2014) Optical neural interfaces. Annu Rev Biomed Eng 16:103-29 [Journal] [PubMed]

Wolpert DM, Diedrichsen J, Flanagan JR (2011) Principles of sensorimotor learning. Nat Rev Neurosci 12:739-51 [Journal] [PubMed]

Zajac FE (1989) Muscle and tendon: properties, models, scaling, and application to biomechanics and motor control. Crit Rev Biomed Eng 17:359-411 [PubMed]

(88 refs)