Motor system model with reinforcement learning drives virtual arm (Dura-Bernal et al 2017)

 Download zip file   Auto-launch 
Help downloading and running models
Accession:194897
"We implemented a model of the motor system with the following components: dorsal premotor cortex (PMd), primary motor cortex (M1), spinal cord and musculoskeletal arm (Figure 1). PMd modulated M1 to select the target to reach, M1 excited the descending spinal cord neurons that drove the arm muscles, and received arm proprioceptive feedback (information about the arm position) via the ascending spinal cord neurons. The large-scale model of M1 consisted of 6,208 spiking Izhikevich model neurons [37] of four types: regular-firing and bursting pyramidal neurons, and fast-spiking and low-threshold-spiking interneurons. These were distributed across cortical layers 2/3, 5A, 5B and 6, with cell properties, proportions, locations, connectivity, weights and delays drawn primarily from mammalian experimental data [38], [39], and described in detail in previous work [29]. The network included 486,491 connections, with synapses modeling properties of four different receptors ..."
Reference:
1 . Dura-Bernal S, Neymotin SA, Kerr CC, Sivagnanam S, Majumdar A, Francis JT, Lytton WW (2017) Evolutionary algorithm optimization of biological learning parameters in a biomimetic neuroprosthesis. IBM Journal of Research and Development (Computational Neuroscience special issue) 61(2/3):6:1-6:14
Citations  Citation Browser
Model Information (Click on a link to find other models with that property)
Model Type: Realistic Network;
Brain Region(s)/Organism:
Cell Type(s): Abstract Izhikevich neuron;
Channel(s):
Gap Junctions:
Receptor(s): GabaA; GabaB; NMDA; AMPA;
Gene(s):
Transmitter(s): Glutamate; Gaba;
Simulation Environment: NEURON; Python;
Model Concept(s): Learning; Reinforcement Learning; Reward-modulated STDP; STDP; Motor control; Sensory processing;
Implementer(s): Dura-Bernal, Salvador [salvadordura at gmail.com]; Kerr, Cliff [cliffk at neurosim.downstate.edu];
Search NeuronDB for information about:  GabaA; GabaB; AMPA; NMDA; Gaba; Glutamate;
import pylab
import pickle
import os


def getTarget0err(f):
  stdin,stdout = os.popen2("tail -n 4 "+f)
  stdin.close()
  lines = stdout.readlines(); stdout.close()
  line = lines[0].split()
  return float(line[5])

def errorFromShell(filestem, readTarget0ErrFromRunFile, maxGens, maxCands, maxTargets):

    minVal = 1
    minValtarg0 = 1
    for igen in range(maxGens):
            for icand in range(maxCands):
                    error = []
                    for itarget in range(maxTargets):
                            try:
                                    errfilename = '%s/gen_%s_cand_%d_target_%d_error' % (filestem,igen,icand,itarget)
                                    runfilename = '%s/gen_%s_cand_%d.run' % (filestem,igen,icand)
                                    with open(errfilename, 'r') as f:
                                            error.append(pickle.load(f))
                                            print 'gen=%d, cand=%d, target=%d: error = %f' % (igen, icand, itarget, error[-1])
                            except:
                                    print('not found file: stem=%s, gen=%d, cand=%d, targ=%d' % (filestem,igen,icand,itarget))
                    avgErr = pylab.mean(error)
                    if readTarget0ErrFromRunFile:
                            try:
                                    target0err = getTarget0err(runfilename)
                                    print "error target 0: %.2f \n" % (target0err)  
                                    if target0err < minValtarg0:
                                            minValtarg0 = target0err
                                            minCandtarg0 = icand
                                            minGentarg0 = igen     
                            except:       
                                    print "error reading file:", runfilename        

                    print "avg error: %.2f \n" % (avgErr)
                    if avgErr < minVal:
                            minVal = avgErr
                            minCand = icand
                            minGen = igen

    print "Min error = %f ; gen = %d ; cand = %d \n" % (minVal, minGen, minCand)
    if readTarget0ErrFromRunFile: 
            print "Min error (target 0) = %f ; gen = %d ; cand = %d \n" % (minValtarg0, minGentarg0, minCandtarg0)

def errorFromPickle(filestem, maxGens, maxCands):
    from collections import OrderedDict
    minVals = OrderedDict
    minVals = {'error0_pre': [1,0,0], 'error1_pre': [1,0,0], 'error_pre': [1,0,0], 'errord_pre': [1,0,0], \
        'error0_post': [1,0,0], 'error1_post': [1,0,0], 'error_post': [1,0,0], 'errord_post': [1,0,0], \
        'error0_lesion': [1,0,0], 'error1_lesion': [1,0,0], 'error_pre': [1,0,0], 'errord_tot': [1,0,0], 'error_fitness': [1,0,0]}
    
    for igen in range(maxGens):
        for icand in range(maxCands):
            error = []
            try:
                errfilename = '%s/gen_%d_cand_%d_target_0_error' % (filestem,igen,icand)
                with open(errfilename, 'r') as f:
                        error = pickle.load(f)
                        #print 'gen=%d, cand=%d, errord_tot = %f, error_fitness = %f' % (igen, icand, error['errord_tot'], error['error_fitness'])
                for key,val in error.iteritems():
                    if val < minVals[key][0]: 
                        minVals[key][0] = val
                        minVals[key][1] = igen
                        minVals[key][2] = icand
            except:
                    print('not found file: stem=%s, gen=%d, cand=%d' % (filestem,igen,icand))

    for key,val in minVals.iteritems():
        print "%s = %f ; gen = %d ; cand = %d \n" % (key, val[0], val[1], val[2])
   
   

# set params
filestem = '../data/15aug29_evolutionStrategy'
maxGens = 540
maxCands = 40
maxTargets = 1
readTarget0ErrFromRunFile = 1

# call function to obtain errors from shell output
#errorFromShell(filestem, readTarget0ErrFromRunFile, maxGens, maxCands, maxTargets)

# call function to obtain errors from pickle file
errorFromPickle(filestem, maxGens, maxCands)