Reward modulated STDP (Legenstein et al. 2008)

 Download zip file 
Help downloading and running models
Accession:116837
"... This article provides tools for an analytic treatment of reward-modulated STDP, which allows us to predict under which conditions reward-modulated STDP will achieve a desired learning effect. These analytical results imply that neurons can learn through reward-modulated STDP to classify not only spatial but also temporal firing patterns of presynaptic neurons. They also can learn to respond to specific presynaptic firing patterns with particular spike patterns. Finally, the resulting learning theory predicts that even difficult credit-assignment problems, where it is very hard to tell which synaptic weights should be modified in order to increase the global reward for the system, can be solved in a self-organizing manner through reward-modulated STDP. This yields an explanation for a fundamental experimental result on biofeedback in monkeys by Fetz and Baker. In this experiment monkeys were rewarded for increasing the firing rate of a particular neuron in the cortex and were able to solve this extremely difficult credit assignment problem. ... In addition our model demonstrates that reward-modulated STDP can be applied to all synapses in a large recurrent neural network without endangering the stability of the network dynamics."
Reference:
1 . Legenstein R, Pecevski D, Maass W (2008) A learning theory for reward-modulated spike-timing-dependent plasticity with application to biofeedback. PLoS Comput Biol 4:e1000180 [PubMed]
Citations  Citation Browser
Model Information (Click on a link to find other models with that property)
Model Type: Realistic Network;
Brain Region(s)/Organism: Neocortex;
Cell Type(s):
Channel(s):
Gap Junctions:
Receptor(s):
Gene(s):
Transmitter(s):
Simulation Environment: Python; PCSIM;
Model Concept(s): Pattern Recognition; Spatio-temporal Activity Patterns; Reinforcement Learning; STDP; Biofeedback; Reward-modulated STDP;
Implementer(s): 
#
# 
#   Dejan Pecevski, dejan@igi.tugraz.at, 
#   November, 2008
#
#

This model contains the scripts in Python and other necessary files to
reproduce the results reported in:

    Legenstein R, Pecevski D, Maass W 2008 A Learning Theory
    for Reward-Modulated Spike-Timing-Dependent Plasticity with 
    Application to Biofeedback. PLoS Computational Biology 4(10): e1000180, Oct, 2008 
    doi:10.1371/journal.pcbi.1000180
    
To perform the simulations and produce the figures you need to:

1. Install the Parallel Circuit SIMulator - PCSIM:
    See the instructions on http://www.igi.tugraz.at/pcsim on how to do that.
    Checkout the newest revision from the repository. 

2. Set the RMSTDP_HOME environment variable to the directory where
   this README file resides.

3. Install additional python packages for scientific computing:
    numpy 1.1.1 
    scipy 0.6.0
    matplotlib 0.98.3      
    pygsl 1.20
    mpi4py 0.6.0
    pytables 2.0.4
    ipython 0.9.1

    and all dependent packages from these.

4. You need to compile a pcsim extension module used in the
   simulations.
   To do this:

   - Goto the subdirectory "packages/reward_gen".

   - Edit the line 5 in module_recipe.cmake

     SET( PCSIM_SOURCE_DIR "$ENV{HOME}/pcsim" )

     so that PCSIM_SOURCE_DIR variable is set to the location of your
     installation of PCSIM.
     The default already set value is ${HOME}/pcsim.

   - Execute:

   	    python pcsim_extension.py build

5. Now you are ready to go. Each directory contains the files for one
   simulation

   from 1 to 5, as they are enumerated in the paper, and also
   additional simulations
   reported in the supplementary figures. 

   In each directory there is a README file explaining how to run the
   scripts in the directory and which figures are produced from the
   scripts.