References and models cited by this paper

References and models that cite this paper

Bakker B (2002) Reinforcement learning with long short-term memory Neural information processing systems, Dietterich T:Becker S:Ghahramani Z, ed. pp.1475

Balsam PD, Drew MR, Yang C (2002) Timing at the start of associative learning Learning And Motivation 33:141-155

Barto AG (1995) Adaptive critics and the basal ganglia Models of Information Processing in the Basal Ganglia, Houk JC:Davis JL:Beiser DG, ed. pp.1

Bertin M, Schweighofer N, Doya K (2007) Multiple model-based reinforcement learning explains dopamine neuronal activity. Neural Netw 20:668-75 [Journal] [PubMed]

Beylin AV, Gandhi CC, Wood GE, Talk AC, Matzel LD, Shors TJ (2001) The role of the hippocampus in trace conditioning: temporal discontinuity or task difficulty? Neurobiol Learn Mem 76:447-61 [Journal] [PubMed]

Braver TS, Cohen JD (2000) On the control of control: The role of dopamine in regulating prefrontal function and working memory Control of cognitive processes: Attention and performance XVIII, Monsell S:Driver J, ed. pp.713

Brody CD, Hernández A, Zainos A, Romo R (2003) Timing and neural encoding of somatosensory parametric working memory in macaque prefrontal cortex. Cereb Cortex 13:1196-207 [PubMed]

Brown J, Bullock D, Grossberg S (1999) How the basal ganglia use parallel excitatory and inhibitory learning pathways to selectively respond to unexpected rewarding cues. J Neurosci 19:10502-11 [PubMed]

Buhusi CV, Meck WH (2000) Timing for the absence of a stimulus: the gap paradigm reversed. J Exp Psychol Anim Behav Process 26:305-22 [PubMed]

Buhusi CV, Meck WH (2005) What makes us tick? Functional and neural mechanisms of interval timing. Nat Rev Neurosci 6:755-65 [Journal] [PubMed]

Church RM (2003) A concise introduction to scalar timing theory Functional and neural mechanisms of interval timing, Meck WH, ed. pp.3

Clark RE, Squire LR (1998) Classical conditioning and brain systems: the role of awareness. Science 280:77-81 [PubMed]

Daw ND, Courville AC, Touretzky DS (2003) Timing and partial observability in the dopamine system Neural information processing systems, BeckerS:Thrun S:Obermayer K, ed. pp.83

Daw ND, Courville AC, Tourtezky DS, Touretzky DS (2006) Representation and timing in theories of the dopamine system. Neural Comput 18:1637-77 [Journal] [PubMed]

Daw ND, Doya K (2006) The computational neurobiology of learning and reward. Curr Opin Neurobiol 16:199-204 [Journal] [PubMed]

Dormont JF, Condé H, Farin D (1998) The role of the pedunculopontine tegmental nucleus in relation to conditioned motor performance in the cat. I. Context-dependent and reinforcement-related single unit activity. Exp Brain Res 121:401-10 [PubMed]

Doya K (1999) What are the computations of the cerebellum, the basal ganglia and the cerebral cortex? Neural Netw 12:961-974 [PubMed]

Doya K (2000) Complementary roles of basal ganglia and cerebellum in learning and motor control. Curr Opin Neurobiol 10:732-9 [PubMed]

Dragoi V, Staddon JE, Palmer RG, Buhusi CV (2003) Interval timing as an emergent learning property. Psychol Rev 110:126-44 [PubMed]

Durstewitz D (2004) Neural representation of interval time. Neuroreport 15:745-9 [PubMed]

Eck D, Schmidhuber A (2002) Learning the long-term structure of the blues. Artificial Neural Networks-ICANN 2002:284-289

Fiorillo CD, Newsome WT, Schultz W (2008) The temporal precision of reward prediction in dopamine neurons. Nat Neurosci 121: -10

Fiorillo CD, Tobler PN, Schultz W (2003) Discrete coding of reward probability and uncertainty by dopamine neurons. Science 299:1898-902 [Journal] [PubMed]

Florian RV (2007) Reinforcement learning through modulation of spike-timing-dependent synaptic plasticity. Neural Comput 19:1468-502 [Journal] [PubMed]

Fukuda M, Ono T, Nakamura K, Tamura R (1990) Dopamine and ACh involvement in plastic learning by hypothalamic neurons in rats. Brain Res Bull 25:109-14 [PubMed]

Fukuda M, Ono T, Nishino H, Nakamura K (1986) Neuronal responses in monkey lateral hypothalamus during operant feeding behavior. Brain Res Bull 17:879-83 [PubMed]

Funahashi S, Bruce CJ, Goldman-Rakic PS (1989) Mnemonic coding of visual space in the monkey's dorsolateral prefrontal cortex. J Neurophysiol 61:331-49 [Journal] [PubMed]

Gallistel CR, Gibbon J (2000) Time, rate, and conditioning. Psychol Rev 107:289-344 [PubMed]

Gers FA, Schmidhuber J, Cummins F (2000) Learning to forget: continual prediction with LSTM. Neural Comput 12:2451-71 [PubMed]

Gers FA, Schraudolph N, Schmidhuber J (2002) Learning precise timing with LSTM recurrent networks J Mach Learn Res 3:115-143

Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9:1735-80 [PubMed]

Hollerman JR, Schultz W (1998) Dopamine neurons report an error in the temporal prediction of reward during learning. Nat Neurosci 1:304-9 [Journal] [PubMed]

Hopson JW (2003) Timing without a clock Functional and neural mechanisms of interval timing :23-60

Houk JC, Adams JL, Barto AGA (1995) A model of how the basal ganglia generate and use neural signals that predict reinforcement. Models Of Information Processing In The Basal Ganglia, Houk JC:Davis JL:Beiser DG, ed. pp.249

Ivry RB, Schlerf JE (2008) Dedicated and intrinsic models of time perception. Trends Cogn Sci 12:273-80 [Journal] [PubMed]

Izhikevich EM (2007) Solving the distal reward problem through linkage of STDP and dopamine signaling. Cereb Cortex 17:2443-52 [Journal] [PubMed]

Joel D, Weiner I (2000) The connections of the dopaminergic system with the striatum in rats and primates: an analysis with respect to the functional and compartmental organization of the striatum. Neuroscience 96:451-74 [PubMed]

Karmarkar UR, Buonomano DV (2007) Timing in the absence of clocks: encoding time in neural network states. Neuron 53:427-38 [Journal] [PubMed]

Kolodziejski C, Porr B, Wörgötter F (2009) On the asymptotic equivalence between differential Hebbian and temporal difference learning. Neural Comput 21:1173-202 [Journal] [PubMed]

Komura Y, Tamura R, Uwano T, Nishijo H, Kaga K, Ono T (2001) Retrospective and prospective coding for predicted reward in the sensory thalamus. Nature 412:546-9 [Journal] [PubMed]

Laubach M (2005) Who's on first? What's on second? The time course of learning in corticostriatal systems Trends In Neurosciences 28:508-511

Lebedev MA, O'Doherty JE, Nicolelis MA (2008) Decoding of temporal intervals from cortical ensemble activity. J Neurophysiol 99:166-86 [Journal] [PubMed]

Levitt H (1971) Transformed up-down methods in psychoacoustics. J Acoust Soc Am 49:Suppl 2:467+ [PubMed]

Lewis PA (2002) Finding the timer. Trends Cogn Sci 6:195-196 [PubMed]

Ljungberg T, Apicella P, Schultz W (1992) Responses of monkey dopamine neurons during learning of behavioral reactions. J Neurophysiol 67:145-63 [Journal] [PubMed]

Lucchetti C, Bon L (2001) Time-modulated neuronal activity in the premotor cortex of macaque monkeys. Exp Brain Res 141:254-60 [Journal] [PubMed]

Lucchetti C, Ulrici A, Bon L (2005) Dorsal premotor areas of nonhuman primate: functional flexibility in time domain. Eur J Appl Physiol 95:121-30 [Journal] [PubMed]

Ludvig EA, Sutton RS, Kehoe EJ (2008) Stimulus representation and the timing of reward-prediction errors in models of the dopamine system. Neural Comput 20:3034-54 [Journal] [PubMed]

Ludvig EA, Sutton RS, Verbeek E, Kehoe EJ (2009) A computational model of hippocampal function in trace conditioning Neural Information Processing Systems, Koller D:Schuurmans D:Bengio Y:Bottou L, ed. pp.993

Mirenowicz J, Schultz W (1994) Importance of unpredictability for reward responses in primate dopamine neurons. J Neurophysiol 72:1024-7 [Journal] [PubMed]

Montague PR, Dayan P, Sejnowski TJ (1996) A framework for mesencephalic dopamine systems based on predictive Hebbian learning. J Neurosci 16:1936-47 [PubMed]

Montague PR, Hyman SE, Cohen JD (2004) Computational roles for dopamine in behavioural control. Nature 431:760-7 [Journal] [PubMed]

Morris G, Arkadir D, Nevet A, Vaadia E, Bergman H (2004) Coincident but distinct messages of midbrain dopamine and striatal tonically active neurons. Neuron 43:133-43 [Journal] [PubMed]

Morris G, Nevet A, Arkadir D, Vaadia E, Bergman H (2006) Midbrain dopamine neurons encode decisions for future action. Nat Neurosci 9:1057-63 [Journal] [PubMed]

O'Reilly RC, Frank MJ (2006) Making working memory work: a computational model of learning in the prefrontal cortex and basal ganglia. Neural Comput 18:283-328 [Journal] [PubMed]

Otani S, Daniel H, Roisin MP, Crepel F (2003) Dopaminergic modulation of long-term synaptic plasticity in rat prefrontal neurons. Cereb Cortex 13:1251-6 [PubMed]

Pan WX, Schmidt R, Wickens JR, Hyland BI (2005) Dopamine cells respond to predicted events during classical conditioning: evidence for eligibility traces in the reward-learning network. J Neurosci 25:6235-42 [Journal] [PubMed]

Potjans W, Morrison A, Diesmann M (2009) A spiking neural network model of an actor-critic learning agent. Neural Comput 21:301-39 [Journal] [PubMed]

Reynolds JN, Wickens JR (2002) Dopamine-dependent plasticity of corticostriatal synapses. Neural Netw 15:507-21 [PubMed]

Rivest F, Bengio Y, Kalaska JF (2005) Brain inspired reinforcement learning Neural information processing systems, Saul LK:Weiss Y:Bottou L, ed. pp.1129

Roberts PD, Santiago RA, Lafferriere G (2008) An implementation of reinforcement learning based on spike timing dependent plasticity. Biol Cybern 99:517-23 [Journal] [PubMed]

Roesch MR, Calu DJ, Schoenbaum G (2007) Dopamine neurons encode the better option in rats deciding between differently delayed or sized rewards. Nat Neurosci 10:1615-24 [Journal] [PubMed]

Romo R, Brody CD, Hernández A, Lemus L (1999) Neuronal correlates of parametric working memory in the prefrontal cortex. Nature 399:470-3 [Journal] [PubMed]

Rougier NP, Noelle DC, Braver TS, Cohen JD, O'Reilly RC (2005) Prefrontal cortex and flexible cognitive control: rules without symbols. Proc Natl Acad Sci U S A 102:7338-43 [Journal] [PubMed]

Samejima K, Ueda Y, Doya K, Kimura M (2005) Representation of action-specific reward values in the striatum. Science 310:1337-40 [Journal] [PubMed]

Schultz W, Apicella P, Ljungberg T (1993) Responses of monkey dopamine neurons to reward and conditioned stimuli during successive steps of learning a delayed response task. J Neurosci 13:900-13 [PubMed]

Schultz W, Apicella P, Romo R, Scarnati E (1995) Context-dependent activity in primate striatum reflecting past and future behavioral events Models Of Information Processing In The Basal Ganglia, Houk JC:Davis JL:Beiser DG, ed. pp.11

Schultz W, Apicella P, Scarnati E, Ljungberg T (1992) Neuronal activity in monkey ventral striatum related to the expectation of reward. J Neurosci 12:4595-610 [PubMed]

Schultz W, Dayan P, Montague PR (1997) A neural substrate of prediction and reward. Science 275:1593-9 [PubMed]

Suri RE, Schultz W (1998) Learning of sequential movements by neural network model with dopamine-like reinforcement signal. Exp Brain Res 121:350-4 [PubMed]

Suri RE, Schultz W (1999) A neural network model with dopamine-like reinforcement signal that learns a spatial delayed response task. Neuroscience 91:871-90 [PubMed]

Sutton RS, Barto AG (1998) Reinforcement learning: an introduction [Journal]

Thibaudeau G, Potvin O, Allen K, Doré FY, Goulet S (2007) Dorsal, ventral, and complete excitotoxic lesions of the hippocampus in rats failed to impair appetitive trace conditioning. Behav Brain Res 185:9-20 [Journal] [PubMed]

Todd MT, Niv Y, Cohen JD (2009) Learning to use working memory in partially observable environments through dopaminergic reinforcement Neural information processing systems, Koller D:Schuurmans D:Bengio Y:Bottou L, ed. pp.1689

Wickens J, Kotter R (1995) Cellular models of reinforcement Models ofinformation processing in the basal ganglia, Houk JC:Davis JL:Beiser DG, ed.