Time and the brain: how our perception makes us time travel ⏳

By Laurent Perrinet, CNRS researcher in Computational Neuroscience, Aix-Marseille University (AMU)

When we observe an hourglass, when we fix our gaze on the grains of sand falling, we have the feeling that time is flowing continuously. We think this has been the case since the birth of the world, and that nothing can contradict this universal truth.


Yet, our sensory perceptions and the neurons that give rise to them have quite a different way of pacing time. A subjective and sensual one, in the literal sense of the term.

The example of vision

To better tell you about this "sensory time," I'll use the example of vision. It works immediately and effortlessly, with rapid, efficient, automatic learning. No manual is needed to learn how to see! But in reality, the visual system has to overcome many difficulties to achieve this efficiency. Difficulties become apparent, for example, when working on computer vision systems, such as making a smartphone smarter or future cars autonomous.

Let's take, for instance, what's called the flash lag illusion. Look at this scene: the red dot scrolls across the screen, and you will then see a green flash appear when the red dot is vertically aligned with the green point.


Most of you will perceive that the position of the red dot is shifted to the right compared to the flash, in the direction of the movement. Look again: similarly, the flash is perceived delayed relative to the moving point. Yet, if you watch the video in slow motion, you'll see that the physical reality is different. This simple setup illustrates that, instead of being synchronized, visual objects can be perceived at different subjective moments, "traveling" through the time of our senses.

Amazingly, this illusion is universal, and it is somehow embedded in our senses. Where does this "sensory time" come from? Does it have the same linear and continuous form that is generally attributed to time? And how does its definition shed light on the mysteries of the brain?

The brain, a black interior

Some imagine vision as producing an "internal luminous screen." But in reality, apart from the light that reaches the retina, the outgrowth of our brain lining the back of the eye, there is no light in the brain. Complete darkness. Safely encased in the hermetically sealed space of the skull, the brain is shielded from any direct contact with the outside world. Inside, its roughly 10 billion neurons form large networks, organized over multiple scales—from simple neuron populations to the networks of brain areas.

We know that all information flows there through electrochemical signals distributed across the membranes of nerve cells. These signals are constantly exchanged from one neuron to another, within each network, thanks to numerous synapses. These signals, and only these, are what give you, at this very moment, simultaneous access to your senses, thoughts, and actions. The next step is to discover how this network can organize itself over time, and how information flows are coordinated and synchronized.

On the timescale of perception, there is no centralized clock in the brain that provides synchronous beats to the different parts, like a conductor. Once again, the brain cannot be compared to a computer. The evidence is clear: like a jazz group improvising on a single theme, this capability is inherent within the brain and emerges from distributed and self-organized interactions. But what are the processes at play?

Inevitable transmission delays

Let's return to the anatomy of the visual system. Due to the physiology of nerve cells, the speed of information transmission in our brain varies along different transmission pathways, reaching up to 62 mph (100 km/h) for the fastest.

Given the volume of the cranial cavity, this inevitably results in transmission delays: for instance, an image that illuminates the retina excites the primary visual cortex only about 50 milliseconds later. There, the visual information is transformed and distributed to other brain areas, which requires an additional 50 milliseconds. Finally, the transmitted information might generate muscle activity and, for example, induce an eye saccade after a total time of about 150 milliseconds.


Let's try to visualize these propagation delays with a simple task. You hold a ball in your right hand, and you watch it fall into your left hand from a height of 4 inches (10 cm): its fall takes about 150 milliseconds. Knowing that the image is delayed by 50 to 100 milliseconds in your visual cortex, this means that when your left hand has caught the ball, the image of the ball that the cortex receives is still in the middle of its trajectory!

In other words, just like stars whose light reaches us only after traveling for years, it is a past image of the ball that reaches our visual cortex. For the brain, this is a real problem. Since decisions and actions take time, in order to close your hand at the right moment to catch the ball, the decision must be made in advance. The future action, as formed in the present, must therefore be constructed from the past... Complicated, isn't it?

A temporal puzzle

Here we face a real temporal puzzle. On one side, absolute, external time is inaccessible to the neurons involved in catching the ball, except for the sensory neurons. On the other, subjective, internal time relies on the proper functioning of the brain and on the synchronization of past, present, and future information. This scientific puzzle seems too complex to solve...

Let's take a step back: in general, physical systems transform by exchanging energy and matter with their environment. In any system, according to the second law of thermodynamics, disorder, as measured by entropy, must increase. That's why there's an asymmetry in the flow of time, what we call the arrow of time. As a result, if someone films a game of billiards and then plays the movie backward, we find it odd. Yet, among all physical systems, there's a class that has acquired the ability to reverse this arrow of time: living organisms.


Compared to billiard balls, and more generally to passive systems, living systems are those whose organization enables them to preserve their structure for as long as possible. Such a system can thus resist the constant flow of randomness imposed by the arrow of time by interacting with its environment through various processes. Since these processes counteract randomness, they are called predictive. In theory, they can combine and interact across different scales of space and time: natural selection for a species, learning for an individual, or simple prediction, as is relevant here.

Let's return to our visual illusion. We explained it by the transmission delays of information, on the order of 50 to 100 milliseconds, in the visual system. The perceptual system would thus do its best to compensate for this systematic delay and predict the trajectory of the observed elements. And the image of a moving point would, in this way, be projected ahead of its physical position.

The necessary manipulation of information

Ultimately, our visual system doesn't just interpret the image transmitted by the retina but tries to bring it closer to what it believes it perceives at the present moment: knowing the transmission delays and the speed of the moving point, it "manipulates" its position along its trajectory, thus "advancing" the red point to its current position. However, one problem remains: how can we explain that the green point, at the moment of the flash, isn't similarly shifted in time? In other words, where does the sensory processing differential between the moving point and the light flash come from?

As we just mentioned, our visual system has several predictive systems based on information acquired from experience. Our brain can indeed learn that an object is likely to follow a coherent trajectory (as with the ball or the moving point), or that the nose is in the middle of the face, that natural light generally comes from above, etc.

The idea of such a predictive brain armed with “a priori” knowledge about the structure of the world seems bold and appealing. But can we formalize it, turn it into a mathematical theory, and thus provide a unified conceptual framework for how the brain works? The answer is yes, if we believe the research of the British scientist Karl Friston.

According to this neuroscientist, a theory of a predictive brain fits within the broader theoretical framework of "free energy minimization." This, according to its author, is a “mathematical formulation of how biological agents resist the natural tendency toward disorder" and "maintain their state in a changing environment." To do this, they must minimize entropy, and thus "the long-term average of surprise...", which amounts to minimizing free energy.

Sorted information for better prediction

In short, it's about a quantity of information that measures a system's degree of surprise, which can be measured simply in bits, much like the size of a computer file. And within this new theoretical framework, every behavior (action, perception, learning...) can be described as the minimization of the amount of surprise, in order to determine the best direction to counter the randomness of time. More importantly, the principle of free energy minimization allows us to describe and predict phenomena that were previously difficult to explain, both for animal and human behavior and for brain function.

However, there was no explicit model of sensory time. This is what we've attempted to sketch out here. Such sensory time helps explain the flash lag illusion by differentiating between the predictable moving point and the flash, whose timing is unpredictable.

But this model is also useful to the brain for predicting more complex trajectories: for instance, a ball whose reappearance is anticipated after passing behind a wall and briefly becoming invisible. With this model, which possesses a hierarchical processing similar to that found among different brain areas, we open up to other facets of time representation in the brain, or even to abnormal forms, such as those observed in schizophrenics.