Information, Entropy and Self-organization

I follow relentlessly a curation of complex system and complexity science at the excellent marketing platform that is It is called Complexity Digest and it is a treasure trove of resources in those subjects, posting on an almost daily basis research papers, conference announcements, videos, courses and other resources.

I found there this  paper (essay) published open-access in the journal Entropy called Information and Self-Organization. The paper title and its content is appropriate to review and share here in this Blog. It is at the heart and matter of The Information Age (in conjunction with Artificial Intelligence and Data Science topics).

This an editorial essay listing and outlining a collection of papers published within a Special Issue edition of Entropy. It makes for a nice read and explains how modern Information Theory or Complex Systems subjects intersects and works in interdisciplinary fashion with a diverse range of disciplines ranging from Physics, Chemistry, Neurosciences and Cognitive Sciences, Economics and Public Policy, Evolutionary Game Theory, Psychology, Social Organization and urbanization studies, etc. It is intriguing how such apparently completely disparate subjects could in some way do work interdisciplinary, but this is true only within a mindset properly set up by Information Theory and Complex Systems research.

I will highlight some of the main passages from the essay. Later there will be some concluding remarks.

9783662078938Information and Self-Organization – A Macroscopic Approach to Complex Systems

Information and Self-Organization


The process of “self-organization” takes place in open and complex systems that acquire spatio-temporal or functional structures without specific ordering instructions from the outside. In domains such as physics, chemistry or biology, the phrase, “far from equilibrium”, refers to systems that are “far from thermal equilibrium”, while in other disciplines, the term refers to the property of being “away from the resting state”. Such systems are “complex” in the sense that they are composed of many interacting components, parts, elements, etc., and “open” in the sense that they exchange with their environment matter, energy, and information. Here, “information” may imply Shannon information [1], as a measure of the capacity of a channel through which a message passes, pragmatic information, as the impact of a message on recipients, or semantic information, as the meaning conveyed by a message.


An attempt to bring these lines of thought together was made by Hermann Haken in his 1988 book Information and Self-Organization [2]. In the meantime, a number of authors have studied the interplay between information and self-organization in a variety of fields. Though the selection of the relevant authors and topics is surely not complete, we believe that this special issue mirrors the state of these interdisciplinary approaches fairly well. In fact, the various papers of this Special Issue expose the different ways processes of self-organization are linked with the various forms of information. As will be seen below, a study of such links has consequences on a number of research domains, ranging from physics and chemistry, through the life sciences and cognitive science, including human behavior and action, to our understanding of society, economics, and the dynamics of cities and urbanization.


Atmanspacher responds to the above questions by reference to Grassberger’s and Atlan’s views that the concept of complexity is essentially subjective and can only be defined by reference to an observer. From this epistemological view of complexity follows the challenge “to understand the relation between the complexity of a pattern or a symbol sequence and the meaning” it conveys to the observer (or put alternatively, the meaning a subjective observer extracts from the pattern or sequence). Atmanspacher suggests meeting this challenge by reference to Shannon’s purely syntactic information and its required extension by Weaver into semantic and pragmatic forms of information—in particular into pragmatic information as elaborated by Weizsäcker [4]. The rational: “If the meaning of some input into a system is understood, then it triggers action and changes the structure or behavior of the system.” From this view follows an intimate connection between complexity and meaning mediated as it is by the meaningful pragmatic information. Atmanspacher closes the article by a suggestion that although the notion of meaning is “usually employed in the study of cognitive systems, also physical systems allow (though not require) a description in terms of pragmatic information.” He illustrates this claim by reference to the case study of laser systems far from thermal equilibrium.
Information is a confusing notion: on the one hand, it is commonly used in everyday language as well as in scientific discussions, on the other, as Rainer Feistel and Werner Ebeling write in their contribution [5], “to the present day, the information specialists … have not agreed yet on a generally accepted, comprehensive and rigorous definition of what ‘information’ actually is”. As a consequence, studies on information—from Shannon’s seminal theory [1] and onward—tend to start with what their authors define as information. Feistel and Ebeling’s contribution is of no exception and starts by introducing a distinction between two new notions of information—structural information, “associated with arbitrary physical processes or structures” and symbolic information, the self-organized emergent property of socio-culturally agreed conventions. As an example, they refer to the evolution of human spoken language in which structural information in the form of “sound waves produced involuntarily by human babies…” are being transformed, by what they describe as ritualization, into the conventions of symbolic information of spoken languages. By “ritualization” they refer to a process of self-organization that gives rise to the emergence of symbolic information.
In their paper, Haken and Portugali [7] stress a prerequisite for self-organization: the spontaneous formation of structures and functions by a system requires an exchange of energy, information and/or matter with its surrounding. In other words, the system must be open. This requirement holds both for the animate world (e.g., a flower) and the inanimate world (e.g., a fluid forming movement patterns). With respect to information, Haken and Portugali discuss the concepts of Shannon information, pragmatic information and semantic information as well as their interrelation. The latter is highlighted by the new concept of information adaptation, according to which the various forms of information condition each other [8]. To simultaneously deal with information and self-organization, Haken and Portugali base their approach on Jaynes’ maximum (information) entropy principle and establish connections with basic concepts of synergetics, i.e., order parameters and the slaving principle. Order parameters are both macroscopic descriptors of a system and determine the behavior of the numerous individual elements by means of the “slaving principle”. This implies a considerable reduction of complexity. In this approach, order parameters move in an “attractor landscape”. Since information is processed by the human brain, it appears natural to include neuronal self-organization and its perceptual correlates in the Haken and Portugali approach. The best studied process is surely visual perception, i.e., pattern recognition. This allows the authors to exemplify the concepts of order parameters, and the slaving principle. As a byproduct, the equations of the Synergetic Computer for pattern recognition are derived in a new way. The authors deal also with the invariance problem: how can we (or an advanced computer) recognize objects irrespective of their position in space, illumination etc.? Using their concept of “quasi-attractors”, the authors deal with the recognition of ambivalent figures, hybrid images and scenes. A treatment of saccadic eye movements is sketched. Their paper closes with a discussion of exploratory behavior of rats, some applications to urbanism (“synchronisation urge”) and hints at ties to consciousness research.
Emergence, self-organization, and complexity are among the basic properties of complex systems. In several previous studies Gershenson and Fernandez have proposed measures of emergence, self-organization, and complexity based on Shannon’s information theory. They interpreted Shannon’s information as a measure of emergence, the inverse of Shannon’s information as a measure of self-organization, while the balance between the two is interpreted as a measure of complexity. They developed these measures using discrete Shannon information with disadvantages such that in the continuous domain, Shannon’s information entropy is “a proxy of the average uncertainty for a probability distribution with a given parameter set, rather than a proxy of the system’s average uncertainty”.
Thompson and Quian [13] base their axiomatic approach to stochastic non-equilibrium thermodynamics on abstract mathematical concepts. In particular, they assume the existence of a well-defined stationary probability distribution on a countable state space and illustrate their procedure concerning criticality by several examples. Central to their approach is the relation between a potential H(x) of a state x and the stationary probability peq(x):



The authors relate this equation to Boltzmann’s distribution function of systems in thermal equilibrium, his entropy formula, and eventually peq to a distribution function using the definition of free energy. Based on their foregoing work, Graham and Haken [14] have shown that—under rather general conditions—the relation (1) holds also for systems far from thermal equilibrium and can be applied, e.g., to non-equilibrium phase transitions. Relations of the type (1) can also be derived by means of Jaynes’ maximum (information) entropy principle, that is utilized in the contribution by Haken and Portugali (this issue). The relation (1) is basic to Friston’s [15] free energy principle for biological systems.

From our point of view, a little caveat should be observed with respect to calling H(x) “energy”. Actually, in nonequilibrium systems, peq(x) is determined by rate constants, whereas in equilibrium states, constants determining energies are involved.
Finally and concluding the list of contributions to the Special Issue of Entropy on Complex Systems:
Tsuda, Yamaguti, and Watanabe [16] deal with the development of the human brain as self-organizing process leading to functional differentiation of neurons and cortical modules. To this end they present numerical treatments of specific models such as one-dimensional maps, e.g., a time-discrete version of the well-known Kuramoto model. The general frame is provided by deterministic dynamical systems subjected to constraints as previously applied to neural systems by Tsuda and co-authors. In its development, the structural and functional differentiation of an individual brain is promoted in particular by the intentions and actions of surrounding people, which according to Tsuda et al. become constraints of the self-organization of neural dynamics. The authors derive and discuss also mutual information as well as transfer entropies. Their results imply a hierarchy between two modules 1 and 2 in accordance with synergetics where slaving modes of module 1 behave cooperatively forming few order parameters, whereas slaved modes of module 2 show more varieties of interaction. The authors conclude that a developed system may express a conscious mind for module 1 and a more unconscious mind for module 2. In our opinion, this fits nicely—at least in principle, with Leopold’s experimental finding [17], on vision.



The broad and interesting list of contributing papers to this beginning of 2017 Special Issue of Entropy is highly recommended. I for my self may read some of the papers in the list, and maybe will further review them here for this Blog. I hope sincerely that this to be also  to the enjoyment and utility of all the readers and followers of this Blog.

featured image: Information and Self-organization by Hermann Haken


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s