Abstract
We propose a general framework to simulate stochastic trajectories with arbitrarily long memory dependence and efficiently evaluate large deviation functions associated to time-extensive observables. This extends the 'cloning' procedure of Giardiná et al (2006 Phys. Rev. Lett. 96 120603) to non-Markovian systems. We demonstrate the validity of this method by testing non-Markovian variants of an ion-channel model and the totally asymmetric exclusion process, recovering results obtainable by other means.
Export citation and abstract BibTeX RIS
1. Introduction
The theory of large deviations has been widely applied to equilibrium statistical mechanics [1]. Far from equilibrium, we can still deploy the same formalism, but targeting trajectories in space–time rather than static configurations [2]. Instead of asking for the probability of observing a configuration with a given energy, we require the probability of a trajectory of duration T with a value J of a time-extensive observable. For this purpose, the details of the time evolution take on a major role.
When the stochastic dynamics of a model system are Markovian, i.e., memoryless, we can specify the rules for its evolution in time by means of the constant rates of transitions from configuration xi to configuration xj. The full set of rates encodes inter-event times with exponential waiting-time distributions, which indeed possess the memoryless property. However, to model real-world systems, such a simplified description may not be appropriate. In fact, non-exponential waiting times seem to be relevant in many contexts, see, e.g., [3–7].
Large deviation functionals have been computed in selected non-Markovian systems (e.g., assuming the so-called temporal additivity principle [8], or by defining hidden variables [9]) and have revealed structure hidden in the stationary state. Analytical progress is difficult and simulations are necessary to explore systematically more realistic models with memory. However, to our knowledge, numerical schemes able to efficiently probe large deviation functionals have been discussed in general only for memoryless systems [10–14]. In this letter, we fill this literature gap and provide a general numerical method to generate trajectories corresponding to arbitrarily rare values of J, based on the 'cloning' procedure of [10]. The text is organised as follows. In section 2, we set up the general formalism, while in section 3 we present the simulation scheme for non-Markovian systems and the numerical method to evaluate large deviation functionals. Section 4 deals with the special case of the semi-Markov process, where the formalism has a particularly lucid interpretation in terms of the generalised master equation (GME). In section 5 we test the method in some examples of increasing complexity, where the large deviation functions can be computed exactly. We conclude with a discussion in section 6.
2. Thermodynamics of trajectories
A trajectory or history of a stochastic process starting at time t0 and ending at time t, on a configuration space , is defined as the sequence
where is the initial configuration, , and ti (for ) denotes the instant where the system jumps from configuration to xi. Specifically, we are interested in the probability density that a trajectory is observed. Hereafter, we will consider a discrete configuration space , although most of the arguments presented remain valid in the continuous case. In general, we can separate into a product over the contributions of each jump, multiplied by the probability that the configuration at t0 is x0, i.e.
where the generic factor is the waiting-time probability density (WTD) that the transition from to xn occurs during the infinitesimal interval given the history w(tn−1); it obeys the normalisation . Also, is the survival probability of the configuration xn for the interval . The special case without dependence on the history before the last jump will be considered in section 4. The probability that the system has configuration at is
We characterise a non-equilibrium system by means of time-extensive functionals of the trajectory , that can be written as the sum of elementary contributions corresponding to configuration changes. Alternatively it is possible to consider 'static' contributions . Functionals of these types may represent physical observables integrated over the observation time T = t − t0, e.g., the dynamical activity in a glassy system [15], the moles of metabolites produced in a biochemical pathway [16, 17], the customers served in a queuing network [18], the flow in an interacting particle system [19], or certain quantities in stochastic thermodynamics [20, 21]. Hereafter, we will refer to as the time-integrated current and to its empirical average simply as current.
The latter observable still fluctuates in time, although it is doomed to converge to its ensemble average in the limit (with t0 fixed and finite). The consequent computational difficulty in obtaining the probability of having a given value of , can be alleviated by introducing the 'canonical' density , the 'partition function'
and the scaled cumulant generating function (SCGF)
where s is an intensive conjugate field. Assuming a large deviation principle, i.e.
the rate function can be obtained by a Legendre–Fenchel transform
when the SCGF is differentiable [22]. Equation (4) can be written as
where
can be seen as the 'biased' WTD of a stochastic dynamics that does not conserve total probability. This immediately suggests that we can access the SCGF by computing the exponential rate of divergence of an ensemble of trajectories.
3. A numerical approach
3.1. Non-Markovian stochastic simulation
The WTD can be expressed in terms of a time-dependent rate or hazard , which is the probability density that there is a jump from to xn in , conditioned on having no transitions during the interval ,
For brevity we define , which is the value of the age, i.e. the time elapsed since the last jump, at which the next jump takes place (see figure 1). Roughly speaking, the hazard is the likelihood of having an almost immediate transition from a state known to be of age τ, to a state xn, and, crucially, can also depend on the history . From equation (10), summing over , and defining and , we get
The age-dependent sum is also referred to as the escape rate from and, using equation (11), can be written as a logarithmic derivative
Integrating equation (12) with initial condition gives:
These let us cast equation (10) in the more convenient form
where
is the probability that the system jumps into the state xn, given that it escapes the state at age τ. It is important to notice that the normalisation conditions and are satisfied. Hence, we can sample a random waiting time τ, according to the density and, after that, a random arrival configuration xn, according to the probability mass . This suggests a standard Monte Carlo algorithm for the generation of a trajectory (1):
- (1)Initialise the system to a configuration x0 and a time t0. Set a counter to n = 1.
- (2)Draw a value τ according to the density (11) and update the time to .
- (3)Update the system configuration to xn, with probability given by (16).
- (4)Update n to and repeat from (2) until tn reaches the desired simulation time.
3.2. The cloning step
We now need to take into account the effect of the factor on the dynamics, which is to increment (if ) or decrement (if ) the 'weight' of a trajectory, within an ensemble. This can be implemented by means of the so-called 'cloning' method, which consists of assigning to each trajectory of the ensemble a population of identical clones proportional to its weight. As reviewed, e.g., in [23, 24], the idea is not new. It seems to be born in the context of quantum physics and can be traced back to Enrico Fermi [25]. Noticeably, cloning was proposed as a general scheme for the evaluation of large deviation functionals of non-equilibrium Markov processes in [10] (with further continuous-time implementation in [11]) and has also been extensively applied within equilibrium statistical physics [26–28]. Here, we refine and extend this idea for the case of non-Markovian processes. One of the devices used in [10, 11] is to define modified transition probabilities—valid only under the Markovian assumption—and a modified cloning factor, encoding the contraction or expansion of the trajectory weight. In fact, it is implicit in the original work that the redefinition of such quantities is unnecessary; in some cases it may also be inconvenient, see section 4. An arguably more natural choice, especially for non-Markovian dynamics, is to focus on the WTDs. Specifically, equations (8) and (9) suggest the following procedure:
- (1)Set up an ensemble of N clones and initialise each with a given time t0, a random configuration x0, and a counter n = 0. Set a variable C to zero. For each clone, draw a time τ until the next jump from the density , and then choose the clone with the smallest value of t = t0 + τ.
- (2)For the chosen clone, update n to and then the configuration from to xn according to the probability mass .
- (3)Generate a new waiting time τ for the updated clone according to and increment its value of t to .
- (4)Cloning step. Compute , where u is drawn from a uniform distribution on .
- (1)If y = 0, prune the current clone. Then replace it with another one, uniformly chosen among the remaining .
- (2)If , produce y copies of the current clone. Then, prune a number y of elements, uniformly chosen among the existing N + y.
- (5)Increment C to . Choose the clone with the smallest t, and repeat from (2) until t − t0 for the chosen clone reaches the desired simulation time T.
The SCGF is finally recovered as −C/T for large T. The net effect of step (4) is to maintain a constant population of samples whose mean current does not decay to .
4. Semi-Markov systems
A common framework for modelling systems with finite-range memory is the formalism of semi-Markov processes, also referred to as continuous-time random walks (CTRWs) in the physics literature. CTRWs were introduced to model transport on lattices [29, 30] and later used in many other contexts, e.g., to describe quantum dots [31], temporal networks [32], animal movements [33, 34], biochemical reactions [35], and single-molecule kinetics [36, 37]. In such systems, the probability of having a jump depends only on the departure state and on the age, meaning that the memory is lost after each jump and the dependence on the previous history is removed. Under this assumption, the probability density of observing a trajectory (1) is
where the primed WTD can differ from the others3 . When has no dependence on τ (as well as, of course, no dependence on , due to the semi-Markov dynamics), we can write and the process is said to satisfy direction-time independence (DTI)4 .
In a slight shift in notation we now use xi and xj as configuration labels. The probability for the system to be in the configuration xi at time t, in a semi-Markov process, follows a convenient differential equation, which is referred to as the GME:
where is the memory kernel, taking into account the configuration at time , and depends on the primed WTDs (see details in the online supplementary material). The memory kernel is defined through an equation similar to (10), but in the Laplace domain
with .
The statistics of time-extensive variables in semi-Markov processes have been studied in [31, 38, 40] and compared to memoryless processes. In systems described by a standard master equation, one strategy is to analyse a process that obeys a modified rate equation, obtained replacing the time-independent rates , with the products , which are referred to as 'biased' rates. This is particularly simple when can only take values , i.e., at each step, the total current varies, at most, by one unit. In semi-Markov systems it is possible to investigate the statistics of J in a similar, but more general, way. Instead of the standard master equation, we deploy the GME (18). The probability of having a configuration xi with total current J at time t, under the constraint that the current can only grow or decrease by one unit at each jump, obeys the following GME:
We now make the assumption that the memory kernels are independent of the total integrated current J (only depending on the current increment), i.e.
where . The system is diagonalised with respect to the current subspace by means of the discrete Laplace transform
and is then equivalent to
which can be represented in a more compact form as
where is a linear s-dependent integral operator and has components . The limit as of (where is a row vector with all entries equal to one) is the SCGF of J. Clearly, equation (24) does not conserve the product , except for s = 0 when . The dynamics described by equation (23) is equivalent to the dynamics described by the GME (20), with the memory kernels, corresponding to jumps that contribute a unit c in the total current, multiplied by a factor . From linearity, it follows that the Laplace-transformed kernels are
This confirms that the modified dynamics can be simulated biasing the WTDs , i.e., multiplying them by .
The Markovian case is recovered for . Using this kernel, equations (23) and (24) can be written as
where is the s-modified stochastic generator of the Markov process with time independent rates and components
where is the rate of escape from5 xi. This shows that biasing the rates is consistent with biasing the WTDs (see also some related discussions in [15]). However, from a numerical point of view, the latter choice remains convenient even for the Markovian case, as it avoids us having to define the modified transition probabilities of [11]. To see this, we consider the biased Markovian WTD
which is the product of an exponential probability density , a time-independent probability mass , and a simple cloning factor . These specify the two steps of the standard Doob–Gillespie algorithm for Markov processes [41], followed by a cloning step of weight . Another legitimate choice is to define , set , and write
With such an arrangement, we recognise the algorithm of [11], i.e., at each step, the configuration evolves according to a stochastic generator with rates , and the ensemble is modified with the cloning factor . As the cloning factor here is exponential in time, during long intervals the relative number of new clones can be large. This can cause major finite-ensemble errors, which are shown to be important, e.g., in [9, 42]. Conversely, an implementation based on equation (29) seems to be one way to reduce (but not completely eliminate) such a problem.
5. Examples
We now test our procedure against three non-Markovian models, whose exact large deviations are known from the literature or can be deduced from Markovian models.
5.1. Semi-Markov models for ion-channel gating with and without DTI
The current through an ion channel in a cellular membrane can be modelled with only two states, corresponding to the gate being singly occupied (x1) or empty (x0); an ion can enter or leave this channel via the left (L) or right (R) boundary and non-exponential waiting times lead to a complex behaviour [43]. Specifically, we denote the WTD for a particle succeeding in entering (or leaving) through the boundary L by (or ) with respective density (or ) for the boundary R. The rightwards current is measured by a counter that increases (decreases) by one when a particle enters (leaves) the system through the boundary L. Its exact SCGF is obtained numerically in [38] as the leading pole of the time-Laplace transform of , for the DTI-case with , and the particular choice , where
with as the Gamma function; the Markovian case is recovered for k = 1. Notably, the cloning method of section 3 can be implemented for any WTD, as only a bias of , for ions leaving the channel leftwards, and a bias of , for ions entering from left, are needed. Figure 2(a) shows that this method reproduces, within numerical accuracy, the solution given in [38].
Download figure:
Standard image High-resolution imageIn a quest for more general classes of models to illustrate the power of our approach, we now relax the constraint of DTI and assume that each transition can be triggered independently by two mechanisms, corresponding to the two boundaries. We still assume that memory of the previous history is lost as soon as the system changes state, thus preserving the semi-Markov nature. At the instant when the gate is emptied, a particle attempts to enter the system from the left boundary after a waiting time with density distribution , while another particle attempts to arrive from the right boundary after a time distributed according to . The waiting times and , as well as the densities and are defined similarly. In order to have a right (left) jump during the interval , we also require that the left (right) mechanism remains silent until time τ. Consequently, the WTDs are
where are survival probabilities, with ρ denoting the mechanism L or R. As a concrete choice, we again assign a Gamma probability distribution to the waiting time of each event
so that the survival probabilities are
where is the upper incomplete Gamma function. The time to the next jump, given that the system just reached state xj (i.e., its age is zero) is and is associated to the total survival probability
Once the transition time is known, either the left or right trigger is chosen, according to the age-dependent rates
where is the lower incomplete Gamma function. The SCGF of the left current is computed by biasing the WTDs and with , respectively.
While the implementation of the method of section 3.2 remains straightforward for this model, a general solution for the exact SCGF is missing. We thus specialise to the case with , for which the Laplace transform of is a rational function of ν, viz.
where we defined for convenience and
This can be though of as resulting from a walker spending exponentially distributed times in three hidden stages, as in [44]. Hence, the two-state semi-Markov process with WTD (32) and (33) can be seen as a six-state Markov process, see online supplementary material for more details and an alternative formulation. The linearity of the Laplace transform permits the distinction of the left and right contributions in (38), hence it remains easy to bias the rates that correspond to a change in J and the SCGF can be exactly found as the leading eigenvalue of a modified Markovian stochastic generator (see again supplementary material). Figure 2(b) shows convincing agreement of our method with this exact approach.
5.2. Totally asymmetric exclusion process (TASEP) with history dependence
More general non-Markovian systems are those whose WTDs depend on events which occurred during the whole observation time. Systems in this class are the 'elephant' random walk [45] and its analogues, where the transition probabilities at time t depend on the history through the time-averaged current . We focus here on an interacting particle system with such current-dependent rates, namely the TASEP of [46].
Non-Markovian interacting particle systems can be described by assigning a trigger for jump attempts with WTD and a corresponding survival function to each elementary event i that controls the particle dynamics. The probability density that the next transition is of type i and occurs in the time interval , given that, for each j, a time τj has elapsed since the last event of type j, is given by6
where and . With exact expressions for these WTDs, we can implement the algorithms of section 3.
The TASEP consists of a one-dimensional lattice of length L, where each lattice site l, , can be either empty () or occupied by a particle (). Particles on a site are driven rightwards: they attempt a bulk jump to site with WTD , the attempt being successful if , as in [47–49]. With open boundaries, a particle that reaches the rightmost site L leaves the system with WTD . Also, as soon as , a further boundary mechanism turns on and particles arrive on the leftmost site with WTD . The special choice , , and corresponds to the standard Markovian TASEP with constant left, bulk and right rates α, p, and β.
We now assume that only the left boundary has a non-exponential WTD, while the particle triggers have exponential WTDs with rate 1 for free particles in the bulk, and rate β for the particle on the rightmost site. Consequently the inter-event time density distribution, conditioned on a time τ0 having elapsed since the last arrival, is
The probability mass distribution, conditioned on an age τ and elapsed time τ0 is
where η0 and ηL encode the exclusion rules at boundaries, , and n is the number of free particles in the bulk, which depends on the lattice configuration before the jump.
Let us impose now that the arrival rate α depends linearly on the input current , i.e., , which defines a time-dependent rate . Similar functional dependence (but on the instantaneous output current) has been used to model ribosome recycling in protein translation [50, 51]. Generically, such rates describe a simple form of positive feedback (for ), whose effect on the stationary state of the TASEP is to shrink the low-density phase [46, 51]. The current fluctuations are also altered; the rate function in this phase has already been computed, for our model, by means of the temporal additivity principle [8, 46], hence this provides a testing ground for the cloning method of section 3. The particle arrival mechanism starts when the leftmost site is emptied, when we set an age of . Denoting by q the current immediately after the last arrival, which occurred at , the value at age τ can be expressed as , hence the trigger hazard is
where τ is the trigger age. Initial values of τ0 and are chosen to be 1 and 0, respectively. This allows us to derive the trigger survival probability and trigger WTD (see also [46]) which are, respectively
and . Using these in equations (41) and (44) allows us to generate the trajectories, and the large deviation function can be evaluated by applying a bias to the arrival WTD and following the algorithm of section 3. The results are plotted in figure 3 and validated by the exact numerical calculation of [46] (which assumes the temporal additivity principle).
Download figure:
Standard image High-resolution imageIt is worth noting that, for large negative values of s, the SCGF displays a linear branch with slope . If remains linear with the same slope for , its Legendre–Fenchel transform will be defined only for . This appears to be related to the dynamical phase transition seen in the Markovian TASEP, where large current fluctuations require correlations on the scale of the system size and the rate function, in the corresponding regime, diverges with L [52]. Indeed, space–time diagrams (not shown) of the density profile from the cloning simulations seem to suggest that the correlation length increases as s becomes more negative. It would be interesting to further reduce the finite-ensemble errors (as discussed in section 6) in order to probe larger negative values of s.
6. Discussion
We have demonstrated that the 'cloning' algorithm for the evaluation of large deviations can be applied consistently for both Markovian and non-Markovian dynamics. In fact, the cloning/pruning of trajectories at each temporal step can be performed according to a very simple factor multiplying the WTDs, as in equation (9). Our analysis encompasses classes of systems with different memory dependence and exploits the similarities between their different formalisms. The efficacy of this approach is confirmed by numerical results for some of the rare non-Markovian models whose large deviation functions can be obtained exactly.
For general non-Markovian cases, the implementation of our procedure is not much harder than the exact simulation of the original trajectories. In Markov processes, the procedure is equivalent to those of [10, 11], where biased dynamics involving alternative rates or transition probabilities have been defined. We expect that, to minimise finite-ensemble effects, an optimal choice of modified WTDs and cloning factors exists for both non-Markovian and Markovian systems, along the lines of the feedback control of [14]. Further developments can thus be anticipated.
We also mention that the discrete-time case of [10] is interesting as the jumps and the cloning steps occur simultaneously for each ensemble element. This feature can be used to prevent a single clone replacing a macroscopic fraction of the ensemble, thus reducing finite size effects (see online supplementary material). In continuous time an equivalent strategy is to mimic the discrete-time steps, as in, e.g., [53], so each trajectory evolves independently for a constant interval in this case, the product of the cloning factors encountered during the interval, as well as the time elapsed since the last jump must be stored. This permits the application of the cloning step to all clones simultaneously.
Large deviation functionals are often hard to obtain analytically, and such a difficulty is exacerbated in non-Markovian systems, which better describe real-world situations; we believe that the results of this work open up a promising avenue for numerical studies.
Acknowledgments
It is a pleasure to thank Raúl J Mondragón, Stefan Grosskinsky, Oscar Bandtlow, and Arturo Narros for many helpful discussions.
Footnotes
- 3
A natural situation is when we observe a portion of a trajectory that started before t0. In this case, , as it depends on the time of the last jump before t0, being conditioned on the survival until t0. Also, in this case, the probability that x0 survives the time is .
- 4
- 5
- 6
In a slight abuse of the notation, we continue to use the full w(t) as a parameter but explicitly show the conditioning on the τi.