NEURAL NETWORKS WITH FINITE-WIDTH ACTION POTENTIALS
Fariel Shafee
The integrate-and-fire type of dynamic neural networks are studied in models with hitherto neglected finite widths to make them more realistic. Pulse shapes of arbitrary widths and shapes are considered and differences in the evolving firing patterns are noted. The convergence to limit cycle is examined both analytically and in simulation experiments. We present a proof of a theorem regarding the timer period of oscillation analogous to that of Hopfield and Herz for nonleaking models with finite width action potentials. It is pointed out that the concept of simultaneity in such networks may be subject to bin widths, i.e. window apertures of cumulative counts, in simulation experiments.The relation to Kolmogorov complexity is briefly discussed. The effects of varying the shape of the action potential and its width are examined in some detail. The roles of the magnitude of a leaking resistance connected to the neurons, a common external current and synaptic currents between neighbors are also examined and found to bear important consequences in the dynamical development of the networks.
hopnet.tex
\documentclass[aps,pre,twocolumn,floatfix,showpacs,showkeys]{revtex4}
%\documentstyle[aps,twocolumn,floats]{revtex}
%\documentstyle[12pt,epsf]{article}
%\usepackage{graphics}
\usepackage{amsmath,amsfonts,amssymb,mathrsfs,graphicx}
\bibliographystyle{apsrev}
\keywords{neural network; action potential; dynamic memory}
\begin{document}
%\tighten
%\draft
%\twocolumn[\hsize\textwidth\columnwidth\hsize\csname
%@twocolumnfalse\endcsname
\title{NEURAL NETWORKS WITH FINITE-WIDTH ACTION POTENTIALS}


\author{Fariel Shafee}
\affiliation{ Department of Physics\\ Princeton University\\
Princeton, NJ 08540\\ USA.} \email{fshafee@princeton.edu }

\begin {abstract}
The integrate-and-fire type of dynamic neural networks are studied
in models with hitherto neglected finite widths to make them more
realistic. Pulse shapes of arbitrary widths and shapes are
considered and differences in the evolving firing patterns are
noted. The convergence to limit cycle is examined both analytically
and in simulation experiments. We present a proof of a theorem
regarding the timer period of oscillation analogous to that of
Hopfield and Herz for nonleaking models with finite width action
potentials. It is pointed out that the concept of simultaneity in
such networks may be subject to bin widths, i.e. window apertures of
cumulative counts, in simulation experiments.The relation to
Kolmogorov complexity is briefly discussed. The effects of varying
the shape of the action potential and its width are examined in some
detail. The roles of the magnitude of a leaking resistance connected
to the neurons, a common external current and synaptic currents
between neighbors are also examined and found to bear important
consequences in the dynamical development of the networks.

\end{abstract}

\pacs{PACS numbers: 07.05.Mh,84.35.+i, 87.18.Sn} \vspace*{1cm}
%]

\maketitle

\section{Introduction}

Models of neural networks are important to understand biological
information processing and storage systems, and also because their
relation to many physical systems shed new light on those
nonbiological complexes. It is well-known that Hopfield�s
associative memory networks \cite{JH1,JH2} share many common
features with spin glasses \cite{AN1,ST1}. Artificial neural
networks may one day pave the way for creating computers which are
more akin to organic pattern recognition systems, and hence more
efficient in tasks at present not addressable by mechanistic
serial processing.

Associative memory networks \cite{AM1,DE1,GU1,SO1,HO1,ZE1,SH1} are
usually conceived as static patterns holding memories of inputs,
but in recent years there is growing interest in systems which too
hold memories of inputs in a dynamic fashion, in the form of
phase-locked oscillations. Several different type of models have
been presented \cite{Ho1,AO1} with different assumptions about
their {\em modus operandi}. The differences may be in the ways the
neurons are triggered, in the ways they convey their information
to other neurons, in the fashion they are interconnected, and
other conjectures about the role of the environment.

Among these models are included a set of relatively simple
networks of integrate-and-fire neurons conceived by Hopfield and
Herz \cite{JH2}, whose work shows that many interesting nontrivial
predictions regarding the dynamic behavior of such networks could
be made from their simple properties. However, in their work, to
maintain simplicity, they assumed that the action potential is in
the form of a delta function in time, i.e. it is instantaneous.
But in some of the models they present, when a neuron fires on
attaining threshold from part of the action potential (AP) from a
neighbor, the remaining excess charge from the neighbor may
remains stored and contributes to the next firing of the receiver,
which in a sense mimics the effects of a finite width, without
involving quantitative complications that an explicit finite-width
model would entail.

A more realistic approach, however, cannot avoid dealing with all
the consequences of the finite width of the AP, which is a
biological fact and must also be an inevitable feature of most
physical realizations.{\em A priori}, these may be in the form of
changed periods, different convergence behavior to limit cycles or
modified rates of convergence, different paths to globalization,
or its absence altogether etc.

The width introduces its own time scale that can be expected to
interact with the time scale of the zero-width period obtained in
the H\&H models, which in turn depends on the strength of coupling
among the neurons and the environmental current. It also
introduces the possibility of a more complicated distribution of
the action potential current to the receiving neuron, affecting
its attainment of threshold, i.e. its period and properties
related to convergence to limit cycles.

We have, therefore, investigated the consequences of introducing
finite widths into the integrate-and-fire models. Where possible
we have tried to use analytical methods. Where that turns out to
be impossible, we have used simulation. In the next section we
first describe different types of integrate-and-fire models to
establish the concepts and notation used in this work. In section
3 we show that the convergence to periodic behavior for Class C
models can be analytically proved even in the finite width case.
In section 4 we present our simulations with class C models using
APs of various finite widths and different shapes. In section 5 we
do the same with the leaking model A. Finally in section 6 we
present our conclusions.




\section{INTEGRATE-AND-FIRE  MODELS}


Integrate-and-fire neurons are usually considered in four
different scenarios \cite{JH2}. In each case every neuron in the
network receives some charge from other neurons in the
neighborhood, and also from an environmental current source. This
raises the electrostatic potential of the neuron. When the neuron
has accumulated sufficient charge and its potential exceeds a
threshold, it fires, i.e. it gives forth an AP to the other
neurons, which too may be triggered in due course. If all neurons
attain the same periodicity, the phase difference among them must
become constant, i.e. the system becomes phase-locked. It is also
possible for some of the neurons to fire simultaneously
(synchronicity) if they reach the limit cycle together.

Let $u_i(t)$ be the potential of the $i$-th neuron at time $t$,
$I_i(t)$ the external current input into this neuron, $R$ its
leaking resistance and $T_{ij} f_j(t)$ the action current getting
into neuron $i$ from neuron $j$.  We can then write the current
conservation equation as:

\begin{equation} \label{eq1}
 C du_i/dt = -u_i(t)/R +  \Sigma_j T_{ij} f(t) + I_i(t)
\end {equation}

For simplicity with no loss of generality we take the threshold
for firing to be $u=1$, by renormalizing $u$ to the dimensionless
quantity $u/u_{threshold}$. We can also divide the equation by $C$
and absorb it in $R$ and $I$, in which case $R$ actually
represents $CR$, the decay time constant, and $I$ represents $I/(C
u_{threshold})$ which has the dimension of inverse time. In other
words, in our notation $R$ and $1/I$ will set two different time
scales. In most of our work we shall assign the value $1$ to $I$
to represent the scale of an external clock against which all
internal dynamics is timed.

In the H\&H work the action pulse shape $f(t)$ is a Dirac delta
function, i.e. it is a zero-width normalized pulse:

\begin{equation} \label{eq2}
f_j(t) =  \delta (t - t_j)
\end{equation}

where $t_j$ is the time of the last firing of neuron $j$.

\par

The four models differ in the treatment of the leaking as
represented by $R$ and in the way they handle the after-fire
potential $u_i$.

In Model A, $R$ is finite and may be taken to be equal to $1$.
This establishes the time scale $C R$. In this model if neuron $i$
receives a synaptic charge in excess of that required to take the
potential to the threshold, then, after firing, $u_i(t)$ resets to
a value corresponding to the excess charge (we omit the implied
symbol for summation over neighbors $j$ for clarity):

\begin{equation} \label{eq3}
u_i(t^+)  = [u_i(t^-) +  \Delta T_{ij}  �  1]  + (1 -
\Delta)T_{ij} = (1 -\Delta)T_{ij}
\end{equation}

where

\begin{equation} \label{eq4}
\Delta T_{ij}  = 1 - u_i(t^-)
\end{equation}

In Model B, the potential resets to zero after firing,
irrespective of the oncoming synaptic current and the previous
state of the neuron. It too has a leaking $R=1$. Model C is the
version of A that does not leak, i.e. $R =\infty$, and Model D is
the non-leaking version of B.

One can integrate the model C potential since the last firing at
$t_0$ gives, assuming for simplicity constant external current
$I$:

\begin{equation} \label {eq5}
u_i(t)  = u_i(t^+)  + \Sigma ^\prime_j T_{ij}   + I (t - t_0)
\end{equation}

where the summation is over only those other neurons $j$ which
have fired since $t_0$.

It is convenient to use a simple topology of the network by
assuming only nearest neighbor interactions. Then, after
phase-locking has been established with period $P$, we must have:

\begin{equation} \label{eq6}
u(t_0+P) = 1 =  \Sigma_j T_{ij}   + I P
\end{equation}


So
\begin{equation} \label{eq7}
P = (1 - A)/I
\end{equation}

where for notational convenience, assuming constant $T_{ij}$, we
get:

\begin{equation} \label{eq8}
A=   \Sigma_j T_{ij} = Z \alpha
\end{equation}

$Z$ being the co-ordination number of the neural lattice, i.e. the
number of nearest neighbors. In Model A it is not possible to
integrate Eq.\ref{eq1},  because the integral depends on the
specific times the contributions from the different neighbors are
received. This uncertainty results from the leakage, which is
proportional to the instantaneous potential.



\section{ CONVERGENCE TO PERIODICITY IN THE FINITE WIDTH CASE}

When the action potential occupies a finite width we have to use
the pulse form factor $f$ with
\begin{equation} \label{eq9}
\int_{t_0}^{t_0+\Delta t} dt f(t)=1
\end{equation}

Here $t_o$ is the time of the beginning of a firing and $\Delta t$
is the width of the action potential and the consequent synaptic
current.

As in the case of zero width action potentials, we can show that
in the finite width case too the system moves to a limit cycle
with phase locking among the neurons.  We shall do it for the
nonleaking model C. We present the proof in three parts.

\par

I: The time between the beginning of two successive firings of the
same neuron cannot be less than $P= (1 -  A)/I$, where the
variables are as defined in Eq. \ref{eq7} and Eq. \ref{eq8}.

\par

Proof: Let $i$ be the neuron which has the smallest interval
between two firings (there may be other neurons with the same
least interval). Let the two firings begin at times $t_1$ and
$t_2$. Let us consider any neighbor $j$ of $i$. Since, $i$ has the
least interval between successive firings, $j$ cannot begin to
fire twice within this interval. Hence, $i$ can receive at most
fractional charges from two firings of $j$ within this interval,
of which the beginning of the second firing $t^\prime_2$ can be
within the interval, but not the earlier one $t^\prime_1$. So

\begin{equation} \label{eq10}
t_2 -  t_1 \leq t^\prime_2  -  t^\prime_1
\end {equation}

 i.e.

\begin{equation} \label {eq11}
t_2 -  t^\prime_2 \leq t_1  - t^\prime_1
\end{equation}


The total contribution received from $j$ by $i$ within this
interval is:

\begin{equation}\label{eq12}
\alpha_j= \int_0^{s_2 \Delta t} f(t) dt +\int_{s_1 \Delta t}
^{\Delta t} f(t) dt
\end{equation}


where

\begin{equation} \label{eq13}
 s_{1,2}
=(t_{1,2}- t^\prime_{1,2})/\Delta t.
\end{equation}

As, $s_2 \leq s1 $, we see that

\begin{equation} \label{eq14}
\alpha_j \leq \alpha
\end{equation}

ensuring that the period for $i$, $P_i \geq P$.


II: The system converges to a limit cycle for the finite width
case:

Proof:  The Lyapunov function  $E$ is defined by \cite{JH2}
 $E= -\Sigma u_i$. Then its change after an interval $P$ given by
 Eq.
\ref{eq7} is

\begin{eqnarray}\label{eqa15}
E(t+P) -  E(t)  =  \Sigma_i [- u_i(t+P) ] - \Sigma_i [-
u_i(t)]\nonumber\\
 = \Sigma_i(- I P - \Sigma_j T_{ij} \int_t^{t+P}  f_j(t)
 dt\nonumber\\
                    + \Sigma_i \int_t^{t+P}  f_i(t) dt\nonumber \\
= - (1 -  A) N + (1 -  A) \Sigma_i \int_t^{t+P}f_i(t) dt\nonumber \\
 = - (1 - A)[ N -\Sigma_i \int_t^{t+P}f_i(t) dt]
\end{eqnarray}

But no neuron can begin to fire twice in an interval of less than
$P$; so $E(t+P) - E(t)$ is nonpositive as in the case of zero
width AP. Hence, like the zero width case, we have a convergence
to a limit cycle.

III:  If there is a periodicity, the period must be  $P =(1 -
A)/I$. Proof: Firstly, by part $1$, it cannot be less than $P$,
because no neuron can ever begin to fire again within an interval
less than $P$. Indeed if we now assume that a periodicity $T$ has
been established, i.e. all neurons have become phase locked with
equal phase difference in all periods, then, if $t_i$ is the last
time neuron $i$ began to fire: $u_i(t_i+T) = 1 = \Sigma_j T_{ij}
\int_t^{ t+T} f_i(t) dt + I T  = A + [(1 - A)/P ] T $.

The last equality follows from the fact that if $i$ receives a
fraction of a pulse from a neighbor $j$ at the beginning of the
cycle, it will receive the rest at the last part of its cycle
because of phase lock. i.e. $\int_t^{t+P}  f_j(t) dt = 1$ for all
$j$. So $T = P$.


\section{ RATE OF CONVERGENCE TO LIMIT CYCLE}

An interesting observation can be made for the speed of
convergence to the limit cycle with period $P$. Let us consider,
for simplicity, finite pulses of square shape. Then the
contribution from a neighbor $j$ to a neuron $i$ will be
proportional to the fractional duration of the pulse received by
$i$ within its (variable period) cycle. Again, for simplicity let
us consider only one neighbor. Then, if $P_n$ is the period of the
$n$-th cycle, and $f_n$ is the duration of the pulse received by
$i$, and $w$ is the width of the pulse, then

\begin{eqnarray} \label{eq16}
P_{n+1}  = [ 1 � (A/w) ( w - f_n) - A f_{n+1}/w]/I\nonumber\\
        =  P_0 + A/(w I) [ f_n -� f_{n+1}]
\end{eqnarray}

Now, writing

\begin{equation}  \label{eq17}
 \Delta P_n = P_{n} -� P
 \end{equation}
 we get
\begin{eqnarray}\label{eqa18}
\Delta P_{n+1}  / \Delta P_n    = A/ [ I w + A]
\end{eqnarray}

We can see that convergence to the limit cycle is a geometric
sequence and the rate of convergence depends on the $A/ (I w +A)$
ratio, which of course is dimensionless in our choice of notation.
For fixed $w$ a higher current would bring in faster convergence.
It may seem that for the zero width case we have no convergence at
all. But actually in this singular case we cannot split the pulse
into two parts as in Eq. \ref{eq16}, so that we must have $f_n=1$,
and $\Delta P_n =0$  for all $n$, i.e. the system goes into phase
lock after the completion of the first set of firings. As we
pointed out in the beginning, this relation is derived assuming a
single neighbor. If $Z > 1$, then in general the different
neighbors can contribute with different phase differences, which
may keep changing until phase lock. Eq. \ref{eq17} will then need
to be modified to account for these differences. If $r$ of the $Z$
neighbors contribute fully within a single period, then we have:
\begin{equation} \label{eq19}
\Delta P_{n+1}  / \Delta P_n   = A^\prime / ( I w + A^\prime)
\end{equation}

 where we have used the symbol

 \begin{equation} \label{eq20}
 A^\prime = (1 - r/Z) A
 \end{equation}

Therefore, when all the neighbors contribute their full action
potential within a single cycle, the period becomes $P$. However,
the inverse is not true, i.e. it is possible to distribute the
charge from a single AP to succeeding cycles of the neighbor and
the whole system can still be in total phase-lock with period $P$,
because the R.H.S. of Eq. \ref{eq19} becomes indeterminate. In the
case of pulses of arbitrary shapes it is easy to see that  we get:

\begin{equation}  \label{eq21}
\Delta P_{n+1} = \Sigma_j (\alpha_j /I) + \int_{P_{n+1}}^{P_n}f(t)
dt
\end{equation}
where the sum runs over only those neighbors which do not yet
deliver their full synaptic current in a single cycle. This is an
integral equation depending on the shape function $f(t)$. Despite
the arbitrariness of $f(t)$ we can see that if $\Delta
P_{n+1}\rightarrow \Delta P_n$, then both must approach zero, i.e.
the sequence must be convergent to the limit cycle. If $ \Delta
P_{n+1} / \Delta P_n \rightarrow r < 1 $, then too the convergence
is obvious. If as a first approximation we replace the arbitrary
pulse by a square one of same height but a width giving unit area,
then we would expect a geometric convergence similar to
Eq.\ref{eqa18}- Eq. \ref{eq20}. The details of the pulse shape may
produce smaller perturbations on the rate of convergence without
affecting the general pattern.

\section{NONLEAKING  NETWORKS WITH FINITE WIDTH A.P.}

We simulate a square lattice with periodic boundary conditions to
reduce finite size effects. It has been noted that a 40 X 40
lattice is sufficient to demonstrate all the important
characteristics. At each step the neurons are updated according to
the charge they receive from their neighbors and the external
current during the time loop and then, if the potential reaches
threshold, it fires. The effect of the finite width of the action
potential can be two-fold: (1) reduction of u over a number of
time loops corresponding to the width of the AP after a neuron
begins to fire, and (2) the arrival of the synaptic current
$T_{ij} f(t)$ over a number of time loops according to the shape
of the packet $f(t)$. We do not expect any role for ectopic pulses
resulting from recharging to threshold before the previous firing
is completed, though the synaptic current arriving at a neuron
while it discharges will be retained for the next firing. Hence,
in the nonleaking models it is unnecessary to model the actual
shape of the fall of u. The results will be indistinguishable from
those of a zero-width model in such models, though the situation
in leaking models will be quite different because the leakage will
depend on the falling potential.  The distribution of the synaptic
current over multiple time loops, however, will carry the
nontrivial differential characteristics of finite width action
potentials in all models. In this section we consider only a
nonleaking model. We have in our algorithm provision for
generating pulses of various shapes in the form of discretized
$f(t_i)$ over the time loops during which the action pulse works.
Arbitrary shapes can also be given as input. However, we have only
done the simulations with square and isosceles triangular shapes
for simplicity.

\subsection{ Effect of Pulse Shape:}

In Fig.\ref{fig1}  and Fig. \ref{fig2} we present our simulations
with $A= 0.96$ (i.e. $T_{ij} = 0.24$), $I= 1$ and $width = 0.01$
(which is $1/4$ of $P$) using first a square pulse
(Fig.\ref{fig1}), and then a triangular one (Fig.\ref{fig2}). We
note that indeed in both cases the period $0.04$ is equal to $P [
= (1 - A)/I]$, as proved earlier. However, there is a slight
difference between them in their rate of convergence to the limit
cycle; the square pulses almost converge after $t= 4$, whereas for
the triangular ones a similar level of convergence happens after
$t=6$. This can be expected on account of the simplicity of the
square pulse allowing it to adapt itself more easily for
phase-locking.
\begin{figure}[ht!]
\includegraphics[width=8cm]{Fg1t.eps}
 \caption{\label{fig1}Square pulse with width $w=
0.01$, $A= 0.96$, $I=1$.}
\end{figure}
\begin{figure}[ht!]
\includegraphics[width=8cm]{Fg2t.eps}
\caption{\label{fig2}same as Fig. 1, but with a triangular pulse.}
\end{figure}

\subsection{ Convergence Rate and the Value of A:}

In Fig.\ref{fig3} and Fig.\ref{fig4}  we show the effect of
changing the value $A$, keeping the same ratio of the width with
the corresponding period $P$. Fig.\ref{fig3} is for a square pulse
and Fig.{\ref{fig4} is for a triangular one. Of course varying $A$
changes the period. But we have also seen that the rate of
convergence depends on the relative magnitudes of $A$, $I$ and $w$
according to Eq.\ref{eqa18}- Eq. \ref{eq21}. A higher contribution
from the neighboring neurons than the constant external driver
seems to delay convergence, which may be expected on account of
the more intricate interactions in phase adaptation of the
neighboring neurons than in their susceptibility for acting
cohesively with a common external driver. For $A= 0.24$, i.e.
$\alpha=0.06$, we get practically immediate convergence after all
the neurons have fired for the first time. This feature is common
for both shapes of pulses.
\begin{figure}[ht!]
\includegraphics[width=8cm]{Fg3t.eps}
\caption{\label{fig3}Square pulse, with $A= 0.24$, $I=1$, $w=
0.2$}
\end{figure}
\begin{figure}[ht!]
\includegraphics[width=8cm]{Fg4t.eps}
\caption{\label{fig4}same as Fig. 3, but with a triangular A.P.
pulse.}
\end{figure}

\subsection{ Effect of Region of Initial Excitation:}

The network can be made to receive an initial excitation at the
periphery, or throughout the network. In Fig.\ref{fig5} we repeat
the experiment of Fig.\ref{fig2}, with the same parameters, but
with only the peripheral neurons initially excited. We see that
the general pattern remains the same, and cannot be distinguished
from the whole net excitation of Fig. \ref{fig2}. However, in Fig.
\ref{fig6} we have done the same with the smaller $A (=0.24)$
case, first studied with whole net excitation in Fig.\ref{fig4}.
There is remarkable synchronicity in this case, the great majority
of the neurons firing together periodically. We can understand
what is happening here if we picture the initial development of
the system. All the inner neurons are initially uncharged and they
do not receive pulses from neighbors until their first firing at
$t=1$, which is caused simultaneously by the common driver $I$.
Henceforth, all these inner neurons remain phase locked. By
contrast for $A= 0.96$, the external current has little role in
charging up the neurons, and hence the neurons get  phase-locked
only after their mutual interactions, making the situation similar
to the whole net excitation.


\begin{figure}[ht!]
\includegraphics[width=8cm]{Fg5t.eps}
\caption{\label{fig5}As Fig. 2, but with peripheral initial
excitation.}
\end{figure}
\begin{figure}[ht!]
\includegraphics[width=8cm]{Fg6t.eps}
\caption{\label{fig6}As Fig. 4, but with peripheral initial
excitation.}
\end{figure}

\section{Synchronicity, Bin Size and Dynamic Entropy:}

While the synchronicity just described, induced by the common
external agent, may indeed be genuine, the other synchronicities may
be more suspect. Since in simulation experiments the time loops must
occur after a small but finite duration, the procedure puts many
events of different exact times in the same bin. This is revealed if
we change the bin size in the experiment. Neurons which appear in
the same bin seem to split up when the resolution is increased. In
Fig.\ref{fig7} we show the results of lumping $10$ loops of
Fig.\ref{fig2} into one loop. Apart from the proportionately higher
bin counts, convergence also appears to arrive faster, which of
course is only an artifact.

\begin{figure}[ht!]
\includegraphics[width=8cm]{Fg7t.eps}
\caption{\label{fig7}As Fig. 2, but with a time step 10 times
greater.}
\end{figure}

Because of the built-in time scales related to charging rate,
affinities with nearest neighbors, and the width of the action
pulses, it is not expected that such neural networks would show
self-similarity, i.e. a fractal structure. It is obvious that if the
time bin size is made small enough, the amplitude, i.e. the number
of firings per bin, would normally be reduced to a sequence of units
and zeros, though, because of the discrete nature of the upgradings
in a computer simulation, there may be occasional coincidences. A
two-letter alphabet can then define any time sequence of an
arbitrary length.

In a time series Kolmogorov-Sinai entropy \cite{KO1, SI1} is
occasionally used as a characteristic measure of complexity of the
series. It is a measure of the size of the algorithm needed to
reproduce the series. A completely random sequence has infinite K-S
entropy, a chaotic system has finite entropy, and a completely
periodic system has zero entropy. The latter follows from the fact
that in amplitude space the trajectory of the system follows the
same path repeatedly; for a sequence of $N$ such steps:

\begin {equation}\label{eqnew1}
K_N = - \sum_{i_0,i_1,...,i_N}   p_{i_0,i_1,....i_N}log
(p_{i_0,i_1,...,i_N})
\end{equation}

and hence for a phase-locked periodic system, only a particular
sequence of amplitudes $\{{i_0,i_1,...,i_N}\}$ will have probability
1, and all other sequences will have probability zero, giving zero
$K_N$ and hence zero K-S entropy. The K-S entropy is the difference

\begin {equation}\label{eqnew2}
S_{KS} = \lim_{\epsilon \rightarrow 0, N \rightarrow \infty}
(K_{N+1} - K_N)
\end{equation}

where $\epsilon$ is the size of size of a unique state in the
embedding phase space. In this case since we have a discrete phase
space with integral amplitudes, and hence the transition $\epsilon
\rightarrow 0$ is also discrete. A chaotic or random case might
produce problems in such a case, but not in a purely deterministic
situation as this one.

Entropy would be null irrespective of the complexity of the pattern,
and simplifying the pattern by summing bins together in any
well-defined nonrandom fashion would not change the entropy from its
null value.

For the initial time domain containing the transition to the
phase-locking state, since the system is fully deterministic,
despite the apparent random pattern of the firings it follows a
unique trajectory, given a unique initial state, and hence there is
only point in $N$-dimensional or in $N+1$-dimensional embedding
space with nonzero probability, again  giving zero entropy.


\section{LEAKING NETWORKS WITH FINITE WIDTH A.P.}

In Fig.\ref{fig8} we have shown the results of simulating a neural
network similar to that of  Fig. \ref{fig2}, but with a finite
value of the leaking resistance $R$. Compared to Fig. \ref{fig2}
we see a higher time period $T= 0.047$, though in this case the
convergence seems to have taken place much earlier at $t=1.7$.
 For $R=2$, we get $T= 0.069$, and
the limit cycle seems to appear only around $t= 3.5$. The smaller
resistance of course dampens the charging more and this results in
the higher time period. It also effectively reduces the effect of
the charging external current I and, as we have commented earlier,
this weakening of a common driver increases the role of the
complex phase adjustments of the neighbors, which in turn delays
the convergence to the limit cycle. If we integrate Eq. \ref{eq1}
assuming like H \&H that all the synaptic current from the
neighbors arrive at the beginning of the cycle, and that we have
narrow widths as a first approximation, but keeping R as a free
time scale, we get
\begin{figure}[ht!]
\includegraphics[width=8cm]{Fg8t.eps}
\caption{\label{fig8}As Fig. 2, but with leaking neurons with $R=
10$.}
\end{figure}

\begin{equation} \label{eq22}
T = R \log [(IR - A)/(IR  -  1)]
\end{equation}

Again, this is derived assuming that each neuron gets the full
synaptic current from the neighbors at the very beginning of the
cycle and a narrow width as in H\&H. We notice that though $IR$ is
a dimensionless quantity here, the period $T$ is controlled by the
scale of $R$ if $IR$ is constant, i.e., if we vary both $I$ and
$R$ keeping $IR$ constant, then

\begin{equation}\label{eq23}
 T_1/T_2 = R_1/R_2
\end{equation}

So, for $I=1$ which we have used throughout as an external time
scale reference, making $R=1$ should give a singularity. Indeed
when we tried to simulate a network with $I=1$ and $R=1$, not a
single neuron fired. However, if we change $R$ to a slightly
higher value, e.g. $R= 1.0001$, other parameters remaining
identical, we see a very interesting phenomenon. The neurons now
get sufficient accumulated charge from their initial random
charges, the external current and the synaptic current from
neighbors to defeat the damping of the potential due to leakage
and reach firing threshold. However, as the damping is severe near
the critical value R=1, the differentiation of the initial phases
becomes relatively unimportant and many neurons move towards
synchronicity. In Fig. \ref{fig9} we see the first bunch of
firings, with some dispersion, whereas in Fig.\ref{fig10}, which
gives a more extended picture, the synchronicity appears to become
more established in successive firings. This is in contrast to the
case of weaker damping (Fig.\ref{fig8}) described above, where
periodicity establishes phase-locking, but not synchronicity. Eq.
\ref{eq22} seems to be more accurate for large $R$ with small $T$,
than for small $R$ near criticality (Table 1).
\begin{figure}[ht!]
\includegraphics[width=8cm]{Fg9t.eps}
\caption{\label{fig9}First firing of a net of leaking neurons,
with a resistance just above the critical value of $R=1$.}
\end{figure}
\begin{figure}[ht!]
\includegraphics[width=8cm]{Fg10t.eps}
\caption{\label{fig10}As above, showing successive peaks with
growing synchronicity. The first firings shown in detail in the
previous figure are the small clump at around $t=2.8$.}

\end{figure}


\begin{table}
\caption{\bf Time period for leaking networks}
\begin{ruledtabular}
\begin{tabular}{|c|c|c|c|c|c|}
$R$ & 1.2 & 1.5 & 2.0 & 5.0 & 10.0\\
\hline
$T_{formula}$ & 0.22 & 0.11& 0.078 & 0.050  & 0.044\\
$T_{sim}[w=0.001]$ & 0.095 & 0.078  & 0.068 & 0.051& 0.048 \\
$T_{sim}[w=0.002]$ & 0.21  & 0.11  & 0.077 & 0.049 & 0.045 \\
\end{tabular}
\end{ruledtabular}
\end{table}

It is understandable that for large periods the discrepancy
between Eq. \ref{eq22} and the results of the simulation increase.
The neighboring neurons can then distribute their synaptic
contributions over a larger duration, and any signal arriving late
is attenuated by $R$ less than those arriving at the beginning of
the cycle, which we have assumed. So, for large $T$, the effective
$A$ in Eq. \ref{eq22} is higher than the real $A$, which decreases
the period to some extent. We also note that if the width of the
synaptic current pulse is changed there may be a change in the
period, because a wider pulse is more liable to get attenuated by
the damping $R$.  The last row of the table presents the results
of simulating with $w=0.002$, which gives almost perfect agreement
with Eq. \ref{eq22}.

\section{Conclusions}

In this work we find that many of the results of an over-simple
zero-width integrate-and-fire neural network are valid for more
realistic finite-width models . We have proved analytically that
even with a finite width, the system converges to a limit cycle
with a period identical to the zero-width case. However, the
problem of convergence is more complicated for the case of  finite
width AP spikes, and we have shown that probably only asymptotic
convergence ensues as a geometric sequence, which depends on the
parameters of the model, i.e. the width, the external current and
the synaptic coupling with the neighbors. The shape of the pulse
does not seem to be the determining factor in controlling
convergence, though it might have some perturbative modifications.
The question of synchronicity seems to be complicated by the
inevitable finiteness of the time steps in a simulation and opens
up the possibility of investigating any fractal properties
associated with such networks. The radical difference between
peripheral and whole net excitation also seems interesting and
worth further investigation. Stochastic and mean field
approximations may give analytic results where exact methods may
be limited, as in the case of leaking neurons, though the simple
formula Eq. \ref{eq22} for the period, derived with drastic
assumptions give remarkably accurate predictions. The transition
to sharp synchronicity in almost critically damped networks may
therefore be amenable to more precise analysis.
\begin{acknowledgments}

%\vspace{1.0cm} \centerline{\bf Acknowledgments} \vspace{0.5cm}
The author would like to thank Professor J.H. Hopfield for useful
discussions.
\end{acknowledgments}
\begin{thebibliography}{}
\bibitem{JH1}J.J. ~Hopfield,Proc. Natl. Acad. Sci.  {\bf 79}, 2554 (1982)
\bibitem{JH2}J.J. ~Hopfield  and A.V.M. ~Herz, \textit {ibid} {\bf 92}, 6655 (1995)
\bibitem{AN1}P.W. ~Anderson, Mat. Res. Bull. {\bf 5}, 549 (1970)
\bibitem{ST1} D.L. ~Stein(ed.), \textit {Spin Glasses and Biology }(World Scientific,
1992)
\bibitem{AM1} D.J. ~Amit,H. Gutfreund and H. Sampolinsky, Phys.
Rev. Lett. {\bf 55}, 1530 (1985)
\bibitem{DE1} B. ~Derida, E. Gardner and A. Zippelius,  Europhys.
Lett. {\bf 4}, 167 (1987)
\bibitem {GU1}H. ~Gutfreund  and M. Mezard, Phys. Rev. Lett. {\bf 61},
 235 (1988)
\bibitem {SO1} H. ~Sompolinsky and I. Kantner, Phy. Rev. Lett.
{\bf 57}, 2861 (1986)
\bibitem {HO1} D. ~Horn, Physica A{\bf 200}, 594 (1993)
\bibitem {ZE1} F. ~Zertuche, R. Lopes-Pena and H. Waelbroech, J. Phys.
A {\bf 27}, 5879 (1994)
\bibitem {SH1} T.L.H. ~ Watkin and D. Sherrington, J. Phys. A
{\bf 24}, 5427 (1991)
\bibitem {KO1}A. ~Kolmogorov, Dokl. Acad.
Nauk. SSSR {\bf 124}, 754 (1959)
\bibitem {SI1} Ya. G. ~Sinai, Dokl. Acad. Nauk. SSSR {\bf124}, 768
(1959)
\bibitem{Ho1}F.C. ~Hoppensteadt and E.M. ~Izhikevich, Biological
Cybernetics {\bf 75}, 117 (1997)
\bibitem{AO1} T. ~Aonishi, Osaka University preprint, arXiv:cond-mat/9808121 (1998)
\end{thebibliography}
\end{document}
\end

cond-mat0111151.extracted_bib
\begin{thebibliography}{}
\bibitem{JH1}J.J. ~Hopfield,Proc. Natl. Acad. Sci.  {\bf 79}, 2554 (1982)
\bibitem{JH2}J.J. ~Hopfield  and A.V.M. ~Herz, \textit {ibid} {\bf 92}, 6655 (1995)
\bibitem{AN1}P.W. ~Anderson, Mat. Res. Bull. {\bf 5}, 549 (1970)
\bibitem{ST1} D.L. ~Stein(ed.), \textit {Spin Glasses and Biology }(World Scientific,
1992)
\bibitem{AM1} D.J. ~Amit,H. Gutfreund and H. Sampolinsky, Phys.
Rev. Lett. {\bf 55}, 1530 (1985)
\bibitem{DE1} B. ~Derida, E. Gardner and A. Zippelius,  Europhys.
Lett. {\bf 4}, 167 (1987)
\bibitem {GU1}H. ~Gutfreund  and M. Mezard, Phys. Rev. Lett. {\bf 61},
 235 (1988)
\bibitem {SO1} H. ~Sompolinsky and I. Kantner, Phy. Rev. Lett.
{\bf 57}, 2861 (1986)
\bibitem {HO1} D. ~Horn, Physica A{\bf 200}, 594 (1993)
\bibitem {ZE1} F. ~Zertuche, R. Lopes-Pena and H. Waelbroech, J. Phys.
A {\bf 27}, 5879 (1994)
\bibitem {SH1} T.L.H. ~ Watkin and D. Sherrington, J. Phys. A
{\bf 24}, 5427 (1991)
\bibitem {KO1}A. ~Kolmogorov, Dokl. Acad.
Nauk. SSSR {\bf 124}, 754 (1959)
\bibitem {SI1} Ya. G. ~Sinai, Dokl. Acad. Nauk. SSSR {\bf124}, 768
(1959)
\bibitem{Ho1}F.C. ~Hoppensteadt and E.M. ~Izhikevich, Biological
Cybernetics {\bf 75}, 117 (1997)
\bibitem{AO1} T. ~Aonishi, Osaka University preprint, arXiv:cond-mat/9808121 (1998)
\end{thebibliography}