Entropy (information theory)

4889: 1249:. For example, consider the transmission of sequences comprising the 4 characters 'A', 'B', 'C', and 'D' over a binary channel. If all 4 letters are equally likely (25%), one cannot do better than using two bits to encode each letter. 'A' might code as '00', 'B' as '01', 'C' as '10', and 'D' as '11'. However, if the probabilities of each letter are unequal, say 'A' occurs with 70% probability, 'B' with 26%, and 'C' and 'D' with 2% each, one could assign variable length codes. In this case, 'A' would be coded as '0', 'B' as '10', 'C' as '110', and 'D' as '111'. With this representation, 70% of the time only one bit needs to be sent, 26% of the time two bits, and only 4% of the time 3 bits. On average, fewer than 2 bits are required since the entropy is lower (owing to the high prevalence of 'A' followed by 'B' – together 96% of characters). The calculation of the sum of probability-weighted log probabilities measures and captures this effect. 4282: 4884:{\displaystyle {\begin{aligned}&\operatorname {I} (p_{1}p_{2})&=\ &\operatorname {I} (p_{1})+\operatorname {I} (p_{2})&&\quad {\text{Starting from property 3}}\\&p_{2}\operatorname {I} '(p_{1}p_{2})&=\ &\operatorname {I} '(p_{1})&&\quad {\text{taking the derivative w.r.t}}\ p_{1}\\&\operatorname {I} '(p_{1}p_{2})+p_{1}p_{2}\operatorname {I} ''(p_{1}p_{2})&=\ &0&&\quad {\text{taking the derivative w.r.t}}\ p_{2}\\&\operatorname {I} '(u)+u\operatorname {I} ''(u)&=\ &0&&\quad {\text{introducing}}\,u=p_{1}p_{2}\\&(u\operatorname {I} '(u))'&=\ &0&&\quad {\text{combining terms into one}}\ \\&u\operatorname {I} '(u)-k&=\ &0&&\quad {\text{integrating w.r.t}}\ u,{\text{producing constant}}\,k\\\end{aligned}}} 14169: 17145: 17135: 3155: 561: 153: 43: 6260: 10459: 11594: 3475: 3744: 5926: 6012: 9034:. In practical use, this is generally not a problem, because one is usually only interested in compressing certain types of messages, such as a document in English, as opposed to gibberish text, or digital photographs rather than noise, and it is unimportant if a compression algorithm makes some unlikely or uninteresting sequences larger. 592:, and a receiver. The "fundamental problem of communication" – as expressed by Shannon – is for the receiver to be able to identify what data was generated by the source, based on the signal it receives through the channel. Shannon considered various ways to encode, compress, and transmit messages from a data source, and proved in his 10024: 11332: 3221: 9008:

scheme is lossless – one in which you can always recover the entire original message by decompression – then a compressed message has the same quantity of information as the original but communicated in fewer characters. It has more information (higher entropy) per character. A compressed message has

7064:

satisfying the above properties must be a constant multiple of Shannon entropy, with a non-negative constant. Compared to the previously mentioned characterizations of entropy, this characterization focuses on the properties of entropy as a function of random variables (subadditivity and additivity),

12370:

could take) have trivially low entropy and their sum would become big. But the key insight was showing a reduction in entropy by non negligible amounts as one expands H leading inturn to unbounded growth of a mathematical object over this random variable is equivalent to showing the unbounded growth

9203:

It is important not to confuse the above concepts. Often it is only clear from context which one is meant. For example, when someone says that the "entropy" of the English language is about 1 bit per character, they are actually modeling the English language as a stochastic process and talking about

9025:

than one bit of information per bit of message can be attained by employing a suitable coding scheme. The entropy of a message per bit multiplied by the length of that message is a measure of how much total information the message contains. Shannon's theorem also implies that no lossless compression

688:

The core idea of information theory is that the "informational value" of a communicated message depends on the degree to which the content of the message is surprising. If a highly likely event occurs, the message carries very little information. On the other hand, if a highly unlikely event occurs,

10725:

Although the analogy between both functions is suggestive, the following question must be set: is the differential entropy a valid extension of the Shannon discrete entropy? Differential entropy lacks a number of properties that the Shannon discrete entropy has – it can even be negative –

3217:

The entropy of the unknown result of the next toss of the coin is maximized if the coin is fair (that is, if heads and tails both have equal probability 1/2). This is the situation of maximum uncertainty as it is most difficult to predict the outcome of the next toss; the result of each toss of the

300:

quantifies the average level of uncertainty or information associated with the variable's potential states or possible outcomes. This measures the expected amount of information needed to describe the state of the variable, considering the distribution of probabilities across all potential states.

11318: 9332:

using exclusive or. If the pad has 1,000,000 bits of entropy, it is perfect. If the pad has 999,999 bits of entropy, evenly distributed (each individual bit of the pad having 0.999999 bits of entropy) it may provide good security. But if the pad has 999,999 bits of entropy, where the first bit is

9249:

and using it in place of the text of the book whenever one wants to refer to the book. This is enormously useful for talking about books, but it is not so useful for characterizing the information content of an individual book, or of language in general: it is not possible to reconstruct the book

9211:

If very large blocks are used, the estimate of per-character entropy rate may become artificially low because the probability distribution of the sequence is not known exactly; it is only an estimate. If one considers the text of every book ever published as a sequence, with each symbol being the

3748:

Uniform probability yields maximum uncertainty and therefore maximum entropy. Entropy, then, can only decrease from the value associated with uniform probability. The extreme case is that of a double-headed coin that never comes up tails, or a double-tailed coin that never results in a head. Then

8967:

Shannon's definition of entropy, when applied to an information source, can determine the minimum channel capacity required to reliably transmit the source as encoded binary digits. Shannon's entropy measures the information contained in a message as opposed to the portion of the message that is

8926:

of Shannon's information theory: the thermodynamic entropy is interpreted as being proportional to the amount of further Shannon information needed to define the detailed microscopic state of the system, that remains uncommunicated by a description solely in terms of the macroscopic variables of

3510: 1252:

English text, treated as a string of characters, has fairly low entropy; i.e. it is fairly predictable. We can be fairly certain that, for example, 'e' will be far more common than 'z', that the combination 'qu' will be much more common than any other combination with a 'q' in it, and that the

8945:(from 1961) and co-workers have shown, to function the demon himself must increase thermodynamic entropy in the process, by at least the amount of Shannon information he proposes to first acquire and store; and so the total thermodynamic entropy does not decrease (which resolves the paradox). 5738: 5475: 2180: 8931:. Adding heat to a system increases its thermodynamic entropy because it increases the number of possible microscopic states of the system that are consistent with the measurable values of its macroscopic variables, making any complete state description longer. (See article: 11754: 6255:{\displaystyle \mathrm {H} _{n}\left({\frac {1}{n}},\ldots ,{\frac {1}{n}}\right)=\mathrm {H} _{k}\left({\frac {b_{1}}{n}},\ldots ,{\frac {b_{k}}{n}}\right)+\sum _{i=1}^{k}{\frac {b_{i}}{n}}\,\mathrm {H} _{b_{i}}\left({\frac {1}{b_{i}}},\ldots ,{\frac {1}{b_{i}}}\right).} 8382: 5730: 8968:

determined (or predictable). Examples of the latter include redundancy in language structure or statistical properties relating to the occurrence frequencies of letter or word pairs, triplets etc. The minimum channel capacity can be realized in theory by using the

3501:, then there is less uncertainty. Every time it is tossed, one side is more likely to come up than the other. The reduced uncertainty is quantified in a lower entropy: on average each toss of the coin delivers less than one full bit of information. For example, if 15080:

Compare: Boltzmann, Ludwig (1896, 1898). Vorlesungen über Gastheorie : 2 Volumes – Leipzig 1895/98 UB: O 5262-6. English version: Lectures on gas theory. Translated by Stephen G. Brush (1964) Berkeley: University of California Press; (1995) New York: Dover

9043:

estimates the world's technological capacity to store and communicate optimally compressed information normalized on the most effective compression algorithms available in the year 2007, therefore estimating the entropy of the technologically available sources.

11139: 10454:{\displaystyle \eta (X)=-\sum _{i=1}^{n}{\frac {p(x_{i})\log _{b}(p(x_{i}))}{\log _{b}(n)}}=\sum _{i=1}^{n}{\frac {\log _{b}(p(x_{i})^{-p(x_{i})})}{\log _{b}(n)}}=\sum _{i=1}^{n}\log _{n}(p(x_{i})^{-p(x_{i})})=\log _{n}(\prod _{i=1}^{n}p(x_{i})^{-p(x_{i})}).} 9805: 11017: 10013: 9101:

The authors estimate humankind technological capacity to store information (fully entropically compressed) in 1986 and again in 2007. They break the information into three categories—to store information on a medium, to receive information through one-way

1082:

Entropy measures the expected (i.e., average) amount of information conveyed by identifying the outcome of a random trial. This implies that rolling a die has higher entropy than tossing a coin because each outcome of a die toss has smaller probability

11589:{\displaystyle {\begin{aligned}\sum _{i=-\infty }^{\infty }f(x_{i})\Delta &\to \int _{-\infty }^{\infty }f(x)\,dx=1\\\sum _{i=-\infty }^{\infty }f(x_{i})\Delta \log(f(x_{i}))&\to \int _{-\infty }^{\infty }f(x)\log f(x)\,dx.\end{aligned}}} 12736: 7575: 9278:. The first 128 symbols of the Fibonacci sequence has an entropy of approximately 7 bits/symbol, but the sequence can be expressed using a formula and this formula has a much lower entropy and applies to any length of the Fibonacci sequence. 13434: 11792:

in general a good measure of uncertainty or information. For example, the differential entropy can be negative; also it is not invariant under continuous co-ordinate transformations. This problem may be illustrated by a change of units when

11144: 3191:

Here, the entropy is at most 1 bit, and to communicate the outcome of a coin flip (2 possible values) will require an average of at most 1 bit (exactly 1 bit for a fair coin). The result of a fair die (6 possible values) would have entropy

14156:

loss, that minimizes the average cross entropy between ground truth and predicted distributions. In general, cross entropy is a measure of the differences between two datasets similar to the KL divergence (also known as relative entropy).

10697: 7289: 6836: 9573: 7870: 615:. The analogy results when the values of the random variable designate energies of microstates, so Gibbs's formula for the entropy is formally identical to Shannon's formula. Entropy has relevance to other areas of mathematics such as 12238: 11923: 7148:

The Shannon entropy satisfies the following properties, for some of which it is useful to interpret entropy as the expected amount of information learned (or uncertainty eliminated) by revealing the value of a random variable

3470:{\displaystyle {\begin{aligned}\mathrm {H} (X)&=-\sum _{i=1}^{n}{p(x_{i})\log _{b}p(x_{i})}\\&=-\sum _{i=1}^{2}{{\frac {1}{2}}\log _{2}{\frac {1}{2}}}\\&=-\sum _{i=1}^{2}{{\frac {1}{2}}\cdot (-1)}=1.\end{aligned}}} 9265:

The Fibonacci sequence is 1, 1, 2, 3, 5, 8, 13, .... treating the sequence as a message and each number as a symbol, there are almost as many symbols as there are characters in the message, giving an entropy of approximately

3062: 4058: 13794: 8910:

times the Shannon entropy), Boltzmann's equation results. In information theoretic terms, the information entropy of a system is the amount of "missing" information needed to determine a microstate, given the macrostate.

2006: 1692: 5960:

elements each, the entropy of the whole ensemble should be equal to the sum of the entropy of the system of boxes and the individual entropies of the boxes, each weighted with the probability of being in that particular

481: 12945: 1229:, when the event outcome is known ahead of time, and the entropy is zero bits. When the entropy is zero bits, this is sometimes referred to as unity, where there is no uncertainty at all – no freedom of choice – no 10889: 1253:

combination 'th' will be more common than 'z', 'q', or 'qu'. After the first few letters one can often guess the rest of the word. English text has between 0.6 and 1.3 bits of entropy per character of the message.

15115:"Translation of Ludwig Boltzmann's Paper "On the Relationship between the Second Fundamental Theorem of the Mechanical Theory of Heat and Probability Calculations Regarding the Conditions for Thermal Equilibrium"" 13904: 8183: 6581: 568:— with two coins there are four possible outcomes, and two bits of entropy. Generally, information entropy is the average amount of information conveyed by an event, when considering all possible outcomes. 13551: 9419: 3739:{\displaystyle {\begin{aligned}\mathrm {H} (X)&=-p\log _{2}(p)-q\log _{2}(q)\\&=-0.7\log _{2}(0.7)-0.3\log _{2}(0.3)\\&\approx -0.7\cdot (-0.515)-0.3\cdot (-1.737)\\&=0.8816<1.\end{aligned}}} 5921:{\displaystyle \mathrm {H} _{n}{\bigg (}\underbrace {{\frac {1}{n}},\ldots ,{\frac {1}{n}}} _{n}{\bigg )}<\mathrm {H} _{n+1}{\bigg (}\underbrace {{\frac {1}{n+1}},\ldots ,{\frac {1}{n+1}}} _{n+1}{\bigg )}.} 9839:

with a non-uniform distribution will have less entropy than the same set with a uniform distribution (i.e. the "optimized alphabet"). This deficiency in entropy can be expressed as a ratio called efficiency:

6681: 2915: 7396: 5314: 9192:(The "rate of self-information" can also be defined for a particular sequence of messages or symbols generated by a given stochastic process: this will always be equal to the entropy rate in the case of a 8728: 1078: 11025: 8996:

of a data source is the average number of bits per symbol needed to encode it. Shannon's experiments with human predictors show an information rate between 0.6 and 1.3 bits per character in English; the

11613: 8785:. In classical thermodynamics, entropy is defined in terms of macroscopic measurements and makes no reference to any probability distribution, which is central to the definition of information entropy. 7031: 10900: 9262:

that outputs the sequence. A code that achieves the entropy rate of a sequence for a given model, plus the codebook (i.e. the probabilistic model), is one such program, but it may not be the shortest.

1518: 12279:

as measures. It is defined for any measure space, hence coordinate independent and invariant under co-ordinate reparameterizations if one properly takes into account the transformation of the measure

14433: 12538: 5214:) does not matter in the definition of entropy. Entropy only takes into account the probability of observing a specific event, so the information it encapsulates is information about the underlying 7931: 8598: 8105: 6903: 2632: 8252: 875: 11337: 9250:

from its identifier without knowing the probability distribution, that is, the complete text of all the books. The key idea is that the complexity of the probabilistic model must be considered.

5617: 4287: 3515: 3226: 14138:

distributions. The idea is that the distribution that best represents the current state of knowledge of a system is the one with the largest entropy, and is therefore suitable to be the prior.

13649: 8025: 13285: 9254:

is a theoretical generalization of this idea that allows the consideration of the information content of a sequence independent of any particular probability model; it considers the shortest

14516: 6398: 4949: 2533: 4127: 1907: 3100: 2700: 12309:

Intuitively the idea behind the proof was if there is low information in terms of the Shannon entropy between consecutive random variables (here the random variable is defined using the

9645: 5047: 2263: 391: 13997:

techniques arise largely from statistics and also information theory. In general, entropy is a measure of uncertainty and the objective of machine learning is to minimize uncertainty.

9846: 2743: 6452: 13984: 7690: 1001: 693:

be the winning number of a lottery provides very little information, because any particular chosen number will almost certainly not win. However, knowledge that a particular number

2322: 1592: 13167: 13001: 11958: 8842: 8458: 7736: 1366: 7109: 6949: 5536: 4983: 1825: 4080: 3973: 1418: 5612: 2412: 8217: 4275: 2784: 10507: 9837: 2001: 1977: 1318: 343: 12570: 12357: 7456: 10568: 7134: 7062: 2442: 1542: 678: 8744:

At an everyday practical level, the links between information entropy and thermodynamic entropy are not evident. Physicists and chemists are apt to be more interested in

8426: 5580: 4228: 12397: 11835: 1213: 13326: 6479: 6404:

and related quantities inherit simple relation, in turn. The measure theoretic definition in the previous section defined the entropy as a sum over expected surprisals

2986: 14468: 9622: 3978: 2816: 14376: 14042: 11990: 10784: 9319: 501: 14124:. The information gain is used to identify which attributes of the dataset provide the most information and should be used to split the nodes of the tree optimally. 10576: 7616: 7168: 6715: 5111: 4185: 4153: 1795: 9128:. A diversity index is a quantitative statistical measure of how many different types exist in a dataset, such as species in a community, accounting for ecological 5085: 1600: 1149: 1115: 895: 521: 9457: 7747: 3120: 2654: 820: 791: 762: 9321:

guesses to break by brute force. Entropy fails to capture the number of guesses required if the possible keys are not chosen uniformly. Instead, a measure called

6707: 6607: 5009: 396: 14122: 14102: 14082: 14062: 8865: 8241: 7636: 3144: 2981: 2961: 2937: 2836: 2559: 2470: 2362: 2342: 1953: 1933: 932: 733: 319: 8949:

imposes a lower bound on the amount of heat a computer must generate to process a given amount of information, though modern computers are far less efficient.

10800: 1294: 12091: 14922: 11313:{\displaystyle \mathrm {H} ^{\Delta }=-\sum _{i=-\infty }^{\infty }f(x_{i})\Delta \log(f(x_{i}))-\sum _{i=-\infty }^{\infty }f(x_{i})\Delta \log(\Delta ).} 6454:

for an extremal partition. Here the logarithm is ad hoc and the entropy is not a measure in itself. At least in the information theory of a binary string,

15008: 14672: 14607: 2841: 15838: 14614: 13656: 11817:. The argument of the logarithm must be dimensionless, otherwise it is improper, so that the differential entropy as given above will be improper. If 14679: 8777:

for even tiny amounts of substances in chemical and physical processes represent amounts of entropy that are extremely large compared to anything in

15684: 13799:

Rearranging gives the upper bound. For the lower bound one first shows, using some algebra, that it is the largest term in the summation. But then,

3749:

there is no uncertainty. The entropy is zero: each toss of the coin delivers no new information as the outcome of each coin toss is always certain.

564:

Two bits of entropy: In the case of two fair coin tosses, the information entropy in bits is the base-2 logarithm of the number of possible outcomes

8492: 1006: 6492:

to that of subsets of a universal set. Information is quantified as "dits" (distinctions), a measure on partitions. "Dits" can be converted into

14302:– a generalization of Shannon entropy; it is one of a family of functionals for quantifying the diversity, uncertainty or randomness of a system. 15301:"A tribute to Claude Shannon (1916–2001) and a plea for more rigorous use of species richness, species diversity and the 'Shannon–Wiener' Index" 12806: 1425: 14947: 10534:

The Shannon entropy is restricted to random variables taking discrete values. The corresponding formula for a continuous random variable with

15969: 60: 13805: 15610: 15368: 13287:. Now use this to bound the right side of Shearer's inequality and exponentiate the opposite sides of the resulting inequality you obtain. 8117: 6515: 2564: 275: 14197:– is a measure of the average number of bits needed to identify an event from a set of possibilities between two probability distributions 825: 16637: 16448: 13445: 11992:, which would in general be infinite. This is expected: continuous variables would typically have infinite entropy when discretized. The 9351: 9294:

is unmeasurable. For example, a 128-bit key that is uniformly and randomly generated has 128 bits of entropy. It also takes (on average)

8941:

can (hypothetically) reduce the thermodynamic entropy of a system by using information about the states of individual molecules; but, as

8867:

is the thermodynamic entropy of a particular macrostate (defined by thermodynamic parameters such as temperature, volume, energy, etc.),

627:

establishing that entropy should be a measure of how informative the average outcome of a variable is. For a continuous random variable,

107: 5470:{\displaystyle \mathrm {H} _{n}\left(p_{1},p_{2},\ldots p_{n}\right)=\mathrm {H} _{n}\left(p_{i_{1}},p_{i_{2}},\ldots ,p_{i_{n}}\right)} 14746: 11996:

is really a measure of how much easier a distribution is to describe than a distribution that is uniform over its quantization scheme.

6615: 2175:{\displaystyle \mathrm {H} (X|Y)=-\sum _{x,y\in {\mathcal {X}}\times {\mathcal {Y}}}p_{X,Y}(x,y)\log {\frac {p_{X,Y}(x,y)}{p_{Y}(y)}},} 79: 16337: 12359:. And in an interval the sum over that interval could become arbitrary large. For example, a sequence of +1's (which are values of 7325: 2475: 11749:{\displaystyle h=\lim _{\Delta \to 0}\left(\mathrm {H} ^{\Delta }+\log \Delta \right)=-\int _{-\infty }^{\infty }f(x)\log f(x)\,dx,} 8664: 7316:: it is attained by the uniform probability distribution. That is, uncertainty is maximal when all possible events are equiprobable: 16843: 16666: 16460: 8873:

is the number of microstates (various combinations of particles in various energy states) that can yield the given macrostate, and

1848: 12378:

The proof is quite involved and it brought together breakthroughs not just in novel use of Shannon Entropy, but also its used the

86: 16151: 9166:

of an individual message or symbol taken from a given probability distribution (message or sequence seen as an individual event),

6957: 16848: 16425: 15245: 14638: 14573: 3210:

Consider tossing a coin with known, not necessarily fair, probabilities of coming up heads or tails; this can be modelled as a

577: 14381: 12447: 12394: 7429:

simultaneously) is equal to the information revealed by conducting two consecutive experiments: first evaluating the value of

15802: 15778: 15764: 15747: 15694: 15604: 15543: 15449: 14878: 14819: 14792: 8377:{\displaystyle \mathrm {H} (\lambda p_{1}+(1-\lambda )p_{2})\geq \lambda \mathrm {H} (p_{1})+(1-\lambda )\mathrm {H} (p_{2})} 15887: 7879: 5725:{\displaystyle \mathrm {H} _{n}(p_{1},\ldots ,p_{n})\leq \mathrm {H} _{n}\left({\frac {1}{n}},\ldots ,{\frac {1}{n}}\right)} 937: 93: 16578: 15405: 15276: 11829:(i.e. "bin size") and therefore has the same units, then a modified differential entropy may be written in proper form as: 8632: 8534: 8054: 6844: 5293:, so that changing the values of the probabilities by a very small amount should only change the entropy by a small amount. 608: 15661: 15222: 9156:

There are a number of entropy-related concepts that mathematically quantify information content of a sequence or message:

14700: 11993: 11783: 10727: 9333:

fixed and the remaining 999,999 bits are perfectly random, the first bit of the ciphertext will not be encrypted at all.

207: 15877: 14968: 14104:, quantifies the expected information, or the reduction in entropy, from additionally knowing the value of an attribute 13569: 7971: 16955: 16693: 16632: 16443: 16393: 16216: 15850: 15686:

The Cross-Entropy Method: A Unified Approach to Combinatorial Optimization, Monte-Carlo Simulation and Machine Learning

14550: 14246: 14141: 14004: 9345:

of text. For an order-0 source (each character is selected independent of the last characters), the binary entropy is:

9014: 8958: 8628: 600:

compressed onto a perfectly noiseless channel. Shannon strengthened this result considerably for noisy channels in his

593: 234: 75: 13188: 9445:(one in which the probability of selecting a character is dependent only on the immediately preceding character), the 16076: 16061: 15962: 15827: 15481: 15086: 14740: 14473: 11134:{\displaystyle \mathrm {H} ^{\Delta }:=-\sum _{i=-\infty }^{\infty }f(x_{i})\Delta \log \left(f(x_{i})\Delta \right)} 8501:

in information theory came from the close resemblance between Shannon's formula and very similar known formulae from

6332: 5735:

Increasing number of outcomes: for equiprobable events, the entropy should increase with the number of outcomes i.e.

4901: 218: 126: 9800:{\displaystyle \mathrm {H} ({\mathcal {S}})=-\sum _{i}p_{i}\sum _{j}p_{i}(j)\sum _{k}p_{i,j}(k)\ \log \ p_{i,j}(k).} 9328:

Other problems may arise from non-uniform distributions used in cryptography. For example, a 1,000,000-digit binary

17179: 17068: 11012:{\displaystyle \int _{-\infty }^{\infty }f(x)\,dx=\lim _{\Delta \to 0}\sum _{i=-\infty }^{\infty }f(x_{i})\Delta ,} 10468:. This formulation is also referred to as the normalized entropy, as the entropy is divided by the maximum entropy 6505: 4085: 697:

win a lottery has high informational value because it communicates the occurrence of a very low probability event.

268: 14208:– a coding scheme that assigns codes to symbols so as to match code lengths with the probabilities of the symbols. 12386: 12243:

In this form the relative entropy generalizes (up to change in sign) both the discrete entropy, where the measure

10008:{\displaystyle \eta (X)={\frac {H}{H_{max}}}=-\sum _{i=1}^{n}{\frac {p(x_{i})\log _{b}(p(x_{i}))}{\log _{b}(n)}}.} 3067: 2666: 17194: 17189: 17078: 16916: 16767: 16686: 16480: 15936: 15863: 12015: 9010: 6315: 5014: 1245:

Information theory is useful to calculate the smallest amount of information required to convey a message, as in

17: 14913: 12285:. The relative entropy, and (implicitly) entropy and differential entropy, do depend on the "reference" measure 5177:

bit of information, which is approximately 0.693 nats or 0.301 decimal digits. Because of additivity,

2185: 348: 17051: 16671: 16465: 16253: 8933: 8793: 8114:

The entropy of two simultaneous events is no more than the sum of the entropies of each individual event i.e.,

4241: 2705: 192: 64: 6407: 3831:: an increase in the probability of an event decreases the information from an observed event, and vice versa. 17148: 16184: 15908: 14960:

Chakrabarti, C. G., and Indranil Chakrabarty. "Shannon entropy: axiomatic characterization and application."

14643: 14578: 14241: 12428: 9290:, entropy is often roughly used as a measure of the unpredictability of a cryptographic key, though its real 5166: 16813: 15918: 13940: 12407:

While the use of Shannon Entropy in the proof is novel it is likely to open new research in this direction.

11759:

which is, as said before, referred to as the differential entropy. This means that the differential entropy

8901:. When these probabilities are substituted into the above expression for the Gibbs entropy (or equivalently 7641: 6484:

Motivated by such relations, a plethora of related and competing quantities have been defined. For example,

17138: 16583: 16141: 15955: 15833: 14985: 14256: 14131: 12005: 10535: 8749: 2268: 1565: 1215:(about 1.58496) bits of information because it can have one of three values.) The minimum surprise is when 601: 244: 14676: 14611: 13119: 12953: 12372: 12303: 11937: 11770:. Rather, it differs from the limit of the Shannon entropy by an infinite offset (see also the article on 8802: 8431: 7695: 1323: 17184: 16131: 16126: 15903: 12010:

Another useful measure of entropy that works equally well in the discrete and the continuous case is the

8886:. It is assumed that each microstate is equally likely, so that the probability of a given microstate is 7068: 6908: 5483: 4954: 1800: 261: 249: 4065: 3958: 1371: 100: 17073: 17000: 16838: 16818: 16762: 16420: 16211: 16014: 15882: 14003:

algorithms use relative entropy to determine the decision rules that govern the data at each node. The

9218:

published books, and each book is only published once, the estimate of the probability of each book is

9148:

index with parameter equal to 1. The Shannon index is related to the proportional abundances of types.

9107: 8509: 2445: 1174:, for which one outcome is not expected over the other. In this case a coin flip has an entropy of one 612: 223: 12731:{\displaystyle P_{i}(A)=\{(x_{1},\ldots ,x_{i-1},x_{i+1},\ldots ,x_{d}):(x_{1},\ldots ,x_{d})\in A\}.} 8748:

in entropy as a system spontaneously evolves away from its initial conditions, in accordance with the

7570:{\displaystyle \mathrm {H} (X,Y)=\mathrm {H} (X|Y)+\mathrm {H} (Y)=\mathrm {H} (Y|X)+\mathrm {H} (X).} 5588: 2379: 17174: 17083: 17024: 16950: 16798: 16388: 16383: 16238: 16081: 15591:. Advances in Intelligent Systems and Computing. Vol. 652. Singapore: Springer. pp. 31–36. 14149: 9197: 8998: 8988:.) In practice, compression algorithms deliberately include some judicious redundancy in the form of 8191: 2748: 1715: 10471: 9818: 8788:

The connection between thermodynamics and what is now known as information theory was first made by

4260: 1982: 1958: 1299: 324: 17088: 16661: 16455: 16156: 14293: 14216: 14189: 12327: 5215: 3822: 1274: 596:

that the entropy represents an absolute mathematical limit on how well data from the source can be

31: 13429:{\displaystyle {\frac {2^{n\mathrm {H} (q)}}{n+1}}\leq {\tbinom {n}{k}}\leq 2^{n\mathrm {H} (q)},} 10551: 7117: 7045: 2421: 1525: 822:

is close to 0, the surprisal of the event is high. This relationship is described by the function

634: 17029: 16400: 16287: 16243: 16056: 16039: 16029: 14200: 12742: 8946: 8391: 6326: 5541: 4190: 3201: 53: 15584: 15357: 10702:

This is the differential entropy (or continuous entropy). A precursor of the continuous entropy

4277:

be the information function which one assumes to be twice continuously differentiable, one has:

1185: 16654: 16405: 16189: 16034: 15038:

Aczél, J.; Forte, B.; Ng, C. T. (1974). "Why the Shannon and Hartley entropies are 'natural'".

14284: 14000: 10692:{\displaystyle \mathrm {H} (X)=\mathbb {E} =-\int _{\mathbb {X} }f(x)\log f(x)\,\mathrm {d} x.} 10570:

on the real line is defined by analogy, using the above form of the entropy as an expectation:

8738: 7284:{\displaystyle \mathrm {H} _{n+1}(p_{1},\ldots ,p_{n},0)=\mathrm {H} _{n}(p_{1},\ldots ,p_{n})} 6831:{\displaystyle \mathrm {H} _{n+1}(p_{1},\ldots ,p_{n},0)=\mathrm {H} _{n}(p_{1},\ldots ,p_{n})} 6457: 1179: 16926: 15289:, 332(6025); free access to the article through here: martinhilbert.net/WorldInfoCapacity.html 14438: 9591: 2789: 689:

the message is much more informative. For instance, the knowledge that some particular number

17058: 15898: 14809: 14782: 14349: 14288: 14280: 14009: 12555: 12031: 11963: 11771: 10769: 10465: 9568:{\displaystyle \mathrm {H} ({\mathcal {S}})=-\sum _{i}p_{i}\sum _{j}\ p_{i}(j)\log p_{i}(j),} 9297: 9291: 9251: 8985: 8919: 8915: 8502: 7865:{\displaystyle \mathrm {H} (X)+\mathrm {H} (f(X)|X)=\mathrm {H} (f(X))+\mathrm {H} (X|f(X)),} 6489: 4895: 1701: 589: 486: 14730: 7586: 5090: 4164: 4132: 1765: 16742: 16204: 16166: 15987: 15312: 15159: 14265: 14251: 13063:

occurs with equal probability. Then (by the further properties of entropy mentioned above)

10529: 9031: 8522: 7933:, the entropy of a variable can only decrease when the latter is passed through a function. 5123: 5052: 1120: 1086: 880: 628: 506: 187: 167: 14665: 9124:

Entropy is one of several ways to measure biodiversity, and is applied in the form of the

9030:

messages. If some messages come out shorter, at least one must come out longer due to the

3105: 2639: 796: 767: 738: 8: 16973: 16864: 16823: 16808: 16777: 16772: 16681: 16588: 16521: 16490: 16475: 16258: 14183: 14145: 9137: 8651: 6686: 6586: 6401: 5290: 4988: 4237: 3807: 3479:

However, if we know the coin is not fair, but comes up heads or tails with probabilities

1912: 1842: 1230: 702: 172: 15316: 15163: 12269:

is itself a probability distribution, the relative entropy is non-negative, and zero if

10740:

goes to zero. In the discrete case, the bin size is the (implicit) width of each of the

10018:

Applying the basic properties of the logarithm, this quantity can also be expressed as:

2324:. This quantity should be understood as the remaining randomness in the random variable 17046: 17016: 16995: 16901: 16833: 16727: 16415: 16231: 16221: 16116: 16096: 16091: 15713: 15535: 15517: 15338: 15175: 15063: 15055: 14941: 14835: 14726: 14656: 14591: 14270: 14236: 14221: 14211: 14174: 14127: 14107: 14087: 14067: 14047: 12379: 12310: 11788:

It turns out as a result that, unlike the Shannon entropy, the differential entropy is

9259: 9193: 9185: 8938: 8928: 8883: 8850: 8753: 8613: 8226: 7621: 6322: 3872: 3129: 2966: 2946: 2922: 2821: 2657: 2544: 2455: 2347: 2327: 1938: 1918: 917: 718: 585: 304: 289: 182: 144: 27:

Expected amount of information needed to specify the output of a stochastic data source

16627: 12233:{\displaystyle D_{\mathrm {KL} }(p\|m)=\int \log(f(x))p(dx)=\int f(x)\log(f(x))m(dx).} 7114:

It is worth noting that if we drop the "small for small probabilities" property, then

16990: 16978: 16960: 16828: 16712: 16649: 16495: 16410: 16366: 16327: 16009: 15929:—repository of implementations of Shannon entropy in different programming languages. 15846: 15823: 15798: 15774: 15760: 15752: 15743: 15690: 15653: 15600: 15445: 15330: 15325: 15300: 15285: 15214: 15082: 15067: 14874: 14815: 14788: 14736: 14546: 14168: 14135: 10733:

To answer this question, a connection must be established between the two functions:

9039: 8981: 8782: 7159:

Adding or removing an event with probability zero does not contribute to the entropy:

5971: 5156: 5145: 3211: 3205: 2415: 536: 15539: 15342: 15249: 15179: 905:

is the only function that satisfies а specific set of conditions defined in section

16965: 16921: 16894: 16889: 16747: 16732: 16642: 16551: 16546: 16375: 16108: 16086: 15978: 15645: 15592: 15573:

Probability and Computing, M. Mitzenmacher and E. Upfal, Cambridge University Press

15527: 15435: 15320: 15274:"The World's Technological Capacity to Store, Communicate, and Compute Information" 15206: 15167: 15147: 15126: 15047: 15000: 14660: 14652: 14595: 14587: 14260: 14231: 14205: 13994: 12260: 12250: 11918:{\displaystyle \mathrm {H} =\int _{-\infty }^{\infty }f(x)\log(f(x)\,\Delta )\,dx,} 9255: 9162: 9133: 9129: 9017:

states a lossless compression scheme cannot compress messages, on average, to have

9005: 8962: 8789: 8778: 8655: 8640: 8636: 8468: 8464: 8220: 5131: 3170: 1553: 1279: 1246: 620: 554: 239: 197: 15587:. In Panigrahi, Bijaya Ketan; Hoda, M. N.; Sharma, Vinod; Goel, Shivendra (eds.). 15509: 14299: 3057:{\displaystyle \mathrm {H} _{\mu }(M)=\sup _{P\subseteq M}\mathrm {H} _{\mu }(P).} 16884: 16698: 16622: 16603: 16573: 16541: 16507: 16066: 16004: 15940: 15922: 15842: 15792: 15731: 15486: 15477: 15280: 14972: 14683: 14618: 14540: 12401: 12390: 9125: 9119: 8647: 7137: 6488:'s analysis of a "logic of partitions" defines a competing measure in structures 4053:{\displaystyle \operatorname {I} (p)=\log \left({\tfrac {1}{p}}\right)=-\log(p).} 2818:. (This is a relaxation of the usual conditions for a partition.) The entropy of 297: 15944:

an interdisciplinary journal on all aspects of the entropy concept. Open access.

15771:

Information Measures: Information and its Description in Science and Engineering

15596: 15394: 15273: 13789:{\displaystyle \sum _{i=0}^{n}{\tbinom {n}{i}}q^{i}(1-q)^{n-i}=(q+(1-q))^{n}=1.} 6499: 16676: 16470: 16199: 16194: 16051: 16024: 15996: 15811: 15633: 15194: 14634: 14569: 14310: 12313:(which is a useful mathematical function for studying distribution of primes) 9145: 8973: 8752:, rather than an unchanging probability distribution. As the minuteness of the 8734: 6493: 6485: 3167: 2373: 1545: 573: 550: 546: 532: 14696: 14600: 10515:, as indicated by the insensitivity within the final logarithm above thereto. 8927:

classical thermodynamics, with the constant of proportionality being just the

7065:

rather than the properties of entropy as a function of the probability vector

1687:{\displaystyle \mathrm {H} (X)=-\sum _{x\in {\mathcal {X}}}p(x)\log _{b}p(x),} 17168: 16983: 16931: 16598: 16593: 16568: 16500: 16121: 16019: 15815: 15735: 15657: 15649: 15465: 15402:

Proceedings of the Information Technology & Telecommunications Conference

15334: 15218: 14965: 14346:

This definition allows events with probability 0, resulting in the undefined

14316: 14226: 14194: 14153: 13925:

A nice interpretation of this is that the number of binary strings of length

12416: 9442: 9287: 9171: 8942: 8526: 5226:

Another characterization of entropy uses the following properties. We denote

2940: 616: 177: 15440: 15171: 15004: 14986:"Logical Information Theory: New Logical Foundations for Information Theory" 10757:. As the continuous domain is generalized, the width must be made explicit. 6838:, i.e., adding an outcome with probability zero does not change the entropy. 6504:

Another succinct axiomatic characterization of Shannon entropy was given by

503:

denotes the sum over the variable's possible values. The choice of base for

17104: 16071: 16046: 15947: 15926: 14326: 11607:, requires a special definition of the differential or continuous entropy: 10509:. Furthermore, the efficiency is indifferent to choice of (positive) base 9447: 9342: 9329: 9180: 9001:

can achieve a compression ratio of 1.5 bits per character in English text.

8993: 3885: 476:{\displaystyle \mathrm {H} (X):=-\sum _{x\in {\mathcal {X}}}p(x)\log p(x),} 202: 12940:{\displaystyle \mathrm {H} \leq {\frac {1}{r}}\sum _{i=1}^{n}\mathrm {H} } 901:, which gives 0 surprise when the probability of the event is 1. In fact, 17063: 16941: 16737: 16613: 16563: 14321: 12299: 9175:

of the symbols forming the message or sequence (seen as a set of events),

8969: 7136:

must be a non-negative linear combination of the Shannon entropy and the

6321:

The characterization here imposes an additive property with respect to a

5478: 5169:

of each other. For instance, in case of a fair coin toss, heads provides

14893: 3173:) of a coin flip, measured in bits, graphed versus the bias of the coin 17120: 16911: 16906: 16793: 16752: 16558: 15790: 15709: 15210: 15059: 14305: 14275: 12383: 10884:{\displaystyle f(x_{i})\Delta =\int _{i\Delta }^{(i+1)\Delta }f(x)\,dx} 5152: 5141: 2661: 1749: 1735: 1724: 560: 542: 152: 15933: 15131: 15114: 13899:{\displaystyle {\binom {n}{k}}q^{qn}(1-q)^{n-nq}\geq {\frac {1}{n+1}}} 6400:. Observe that a logarithm mediates between these two operations. The 3154: 15522: 15434:. Lecture Notes in Computer Science. Vol. 1758. pp. 62–77. 10719: 10715: 9325:

can be used to measure the effort required for a brute force attack.

9200:

are also used to compare or relate different sources of information.

9103: 8646:

The Gibbs entropy translates over almost unchanged into the world of

8472: 8178:{\displaystyle \mathrm {H} (X,Y)\leq \mathrm {H} (X)+\mathrm {H} (Y)} 6576:{\displaystyle \mathrm {H} (X,Y)\leq \mathrm {H} (X)+\mathrm {H} (Y)} 3800:. The amount of information acquired due to the observation of event 2449: 1705: 1262: 898: 524: 15531: 15051: 42: 17034: 16879: 16536: 15100:

Geometry of Quantum States: An Introduction to Quantum Entanglement

13546:{\displaystyle \mathrm {H} (q)=-q\log _{2}(q)-(1-q)\log _{2}(1-q).} 10737: 9414:{\displaystyle \mathrm {H} ({\mathcal {S}})=-\sum p_{i}\log p_{i},} 9049: 9021:

than one bit of information per bit of message, but that any value

8989: 597: 3879:

Given two independent events, if the first event can yield one of

16803: 16277: 16226: 15915: 14712: 9245:. As a practical code, this corresponds to assigning each book a 8513: 6676:{\displaystyle \mathrm {H} (X,Y)=\mathrm {H} (X)+\mathrm {H} (Y)} 3806:

follows from Shannon's solution of the fundamental properties of

2910:{\displaystyle \mathrm {H} _{\mu }(P)=\sum _{A\in P}h_{\mu }(A).} 7950:

are two independent random variables, then knowing the value of

7405:

The entropy or the amount of information revealed by evaluating

914:

Hence, we can define the information, or surprisal, of an event

16317: 7391:{\displaystyle \mathrm {H} (p_{1},\dots ,p_{n})\leq \log _{b}n} 15195:"Irreversibility and Heat Generation in the Computing Process" 14962:

International Journal of Mathematics and Mathematical Sciences

8723:{\displaystyle S=-k_{\text{B}}\,{\rm {Tr}}(\rho \ln \rho )\,,} 8185:, with equality if and only if the two events are independent. 5614:

should be maximal if all the outcomes are equally likely i.e.

527:, varies for different applications. Base 2 gives the unit of 17152: 16757: 16350: 16297: 15708:

This article incorporates material from Shannon's entropy on

14296:– a measure of distinguishability between two quantum states. 12302:

used entropy to make a useful connection trying to solve the

11019:

where this limit and "bin size goes to zero" are equivalent.

10746:(finite or infinite) bins whose probabilities are denoted by 10464:

Efficiency has utility in quantifying the effective use of a

6500:

Alternative characterization via additivity and subadditivity

3900:

equiprobable outcomes of the joint event. This means that if

1073:{\displaystyle I(E)=\log _{2}\left({\frac {1}{p(E)}}\right).} 624: 14868: 13045:

We sketch how Loomis–Whitney follows from this: Indeed, let

11928:

and the result will be the same for any choice of units for

6294:

this implies that the entropy of a certain outcome is zero:

16307: 16161: 16146: 16136: 14897: 14807: 13916:

terms in the summation. Rearranging gives the lower bound.

9246: 7962:(since the two don't influence each other by independence): 7026:{\displaystyle \lim _{q\to 0^{+}}\mathrm {H} _{2}(1-q,q)=0} 5970:

The rule of additivity has the following consequences: for

3218:

coin delivers one full bit of information. This is because

607:

Entropy in information theory is directly analogous to the

14518:, not allowing events with probability equal to exactly 0. 14044:, which is equal to the difference between the entropy of 13051:

be a uniformly distributed random variable with values in

10518: 6496:, to get the formulas for conditional entropy, and so on. 3837:: events that always occur do not communicate information. 1513:{\displaystyle \mathrm {H} (X)=\mathbb {E} =\mathbb {E} .} 16282: 16248: 13031:(so the dimension of this vector is equal to the size of 12085:-integral 1, then the relative entropy can be defined as 9281: 8977: 5936:

uniformly distributed elements that are partitioned into

5127: 1723:, and 10, and the corresponding units of entropy are the 1270: 1175: 793:

is close to 1, the surprisal of the event is low, but if

528: 15432:

International Workshop on Selected Areas in Cryptography

15365:

Proc. IEEE International Symposium on Information Theory

14836:

Information theory primer with an appendix on logarithms

14428:{\displaystyle \lim \limits _{x\rightarrow 0}x\log(x)=0} 13290: 12533:{\displaystyle |A|^{d-1}\leq \prod _{i=1}^{d}|P_{i}(A)|} 12427:

A simple example of this is an alternative proof of the

9341:

A common way to define entropy for text is based on the

9188:(message or sequence is seen as a succession of events). 8486: 6302:. This implies that the efficiency of a source set with 15683:

Rubinstein, Reuven Y.; Kroese, Dirk P. (9 March 2013).

14470:

equals 0 in this context. Alternatively one can define

588:

system composed of three elements: a source of data, a

15430:

Pliam, John (1999). "Selected Areas in Cryptography".

14732:

Information Theory, Inference, and Learning Algorithms

13682: 13574: 13373: 11777: 9639:

For a second order Markov source, the entropy rate is

7926:{\displaystyle \mathrm {H} (f(X))\leq \mathrm {H} (X)} 4263: 4011: 3875:

is the sum of the information learned from each event.

1282: 15757:

Information Theory, Inference and Learning Algorithms

14476: 14441: 14384: 14352: 14110: 14090: 14070: 14050: 14012: 13943: 13808: 13659: 13572: 13448: 13329: 13191: 13122: 12956: 12809: 12573: 12450: 12330: 12094: 11966: 11940: 11838: 11616: 11335: 11147: 11028: 10903: 10803: 10772: 10736:

In order to obtain a generally finite measure as the

10579: 10554: 10474: 10027: 9849: 9821: 9648: 9594: 9460: 9354: 9300: 9106:

networks, or to exchange information through two-way

8853: 8805: 8667: 8593:{\displaystyle S=-k_{\text{B}}\sum p_{i}\ln p_{i}\,,} 8537: 8434: 8394: 8255: 8229: 8194: 8120: 8100:{\displaystyle \mathrm {H} (X|Y)\leq \mathrm {H} (X)} 8057: 7974: 7882: 7750: 7698: 7644: 7624: 7589: 7459: 7328: 7171: 7120: 7071: 7048: 6960: 6911: 6898:{\displaystyle \mathrm {H} _{n}(p_{1},\ldots ,p_{n})} 6847: 6718: 6689: 6618: 6589: 6518: 6460: 6410: 6335: 6015: 5741: 5620: 5591: 5544: 5486: 5317: 5093: 5055: 5017: 4991: 4957: 4904: 4285: 4193: 4167: 4135: 4088: 4068: 3981: 3961: 3513: 3224: 3132: 3108: 3070: 2989: 2969: 2949: 2925: 2844: 2824: 2792: 2751: 2708: 2669: 2642: 2627:{\displaystyle h_{\mu }(A)=\mu (A)\sigma _{\mu }(A).} 2567: 2547: 2478: 2458: 2424: 2382: 2350: 2330: 2271: 2188: 2009: 1985: 1961: 1941: 1921: 1851: 1803: 1768: 1603: 1568: 1528: 1428: 1374: 1326: 1302: 1188: 1123: 1089: 1009: 940: 920: 883: 828: 799: 770: 741: 721: 637: 572:

The concept of information entropy was introduced by

509: 489: 399: 351: 327: 307: 15638:

IEEE Transactions on Systems Science and Cybernetics

14164: 9140:. Specifically, Shannon entropy is the logarithm of 9113: 6308:

symbols can be defined simply as being equal to its

870:{\displaystyle \log \left({\frac {1}{p(E)}}\right),} 15464:Indices of Qualitative Variation. AR Wilcox - 1967 15269: 15267: 14761: 12024:as follows. Assume that a probability distribution 2372:Entropy can be formally defined in the language of 67:. Unsourced material may be challenged and removed. 15791:Martin, Nathaniel F.G.; England, James W. (2011). 15725: 15585:"Comparative Analysis of Decision Tree Algorithms" 14510: 14462: 14427: 14370: 14116: 14096: 14076: 14056: 14036: 13978: 13898: 13788: 13644:{\displaystyle {\tbinom {n}{k}}q^{qn}(1-q)^{n-nq}} 13643: 13545: 13428: 13279: 13161: 12995: 12939: 12730: 12532: 12351: 12253:, and the differential entropy, where the measure 12232: 11984: 11952: 11917: 11748: 11588: 11312: 11133: 11011: 10883: 10778: 10691: 10562: 10501: 10453: 10007: 9831: 9810: 9799: 9616: 9567: 9413: 9313: 8859: 8836: 8722: 8592: 8452: 8420: 8376: 8235: 8211: 8177: 8099: 8020:{\displaystyle \mathrm {H} (X|Y)=\mathrm {H} (X).} 8019: 7925: 7864: 7730: 7684: 7630: 7610: 7569: 7390: 7283: 7128: 7103: 7056: 7025: 6943: 6897: 6830: 6701: 6675: 6601: 6575: 6473: 6446: 6392: 6329:is defined in terms of a multiplicative property, 6254: 5920: 5724: 5606: 5574: 5530: 5469: 5105: 5079: 5041: 5003: 4977: 4943: 4883: 4269: 4222: 4179: 4147: 4121: 4074: 4052: 3967: 3738: 3469: 3138: 3114: 3094: 3056: 2975: 2955: 2931: 2909: 2830: 2810: 2778: 2737: 2694: 2648: 2626: 2553: 2527: 2464: 2436: 2406: 2356: 2336: 2316: 2257: 2174: 1995: 1971: 1947: 1927: 1901: 1819: 1789: 1686: 1586: 1536: 1512: 1412: 1360: 1312: 1288: 1207: 1167:of landing on tails. The maximum surprise is when 1143: 1109: 1072: 995: 926: 889: 869: 814: 785: 756: 727: 672: 515: 495: 475: 385: 337: 313: 14915:An introduction to information theory and entropy 14697:"Entropy (for data science) Clearly Explained!!!" 13825: 13812: 10897:can be approximated (in the Riemannian sense) by 10786:. By the mean-value theorem there exists a value 5910: 5836: 5808: 5756: 3064:Finally, the entropy of the probability space is 735:is a function which increases as the probability 545:, and base 10 gives units of "dits", "bans", or " 17166: 15714:Creative Commons Attribution/Share-Alike License 15682: 15298: 15264: 15112: 13280:{\displaystyle \mathrm {H} \leq \log |P_{i}(A)|} 11633: 10945: 8493:Entropy in thermodynamics and information theory 7956:doesn't influence our knowledge of the value of 6962: 3015: 1853: 15564:Aoki, New Approaches to Macroeconomic Modeling. 15033: 15031: 15029: 14629: 14627: 14564: 14562: 14511:{\displaystyle p\colon {\mathcal {X}}\to (0,1]} 14152:often employs a standard loss function, called 8918:(1957), thermodynamic entropy, as explained by 8737:of the quantum mechanical system and Tr is the 8512:the most general formula for the thermodynamic 6393:{\displaystyle P(A\mid B)\cdot P(B)=P(A\cap B)} 5221: 4944:{\displaystyle \operatorname {I} (u)=k\log u+c} 2528:{\displaystyle \sigma _{\mu }(A)=-\ln \mu (A).} 15392: 15148:"Information Theory and Statistical Mechanics" 14545:(Third ed.). Academic Press. p. 51. 12393:in short intervals. Proving it also broke the 12384:averages of modulated multiplicative functions 6508:, Forte and Ng, via the following properties: 3912:bits are needed to encode the first value and 623:. The definition can be derived from a set of 549:". An equivalent definition of entropy is the 15963: 15299:Spellerberg, Ian F.; Fedor, Peter J. (2003). 14864: 14784:Fundamentals in Information Theory and Coding 14780: 13698: 13685: 13590: 13577: 13389: 13376: 13003:is the Cartesian product of random variables 12018:from the distribution to a reference measure 10760:To do this, start with a continuous function 10726:and corrections have been suggested, notably 9208:. Shannon himself used the term in this way. 4122:{\displaystyle \operatorname {I} (u)=k\log u} 3955:Shannon discovered that a suitable choice of 1902:{\displaystyle \lim _{p\to 0^{+}}p\log(p)=0.} 269: 15977: 15582: 15283:, Martin Hilbert and Priscila López (2011), 15037: 15026: 14862: 14860: 14858: 14856: 14854: 14852: 14850: 14848: 14846: 14844: 14624: 14559: 14538: 12722: 12596: 12422: 12116: 11934:. In fact, the limit of discrete entropy as 5569: 5545: 5525: 5487: 5218:, not the meaning of the events themselves. 5189:bits of information, which is approximately 3095:{\displaystyle \mathrm {H} _{\mu }(\Sigma )} 2695:{\displaystyle P\subseteq {\mathcal {P}}(X)} 15835:Information Theory: A Tutorial Introduction 15482:"A Magical Answer to an 80-Year-Old Puzzle" 15395:"Guesswork is not a Substitute for Entropy" 15243: 14838:, National Cancer Institute, 14 April 2007. 12741:The proof follows as a simple corollary of 9336: 6481:lends itself to practical interpretations. 5042:{\displaystyle \operatorname {I} (p)\geq 0} 15970: 15956: 15466:https://www.osti.gov/servlets/purl/4167340 15102:. Cambridge University Press. p. 301. 15097: 14946:: CS1 maint: location missing publisher ( 14532: 13989: 9212:text of a complete book, and if there are 2258:{\displaystyle p_{X,Y}(x,y):=\mathbb {P} } 1597:The entropy can explicitly be written as: 1237:give entropies between zero and one bits. 386:{\displaystyle p\colon {\mathcal {X}}\to } 276: 262: 15689:. Springer Science & Business Media. 15521: 15476: 15439: 15324: 15130: 14841: 14719: 14664: 14599: 11905: 11898: 11736: 11572: 11423: 10934: 10874: 10677: 10641: 10598: 10556: 8716: 8687: 8586: 8467:(negentropy) function is convex, and its 8033:More generally, for any random variables 6583:for jointly distributed random variables 6176: 4971: 4873: 4710: 2738:{\displaystyle \mu (\mathop {\cup } P)=1} 2295: 2224: 1827:, the value of the corresponding summand 1530: 1476: 1447: 1391: 639: 127:Learn how and when to remove this message 15820:The Mathematical Theory of Communication 15740:Elements of Information Theory – 2nd Ed. 15583:Batra, Mridula; Agrawal, Rashmi (2018). 15192: 14983: 14639:"A Mathematical Theory of Communication" 14574:"A Mathematical Theory of Communication" 12788:} such that every integer between 1 and 12415:Entropy has become a useful quantity in 12014:of a distribution. It is defined as the 9151: 6447:{\displaystyle \mu (A)\cdot \ln \mu (A)} 3153: 631:is analogous to entropy. The definition 559: 15393:Malone, David; Sullivan, Wayne (2005). 15199:IBM Journal of Research and Development 15113:Sharp, Kim; Matschinsky, Franz (2015). 14869:Thomas M. Cover; Joy A. Thomas (1991). 14633: 14568: 12410: 12294: 10523: 10519:Entropy for continuous random variables 9048:All figures in entropically compressed 5210:of the events observed (the meaning of 3779:, first define an information function 3102:, that is, the entropy with respect to 14: 17167: 15631: 15546:from the original on 25 September 2023 15355: 15145: 14911: 14808:Han, Te Sun; Kobayashi, Kingo (2002). 14771:, Second edition, John Wiley and Sons. 14725: 13979:{\displaystyle 2^{n\mathrm {H} (k/n)}} 9282:Limitations of entropy in cryptography 8497:The inspiration for adopting the word 7685:{\displaystyle \mathrm {H} (f(X)|X)=0} 996:{\displaystyle I(E)=-\log _{2}(p(E)),} 578:A Mathematical Theory of Communication 15951: 15664:from the original on 16 December 2021 15613:from the original on 19 December 2022 15429: 15225:from the original on 15 December 2021 15014:from the original on 25 December 2022 14811:Mathematics of Information and Coding 14749:from the original on 17 February 2016 14340: 13291:Approximation to binomial coefficient 11141:and expanding the logarithm, we have 10708:is the expression for the functional 8487:Relationship to thermodynamic entropy 7298:The maximal entropy of an event with 7143: 4155:. Additionally, choosing a value for 4062:In fact, the only possible values of 3894:equiprobable outcomes then there are 2317:{\displaystyle p_{Y}(y)=\mathbb {P} } 1587:{\displaystyle \operatorname {I} (X)} 907: 15845:, University of Sheffield, England. 14539:Pathria, R. K.; Beale, Paul (2011). 13162:{\displaystyle (X_{j})_{j\in S_{i}}} 12996:{\displaystyle (X_{j})_{j\in S_{i}}} 11953:{\displaystyle N\rightarrow \infty } 8837:{\displaystyle S=k_{\text{B}}\ln W,} 8453:{\displaystyle 0\leq \lambda \leq 1} 7731:{\displaystyle \mathrm {H} (X,f(X))} 5300:should be unchanged if the outcomes 1361:{\displaystyle p:{\mathcal {X}}\to } 1160:of landing on heads and probability 1117:) than each outcome of a coin toss ( 65:adding citations to reliable sources 36: 15632:Jaynes, Edwin T. (September 1968). 15507: 15374:from the original on 1 January 2014 14703:from the original on 5 October 2021 11999: 11994:limiting density of discrete points 11784:Limiting density of discrete points 11778:Limiting density of discrete points 11763:a limit of the Shannon entropy for 10728:limiting density of discrete points 9588:(certain preceding characters) and 8952: 8388:for all probability mass functions 7692:. Applying the previous formula to 7104:{\displaystyle p_{1},\ldots ,p_{n}} 6944:{\displaystyle p_{1},\ldots ,p_{n}} 5531:{\displaystyle \{i_{1},...,i_{n}\}} 4978:{\displaystyle k,c\in \mathbb {R} } 3752: 1820:{\displaystyle x\in {\mathcal {X}}} 208:Limiting density of discrete points 24: 15832:Stone, J. V. (2014), Chapter 1 of 15720: 15411:from the original on 15 April 2016 14657:10.1002/j.1538-7305.1948.tb00917.x 14592:10.1002/j.1538-7305.1948.tb01338.x 14485: 14247:Information fluctuation complexity 14142:Classification in machine learning 14005:information gain in decision trees 13953: 13816: 13689: 13581: 13450: 13408: 13380: 13341: 13193: 12891: 12811: 12104: 12101: 11947: 11899: 11860: 11855: 11840: 11701: 11696: 11674: 11660: 11655: 11637: 11537: 11532: 11483: 11459: 11454: 11406: 11401: 11383: 11359: 11354: 11301: 11289: 11265: 11260: 11209: 11185: 11180: 11155: 11150: 11123: 11090: 11066: 11061: 11036: 11031: 11003: 10979: 10974: 10949: 10917: 10912: 10857: 10837: 10823: 10773: 10679: 10581: 9824: 9659: 9650: 9471: 9462: 9365: 9356: 8693: 8690: 8354: 8315: 8257: 8196: 8162: 8145: 8122: 8084: 8059: 8001: 7976: 7910: 7884: 7829: 7803: 7769: 7752: 7700: 7646: 7551: 7526: 7509: 7484: 7461: 7330: 7236: 7174: 7122: 7050: 6986: 6905:is invariant under permutation of 6850: 6783: 6721: 6660: 6643: 6620: 6560: 6543: 6520: 6179: 6072: 6018: 5818: 5744: 5673: 5623: 5594: 5384: 5320: 5018: 4905: 4812: 4751: 4669: 4643: 4566: 4506: 4452: 4405: 4358: 4333: 4291: 4264: 4161:is equivalent to choosing a value 4089: 4075:{\displaystyle \operatorname {I} } 4069: 3982: 3968:{\displaystyle \operatorname {I} } 3962: 3519: 3230: 3086: 3073: 3032: 2992: 2847: 2678: 2431: 2392: 2067: 2057: 2011: 1988: 1964: 1812: 1637: 1605: 1569: 1454: 1430: 1413:{\displaystyle p(x):=\mathbb {P} } 1335: 1305: 1182:with equiprobable values contains 490: 433: 401: 360: 330: 25: 17206: 15857: 15508:Tao, Terence (28 February 2016). 14814:. American Mathematical Society. 9114:Entropy as a measure of diversity 8223:in the probability mass function 7441:given that you know the value of 5930:Additivity: given an ensemble of 2367: 1154:Consider a coin with probability 301:Given a discrete random variable 219:Asymptotic equipartition property 76:"Entropy" information theory 17144: 17143: 17134: 17133: 15326:10.1046/j.1466-822X.2003.00015.x 14984:Ellerman, David (October 2017). 14928:from the original on 4 June 2016 14167: 10548:with finite or infinite support 6954:Small for small probabilities: 5607:{\displaystyle \mathrm {H} _{n}} 3924:to encode the second, one needs 3888:outcomes and another has one of 2407:{\displaystyle (X,\Sigma ,\mu )} 1320:and is distributed according to 1296:, which takes values in the set 345:and is distributed according to 321:, which takes values in the set 151: 41: 15726:Textbooks on information theory 15676: 15625: 15576: 15567: 15558: 15510:"The Erdős discrepancy problem" 15501: 15470: 15458: 15423: 15386: 15349: 15305:Global Ecology and Biogeography 15292: 15237: 15186: 15139: 15106: 15091: 15074: 15040:Advances in Applied Probability 14977: 14954: 14905: 14887: 14828: 14064:and the conditional entropy of 9811:Efficiency (normalized entropy) 9225:, and the entropy (in bits) is 9015:Shannon's source coding theorem 8992:to protect against errors. The 8959:Shannon's source coding theorem 8212:{\displaystyle \mathrm {H} (p)} 7042:It was shown that any function 7037: 6316:Redundancy (information theory) 4853: 4794: 4704: 4618: 4481: 4383: 4270:{\textstyle \operatorname {I} } 3871:: the information learned from 2779:{\displaystyle \mu (A\cap B)=0} 1841:, which is consistent with the 683: 235:Shannon's source coding theorem 52:needs additional citations for 15797:. Cambridge University Press. 15794:Mathematical Theory of Entropy 15785:Entropy and Information Theory 15759:, Cambridge University Press, 15712:, which is licensed under the 15244:Mark Nelson (24 August 2006). 14873:. Hoboken, New Jersey: Wiley. 14871:Elements of Information Theory 14801: 14774: 14735:. Cambridge University Press. 14689: 14666:11858/00-001M-0000-002C-4317-B 14505: 14493: 14490: 14457: 14451: 14416: 14410: 14393: 14365: 14359: 14031: 14019: 13971: 13957: 13857: 13844: 13771: 13767: 13755: 13746: 13728: 13715: 13651:is one term of the expression 13623: 13610: 13537: 13525: 13509: 13497: 13491: 13485: 13460: 13454: 13418: 13412: 13351: 13345: 13273: 13269: 13263: 13249: 13236: 13214: 13200: 13197: 13137: 13123: 12971: 12957: 12934: 12912: 12898: 12895: 12853: 12850: 12818: 12815: 12713: 12681: 12675: 12599: 12590: 12584: 12526: 12522: 12516: 12502: 12461: 12452: 12346: 12334: 12224: 12215: 12209: 12206: 12200: 12194: 12185: 12179: 12167: 12158: 12152: 12149: 12143: 12137: 12122: 12110: 11979: 11973: 11944: 11902: 11895: 11889: 11883: 11874: 11868: 11733: 11727: 11715: 11709: 11640: 11626: 11620: 11569: 11563: 11551: 11545: 11521: 11514: 11511: 11498: 11492: 11480: 11467: 11420: 11414: 11390: 11380: 11367: 11304: 11298: 11286: 11273: 11240: 11237: 11224: 11218: 11206: 11193: 11120: 11107: 11087: 11074: 11000: 10987: 10952: 10931: 10925: 10871: 10865: 10854: 10842: 10820: 10807: 10766:discretized into bins of size 10674: 10668: 10656: 10650: 10626: 10623: 10617: 10602: 10591: 10585: 10502:{\displaystyle {\log _{b}(n)}} 10495: 10489: 10445: 10440: 10427: 10417: 10403: 10376: 10357: 10352: 10339: 10329: 10315: 10309: 10266: 10260: 10242: 10237: 10224: 10214: 10200: 10194: 10148: 10142: 10124: 10121: 10108: 10102: 10086: 10073: 10037: 10031: 9996: 9990: 9972: 9969: 9956: 9950: 9934: 9921: 9859: 9853: 9832:{\displaystyle {\mathcal {X}}} 9791: 9785: 9754: 9748: 9719: 9713: 9664: 9654: 9611: 9605: 9559: 9553: 9534: 9528: 9476: 9466: 9370: 9360: 8934:maximum entropy thermodynamics 8713: 8698: 8639:in 1878 after earlier work by 8371: 8358: 8350: 8338: 8332: 8319: 8305: 8292: 8280: 8261: 8206: 8200: 8172: 8166: 8155: 8149: 8138: 8126: 8094: 8088: 8077: 8070: 8063: 8011: 8005: 7994: 7987: 7980: 7920: 7914: 7903: 7900: 7894: 7888: 7856: 7853: 7847: 7840: 7833: 7822: 7819: 7813: 7807: 7796: 7789: 7785: 7779: 7773: 7762: 7756: 7725: 7722: 7716: 7704: 7673: 7666: 7662: 7656: 7650: 7605: 7599: 7561: 7555: 7544: 7537: 7530: 7519: 7513: 7502: 7495: 7488: 7477: 7465: 7435:, then revealing the value of 7366: 7334: 7278: 7246: 7228: 7190: 7014: 6996: 6969: 6892: 6860: 6825: 6793: 6775: 6737: 6670: 6664: 6653: 6647: 6636: 6624: 6570: 6564: 6553: 6547: 6536: 6524: 6441: 6435: 6420: 6414: 6387: 6375: 6366: 6360: 6351: 6339: 5665: 5633: 5074: 5062: 5030: 5024: 4917: 4911: 4828: 4822: 4771: 4767: 4761: 4744: 4685: 4679: 4659: 4653: 4599: 4576: 4539: 4516: 4475: 4462: 4438: 4415: 4377: 4364: 4352: 4339: 4320: 4297: 4244:by the above four properties. 4101: 4095: 4044: 4038: 3994: 3988: 3710: 3701: 3689: 3680: 3658: 3652: 3630: 3624: 3592: 3586: 3564: 3558: 3529: 3523: 3453: 3444: 3323: 3310: 3291: 3278: 3240: 3234: 3089: 3083: 3048: 3042: 3008: 3002: 2901: 2895: 2863: 2857: 2767: 2755: 2726: 2712: 2689: 2683: 2618: 2612: 2599: 2593: 2584: 2578: 2519: 2513: 2495: 2489: 2401: 2383: 2311: 2299: 2288: 2282: 2252: 2228: 2217: 2205: 2163: 2157: 2142: 2130: 2102: 2090: 2029: 2022: 2015: 1996:{\displaystyle {\mathcal {Y}}} 1972:{\displaystyle {\mathcal {X}}} 1890: 1884: 1860: 1778: 1772: 1678: 1672: 1653: 1647: 1615: 1609: 1581: 1575: 1504: 1501: 1495: 1480: 1469: 1466: 1460: 1451: 1440: 1434: 1407: 1395: 1384: 1378: 1355: 1343: 1340: 1313:{\displaystyle {\mathcal {X}}} 1265:, Shannon defined the entropy 1057: 1051: 1019: 1013: 987: 984: 978: 972: 950: 944: 854: 848: 809: 803: 780: 774: 751: 745: 667: 664: 658: 643: 580:", and is also referred to as 467: 461: 449: 443: 411: 405: 380: 368: 365: 338:{\displaystyle {\mathcal {X}}} 193:Conditional mutual information 13: 1: 15146:Jaynes, E. T. (15 May 1957). 14644:Bell System Technical Journal 14579:Bell System Technical Journal 14525: 14242:History of information theory 12352:{\displaystyle \lambda (n+H)} 11960:would also include a term of 10891:the integral of the function 5965: 5011:. Property 1 and 2 give that 3757:To understand the meaning of 3188:represents a result of heads. 1594:is itself a random variable. 1256: 584:. Shannon's theory defines a 15888:Resources in other libraries 15869:Entropy (information theory) 14132:principle of maximum entropy 13557: 12006:Generalized relative entropy 11823:is some "standard" value of 11810:will then have the units of 10563:{\displaystyle \mathbb {X} } 10536:probability density function 8750:second law of thermodynamics 7129:{\displaystyle \mathrm {H} } 7057:{\displaystyle \mathrm {H} } 5222:Alternative characterization 4248: 3149: 2437:{\displaystyle A\in \Sigma } 1537:{\displaystyle \mathbb {E} } 764:of an event decreases. When 673:{\displaystyle \mathbb {E} } 602:noisy-channel coding theorem 245:Noisy-channel coding theorem 7: 15904:Encyclopedia of Mathematics 15597:10.1007/978-981-10-6747-1_4 14964:2005. 17 (2005): 2847-2854 14435:and it can be assumed that 14386: 14160: 13087:denotes the cardinality of 12404:for this specific problem. 12016:Kullback–Leibler divergence 11799:is a dimensioned variable. 9636:as the previous character. 9108:telecommunications networks 8421:{\displaystyle p_{1},p_{2}} 5575:{\displaystyle \{1,...,n\}} 4621:taking the derivative w.r.t 4484:taking the derivative w.r.t 4223:{\displaystyle k=-1/\log x} 10: 17211: 17025:Compressed data structures 16347:RLE + BWT + MTF + Huffman 16015:Asymmetric numeral systems 15822:, Univ of Illinois Press. 15193:Landauer, R. (July 1961). 15098:Życzkowski, Karol (2006). 14912:Carter, Tom (March 2014). 14150:artificial neural networks 13937:many 1's is approximately 13057:and so that each point in 12034:with respect to a measure 12003: 11781: 10527: 9117: 8956: 8764:indicates, the changes in 8510:statistical thermodynamics 8490: 8481: 6683:when the random variables 3199: 2344:given the random variable 1240: 1208:{\displaystyle \log _{2}3} 613:statistical thermodynamics 29: 17129: 17113: 17097: 17015: 16940: 16872: 16863: 16786: 16720: 16711: 16612: 16529: 16520: 16436: 16384:Discrete cosine transform 16374: 16365: 16314:LZ77 + Huffman + context 16267: 16177: 16107: 15995: 15986: 15883:Resources in your library 15589:Nature Inspired Computing 14993:Logic Journal of the IGPL 12763:are random variables and 12429:Loomis–Whitney inequality 12423:Loomis–Whitney inequality 12373:Erdős discrepancy problem 12304:Erdős discrepancy problem 9198:quantities of information 8999:PPM compression algorithm 7447:. This may be written as: 6474:{\displaystyle \log _{2}} 5942:boxes (sub-systems) with 5311:are re-ordered. That is, 17089:Smallest grammar problem 15650:10.1109/TSSC.1968.300117 14463:{\displaystyle 0\log(0)} 14333: 14294:Quantum relative entropy 14257:Kolmogorov–Sinai entropy 14217:Entropy power inequality 14190:Entropy (thermodynamics) 9617:{\displaystyle p_{i}(j)} 9337:Data as a Markov process 8627:is the probability of a 5216:probability distribution 4797:combining terms into one 4386:Starting from property 3 3823:monotonically decreasing 3122:of the sigma-algebra of 2811:{\displaystyle A,B\in P} 1955:taking values from sets 1911:One may also define the 1275:discrete random variable 32:Entropy (disambiguation) 17180:Entropy and information 17030:Compressed suffix array 16579:Nyquist–Shannon theorem 15441:10.1007/3-540-46513-8_5 15172:10.1103/PhysRev.106.620 14371:{\displaystyle \log(0)} 14201:Entropy (arrow of time) 14130:models often apply the 14037:{\displaystyle IG(Y,X)} 13990:Use in machine learning 12800:of these subsets, then 12389:28 October 2023 at the 11985:{\displaystyle \log(N)} 10779:{\displaystyle \Delta } 9314:{\displaystyle 2^{127}} 8922:, should be seen as an 6327:conditional probability 6314:-ary entropy. See also 3202:Binary entropy function 1708:used. Common values of 1546:expected value operator 908:§ Characterization 680:generalizes the above. 496:{\displaystyle \Sigma } 250:Shannon–Hartley theorem 17195:Complex systems theory 17190:Statistical randomness 15742:, Wiley-Interscience, 15358:"Guessing and Entropy" 15356:Massey, James (1994). 14971:5 October 2021 at the 14781:Borda, Monica (2011). 14512: 14464: 14429: 14372: 14285:statistical dispersion 14118: 14098: 14078: 14058: 14038: 14001:Decision tree learning 13980: 13900: 13790: 13680: 13645: 13547: 13430: 13281: 13163: 12997: 12941: 12889: 12732: 12534: 12500: 12353: 12234: 12067:for some non-negative 12040:, i.e. is of the form 11986: 11954: 11919: 11750: 11590: 11463: 11363: 11314: 11269: 11189: 11135: 11070: 11013: 10983: 10885: 10797:in each bin such that 10780: 10693: 10564: 10503: 10455: 10399: 10295: 10177: 10066: 10009: 9914: 9833: 9801: 9624:is the probability of 9618: 9569: 9435:is the probability of 9415: 9315: 8861: 8838: 8724: 8594: 8454: 8422: 8378: 8237: 8213: 8179: 8101: 8021: 7927: 7866: 7732: 7686: 7632: 7612: 7611:{\displaystyle Y=f(X)} 7571: 7392: 7302:different outcomes is 7285: 7130: 7105: 7058: 7027: 6945: 6899: 6832: 6703: 6677: 6603: 6577: 6475: 6448: 6394: 6256: 6158: 5922: 5726: 5608: 5576: 5532: 5471: 5107: 5106:{\displaystyle k<0} 5081: 5043: 5005: 4979: 4945: 4898:leads to the solution 4885: 4271: 4238:base for the logarithm 4224: 4181: 4180:{\displaystyle x>1} 4149: 4148:{\displaystyle k<0} 4123: 4076: 4054: 3969: 3740: 3471: 3429: 3360: 3273: 3197: 3140: 3126:measurable subsets of 3116: 3096: 3058: 2977: 2957: 2933: 2911: 2832: 2812: 2780: 2739: 2696: 2650: 2628: 2555: 2529: 2466: 2438: 2408: 2358: 2338: 2318: 2259: 2176: 1997: 1973: 1949: 1929: 1903: 1821: 1791: 1790:{\displaystyle p(x)=0} 1688: 1588: 1538: 1514: 1414: 1362: 1314: 1290: 1269:(Greek capital letter 1209: 1145: 1111: 1074: 997: 928: 891: 871: 816: 787: 758: 729: 674: 569: 541:gives "natural units" 517: 497: 477: 387: 339: 315: 224:Rate–distortion theory 17059:Kolmogorov complexity 16927:Video characteristics 16304:LZ77 + Huffman + ANS 15634:"Prior Probabilities" 15005:10.1093/jigpal/jzx022 14542:Statistical Mechanics 14513: 14465: 14430: 14373: 14289:nominal distributions 14281:Qualitative variation 14119: 14099: 14079: 14059: 14039: 13981: 13901: 13791: 13660: 13646: 13548: 13431: 13282: 13164: 12998: 12942: 12869: 12733: 12556:orthogonal projection 12535: 12480: 12400:7 August 2023 at the 12354: 12235: 12073:-integrable function 12032:absolutely continuous 11987: 11955: 11920: 11772:information dimension 11751: 11591: 11440: 11340: 11315: 11246: 11166: 11136: 11047: 11014: 10960: 10886: 10781: 10694: 10565: 10504: 10466:communication channel 10456: 10379: 10275: 10157: 10046: 10010: 9894: 9834: 9802: 9619: 9570: 9416: 9316: 9252:Kolmogorov complexity 9152:Entropy of a sequence 8986:Kolmogorov complexity 8972:or in practice using 8920:statistical mechanics 8862: 8839: 8792:and expressed by his 8725: 8595: 8503:statistical mechanics 8455: 8423: 8379: 8238: 8214: 8180: 8102: 8022: 7928: 7867: 7733: 7687: 7633: 7613: 7572: 7417:(that is, evaluating 7393: 7286: 7131: 7106: 7059: 7028: 6946: 6900: 6833: 6704: 6678: 6604: 6578: 6476: 6449: 6395: 6257: 6138: 5923: 5727: 5609: 5577: 5533: 5472: 5108: 5082: 5080:{\displaystyle p\in } 5044: 5006: 4980: 4946: 4896:differential equation 4886: 4272: 4225: 4182: 4150: 4124: 4077: 4055: 3970: 3783:in terms of an event 3741: 3472: 3409: 3340: 3253: 3157: 3141: 3117: 3097: 3059: 2978: 2958: 2934: 2912: 2833: 2813: 2781: 2740: 2697: 2651: 2629: 2556: 2530: 2467: 2439: 2409: 2359: 2339: 2319: 2260: 2177: 1998: 1974: 1950: 1930: 1904: 1822: 1792: 1689: 1589: 1539: 1515: 1415: 1363: 1315: 1291: 1263:Boltzmann's Η-theorem 1210: 1146: 1144:{\displaystyle p=1/2} 1112: 1110:{\displaystyle p=1/6} 1075: 998: 929: 892: 890:{\displaystyle \log } 872: 817: 788: 759: 730: 675: 594:source coding theorem 590:communication channel 563: 518: 516:{\displaystyle \log } 498: 478: 388: 340: 316: 17149:Compression software 16743:Compression artifact 16699:Psychoacoustic model 15783:Gray, R. M. (2011), 15279:27 July 2013 at the 14769:Applied Cryptography 14617:20 June 2014 at the 14474: 14439: 14382: 14350: 14283:– other measures of 14266:Levenshtein distance 14252:Information geometry 14108: 14088: 14068: 14048: 14010: 13941: 13806: 13657: 13570: 13446: 13327: 13189: 13120: 12954: 12807: 12743:Shearer's inequality 12571: 12448: 12411:Use in combinatorics 12328: 12295:Use in number theory 12092: 11964: 11938: 11836: 11614: 11333: 11145: 11026: 10901: 10801: 10770: 10577: 10552: 10530:Differential entropy 10524:Differential entropy 10472: 10025: 9847: 9819: 9646: 9592: 9458: 9441:. For a first-order 9352: 9298: 9032:pigeonhole principle 8947:Landauer's principle 8851: 8803: 8665: 8535: 8523:thermodynamic system 8432: 8392: 8253: 8227: 8192: 8118: 8055: 7972: 7880: 7748: 7696: 7642: 7638:is a function, then 7622: 7587: 7457: 7326: 7169: 7118: 7069: 7046: 6958: 6909: 6845: 6716: 6687: 6616: 6587: 6516: 6458: 6408: 6333: 6013: 5739: 5618: 5589: 5542: 5484: 5315: 5124:units of information 5091: 5053: 5015: 4989: 4955: 4902: 4283: 4261: 4191: 4165: 4133: 4086: 4066: 3979: 3959: 3511: 3222: 3130: 3115:{\displaystyle \mu } 3106: 3068: 2987: 2967: 2947: 2923: 2842: 2822: 2790: 2749: 2706: 2667: 2649:{\displaystyle \mu } 2640: 2565: 2545: 2476: 2456: 2422: 2380: 2348: 2328: 2269: 2186: 2007: 1983: 1959: 1939: 1919: 1849: 1801: 1766: 1601: 1566: 1526: 1426: 1372: 1324: 1300: 1280: 1186: 1121: 1087: 1007: 938: 918: 881: 826: 815:{\displaystyle p(E)} 797: 786:{\displaystyle p(E)} 768: 757:{\displaystyle p(E)} 739: 719: 635: 629:differential entropy 507: 487: 397: 349: 325: 305: 188:Directed information 168:Differential entropy 61:improve this article 30:For other uses, see 17139:Compression formats 16778:Texture compression 16773:Standard test image 16589:Silence compression 15939:31 May 2016 at the 15921:4 June 2016 at the 15841:3 June 2016 at the 15317:2003GloEB..12..177S 15164:1957PhRv..106..620J 14682:10 May 2013 at the 14184:Approximate entropy 14146:logistic regression 12431:: for every subset 11864: 11705: 11541: 11410: 10921: 10861: 9056:Type of Information 9052: 9026:scheme can shorten 8652:von Neumann entropy 6702:{\displaystyle X,Y} 6602:{\displaystyle X,Y} 6402:conditional entropy 5004:{\displaystyle c=0} 4985:. Property 2 gives 4240:. Thus, entropy is 4236:corresponds to the 1913:conditional entropy 1554:information content 703:information content 576:in his 1948 paper " 173:Conditional entropy 17185:Information theory 17047:Information theory 16902:Display resolution 16728:Chroma subsampling 16117:Byte pair encoding 16062:Shannon–Fano–Elias 15769:Arndt, C. (2004), 15480:(1 October 2015). 15246:"The Hutter Prize" 15211:10.1147/rd.53.0183 14727:MacKay, David J.C. 14635:Shannon, Claude E. 14601:10338.dmlcz/101429 14570:Shannon, Claude E. 14508: 14460: 14425: 14400: 14368: 14271:Mutual information 14237:History of entropy 14222:Fisher information 14212:Entropy estimation 14175:Mathematics portal 14128:Bayesian inference 14114: 14094: 14074: 14054: 14034: 13976: 13896: 13786: 13703: 13641: 13595: 13543: 13426: 13394: 13277: 13159: 12993: 12937: 12728: 12530: 12380:Liouville function 12349: 12311:Liouville function 12230: 11982: 11950: 11915: 11847: 11746: 11688: 11647: 11586: 11584: 11524: 11393: 11310: 11131: 11009: 10959: 10904: 10881: 10829: 10776: 10689: 10560: 10499: 10451: 10005: 9829: 9797: 9731: 9702: 9682: 9614: 9565: 9514: 9494: 9411: 9311: 9260:universal computer 9194:stationary process 9186:stochastic process 9089:Telecommunications 9047: 8929:Boltzmann constant 8884:Boltzmann constant 8857: 8834: 8754:Boltzmann constant 8720: 8614:Boltzmann constant 8590: 8450: 8418: 8374: 8233: 8209: 8175: 8097: 8017: 7923: 7862: 7728: 7682: 7628: 7608: 7567: 7388: 7281: 7144:Further properties 7126: 7101: 7054: 7023: 6983: 6941: 6895: 6828: 6699: 6673: 6599: 6573: 6471: 6444: 6390: 6323:partition of a set 6252: 5918: 5907: 5894: 5805: 5798: 5722: 5604: 5572: 5528: 5467: 5167:constant multiples 5103: 5077: 5039: 5001: 4975: 4941: 4881: 4879: 4870:producing constant 4267: 4220: 4177: 4145: 4119: 4072: 4050: 4020: 3965: 3873:independent events 3736: 3734: 3467: 3465: 3198: 3136: 3112: 3092: 3054: 3029: 2973: 2953: 2929: 2907: 2884: 2828: 2808: 2776: 2735: 2692: 2646: 2624: 2551: 2525: 2462: 2434: 2404: 2354: 2334: 2314: 2255: 2172: 2073: 2003:respectively, as: 1993: 1969: 1945: 1925: 1899: 1874: 1817: 1787: 1684: 1643: 1584: 1534: 1510: 1410: 1358: 1310: 1286: 1233:. Other values of 1205: 1178:. (Similarly, one 1141: 1107: 1070: 993: 924: 887: 867: 812: 783: 754: 725: 670: 586:data communication 570: 513: 493: 473: 439: 383: 335: 311: 290:information theory 183:Mutual information 145:Information theory 17162: 17161: 17011: 17010: 16961:Deblocking filter 16859: 16858: 16707: 16706: 16516: 16515: 16361: 16360: 15864:Library resources 15804:978-0-521-17738-2 15779:978-3-540-40855-0 15765:978-0-521-64298-9 15748:978-0-471-24195-9 15696:978-1-4757-4321-0 15606:978-981-10-6747-1 15514:Discrete Analysis 15451:978-3-540-67185-5 15132:10.3390/e17041971 14880:978-0-471-24195-9 14821:978-0-8218-4256-0 14794:978-3-642-20346-6 14385: 14261:dynamical systems 14136:prior probability 14117:{\displaystyle X} 14097:{\displaystyle X} 14077:{\displaystyle Y} 14057:{\displaystyle Y} 13921: 13920: 13894: 13823: 13696: 13588: 13387: 13367: 12867: 12263:. If the measure 11632: 10944: 10270: 10152: 10000: 9886: 9768: 9759: 9722: 9693: 9673: 9517: 9505: 9485: 9247:unique identifier 9099: 9098: 8982:arithmetic coding 8860:{\displaystyle S} 8819: 8783:signal processing 8684: 8554: 8463:Accordingly, the 8236:{\displaystyle p} 7631:{\displaystyle f} 6961: 6325:. Meanwhile, the 6242: 6216: 6174: 6128: 6102: 6060: 6041: 5972:positive integers 5888: 5861: 5843: 5841: 5792: 5773: 5763: 5761: 5715: 5696: 5157:decimal logarithm 5146:natural logarithm 5118: 5117: 4871: 4861: 4857: 4856:integrating w.r.t 4844: 4802: 4798: 4785: 4708: 4695: 4626: 4622: 4609: 4489: 4485: 4448: 4387: 4330: 4019: 3789:with probability 3439: 3393: 3370: 3212:Bernoulli process 3206:Bernoulli process 3139:{\displaystyle X} 3014: 2976:{\displaystyle M} 2963:. The entropy of 2956:{\displaystyle X} 2932:{\displaystyle M} 2869: 2831:{\displaystyle P} 2786:for all distinct 2554:{\displaystyle A} 2465:{\displaystyle A} 2416:probability space 2357:{\displaystyle Y} 2337:{\displaystyle X} 2167: 2038: 1948:{\displaystyle Y} 1928:{\displaystyle X} 1915:of two variables 1852: 1624: 1061: 1003:or equivalently, 927:{\displaystyle E} 858: 728:{\displaystyle E} 713:self-information, 420: 393:, the entropy is 314:{\displaystyle X} 286: 285: 137: 136: 129: 111: 16:(Redirected from 17202: 17175:Data compression 17147: 17146: 17137: 17136: 16966:Lapped transform 16870: 16869: 16748:Image resolution 16733:Coding tree unit 16718: 16717: 16527: 16526: 16372: 16371: 15993: 15992: 15979:Data compression 15972: 15965: 15958: 15949: 15948: 15912: 15808: 15701: 15700: 15680: 15674: 15673: 15671: 15669: 15629: 15623: 15622: 15620: 15618: 15580: 15574: 15571: 15565: 15562: 15556: 15555: 15553: 15551: 15525: 15505: 15499: 15498: 15496: 15494: 15478:Klarreich, Erica 15474: 15468: 15462: 15456: 15455: 15443: 15427: 15421: 15420: 15418: 15416: 15410: 15399: 15390: 15384: 15383: 15381: 15379: 15373: 15362: 15353: 15347: 15346: 15328: 15296: 15290: 15271: 15262: 15261: 15259: 15257: 15248:. Archived from 15241: 15235: 15234: 15232: 15230: 15190: 15184: 15183: 15143: 15137: 15136: 15134: 15110: 15104: 15103: 15095: 15089: 15078: 15072: 15071: 15035: 15024: 15023: 15021: 15019: 15013: 14990: 14981: 14975: 14958: 14952: 14951: 14945: 14937: 14935: 14933: 14927: 14920: 14909: 14903: 14891: 14885: 14884: 14866: 14839: 14834:Schneider, T.D, 14832: 14826: 14825: 14805: 14799: 14798: 14778: 14772: 14765: 14759: 14758: 14756: 14754: 14723: 14717: 14716: 14710: 14708: 14693: 14687: 14675:, archived from 14670: 14668: 14637:(October 1948). 14631: 14622: 14610:, archived from 14605: 14603: 14566: 14557: 14556: 14536: 14519: 14517: 14515: 14514: 14509: 14489: 14488: 14469: 14467: 14466: 14461: 14434: 14432: 14431: 14426: 14399: 14377: 14375: 14374: 14369: 14344: 14232:Hamming distance 14206:Entropy encoding 14177: 14172: 14171: 14123: 14121: 14120: 14115: 14103: 14101: 14100: 14095: 14083: 14081: 14080: 14075: 14063: 14061: 14060: 14055: 14043: 14041: 14040: 14035: 13995:Machine learning 13985: 13983: 13982: 13977: 13975: 13974: 13967: 13956: 13936: 13930: 13915: 13909:since there are 13905: 13903: 13902: 13897: 13895: 13893: 13879: 13874: 13873: 13843: 13842: 13830: 13829: 13828: 13815: 13795: 13793: 13792: 13787: 13779: 13778: 13742: 13741: 13714: 13713: 13704: 13702: 13701: 13688: 13679: 13674: 13650: 13648: 13647: 13642: 13640: 13639: 13609: 13608: 13596: 13594: 13593: 13580: 13558: 13552: 13550: 13549: 13544: 13521: 13520: 13481: 13480: 13453: 13435: 13433: 13432: 13427: 13422: 13421: 13411: 13395: 13393: 13392: 13379: 13368: 13366: 13355: 13354: 13344: 13331: 13319: 13305: 13286: 13284: 13283: 13278: 13276: 13262: 13261: 13252: 13235: 13234: 13233: 13232: 13212: 13211: 13196: 13184: 13169:is contained in 13168: 13166: 13165: 13160: 13158: 13157: 13156: 13155: 13135: 13134: 13116:}. The range of 13115: 13092: 13086: 13084: 13076: 13074: 13062: 13056: 13050: 13041: 13030: 13019: 13013: 13002: 13000: 12999: 12994: 12992: 12991: 12990: 12989: 12969: 12968: 12946: 12944: 12943: 12938: 12933: 12932: 12931: 12930: 12910: 12909: 12894: 12888: 12883: 12868: 12860: 12849: 12848: 12830: 12829: 12814: 12799: 12794:lies in exactly 12793: 12787: 12780: 12762: 12737: 12735: 12734: 12729: 12712: 12711: 12693: 12692: 12674: 12673: 12655: 12654: 12636: 12635: 12611: 12610: 12583: 12582: 12563: 12553: 12539: 12537: 12536: 12531: 12529: 12515: 12514: 12505: 12499: 12494: 12476: 12475: 12464: 12455: 12440: 12395:"parity barrier" 12369: 12358: 12356: 12355: 12350: 12323: 12290: 12284: 12278: 12268: 12261:Lebesgue measure 12258: 12251:counting measure 12248: 12239: 12237: 12236: 12231: 12109: 12108: 12107: 12084: 12078: 12072: 12066: 12039: 12029: 12023: 12012:relative entropy 12000:Relative entropy 11991: 11989: 11988: 11983: 11959: 11957: 11956: 11951: 11933: 11924: 11922: 11921: 11916: 11863: 11858: 11843: 11828: 11822: 11816: 11809: 11798: 11769: 11755: 11753: 11752: 11747: 11704: 11699: 11681: 11677: 11664: 11663: 11658: 11646: 11606: 11602: 11595: 11593: 11592: 11587: 11585: 11540: 11535: 11510: 11509: 11479: 11478: 11462: 11457: 11409: 11404: 11379: 11378: 11362: 11357: 11325: 11319: 11317: 11316: 11311: 11285: 11284: 11268: 11263: 11236: 11235: 11205: 11204: 11188: 11183: 11159: 11158: 11153: 11140: 11138: 11137: 11132: 11130: 11126: 11119: 11118: 11086: 11085: 11069: 11064: 11040: 11039: 11034: 11018: 11016: 11015: 11010: 10999: 10998: 10982: 10977: 10958: 10920: 10915: 10896: 10890: 10888: 10887: 10882: 10860: 10840: 10819: 10818: 10796: 10785: 10783: 10782: 10777: 10765: 10756: 10745: 10713: 10707: 10698: 10696: 10695: 10690: 10682: 10646: 10645: 10644: 10601: 10584: 10569: 10567: 10566: 10561: 10559: 10547: 10514: 10508: 10506: 10505: 10500: 10498: 10485: 10484: 10460: 10458: 10457: 10452: 10444: 10443: 10439: 10438: 10415: 10414: 10398: 10393: 10372: 10371: 10356: 10355: 10351: 10350: 10327: 10326: 10305: 10304: 10294: 10289: 10271: 10269: 10256: 10255: 10245: 10241: 10240: 10236: 10235: 10212: 10211: 10190: 10189: 10179: 10176: 10171: 10153: 10151: 10138: 10137: 10127: 10120: 10119: 10098: 10097: 10085: 10084: 10068: 10065: 10060: 10014: 10012: 10011: 10006: 10001: 9999: 9986: 9985: 9975: 9968: 9967: 9946: 9945: 9933: 9932: 9916: 9913: 9908: 9887: 9885: 9884: 9866: 9838: 9836: 9835: 9830: 9828: 9827: 9806: 9804: 9803: 9798: 9784: 9783: 9766: 9757: 9747: 9746: 9730: 9712: 9711: 9701: 9692: 9691: 9681: 9663: 9662: 9653: 9635: 9629: 9623: 9621: 9620: 9615: 9604: 9603: 9583: 9574: 9572: 9571: 9566: 9552: 9551: 9527: 9526: 9515: 9513: 9504: 9503: 9493: 9475: 9474: 9465: 9440: 9434: 9420: 9418: 9417: 9412: 9407: 9406: 9391: 9390: 9369: 9368: 9359: 9320: 9318: 9317: 9312: 9310: 9309: 9277: 9244: 9224: 9217: 9163:self-information 9143: 9053: 9046: 9037:A 2011 study in 8963:Data compression 8953:Data compression 8909: 8900: 8881: 8872: 8866: 8864: 8863: 8858: 8843: 8841: 8840: 8835: 8821: 8820: 8817: 8790:Ludwig Boltzmann 8779:data compression 8776: 8763: 8729: 8727: 8726: 8721: 8697: 8696: 8686: 8685: 8682: 8656:John von Neumann 8637:J. Willard Gibbs 8626: 8611: 8599: 8597: 8596: 8591: 8585: 8584: 8569: 8568: 8556: 8555: 8552: 8520: 8469:convex conjugate 8465:negative entropy 8459: 8457: 8456: 8451: 8427: 8425: 8424: 8419: 8417: 8416: 8404: 8403: 8383: 8381: 8380: 8375: 8370: 8369: 8357: 8331: 8330: 8318: 8304: 8303: 8276: 8275: 8260: 8242: 8240: 8239: 8234: 8218: 8216: 8215: 8210: 8199: 8184: 8182: 8181: 8176: 8165: 8148: 8125: 8106: 8104: 8103: 8098: 8087: 8073: 8062: 8044: 8038: 8026: 8024: 8023: 8018: 8004: 7990: 7979: 7961: 7955: 7949: 7943: 7932: 7930: 7929: 7924: 7913: 7887: 7871: 7869: 7868: 7863: 7843: 7832: 7806: 7792: 7772: 7755: 7737: 7735: 7734: 7729: 7703: 7691: 7689: 7688: 7683: 7669: 7649: 7637: 7635: 7634: 7629: 7617: 7615: 7614: 7609: 7576: 7574: 7573: 7568: 7554: 7540: 7529: 7512: 7498: 7487: 7464: 7446: 7440: 7434: 7428: 7422: 7416: 7397: 7395: 7394: 7389: 7381: 7380: 7365: 7364: 7346: 7345: 7333: 7315: 7290: 7288: 7287: 7282: 7277: 7276: 7258: 7257: 7245: 7244: 7239: 7221: 7220: 7202: 7201: 7189: 7188: 7177: 7154: 7135: 7133: 7132: 7127: 7125: 7110: 7108: 7107: 7102: 7100: 7099: 7081: 7080: 7063: 7061: 7060: 7055: 7053: 7032: 7030: 7029: 7024: 6995: 6994: 6989: 6982: 6981: 6980: 6950: 6948: 6947: 6942: 6940: 6939: 6921: 6920: 6904: 6902: 6901: 6896: 6891: 6890: 6872: 6871: 6859: 6858: 6853: 6837: 6835: 6834: 6829: 6824: 6823: 6805: 6804: 6792: 6791: 6786: 6768: 6767: 6749: 6748: 6736: 6735: 6724: 6712:Expansibility: 6709:are independent. 6708: 6706: 6705: 6700: 6682: 6680: 6679: 6674: 6663: 6646: 6623: 6608: 6606: 6605: 6600: 6582: 6580: 6579: 6574: 6563: 6546: 6523: 6512:Subadditivity: 6480: 6478: 6477: 6472: 6470: 6469: 6453: 6451: 6450: 6445: 6399: 6397: 6396: 6391: 6313: 6307: 6301: 6293: 6274: 6261: 6259: 6258: 6253: 6248: 6244: 6243: 6241: 6240: 6228: 6217: 6215: 6214: 6202: 6195: 6194: 6193: 6192: 6182: 6175: 6170: 6169: 6160: 6157: 6152: 6134: 6130: 6129: 6124: 6123: 6114: 6103: 6098: 6097: 6088: 6081: 6080: 6075: 6066: 6062: 6061: 6053: 6042: 6034: 6027: 6026: 6021: 6005: 5983: 5959: 5941: 5935: 5927: 5925: 5924: 5919: 5914: 5913: 5906: 5895: 5890: 5889: 5887: 5873: 5862: 5860: 5846: 5840: 5839: 5833: 5832: 5821: 5812: 5811: 5804: 5799: 5794: 5793: 5785: 5774: 5766: 5760: 5759: 5753: 5752: 5747: 5731: 5729: 5728: 5723: 5721: 5717: 5716: 5708: 5697: 5689: 5682: 5681: 5676: 5664: 5663: 5645: 5644: 5632: 5631: 5626: 5613: 5611: 5610: 5605: 5603: 5602: 5597: 5581: 5579: 5578: 5573: 5537: 5535: 5534: 5529: 5524: 5523: 5499: 5498: 5476: 5474: 5473: 5468: 5466: 5462: 5461: 5460: 5459: 5458: 5435: 5434: 5433: 5432: 5415: 5414: 5413: 5412: 5393: 5392: 5387: 5378: 5374: 5373: 5372: 5357: 5356: 5344: 5343: 5329: 5328: 5323: 5310: 5299: 5288: 5280: 5250: 5203:decimal digits. 5202: 5195: 5188: 5182: 5176: 5164: 5150: 5139: 5132:binary logarithm 5112: 5110: 5109: 5104: 5086: 5084: 5083: 5078: 5048: 5046: 5045: 5040: 5010: 5008: 5007: 5002: 4984: 4982: 4981: 4976: 4974: 4950: 4948: 4947: 4942: 4890: 4888: 4887: 4882: 4880: 4872: 4869: 4859: 4858: 4855: 4851: 4842: 4818: 4806: 4800: 4799: 4796: 4792: 4783: 4777: 4757: 4740: 4736: 4735: 4726: 4725: 4709: 4706: 4702: 4693: 4675: 4649: 4640: 4636: 4635: 4624: 4623: 4620: 4616: 4607: 4598: 4597: 4588: 4587: 4572: 4564: 4563: 4554: 4553: 4538: 4537: 4528: 4527: 4512: 4503: 4499: 4498: 4487: 4486: 4483: 4479: 4474: 4473: 4458: 4446: 4437: 4436: 4427: 4426: 4411: 4403: 4402: 4392: 4388: 4385: 4381: 4376: 4375: 4351: 4350: 4328: 4319: 4318: 4309: 4308: 4289: 4276: 4274: 4273: 4268: 4249: 4235: 4229: 4227: 4226: 4221: 4210: 4186: 4184: 4183: 4178: 4160: 4154: 4152: 4151: 4146: 4128: 4126: 4125: 4120: 4081: 4079: 4078: 4073: 4059: 4057: 4056: 4051: 4025: 4021: 4012: 3974: 3972: 3971: 3966: 3952:to encode both. 3951: 3923: 3911: 3899: 3893: 3884: 3870: 3836: 3830: 3820: 3805: 3799: 3788: 3782: 3778: 3753:Characterization 3745: 3743: 3742: 3737: 3735: 3716: 3664: 3648: 3647: 3620: 3619: 3598: 3582: 3581: 3554: 3553: 3522: 3506: 3500: 3490: 3484: 3476: 3474: 3473: 3468: 3466: 3456: 3440: 3432: 3428: 3423: 3399: 3395: 3394: 3386: 3381: 3380: 3371: 3363: 3359: 3354: 3330: 3326: 3322: 3321: 3303: 3302: 3290: 3289: 3272: 3267: 3233: 3187: 3180: 3165: 3145: 3143: 3142: 3137: 3121: 3119: 3118: 3113: 3101: 3099: 3098: 3093: 3082: 3081: 3076: 3063: 3061: 3060: 3055: 3041: 3040: 3035: 3028: 3001: 3000: 2995: 2982: 2980: 2979: 2974: 2962: 2960: 2959: 2954: 2938: 2936: 2935: 2930: 2916: 2914: 2913: 2908: 2894: 2893: 2883: 2856: 2855: 2850: 2837: 2835: 2834: 2829: 2817: 2815: 2814: 2809: 2785: 2783: 2782: 2777: 2744: 2742: 2741: 2736: 2719: 2701: 2699: 2698: 2693: 2682: 2681: 2655: 2653: 2652: 2647: 2633: 2631: 2630: 2625: 2611: 2610: 2577: 2576: 2560: 2558: 2557: 2552: 2534: 2532: 2531: 2526: 2488: 2487: 2471: 2469: 2468: 2463: 2443: 2441: 2440: 2435: 2413: 2411: 2410: 2405: 2376:as follows: Let 2363: 2361: 2360: 2355: 2343: 2341: 2340: 2335: 2323: 2321: 2320: 2315: 2298: 2281: 2280: 2264: 2262: 2261: 2256: 2227: 2204: 2203: 2181: 2179: 2178: 2173: 2168: 2166: 2156: 2155: 2145: 2129: 2128: 2112: 2089: 2088: 2072: 2071: 2070: 2061: 2060: 2025: 2014: 2002: 2000: 1999: 1994: 1992: 1991: 1978: 1976: 1975: 1970: 1968: 1967: 1954: 1952: 1951: 1946: 1934: 1932: 1931: 1926: 1908: 1906: 1905: 1900: 1873: 1872: 1871: 1840: 1836: 1826: 1824: 1823: 1818: 1816: 1815: 1796: 1794: 1793: 1788: 1758: 1747: 1733: 1721: 1713: 1699: 1693: 1691: 1690: 1685: 1665: 1664: 1642: 1641: 1640: 1608: 1593: 1591: 1590: 1585: 1561: 1551: 1543: 1541: 1540: 1535: 1533: 1519: 1517: 1516: 1511: 1479: 1450: 1433: 1419: 1417: 1416: 1411: 1394: 1367: 1365: 1364: 1359: 1339: 1338: 1319: 1317: 1316: 1311: 1309: 1308: 1295: 1293: 1292: 1287: 1268: 1247:data compression 1228: 1221: 1214: 1212: 1211: 1206: 1198: 1197: 1173: 1166: 1159: 1150: 1148: 1147: 1142: 1137: 1116: 1114: 1113: 1108: 1103: 1079: 1077: 1076: 1071: 1066: 1062: 1060: 1043: 1034: 1033: 1002: 1000: 999: 994: 968: 967: 933: 931: 930: 925: 904: 896: 894: 893: 888: 876: 874: 873: 868: 863: 859: 857: 840: 821: 819: 818: 813: 792: 790: 789: 784: 763: 761: 760: 755: 734: 732: 731: 726: 707:also called the 679: 677: 676: 671: 642: 621:machine learning 567: 555:self-information 522: 520: 519: 514: 502: 500: 499: 494: 482: 480: 479: 474: 438: 437: 436: 404: 392: 390: 389: 384: 364: 363: 344: 342: 341: 336: 334: 333: 320: 318: 317: 312: 278: 271: 264: 240:Channel capacity 198:Relative entropy 155: 141: 140: 132: 125: 121: 118: 112: 110: 69: 45: 37: 21: 17210: 17209: 17205: 17204: 17203: 17201: 17200: 17199: 17165: 17164: 17163: 17158: 17125: 17109: 17093: 17074:Rate–distortion 17007: 16936: 16855: 16782: 16703: 16608: 16604:Sub-band coding 16512: 16437:Predictive type 16432: 16357: 16324:LZSS + Huffman 16274:LZ77 + Huffman 16263: 16173: 16109:Dictionary type 16103: 16005:Adaptive coding 15982: 15976: 15941:Wayback Machine 15923:Wayback Machine 15897: 15894: 15893: 15892: 15872: 15871: 15867: 15860: 15843:Wayback Machine 15805: 15728: 15723: 15721:Further reading 15705: 15704: 15697: 15681: 15677: 15667: 15665: 15630: 15626: 15616: 15614: 15607: 15581: 15577: 15572: 15568: 15563: 15559: 15549: 15547: 15532:10.19086/da.609 15506: 15502: 15492: 15490: 15487:Quanta Magazine 15475: 15471: 15463: 15459: 15452: 15428: 15424: 15414: 15412: 15408: 15397: 15391: 15387: 15377: 15375: 15371: 15360: 15354: 15350: 15297: 15293: 15281:Wayback Machine 15272: 15265: 15255: 15253: 15252:on 1 March 2018 15242: 15238: 15228: 15226: 15191: 15187: 15152:Physical Review 15144: 15140: 15111: 15107: 15096: 15092: 15079: 15075: 15052:10.2307/1426210 15036: 15027: 15017: 15015: 15011: 14988: 14982: 14978: 14973:Wayback Machine 14959: 14955: 14939: 14938: 14931: 14929: 14925: 14918: 14910: 14906: 14892: 14888: 14881: 14867: 14842: 14833: 14829: 14822: 14806: 14802: 14795: 14779: 14775: 14766: 14762: 14752: 14750: 14743: 14724: 14720: 14706: 14704: 14695: 14694: 14690: 14684:Wayback Machine 14632: 14625: 14619:Wayback Machine 14567: 14560: 14553: 14537: 14533: 14528: 14523: 14522: 14484: 14483: 14475: 14472: 14471: 14440: 14437: 14436: 14389: 14383: 14380: 14379: 14351: 14348: 14347: 14345: 14341: 14336: 14331: 14173: 14166: 14163: 14109: 14106: 14105: 14089: 14086: 14085: 14069: 14066: 14065: 14049: 14046: 14045: 14011: 14008: 14007: 13992: 13963: 13952: 13948: 13944: 13942: 13939: 13938: 13932: 13926: 13910: 13883: 13878: 13860: 13856: 13835: 13831: 13824: 13811: 13810: 13809: 13807: 13804: 13803: 13774: 13770: 13731: 13727: 13709: 13705: 13697: 13684: 13683: 13681: 13675: 13664: 13658: 13655: 13654: 13626: 13622: 13601: 13597: 13589: 13576: 13575: 13573: 13571: 13568: 13567: 13561:Proof (sketch) 13516: 13512: 13476: 13472: 13449: 13447: 13444: 13443: 13407: 13403: 13399: 13388: 13375: 13374: 13372: 13356: 13340: 13336: 13332: 13330: 13328: 13325: 13324: 13307: 13296: 13293: 13272: 13257: 13253: 13248: 13228: 13224: 13217: 13213: 13207: 13203: 13192: 13190: 13187: 13186: 13178: 13170: 13151: 13147: 13140: 13136: 13130: 13126: 13121: 13118: 13117: 13102: 13094: 13088: 13080: 13078: 13070: 13064: 13058: 13052: 13046: 13040: 13032: 13029: 13021: 13015: 13012: 13004: 12985: 12981: 12974: 12970: 12964: 12960: 12955: 12952: 12951: 12926: 12922: 12915: 12911: 12905: 12901: 12890: 12884: 12873: 12859: 12844: 12840: 12825: 12821: 12810: 12808: 12805: 12804: 12795: 12789: 12782: 12781:are subsets of 12779: 12770: 12764: 12761: 12752: 12746: 12707: 12703: 12688: 12684: 12669: 12665: 12644: 12640: 12625: 12621: 12606: 12602: 12578: 12574: 12572: 12569: 12568: 12564:th coordinate: 12559: 12552: 12544: 12525: 12510: 12506: 12501: 12495: 12484: 12465: 12460: 12459: 12451: 12449: 12446: 12445: 12432: 12425: 12413: 12402:Wayback Machine 12391:Wayback Machine 12368: 12360: 12329: 12326: 12325: 12322: 12314: 12297: 12286: 12280: 12270: 12264: 12254: 12244: 12100: 12099: 12095: 12093: 12090: 12089: 12080: 12074: 12068: 12041: 12035: 12025: 12019: 12008: 12002: 11965: 11962: 11961: 11939: 11936: 11935: 11929: 11859: 11851: 11839: 11837: 11834: 11833: 11824: 11818: 11811: 11800: 11794: 11786: 11780: 11764: 11700: 11692: 11659: 11654: 11653: 11652: 11648: 11636: 11615: 11612: 11611: 11604: 11600: 11583: 11582: 11536: 11528: 11517: 11505: 11501: 11474: 11470: 11458: 11444: 11437: 11436: 11405: 11397: 11386: 11374: 11370: 11358: 11344: 11336: 11334: 11331: 11330: 11323: 11280: 11276: 11264: 11250: 11231: 11227: 11200: 11196: 11184: 11170: 11154: 11149: 11148: 11146: 11143: 11142: 11114: 11110: 11103: 11099: 11081: 11077: 11065: 11051: 11035: 11030: 11029: 11027: 11024: 11023: 11022:We will denote 10994: 10990: 10978: 10964: 10948: 10916: 10908: 10902: 10899: 10898: 10892: 10841: 10833: 10814: 10810: 10802: 10799: 10798: 10795: 10787: 10771: 10768: 10767: 10761: 10755: 10747: 10741: 10709: 10703: 10678: 10640: 10639: 10635: 10597: 10580: 10578: 10575: 10574: 10555: 10553: 10550: 10549: 10538: 10532: 10526: 10521: 10510: 10480: 10476: 10475: 10473: 10470: 10469: 10434: 10430: 10420: 10416: 10410: 10406: 10394: 10383: 10367: 10363: 10346: 10342: 10332: 10328: 10322: 10318: 10300: 10296: 10290: 10279: 10251: 10247: 10246: 10231: 10227: 10217: 10213: 10207: 10203: 10185: 10181: 10180: 10178: 10172: 10161: 10133: 10129: 10128: 10115: 10111: 10093: 10089: 10080: 10076: 10069: 10067: 10061: 10050: 10026: 10023: 10022: 9981: 9977: 9976: 9963: 9959: 9941: 9937: 9928: 9924: 9917: 9915: 9909: 9898: 9874: 9870: 9865: 9848: 9845: 9844: 9823: 9822: 9820: 9817: 9816: 9813: 9773: 9769: 9736: 9732: 9726: 9707: 9703: 9697: 9687: 9683: 9677: 9658: 9657: 9649: 9647: 9644: 9643: 9631: 9625: 9599: 9595: 9593: 9590: 9589: 9579: 9547: 9543: 9522: 9518: 9509: 9499: 9495: 9489: 9470: 9469: 9461: 9459: 9456: 9455: 9436: 9433: 9425: 9402: 9398: 9386: 9382: 9364: 9363: 9355: 9353: 9350: 9349: 9339: 9305: 9301: 9299: 9296: 9295: 9284: 9271: 9267: 9238: 9230: 9226: 9219: 9213: 9154: 9141: 9122: 9120:Diversity index 9116: 8965: 8957:Main articles: 8955: 8939:Maxwell's demon 8914:In the view of 8908: 8902: 8895: 8887: 8880: 8874: 8868: 8852: 8849: 8848: 8816: 8812: 8804: 8801: 8800: 8775: 8765: 8762: 8756: 8733:where ρ is the 8689: 8688: 8681: 8677: 8666: 8663: 8662: 8648:quantum physics 8635:was defined by 8625: 8617: 8610: 8604: 8580: 8576: 8564: 8560: 8551: 8547: 8536: 8533: 8532: 8516: 8495: 8489: 8484: 8433: 8430: 8429: 8412: 8408: 8399: 8395: 8393: 8390: 8389: 8365: 8361: 8353: 8326: 8322: 8314: 8299: 8295: 8271: 8267: 8256: 8254: 8251: 8250: 8228: 8225: 8224: 8195: 8193: 8190: 8189: 8161: 8144: 8121: 8119: 8116: 8115: 8083: 8069: 8058: 8056: 8053: 8052: 8040: 8034: 8000: 7986: 7975: 7973: 7970: 7969: 7957: 7951: 7945: 7939: 7909: 7883: 7881: 7878: 7877: 7839: 7828: 7802: 7788: 7768: 7751: 7749: 7746: 7745: 7699: 7697: 7694: 7693: 7665: 7645: 7643: 7640: 7639: 7623: 7620: 7619: 7588: 7585: 7584: 7550: 7536: 7525: 7508: 7494: 7483: 7460: 7458: 7455: 7454: 7442: 7436: 7430: 7424: 7418: 7406: 7376: 7372: 7360: 7356: 7341: 7337: 7329: 7327: 7324: 7323: 7309: 7303: 7272: 7268: 7253: 7249: 7240: 7235: 7234: 7216: 7212: 7197: 7193: 7178: 7173: 7172: 7170: 7167: 7166: 7150: 7146: 7138:Hartley entropy 7121: 7119: 7116: 7115: 7095: 7091: 7076: 7072: 7070: 7067: 7066: 7049: 7047: 7044: 7043: 7040: 6990: 6985: 6984: 6976: 6972: 6965: 6959: 6956: 6955: 6935: 6931: 6916: 6912: 6910: 6907: 6906: 6886: 6882: 6867: 6863: 6854: 6849: 6848: 6846: 6843: 6842: 6819: 6815: 6800: 6796: 6787: 6782: 6781: 6763: 6759: 6744: 6740: 6725: 6720: 6719: 6717: 6714: 6713: 6688: 6685: 6684: 6659: 6642: 6619: 6617: 6614: 6613: 6588: 6585: 6584: 6559: 6542: 6519: 6517: 6514: 6513: 6502: 6465: 6461: 6459: 6456: 6455: 6409: 6406: 6405: 6334: 6331: 6330: 6309: 6303: 6299: 6295: 6291: 6282: 6276: 6266: 6236: 6232: 6227: 6210: 6206: 6201: 6200: 6196: 6188: 6184: 6183: 6178: 6177: 6165: 6161: 6159: 6153: 6142: 6119: 6115: 6113: 6093: 6089: 6087: 6086: 6082: 6076: 6071: 6070: 6052: 6033: 6032: 6028: 6022: 6017: 6016: 6014: 6011: 6010: 6000: 5991: 5985: 5982: 5974: 5968: 5958: 5949: 5943: 5937: 5931: 5909: 5908: 5896: 5877: 5872: 5850: 5845: 5844: 5842: 5835: 5834: 5822: 5817: 5816: 5807: 5806: 5800: 5784: 5765: 5764: 5762: 5755: 5754: 5748: 5743: 5742: 5740: 5737: 5736: 5707: 5688: 5687: 5683: 5677: 5672: 5671: 5659: 5655: 5640: 5636: 5627: 5622: 5621: 5619: 5616: 5615: 5598: 5593: 5592: 5590: 5587: 5586: 5543: 5540: 5539: 5519: 5515: 5494: 5490: 5485: 5482: 5481: 5454: 5450: 5449: 5445: 5428: 5424: 5423: 5419: 5408: 5404: 5403: 5399: 5398: 5394: 5388: 5383: 5382: 5368: 5364: 5352: 5348: 5339: 5335: 5334: 5330: 5324: 5319: 5318: 5316: 5313: 5312: 5309: 5301: 5297: 5286: 5274: 5265: 5258: 5252: 5248: 5235: 5227: 5224: 5197: 5190: 5184: 5183:tosses provide 5178: 5174: 5170: 5165:and so on) are 5163: 5159: 5148: 5138: 5134: 5092: 5089: 5088: 5054: 5051: 5050: 5016: 5013: 5012: 4990: 4987: 4986: 4970: 4956: 4953: 4952: 4903: 4900: 4899: 4878: 4877: 4868: 4854: 4850: 4845: 4837: 4811: 4804: 4803: 4795: 4791: 4786: 4778: 4770: 4750: 4738: 4737: 4731: 4727: 4721: 4717: 4705: 4701: 4696: 4688: 4668: 4642: 4638: 4637: 4631: 4627: 4619: 4615: 4610: 4602: 4593: 4589: 4583: 4579: 4565: 4559: 4555: 4549: 4545: 4533: 4529: 4523: 4519: 4505: 4501: 4500: 4494: 4490: 4482: 4478: 4469: 4465: 4451: 4449: 4441: 4432: 4428: 4422: 4418: 4404: 4398: 4394: 4390: 4389: 4384: 4380: 4371: 4367: 4346: 4342: 4331: 4323: 4314: 4310: 4304: 4300: 4286: 4284: 4281: 4280: 4262: 4259: 4258: 4231: 4206: 4192: 4189: 4188: 4166: 4163: 4162: 4156: 4134: 4131: 4130: 4087: 4084: 4083: 4067: 4064: 4063: 4010: 4006: 3980: 3977: 3976: 3960: 3957: 3956: 3945: 3937: 3929: 3925: 3917: 3913: 3905: 3901: 3895: 3889: 3880: 3868: 3861: 3854: 3847: 3840: 3834: 3826: 3814: 3801: 3798: 3790: 3784: 3780: 3776: 3767: 3758: 3755: 3733: 3732: 3714: 3713: 3662: 3661: 3643: 3639: 3615: 3611: 3596: 3595: 3577: 3573: 3549: 3545: 3532: 3518: 3514: 3512: 3509: 3508: 3502: 3492: 3486: 3480: 3464: 3463: 3431: 3430: 3424: 3413: 3397: 3396: 3385: 3376: 3372: 3362: 3361: 3355: 3344: 3328: 3327: 3317: 3313: 3298: 3294: 3285: 3281: 3274: 3268: 3257: 3243: 3229: 3225: 3223: 3220: 3219: 3208: 3200:Main articles: 3195: 3190: 3189: 3182: 3174: 3159: 3152: 3131: 3128: 3127: 3107: 3104: 3103: 3077: 3072: 3071: 3069: 3066: 3065: 3036: 3031: 3030: 3018: 2996: 2991: 2990: 2988: 2985: 2984: 2968: 2965: 2964: 2948: 2945: 2944: 2924: 2921: 2920: 2889: 2885: 2873: 2851: 2846: 2845: 2843: 2840: 2839: 2823: 2820: 2819: 2791: 2788: 2787: 2750: 2747: 2746: 2715: 2707: 2704: 2703: 2677: 2676: 2668: 2665: 2664: 2641: 2638: 2637: 2606: 2602: 2572: 2568: 2566: 2563: 2562: 2546: 2543: 2542: 2483: 2479: 2477: 2474: 2473: 2457: 2454: 2453: 2423: 2420: 2419: 2381: 2378: 2377: 2370: 2349: 2346: 2345: 2329: 2326: 2325: 2294: 2276: 2272: 2270: 2267: 2266: 2223: 2193: 2189: 2187: 2184: 2183: 2151: 2147: 2146: 2118: 2114: 2113: 2111: 2078: 2074: 2066: 2065: 2056: 2055: 2042: 2021: 2010: 2008: 2005: 2004: 1987: 1986: 1984: 1981: 1980: 1963: 1962: 1960: 1957: 1956: 1940: 1937: 1936: 1920: 1917: 1916: 1867: 1863: 1856: 1850: 1847: 1846: 1838: 1837:is taken to be 1834: 1828: 1811: 1810: 1802: 1799: 1798: 1767: 1764: 1763: 1762:In the case of 1753: 1739: 1728: 1717: 1716:Euler's number 1709: 1695: 1660: 1656: 1636: 1635: 1628: 1604: 1602: 1599: 1598: 1567: 1564: 1563: 1557: 1549: 1529: 1527: 1524: 1523: 1475: 1446: 1429: 1427: 1424: 1423: 1390: 1373: 1370: 1369: 1334: 1333: 1325: 1322: 1321: 1304: 1303: 1301: 1298: 1297: 1281: 1278: 1277: 1266: 1259: 1243: 1223: 1216: 1193: 1189: 1187: 1184: 1183: 1168: 1161: 1155: 1133: 1122: 1119: 1118: 1099: 1088: 1085: 1084: 1047: 1042: 1038: 1029: 1025: 1008: 1005: 1004: 963: 959: 939: 936: 935: 919: 916: 915: 902: 882: 879: 878: 844: 839: 835: 827: 824: 823: 798: 795: 794: 769: 766: 765: 740: 737: 736: 720: 717: 716: 686: 638: 636: 633: 632: 582:Shannon entropy 565: 557:of a variable. 535:"), while base 508: 505: 504: 488: 485: 484: 432: 431: 424: 400: 398: 395: 394: 359: 358: 350: 347: 346: 329: 328: 326: 323: 322: 306: 303: 302: 298:random variable 282: 133: 122: 116: 113: 70: 68: 58: 46: 35: 28: 23: 22: 18:Shannon entropy 15: 12: 11: 5: 17208: 17198: 17197: 17192: 17187: 17182: 17177: 17160: 17159: 17157: 17156: 17141: 17130: 17127: 17126: 17124: 17123: 17117: 17115: 17111: 17110: 17108: 17107: 17101: 17099: 17095: 17094: 17092: 17091: 17086: 17081: 17076: 17071: 17066: 17061: 17056: 17055: 17054: 17044: 17039: 17038: 17037: 17032: 17021: 17019: 17013: 17012: 17009: 17008: 17006: 17005: 17004: 17003: 16998: 16988: 16987: 16986: 16981: 16976: 16968: 16963: 16958: 16953: 16947: 16945: 16938: 16937: 16935: 16934: 16929: 16924: 16919: 16914: 16909: 16904: 16899: 16898: 16897: 16892: 16887: 16876: 16874: 16867: 16861: 16860: 16857: 16856: 16854: 16853: 16852: 16851: 16846: 16841: 16836: 16826: 16821: 16816: 16811: 16806: 16801: 16796: 16790: 16788: 16784: 16783: 16781: 16780: 16775: 16770: 16765: 16760: 16755: 16750: 16745: 16740: 16735: 16730: 16724: 16722: 16715: 16709: 16708: 16705: 16704: 16702: 16701: 16696: 16691: 16690: 16689: 16684: 16679: 16674: 16669: 16659: 16658: 16657: 16647: 16646: 16645: 16640: 16630: 16625: 16619: 16617: 16610: 16609: 16607: 16606: 16601: 16596: 16591: 16586: 16581: 16576: 16571: 16566: 16561: 16556: 16555: 16554: 16549: 16544: 16533: 16531: 16524: 16518: 16517: 16514: 16513: 16511: 16510: 16508:Psychoacoustic 16505: 16504: 16503: 16498: 16493: 16485: 16484: 16483: 16478: 16473: 16468: 16463: 16453: 16452: 16451: 16440: 16438: 16434: 16433: 16431: 16430: 16429: 16428: 16423: 16418: 16408: 16403: 16398: 16397: 16396: 16391: 16380: 16378: 16376:Transform type 16369: 16363: 16362: 16359: 16358: 16356: 16355: 16354: 16353: 16345: 16344: 16343: 16340: 16332: 16331: 16330: 16322: 16321: 16320: 16312: 16311: 16310: 16302: 16301: 16300: 16292: 16291: 16290: 16285: 16280: 16271: 16269: 16265: 16264: 16262: 16261: 16256: 16251: 16246: 16241: 16236: 16235: 16234: 16229: 16219: 16214: 16209: 16208: 16207: 16197: 16192: 16187: 16181: 16179: 16175: 16174: 16172: 16171: 16170: 16169: 16164: 16159: 16154: 16149: 16144: 16139: 16134: 16129: 16119: 16113: 16111: 16105: 16104: 16102: 16101: 16100: 16099: 16094: 16089: 16084: 16074: 16069: 16064: 16059: 16054: 16049: 16044: 16043: 16042: 16037: 16032: 16022: 16017: 16012: 16007: 16001: 15999: 15990: 15984: 15983: 15975: 15974: 15967: 15960: 15952: 15946: 15945: 15930: 15913: 15891: 15890: 15885: 15880: 15874: 15873: 15862: 15861: 15859: 15858:External links 15856: 15855: 15854: 15851:978-0956372857 15830: 15809: 15803: 15788: 15781: 15767: 15753:MacKay, D.J.C. 15750: 15727: 15724: 15722: 15719: 15703: 15702: 15695: 15675: 15644:(3): 227–241. 15624: 15605: 15575: 15566: 15557: 15500: 15469: 15457: 15450: 15422: 15385: 15348: 15311:(3): 177–179. 15291: 15263: 15236: 15205:(3): 183–191. 15185: 15158:(4): 620–630. 15138: 15105: 15090: 15073: 15046:(1): 131–146. 15025: 14999:(5): 806–835. 14976: 14953: 14904: 14886: 14879: 14840: 14827: 14820: 14800: 14793: 14773: 14760: 14741: 14718: 14688: 14651:(4): 623–656. 14623: 14586:(3): 379–423. 14558: 14552:978-0123821881 14551: 14530: 14529: 14527: 14524: 14521: 14520: 14507: 14504: 14501: 14498: 14495: 14492: 14487: 14482: 14479: 14459: 14456: 14453: 14450: 14447: 14444: 14424: 14421: 14418: 14415: 14412: 14409: 14406: 14403: 14398: 14395: 14392: 14388: 14367: 14364: 14361: 14358: 14355: 14338: 14337: 14335: 14332: 14330: 14329: 14324: 14319: 14314: 14311:Sample entropy 14308: 14303: 14297: 14291: 14278: 14273: 14268: 14263: 14254: 14249: 14244: 14239: 14234: 14229: 14224: 14219: 14214: 14209: 14203: 14198: 14192: 14187: 14180: 14179: 14178: 14162: 14159: 14113: 14093: 14073: 14053: 14033: 14030: 14027: 14024: 14021: 14018: 14015: 13991: 13988: 13973: 13970: 13966: 13962: 13959: 13955: 13951: 13947: 13923: 13922: 13919: 13918: 13907: 13906: 13892: 13889: 13886: 13882: 13877: 13872: 13869: 13866: 13863: 13859: 13855: 13852: 13849: 13846: 13841: 13838: 13834: 13827: 13822: 13819: 13814: 13797: 13796: 13785: 13782: 13777: 13773: 13769: 13766: 13763: 13760: 13757: 13754: 13751: 13748: 13745: 13740: 13737: 13734: 13730: 13726: 13723: 13720: 13717: 13712: 13708: 13700: 13695: 13692: 13687: 13678: 13673: 13670: 13667: 13663: 13638: 13635: 13632: 13629: 13625: 13621: 13618: 13615: 13612: 13607: 13604: 13600: 13592: 13587: 13584: 13579: 13563: 13562: 13554: 13553: 13542: 13539: 13536: 13533: 13530: 13527: 13524: 13519: 13515: 13511: 13508: 13505: 13502: 13499: 13496: 13493: 13490: 13487: 13484: 13479: 13475: 13471: 13468: 13465: 13462: 13459: 13456: 13452: 13437: 13436: 13425: 13420: 13417: 13414: 13410: 13406: 13402: 13398: 13391: 13386: 13383: 13378: 13371: 13365: 13362: 13359: 13353: 13350: 13347: 13343: 13339: 13335: 13292: 13289: 13275: 13271: 13268: 13265: 13260: 13256: 13251: 13247: 13244: 13241: 13238: 13231: 13227: 13223: 13220: 13216: 13210: 13206: 13202: 13199: 13195: 13174: 13154: 13150: 13146: 13143: 13139: 13133: 13129: 13125: 13103:= {1, 2, ..., 13098: 13036: 13025: 13008: 12988: 12984: 12980: 12977: 12973: 12967: 12963: 12959: 12948: 12947: 12936: 12929: 12925: 12921: 12918: 12914: 12908: 12904: 12900: 12897: 12893: 12887: 12882: 12879: 12876: 12872: 12866: 12863: 12858: 12855: 12852: 12847: 12843: 12839: 12836: 12833: 12828: 12824: 12820: 12817: 12813: 12775: 12768: 12757: 12750: 12739: 12738: 12727: 12724: 12721: 12718: 12715: 12710: 12706: 12702: 12699: 12696: 12691: 12687: 12683: 12680: 12677: 12672: 12668: 12664: 12661: 12658: 12653: 12650: 12647: 12643: 12639: 12634: 12631: 12628: 12624: 12620: 12617: 12614: 12609: 12605: 12601: 12598: 12595: 12592: 12589: 12586: 12581: 12577: 12548: 12541: 12540: 12528: 12524: 12521: 12518: 12513: 12509: 12504: 12498: 12493: 12490: 12487: 12483: 12479: 12474: 12471: 12468: 12463: 12458: 12454: 12424: 12421: 12412: 12409: 12364: 12348: 12345: 12342: 12339: 12336: 12333: 12318: 12296: 12293: 12241: 12240: 12229: 12226: 12223: 12220: 12217: 12214: 12211: 12208: 12205: 12202: 12199: 12196: 12193: 12190: 12187: 12184: 12181: 12178: 12175: 12172: 12169: 12166: 12163: 12160: 12157: 12154: 12151: 12148: 12145: 12142: 12139: 12136: 12133: 12130: 12127: 12124: 12121: 12118: 12115: 12112: 12106: 12103: 12098: 12004:Main article: 12001: 11998: 11981: 11978: 11975: 11972: 11969: 11949: 11946: 11943: 11926: 11925: 11914: 11911: 11908: 11904: 11901: 11897: 11894: 11891: 11888: 11885: 11882: 11879: 11876: 11873: 11870: 11867: 11862: 11857: 11854: 11850: 11846: 11842: 11782:Main article: 11779: 11776: 11757: 11756: 11745: 11742: 11739: 11735: 11732: 11729: 11726: 11723: 11720: 11717: 11714: 11711: 11708: 11703: 11698: 11695: 11691: 11687: 11684: 11680: 11676: 11673: 11670: 11667: 11662: 11657: 11651: 11645: 11642: 11639: 11635: 11631: 11628: 11625: 11622: 11619: 11597: 11596: 11581: 11578: 11575: 11571: 11568: 11565: 11562: 11559: 11556: 11553: 11550: 11547: 11544: 11539: 11534: 11531: 11527: 11523: 11520: 11518: 11516: 11513: 11508: 11504: 11500: 11497: 11494: 11491: 11488: 11485: 11482: 11477: 11473: 11469: 11466: 11461: 11456: 11453: 11450: 11447: 11443: 11439: 11438: 11435: 11432: 11429: 11426: 11422: 11419: 11416: 11413: 11408: 11403: 11400: 11396: 11392: 11389: 11387: 11385: 11382: 11377: 11373: 11369: 11366: 11361: 11356: 11353: 11350: 11347: 11343: 11339: 11338: 11309: 11306: 11303: 11300: 11297: 11294: 11291: 11288: 11283: 11279: 11275: 11272: 11267: 11262: 11259: 11256: 11253: 11249: 11245: 11242: 11239: 11234: 11230: 11226: 11223: 11220: 11217: 11214: 11211: 11208: 11203: 11199: 11195: 11192: 11187: 11182: 11179: 11176: 11173: 11169: 11165: 11162: 11157: 11152: 11129: 11125: 11122: 11117: 11113: 11109: 11106: 11102: 11098: 11095: 11092: 11089: 11084: 11080: 11076: 11073: 11068: 11063: 11060: 11057: 11054: 11050: 11046: 11043: 11038: 11033: 11008: 11005: 11002: 10997: 10993: 10989: 10986: 10981: 10976: 10973: 10970: 10967: 10963: 10957: 10954: 10951: 10947: 10943: 10940: 10937: 10933: 10930: 10927: 10924: 10919: 10914: 10911: 10907: 10880: 10877: 10873: 10870: 10867: 10864: 10859: 10856: 10853: 10850: 10847: 10844: 10839: 10836: 10832: 10828: 10825: 10822: 10817: 10813: 10809: 10806: 10791: 10775: 10751: 10700: 10699: 10688: 10685: 10681: 10676: 10673: 10670: 10667: 10664: 10661: 10658: 10655: 10652: 10649: 10643: 10638: 10634: 10631: 10628: 10625: 10622: 10619: 10616: 10613: 10610: 10607: 10604: 10600: 10596: 10593: 10590: 10587: 10583: 10558: 10528:Main article: 10525: 10522: 10520: 10517: 10497: 10494: 10491: 10488: 10483: 10479: 10462: 10461: 10450: 10447: 10442: 10437: 10433: 10429: 10426: 10423: 10419: 10413: 10409: 10405: 10402: 10397: 10392: 10389: 10386: 10382: 10378: 10375: 10370: 10366: 10362: 10359: 10354: 10349: 10345: 10341: 10338: 10335: 10331: 10325: 10321: 10317: 10314: 10311: 10308: 10303: 10299: 10293: 10288: 10285: 10282: 10278: 10274: 10268: 10265: 10262: 10259: 10254: 10250: 10244: 10239: 10234: 10230: 10226: 10223: 10220: 10216: 10210: 10206: 10202: 10199: 10196: 10193: 10188: 10184: 10175: 10170: 10167: 10164: 10160: 10156: 10150: 10147: 10144: 10141: 10136: 10132: 10126: 10123: 10118: 10114: 10110: 10107: 10104: 10101: 10096: 10092: 10088: 10083: 10079: 10075: 10072: 10064: 10059: 10056: 10053: 10049: 10045: 10042: 10039: 10036: 10033: 10030: 10016: 10015: 10004: 9998: 9995: 9992: 9989: 9984: 9980: 9974: 9971: 9966: 9962: 9958: 9955: 9952: 9949: 9944: 9940: 9936: 9931: 9927: 9923: 9920: 9912: 9907: 9904: 9901: 9897: 9893: 9890: 9883: 9880: 9877: 9873: 9869: 9864: 9861: 9858: 9855: 9852: 9826: 9812: 9809: 9808: 9807: 9796: 9793: 9790: 9787: 9782: 9779: 9776: 9772: 9765: 9762: 9756: 9753: 9750: 9745: 9742: 9739: 9735: 9729: 9725: 9721: 9718: 9715: 9710: 9706: 9700: 9696: 9690: 9686: 9680: 9676: 9672: 9669: 9666: 9661: 9656: 9652: 9613: 9610: 9607: 9602: 9598: 9576: 9575: 9564: 9561: 9558: 9555: 9550: 9546: 9542: 9539: 9536: 9533: 9530: 9525: 9521: 9512: 9508: 9502: 9498: 9492: 9488: 9484: 9481: 9478: 9473: 9468: 9464: 9429: 9422: 9421: 9410: 9405: 9401: 9397: 9394: 9389: 9385: 9381: 9378: 9375: 9372: 9367: 9362: 9358: 9338: 9335: 9308: 9304: 9283: 9280: 9269: 9236: 9228: 9190: 9189: 9176: 9167: 9153: 9150: 9146:true diversity 9118:Main article: 9115: 9112: 9097: 9096: 9093: 9090: 9086: 9085: 9082: 9079: 9075: 9074: 9071: 9068: 9064: 9063: 9060: 9057: 8954: 8951: 8906: 8891: 8878: 8856: 8845: 8844: 8833: 8830: 8827: 8824: 8815: 8811: 8808: 8773: 8760: 8735:density matrix 8731: 8730: 8719: 8715: 8712: 8709: 8706: 8703: 8700: 8695: 8692: 8680: 8676: 8673: 8670: 8654:introduced by 8621: 8608: 8601: 8600: 8589: 8583: 8579: 8575: 8572: 8567: 8563: 8559: 8550: 8546: 8543: 8540: 8491:Main article: 8488: 8485: 8483: 8480: 8479: 8478: 8477: 8476: 8449: 8446: 8443: 8440: 8437: 8415: 8411: 8407: 8402: 8398: 8386: 8385: 8384: 8373: 8368: 8364: 8360: 8356: 8352: 8349: 8346: 8343: 8340: 8337: 8334: 8329: 8325: 8321: 8317: 8313: 8310: 8307: 8302: 8298: 8294: 8291: 8288: 8285: 8282: 8279: 8274: 8270: 8266: 8263: 8259: 8245: 8244: 8232: 8208: 8205: 8202: 8198: 8186: 8174: 8171: 8168: 8164: 8160: 8157: 8154: 8151: 8147: 8143: 8140: 8137: 8134: 8131: 8128: 8124: 8111: 8110: 8109: 8108: 8096: 8093: 8090: 8086: 8082: 8079: 8076: 8072: 8068: 8065: 8061: 8047: 8046: 8030: 8029: 8028: 8027: 8016: 8013: 8010: 8007: 8003: 7999: 7996: 7993: 7989: 7985: 7982: 7978: 7964: 7963: 7935: 7934: 7922: 7919: 7916: 7912: 7908: 7905: 7902: 7899: 7896: 7893: 7890: 7886: 7874: 7873: 7872: 7861: 7858: 7855: 7852: 7849: 7846: 7842: 7838: 7835: 7831: 7827: 7824: 7821: 7818: 7815: 7812: 7809: 7805: 7801: 7798: 7795: 7791: 7787: 7784: 7781: 7778: 7775: 7771: 7767: 7764: 7761: 7758: 7754: 7740: 7739: 7727: 7724: 7721: 7718: 7715: 7712: 7709: 7706: 7702: 7681: 7678: 7675: 7672: 7668: 7664: 7661: 7658: 7655: 7652: 7648: 7627: 7607: 7604: 7601: 7598: 7595: 7592: 7580: 7579: 7578: 7577: 7566: 7563: 7560: 7557: 7553: 7549: 7546: 7543: 7539: 7535: 7532: 7528: 7524: 7521: 7518: 7515: 7511: 7507: 7504: 7501: 7497: 7493: 7490: 7486: 7482: 7479: 7476: 7473: 7470: 7467: 7463: 7449: 7448: 7402: 7401: 7400: 7399: 7387: 7384: 7379: 7375: 7371: 7368: 7363: 7359: 7355: 7352: 7349: 7344: 7340: 7336: 7332: 7318: 7317: 7305: 7295: 7294: 7293: 7292: 7280: 7275: 7271: 7267: 7264: 7261: 7256: 7252: 7248: 7243: 7238: 7233: 7230: 7227: 7224: 7219: 7215: 7211: 7208: 7205: 7200: 7196: 7192: 7187: 7184: 7181: 7176: 7161: 7160: 7145: 7142: 7124: 7098: 7094: 7090: 7087: 7084: 7079: 7075: 7052: 7039: 7036: 7035: 7034: 7022: 7019: 7016: 7013: 7010: 7007: 7004: 7001: 6998: 6993: 6988: 6979: 6975: 6971: 6968: 6964: 6952: 6938: 6934: 6930: 6927: 6924: 6919: 6915: 6894: 6889: 6885: 6881: 6878: 6875: 6870: 6866: 6862: 6857: 6852: 6839: 6827: 6822: 6818: 6814: 6811: 6808: 6803: 6799: 6795: 6790: 6785: 6780: 6777: 6774: 6771: 6766: 6762: 6758: 6755: 6752: 6747: 6743: 6739: 6734: 6731: 6728: 6723: 6710: 6698: 6695: 6692: 6672: 6669: 6666: 6662: 6658: 6655: 6652: 6649: 6645: 6641: 6638: 6635: 6632: 6629: 6626: 6622: 6610: 6598: 6595: 6592: 6572: 6569: 6566: 6562: 6558: 6555: 6552: 6549: 6545: 6541: 6538: 6535: 6532: 6529: 6526: 6522: 6501: 6498: 6494:Shannon's bits 6486:David Ellerman 6468: 6464: 6443: 6440: 6437: 6434: 6431: 6428: 6425: 6422: 6419: 6416: 6413: 6389: 6386: 6383: 6380: 6377: 6374: 6371: 6368: 6365: 6362: 6359: 6356: 6353: 6350: 6347: 6344: 6341: 6338: 6297: 6287: 6280: 6263: 6262: 6251: 6247: 6239: 6235: 6231: 6226: 6223: 6220: 6213: 6209: 6205: 6199: 6191: 6187: 6181: 6173: 6168: 6164: 6156: 6151: 6148: 6145: 6141: 6137: 6133: 6127: 6122: 6118: 6112: 6109: 6106: 6101: 6096: 6092: 6085: 6079: 6074: 6069: 6065: 6059: 6056: 6051: 6048: 6045: 6040: 6037: 6031: 6025: 6020: 5996: 5989: 5978: 5967: 5964: 5963: 5962: 5954: 5947: 5928: 5917: 5912: 5905: 5902: 5899: 5893: 5886: 5883: 5880: 5876: 5871: 5868: 5865: 5859: 5856: 5853: 5849: 5838: 5831: 5828: 5825: 5820: 5815: 5810: 5803: 5797: 5791: 5788: 5783: 5780: 5777: 5772: 5769: 5758: 5751: 5746: 5733: 5720: 5714: 5711: 5706: 5703: 5700: 5695: 5692: 5686: 5680: 5675: 5670: 5667: 5662: 5658: 5654: 5651: 5648: 5643: 5639: 5635: 5630: 5625: 5601: 5596: 5583: 5571: 5568: 5565: 5562: 5559: 5556: 5553: 5550: 5547: 5527: 5522: 5518: 5514: 5511: 5508: 5505: 5502: 5497: 5493: 5489: 5465: 5457: 5453: 5448: 5444: 5441: 5438: 5431: 5427: 5422: 5418: 5411: 5407: 5402: 5397: 5391: 5386: 5381: 5377: 5371: 5367: 5363: 5360: 5355: 5351: 5347: 5342: 5338: 5333: 5327: 5322: 5305: 5294: 5270: 5263: 5254: 5244: 5231: 5223: 5220: 5172: 5161: 5136: 5122:The different 5120: 5119: 5116: 5115: 5102: 5099: 5096: 5076: 5073: 5070: 5067: 5064: 5061: 5058: 5038: 5035: 5032: 5029: 5026: 5023: 5020: 5000: 4997: 4994: 4973: 4969: 4966: 4963: 4960: 4940: 4937: 4934: 4931: 4928: 4925: 4922: 4919: 4916: 4913: 4910: 4907: 4892: 4891: 4876: 4867: 4864: 4852: 4849: 4846: 4841: 4838: 4836: 4833: 4830: 4827: 4824: 4821: 4817: 4814: 4810: 4807: 4805: 4793: 4790: 4787: 4782: 4779: 4776: 4773: 4769: 4766: 4763: 4760: 4756: 4753: 4749: 4746: 4743: 4741: 4739: 4734: 4730: 4724: 4720: 4716: 4713: 4703: 4700: 4697: 4692: 4689: 4687: 4684: 4681: 4678: 4674: 4671: 4667: 4664: 4661: 4658: 4655: 4652: 4648: 4645: 4641: 4639: 4634: 4630: 4617: 4614: 4611: 4606: 4603: 4601: 4596: 4592: 4586: 4582: 4578: 4575: 4571: 4568: 4562: 4558: 4552: 4548: 4544: 4541: 4536: 4532: 4526: 4522: 4518: 4515: 4511: 4508: 4504: 4502: 4497: 4493: 4480: 4477: 4472: 4468: 4464: 4461: 4457: 4454: 4450: 4445: 4442: 4440: 4435: 4431: 4425: 4421: 4417: 4414: 4410: 4407: 4401: 4397: 4393: 4391: 4382: 4379: 4374: 4370: 4366: 4363: 4360: 4357: 4354: 4349: 4345: 4341: 4338: 4335: 4332: 4327: 4324: 4322: 4317: 4313: 4307: 4303: 4299: 4296: 4293: 4290: 4288: 4266: 4254: 4253: 4219: 4216: 4213: 4209: 4205: 4202: 4199: 4196: 4176: 4173: 4170: 4144: 4141: 4138: 4118: 4115: 4112: 4109: 4106: 4103: 4100: 4097: 4094: 4091: 4071: 4049: 4046: 4043: 4040: 4037: 4034: 4031: 4028: 4024: 4018: 4015: 4009: 4005: 4002: 3999: 3996: 3993: 3990: 3987: 3984: 3964: 3943: 3935: 3927: 3915: 3903: 3877: 3876: 3866: 3859: 3852: 3845: 3838: 3832: 3794: 3772: 3763: 3754: 3751: 3731: 3728: 3725: 3722: 3719: 3717: 3715: 3712: 3709: 3706: 3703: 3700: 3697: 3694: 3691: 3688: 3685: 3682: 3679: 3676: 3673: 3670: 3667: 3665: 3663: 3660: 3657: 3654: 3651: 3646: 3642: 3638: 3635: 3632: 3629: 3626: 3623: 3618: 3614: 3610: 3607: 3604: 3601: 3599: 3597: 3594: 3591: 3588: 3585: 3580: 3576: 3572: 3569: 3566: 3563: 3560: 3557: 3552: 3548: 3544: 3541: 3538: 3535: 3533: 3531: 3528: 3525: 3521: 3517: 3516: 3462: 3459: 3455: 3452: 3449: 3446: 3443: 3438: 3435: 3427: 3422: 3419: 3416: 3412: 3408: 3405: 3402: 3400: 3398: 3392: 3389: 3384: 3379: 3375: 3369: 3366: 3358: 3353: 3350: 3347: 3343: 3339: 3336: 3333: 3331: 3329: 3325: 3320: 3316: 3312: 3309: 3306: 3301: 3297: 3293: 3288: 3284: 3280: 3277: 3271: 3266: 3263: 3260: 3256: 3252: 3249: 3246: 3244: 3242: 3239: 3236: 3232: 3228: 3227: 3193: 3151: 3148: 3135: 3111: 3091: 3088: 3085: 3080: 3075: 3053: 3050: 3047: 3044: 3039: 3034: 3027: 3024: 3021: 3017: 3013: 3010: 3007: 3004: 2999: 2994: 2972: 2952: 2928: 2906: 2903: 2900: 2897: 2892: 2888: 2882: 2879: 2876: 2872: 2868: 2865: 2862: 2859: 2854: 2849: 2827: 2807: 2804: 2801: 2798: 2795: 2775: 2772: 2769: 2766: 2763: 2760: 2757: 2754: 2734: 2731: 2728: 2725: 2722: 2718: 2714: 2711: 2691: 2688: 2685: 2680: 2675: 2672: 2645: 2623: 2620: 2617: 2614: 2609: 2605: 2601: 2598: 2595: 2592: 2589: 2586: 2583: 2580: 2575: 2571: 2550: 2524: 2521: 2518: 2515: 2512: 2509: 2506: 2503: 2500: 2497: 2494: 2491: 2486: 2482: 2461: 2433: 2430: 2427: 2403: 2400: 2397: 2394: 2391: 2388: 2385: 2374:measure theory 2369: 2368:Measure theory 2366: 2353: 2333: 2313: 2310: 2307: 2304: 2301: 2297: 2293: 2290: 2287: 2284: 2279: 2275: 2254: 2251: 2248: 2245: 2242: 2239: 2236: 2233: 2230: 2226: 2222: 2219: 2216: 2213: 2210: 2207: 2202: 2199: 2196: 2192: 2171: 2165: 2162: 2159: 2154: 2150: 2144: 2141: 2138: 2135: 2132: 2127: 2124: 2121: 2117: 2110: 2107: 2104: 2101: 2098: 2095: 2092: 2087: 2084: 2081: 2077: 2069: 2064: 2059: 2054: 2051: 2048: 2045: 2041: 2037: 2034: 2031: 2028: 2024: 2020: 2017: 2013: 1990: 1966: 1944: 1924: 1898: 1895: 1892: 1889: 1886: 1883: 1880: 1877: 1870: 1866: 1862: 1859: 1855: 1830: 1814: 1809: 1806: 1786: 1783: 1780: 1777: 1774: 1771: 1683: 1680: 1677: 1674: 1671: 1668: 1663: 1659: 1655: 1652: 1649: 1646: 1639: 1634: 1631: 1627: 1623: 1620: 1617: 1614: 1611: 1607: 1583: 1580: 1577: 1574: 1571: 1532: 1509: 1506: 1503: 1500: 1497: 1494: 1491: 1488: 1485: 1482: 1478: 1474: 1471: 1468: 1465: 1462: 1459: 1456: 1453: 1449: 1445: 1442: 1439: 1436: 1432: 1409: 1406: 1403: 1400: 1397: 1393: 1389: 1386: 1383: 1380: 1377: 1357: 1354: 1351: 1348: 1345: 1342: 1337: 1332: 1329: 1307: 1289:{\textstyle X} 1285: 1258: 1255: 1242: 1239: 1204: 1201: 1196: 1192: 1140: 1136: 1132: 1129: 1126: 1106: 1102: 1098: 1095: 1092: 1069: 1065: 1059: 1056: 1053: 1050: 1046: 1041: 1037: 1032: 1028: 1024: 1021: 1018: 1015: 1012: 992: 989: 986: 983: 980: 977: 974: 971: 966: 962: 958: 955: 952: 949: 946: 943: 923: 886: 866: 862: 856: 853: 850: 847: 843: 838: 834: 831: 811: 808: 805: 802: 782: 779: 776: 773: 753: 750: 747: 744: 724: 685: 682: 669: 666: 663: 660: 657: 654: 651: 648: 645: 641: 574:Claude Shannon 551:expected value 512: 492: 472: 469: 466: 463: 460: 457: 454: 451: 448: 445: 442: 435: 430: 427: 423: 419: 416: 413: 410: 407: 403: 382: 379: 376: 373: 370: 367: 362: 357: 354: 332: 310: 284: 283: 281: 280: 273: 266: 258: 255: 254: 253: 252: 247: 242: 237: 229: 228: 227: 226: 221: 213: 212: 211: 210: 205: 200: 195: 190: 185: 180: 175: 170: 165: 157: 156: 148: 147: 135: 134: 49: 47: 40: 26: 9: 6: 4: 3: 2: 17207: 17196: 17193: 17191: 17188: 17186: 17183: 17181: 17178: 17176: 17173: 17172: 17170: 17154: 17150: 17142: 17140: 17132: 17131: 17128: 17122: 17119: 17118: 17116: 17112: 17106: 17103: 17102: 17100: 17096: 17090: 17087: 17085: 17082: 17080: 17077: 17075: 17072: 17070: 17067: 17065: 17062: 17060: 17057: 17053: 17050: 17049: 17048: 17045: 17043: 17040: 17036: 17033: 17031: 17028: 17027: 17026: 17023: 17022: 17020: 17018: 17014: 17002: 16999: 16997: 16994: 16993: 16992: 16989: 16985: 16982: 16980: 16977: 16975: 16972: 16971: 16969: 16967: 16964: 16962: 16959: 16957: 16954: 16952: 16949: 16948: 16946: 16943: 16939: 16933: 16932:Video quality 16930: 16928: 16925: 16923: 16920: 16918: 16915: 16913: 16910: 16908: 16905: 16903: 16900: 16896: 16893: 16891: 16888: 16886: 16883: 16882: 16881: 16878: 16877: 16875: 16871: 16868: 16866: 16862: 16850: 16847: 16845: 16842: 16840: 16837: 16835: 16832: 16831: 16830: 16827: 16825: 16822: 16820: 16817: 16815: 16812: 16810: 16807: 16805: 16802: 16800: 16797: 16795: 16792: 16791: 16789: 16785: 16779: 16776: 16774: 16771: 16769: 16766: 16764: 16761: 16759: 16756: 16754: 16751: 16749: 16746: 16744: 16741: 16739: 16736: 16734: 16731: 16729: 16726: 16725: 16723: 16719: 16716: 16714: 16710: 16700: 16697: 16695: 16692: 16688: 16685: 16683: 16680: 16678: 16675: 16673: 16670: 16668: 16665: 16664: 16663: 16660: 16656: 16653: 16652: 16651: 16648: 16644: 16641: 16639: 16636: 16635: 16634: 16631: 16629: 16626: 16624: 16621: 16620: 16618: 16615: 16611: 16605: 16602: 16600: 16599:Speech coding 16597: 16595: 16594:Sound quality 16592: 16590: 16587: 16585: 16582: 16580: 16577: 16575: 16572: 16570: 16569:Dynamic range 16567: 16565: 16562: 16560: 16557: 16553: 16550: 16548: 16545: 16543: 16540: 16539: 16538: 16535: 16534: 16532: 16528: 16525: 16523: 16519: 16509: 16506: 16502: 16499: 16497: 16494: 16492: 16489: 16488: 16486: 16482: 16479: 16477: 16474: 16472: 16469: 16467: 16464: 16462: 16459: 16458: 16457: 16454: 16450: 16447: 16446: 16445: 16442: 16441: 16439: 16435: 16427: 16424: 16422: 16419: 16417: 16414: 16413: 16412: 16409: 16407: 16404: 16402: 16399: 16395: 16392: 16390: 16387: 16386: 16385: 16382: 16381: 16379: 16377: 16373: 16370: 16368: 16364: 16352: 16349: 16348: 16346: 16341: 16339: 16336: 16335: 16334:LZ77 + Range 16333: 16329: 16326: 16325: 16323: 16319: 16316: 16315: 16313: 16309: 16306: 16305: 16303: 16299: 16296: 16295: 16293: 16289: 16286: 16284: 16281: 16279: 16276: 16275: 16273: 16272: 16270: 16266: 16260: 16257: 16255: 16252: 16250: 16247: 16245: 16242: 16240: 16237: 16233: 16230: 16228: 16225: 16224: 16223: 16220: 16218: 16215: 16213: 16210: 16206: 16203: 16202: 16201: 16198: 16196: 16193: 16191: 16188: 16186: 16183: 16182: 16180: 16176: 16168: 16165: 16163: 16160: 16158: 16155: 16153: 16150: 16148: 16145: 16143: 16140: 16138: 16135: 16133: 16130: 16128: 16125: 16124: 16123: 16120: 16118: 16115: 16114: 16112: 16110: 16106: 16098: 16095: 16093: 16090: 16088: 16085: 16083: 16080: 16079: 16078: 16075: 16073: 16070: 16068: 16065: 16063: 16060: 16058: 16055: 16053: 16050: 16048: 16045: 16041: 16038: 16036: 16033: 16031: 16028: 16027: 16026: 16023: 16021: 16018: 16016: 16013: 16011: 16008: 16006: 16003: 16002: 16000: 15998: 15994: 15991: 15989: 15985: 15980: 15973: 15968: 15966: 15961: 15959: 15954: 15953: 15950: 15943: 15942: 15938: 15935: 15931: 15928: 15924: 15920: 15917: 15914: 15910: 15906: 15905: 15900: 15896: 15895: 15889: 15886: 15884: 15881: 15879: 15876: 15875: 15870: 15865: 15852: 15848: 15844: 15840: 15837: 15836: 15831: 15829: 15828:0-252-72548-4 15825: 15821: 15817: 15813: 15812:Shannon, C.E. 15810: 15806: 15800: 15796: 15795: 15789: 15786: 15782: 15780: 15776: 15772: 15768: 15766: 15762: 15758: 15754: 15751: 15749: 15745: 15741: 15737: 15733: 15730: 15729: 15718: 15717: 15715: 15711: 15698: 15692: 15688: 15687: 15679: 15663: 15659: 15655: 15651: 15647: 15643: 15639: 15635: 15628: 15612: 15608: 15602: 15598: 15594: 15590: 15586: 15579: 15570: 15561: 15545: 15541: 15537: 15533: 15529: 15524: 15519: 15515: 15511: 15504: 15489: 15488: 15483: 15479: 15473: 15467: 15461: 15453: 15447: 15442: 15437: 15433: 15426: 15407: 15403: 15396: 15389: 15370: 15366: 15359: 15352: 15344: 15340: 15336: 15332: 15327: 15322: 15318: 15314: 15310: 15306: 15302: 15295: 15288: 15287: 15282: 15278: 15275: 15270: 15268: 15251: 15247: 15240: 15224: 15220: 15216: 15212: 15208: 15204: 15200: 15196: 15189: 15181: 15177: 15173: 15169: 15165: 15161: 15157: 15153: 15149: 15142: 15133: 15128: 15125:: 1971–2009. 15124: 15120: 15116: 15109: 15101: 15094: 15088: 15087:0-486-68455-5 15084: 15077: 15069: 15065: 15061: 15057: 15053: 15049: 15045: 15041: 15034: 15032: 15030: 15010: 15006: 15002: 14998: 14994: 14987: 14980: 14974: 14970: 14967: 14963: 14957: 14949: 14943: 14924: 14917: 14916: 14908: 14902: 14900: 14895: 14890: 14882: 14876: 14872: 14865: 14863: 14861: 14859: 14857: 14855: 14853: 14851: 14849: 14847: 14845: 14837: 14831: 14823: 14817: 14813: 14812: 14804: 14796: 14790: 14786: 14785: 14777: 14770: 14767:Schneier, B: 14764: 14748: 14744: 14742:0-521-64298-1 14738: 14734: 14733: 14728: 14722: 14714: 14702: 14698: 14692: 14685: 14681: 14678: 14674: 14667: 14662: 14658: 14654: 14650: 14646: 14645: 14640: 14636: 14630: 14628: 14620: 14616: 14613: 14609: 14602: 14597: 14593: 14589: 14585: 14581: 14580: 14575: 14572:(July 1948). 14571: 14565: 14563: 14554: 14548: 14544: 14543: 14535: 14531: 14502: 14499: 14496: 14480: 14477: 14454: 14448: 14445: 14442: 14422: 14419: 14413: 14407: 14404: 14401: 14396: 14390: 14362: 14356: 14353: 14343: 14339: 14328: 14325: 14323: 14320: 14318: 14317:Shannon index 14315: 14312: 14309: 14307: 14304: 14301: 14300:Rényi entropy 14298: 14295: 14292: 14290: 14286: 14282: 14279: 14277: 14274: 14272: 14269: 14267: 14264: 14262: 14258: 14255: 14253: 14250: 14248: 14245: 14243: 14240: 14238: 14235: 14233: 14230: 14228: 14227:Graph entropy 14225: 14223: 14220: 14218: 14215: 14213: 14210: 14207: 14204: 14202: 14199: 14196: 14195:Cross entropy 14193: 14191: 14188: 14185: 14182: 14181: 14176: 14170: 14165: 14158: 14155: 14154:cross-entropy 14151: 14147: 14144:performed by 14143: 14139: 14137: 14133: 14129: 14125: 14111: 14091: 14071: 14051: 14028: 14025: 14022: 14016: 14013: 14006: 14002: 13998: 13996: 13987: 13968: 13964: 13960: 13949: 13945: 13935: 13931:with exactly 13929: 13917: 13913: 13890: 13887: 13884: 13880: 13875: 13870: 13867: 13864: 13861: 13853: 13850: 13847: 13839: 13836: 13832: 13820: 13817: 13802: 13801: 13800: 13783: 13780: 13775: 13764: 13761: 13758: 13752: 13749: 13743: 13738: 13735: 13732: 13724: 13721: 13718: 13710: 13706: 13693: 13690: 13676: 13671: 13668: 13665: 13661: 13653: 13652: 13636: 13633: 13630: 13627: 13619: 13616: 13613: 13605: 13602: 13598: 13585: 13582: 13565: 13564: 13560: 13559: 13556: 13555: 13540: 13534: 13531: 13528: 13522: 13517: 13513: 13506: 13503: 13500: 13494: 13488: 13482: 13477: 13473: 13469: 13466: 13463: 13457: 13442: 13441: 13440: 13423: 13415: 13404: 13400: 13396: 13384: 13381: 13369: 13363: 13360: 13357: 13348: 13337: 13333: 13323: 13322: 13321: 13318: 13314: 13310: 13304: 13300: 13295:For integers 13288: 13266: 13258: 13254: 13245: 13242: 13239: 13229: 13225: 13221: 13218: 13208: 13204: 13182: 13177: 13173: 13152: 13148: 13144: 13141: 13131: 13127: 13114: 13110: 13106: 13101: 13097: 13091: 13083: 13073: 13069:) = log| 13068: 13061: 13055: 13049: 13043: 13039: 13035: 13028: 13024: 13018: 13014:with indexes 13011: 13007: 12986: 12982: 12978: 12975: 12965: 12961: 12927: 12923: 12919: 12916: 12906: 12902: 12885: 12880: 12877: 12874: 12870: 12864: 12861: 12856: 12845: 12841: 12837: 12834: 12831: 12826: 12822: 12803: 12802: 12801: 12798: 12792: 12786: 12778: 12774: 12767: 12760: 12756: 12749: 12744: 12725: 12719: 12716: 12708: 12704: 12700: 12697: 12694: 12689: 12685: 12678: 12670: 12666: 12662: 12659: 12656: 12651: 12648: 12645: 12641: 12637: 12632: 12629: 12626: 12622: 12618: 12615: 12612: 12607: 12603: 12593: 12587: 12579: 12575: 12567: 12566: 12565: 12562: 12557: 12551: 12547: 12519: 12511: 12507: 12496: 12491: 12488: 12485: 12481: 12477: 12472: 12469: 12466: 12456: 12444: 12443: 12442: 12439: 12435: 12430: 12420: 12418: 12417:combinatorics 12408: 12405: 12403: 12399: 12396: 12392: 12388: 12385: 12381: 12376: 12374: 12367: 12363: 12343: 12340: 12337: 12331: 12321: 12317: 12312: 12307: 12305: 12301: 12292: 12289: 12283: 12277: 12273: 12267: 12262: 12257: 12252: 12247: 12227: 12221: 12218: 12212: 12203: 12197: 12191: 12188: 12182: 12176: 12173: 12170: 12164: 12161: 12155: 12146: 12140: 12134: 12131: 12128: 12125: 12119: 12113: 12096: 12088: 12087: 12086: 12083: 12077: 12071: 12064: 12060: 12056: 12052: 12048: 12044: 12038: 12033: 12028: 12022: 12017: 12013: 12007: 11997: 11995: 11976: 11970: 11967: 11941: 11932: 11912: 11909: 11906: 11892: 11886: 11880: 11877: 11871: 11865: 11852: 11848: 11844: 11832: 11831: 11830: 11827: 11821: 11815: 11807: 11803: 11797: 11791: 11785: 11775: 11773: 11767: 11762: 11743: 11740: 11737: 11730: 11724: 11721: 11718: 11712: 11706: 11693: 11689: 11685: 11682: 11678: 11671: 11668: 11665: 11649: 11643: 11629: 11623: 11617: 11610: 11609: 11608: 11579: 11576: 11573: 11566: 11560: 11557: 11554: 11548: 11542: 11529: 11525: 11519: 11506: 11502: 11495: 11489: 11486: 11475: 11471: 11464: 11451: 11448: 11445: 11441: 11433: 11430: 11427: 11424: 11417: 11411: 11398: 11394: 11388: 11375: 11371: 11364: 11351: 11348: 11345: 11341: 11329: 11328: 11327: 11320: 11307: 11295: 11292: 11281: 11277: 11270: 11257: 11254: 11251: 11247: 11243: 11232: 11228: 11221: 11215: 11212: 11201: 11197: 11190: 11177: 11174: 11171: 11167: 11163: 11160: 11127: 11115: 11111: 11104: 11100: 11096: 11093: 11082: 11078: 11071: 11058: 11055: 11052: 11048: 11044: 11041: 11020: 11006: 10995: 10991: 10984: 10971: 10968: 10965: 10961: 10955: 10941: 10938: 10935: 10928: 10922: 10909: 10905: 10895: 10878: 10875: 10868: 10862: 10851: 10848: 10845: 10834: 10830: 10826: 10815: 10811: 10804: 10794: 10790: 10764: 10758: 10754: 10750: 10744: 10739: 10734: 10731: 10729: 10723: 10721: 10717: 10712: 10706: 10686: 10683: 10671: 10665: 10662: 10659: 10653: 10647: 10636: 10632: 10629: 10620: 10614: 10611: 10608: 10605: 10594: 10588: 10573: 10572: 10571: 10545: 10541: 10537: 10531: 10516: 10513: 10492: 10486: 10481: 10477: 10467: 10448: 10435: 10431: 10424: 10421: 10411: 10407: 10400: 10395: 10390: 10387: 10384: 10380: 10373: 10368: 10364: 10360: 10347: 10343: 10336: 10333: 10323: 10319: 10312: 10306: 10301: 10297: 10291: 10286: 10283: 10280: 10276: 10272: 10263: 10257: 10252: 10248: 10232: 10228: 10221: 10218: 10208: 10204: 10197: 10191: 10186: 10182: 10173: 10168: 10165: 10162: 10158: 10154: 10145: 10139: 10134: 10130: 10116: 10112: 10105: 10099: 10094: 10090: 10081: 10077: 10070: 10062: 10057: 10054: 10051: 10047: 10043: 10040: 10034: 10028: 10021: 10020: 10019: 10002: 9993: 9987: 9982: 9978: 9964: 9960: 9953: 9947: 9942: 9938: 9929: 9925: 9918: 9910: 9905: 9902: 9899: 9895: 9891: 9888: 9881: 9878: 9875: 9871: 9867: 9862: 9856: 9850: 9843: 9842: 9841: 9815:A source set 9794: 9788: 9780: 9777: 9774: 9770: 9763: 9760: 9751: 9743: 9740: 9737: 9733: 9727: 9723: 9716: 9708: 9704: 9698: 9694: 9688: 9684: 9678: 9674: 9670: 9667: 9642: 9641: 9640: 9637: 9634: 9628: 9608: 9600: 9596: 9587: 9582: 9562: 9556: 9548: 9544: 9540: 9537: 9531: 9523: 9519: 9510: 9506: 9500: 9496: 9490: 9486: 9482: 9479: 9454: 9453: 9452: 9450: 9449: 9444: 9443:Markov source 9439: 9432: 9428: 9408: 9403: 9399: 9395: 9392: 9387: 9383: 9379: 9376: 9373: 9348: 9347: 9346: 9344: 9334: 9331: 9326: 9324: 9306: 9302: 9293: 9289: 9288:cryptanalysis 9279: 9275: 9263: 9261: 9257: 9253: 9248: 9242: 9234: 9223: 9216: 9209: 9207: 9201: 9199: 9195: 9187: 9183: 9182: 9177: 9174: 9173: 9172:joint entropy 9168: 9165: 9164: 9159: 9158: 9157: 9149: 9147: 9139: 9135: 9131: 9127: 9126:Shannon index 9121: 9111: 9109: 9105: 9094: 9091: 9088: 9087: 9083: 9080: 9077: 9076: 9072: 9069: 9066: 9065: 9061: 9058: 9055: 9054: 9051: 9045: 9042: 9041: 9035: 9033: 9029: 9024: 9020: 9016: 9012: 9007: 9002: 9000: 8995: 8991: 8987: 8983: 8979: 8975: 8971: 8964: 8960: 8950: 8948: 8944: 8940: 8936: 8935: 8930: 8925: 8921: 8917: 8912: 8905: 8899: 8894: 8890: 8885: 8877: 8871: 8854: 8831: 8828: 8825: 8822: 8813: 8809: 8806: 8799: 8798: 8797: 8795: 8791: 8786: 8784: 8780: 8772: 8768: 8759: 8755: 8751: 8747: 8742: 8740: 8736: 8717: 8710: 8707: 8704: 8701: 8678: 8674: 8671: 8668: 8661: 8660: 8659: 8657: 8653: 8649: 8644: 8642: 8638: 8634: 8633:Gibbs entropy 8630: 8624: 8620: 8615: 8607: 8587: 8581: 8577: 8573: 8570: 8565: 8561: 8557: 8548: 8544: 8541: 8538: 8531: 8530: 8529: 8528: 8527:Gibbs entropy 8524: 8519: 8515: 8511: 8506: 8504: 8500: 8494: 8474: 8470: 8466: 8462: 8461: 8447: 8444: 8441: 8438: 8435: 8413: 8409: 8405: 8400: 8396: 8387: 8366: 8362: 8347: 8344: 8341: 8335: 8327: 8323: 8311: 8308: 8300: 8296: 8289: 8286: 8283: 8277: 8272: 8268: 8264: 8249: 8248: 8247: 8246: 8230: 8222: 8203: 8187: 8169: 8158: 8152: 8141: 8135: 8132: 8129: 8113: 8112: 8091: 8080: 8074: 8066: 8051: 8050: 8049: 8048: 8043: 8037: 8032: 8031: 8014: 8008: 7997: 7991: 7983: 7968: 7967: 7966: 7965: 7960: 7954: 7948: 7942: 7937: 7936: 7917: 7906: 7897: 7891: 7875: 7859: 7850: 7844: 7836: 7825: 7816: 7810: 7799: 7793: 7782: 7776: 7765: 7759: 7744: 7743: 7742: 7741: 7719: 7713: 7710: 7707: 7679: 7676: 7670: 7659: 7653: 7625: 7602: 7596: 7593: 7590: 7582: 7581: 7564: 7558: 7547: 7541: 7533: 7522: 7516: 7505: 7499: 7491: 7480: 7474: 7471: 7468: 7453: 7452: 7451: 7450: 7445: 7439: 7433: 7427: 7421: 7414: 7410: 7404: 7403: 7385: 7382: 7377: 7373: 7369: 7361: 7357: 7353: 7350: 7347: 7342: 7338: 7322: 7321: 7320: 7319: 7313: 7308: 7301: 7297: 7296: 7273: 7269: 7265: 7262: 7259: 7254: 7250: 7241: 7231: 7225: 7222: 7217: 7213: 7209: 7206: 7203: 7198: 7194: 7185: 7182: 7179: 7165: 7164: 7163: 7162: 7158: 7157: 7156: 7153: 7141: 7139: 7112: 7096: 7092: 7088: 7085: 7082: 7077: 7073: 7020: 7017: 7011: 7008: 7005: 7002: 6999: 6991: 6977: 6973: 6966: 6953: 6936: 6932: 6928: 6925: 6922: 6917: 6913: 6887: 6883: 6879: 6876: 6873: 6868: 6864: 6855: 6840: 6820: 6816: 6812: 6809: 6806: 6801: 6797: 6788: 6778: 6772: 6769: 6764: 6760: 6756: 6753: 6750: 6745: 6741: 6732: 6729: 6726: 6711: 6696: 6693: 6690: 6667: 6656: 6650: 6639: 6633: 6630: 6627: 6612:Additivity: 6611: 6596: 6593: 6590: 6567: 6556: 6550: 6539: 6533: 6530: 6527: 6511: 6510: 6509: 6507: 6497: 6495: 6491: 6487: 6482: 6466: 6462: 6438: 6432: 6429: 6426: 6423: 6417: 6411: 6403: 6384: 6381: 6378: 6372: 6369: 6363: 6357: 6354: 6348: 6345: 6342: 6336: 6328: 6324: 6319: 6317: 6312: 6306: 6290: 6286: 6279: 6273: 6269: 6249: 6245: 6237: 6233: 6229: 6224: 6221: 6218: 6211: 6207: 6203: 6197: 6189: 6185: 6171: 6166: 6162: 6154: 6149: 6146: 6143: 6139: 6135: 6131: 6125: 6120: 6116: 6110: 6107: 6104: 6099: 6094: 6090: 6083: 6077: 6067: 6063: 6057: 6054: 6049: 6046: 6043: 6038: 6035: 6029: 6023: 6009: 6008: 6007: 6004: 5999: 5995: 5988: 5981: 5977: 5973: 5957: 5953: 5946: 5940: 5934: 5929: 5915: 5903: 5900: 5897: 5891: 5884: 5881: 5878: 5874: 5869: 5866: 5863: 5857: 5854: 5851: 5847: 5829: 5826: 5823: 5813: 5801: 5795: 5789: 5786: 5781: 5778: 5775: 5770: 5767: 5749: 5734: 5718: 5712: 5709: 5704: 5701: 5698: 5693: 5690: 5684: 5678: 5668: 5660: 5656: 5652: 5649: 5646: 5641: 5637: 5628: 5599: 5584: 5566: 5563: 5560: 5557: 5554: 5551: 5548: 5520: 5516: 5512: 5509: 5506: 5503: 5500: 5495: 5491: 5480: 5463: 5455: 5451: 5446: 5442: 5439: 5436: 5429: 5425: 5420: 5416: 5409: 5405: 5400: 5395: 5389: 5379: 5375: 5369: 5365: 5361: 5358: 5353: 5349: 5345: 5340: 5336: 5331: 5325: 5308: 5304: 5295: 5292: 5284: 5283: 5282: 5278: 5273: 5269: 5262: 5257: 5247: 5243: 5239: 5234: 5230: 5219: 5217: 5213: 5209: 5204: 5201: 5194: 5187: 5181: 5168: 5158: 5154: 5147: 5143: 5133: 5129: 5125: 5114: 5100: 5097: 5094: 5071: 5068: 5065: 5059: 5056: 5036: 5033: 5027: 5021: 4998: 4995: 4992: 4967: 4964: 4961: 4958: 4938: 4935: 4932: 4929: 4926: 4923: 4920: 4914: 4908: 4897: 4874: 4865: 4862: 4847: 4839: 4834: 4831: 4825: 4819: 4815: 4808: 4788: 4780: 4774: 4764: 4758: 4754: 4747: 4742: 4732: 4728: 4722: 4718: 4714: 4711: 4698: 4690: 4682: 4676: 4672: 4665: 4662: 4656: 4650: 4646: 4632: 4628: 4612: 4604: 4594: 4590: 4584: 4580: 4573: 4569: 4560: 4556: 4550: 4546: 4542: 4534: 4530: 4524: 4520: 4513: 4509: 4495: 4491: 4470: 4466: 4459: 4455: 4443: 4433: 4429: 4423: 4419: 4412: 4408: 4399: 4395: 4372: 4368: 4361: 4355: 4347: 4343: 4336: 4325: 4315: 4311: 4305: 4301: 4294: 4279: 4278: 4256: 4255: 4251: 4250: 4247: 4246: 4245: 4243: 4242:characterized 4239: 4234: 4217: 4214: 4211: 4207: 4203: 4200: 4197: 4194: 4174: 4171: 4168: 4159: 4142: 4139: 4136: 4116: 4113: 4110: 4107: 4104: 4098: 4092: 4060: 4047: 4041: 4035: 4032: 4029: 4026: 4022: 4016: 4013: 4007: 4003: 4000: 3997: 3991: 3985: 3975:is given by: 3953: 3949: 3941: 3933: 3921: 3909: 3898: 3892: 3887: 3883: 3874: 3865: 3858: 3851: 3844: 3839: 3833: 3829: 3824: 3818: 3813: 3812: 3811: 3809: 3804: 3797: 3793: 3787: 3775: 3771: 3766: 3762: 3750: 3746: 3729: 3726: 3723: 3720: 3718: 3707: 3704: 3698: 3695: 3692: 3686: 3683: 3677: 3674: 3671: 3668: 3666: 3655: 3649: 3644: 3640: 3636: 3633: 3627: 3621: 3616: 3612: 3608: 3605: 3602: 3600: 3589: 3583: 3578: 3574: 3570: 3567: 3561: 3555: 3550: 3546: 3542: 3539: 3536: 3534: 3526: 3505: 3499: 3495: 3489: 3483: 3477: 3460: 3457: 3450: 3447: 3441: 3436: 3433: 3425: 3420: 3417: 3414: 3410: 3406: 3403: 3401: 3390: 3387: 3382: 3377: 3373: 3367: 3364: 3356: 3351: 3348: 3345: 3341: 3337: 3334: 3332: 3318: 3314: 3307: 3304: 3299: 3295: 3286: 3282: 3275: 3269: 3264: 3261: 3258: 3254: 3250: 3247: 3245: 3237: 3215: 3213: 3207: 3203: 3185: 3178: 3172: 3169: 3163: 3156: 3147: 3133: 3125: 3109: 3078: 3051: 3045: 3037: 3025: 3022: 3019: 3011: 3005: 2997: 2970: 2950: 2942: 2941:sigma-algebra 2926: 2917: 2904: 2898: 2890: 2886: 2880: 2877: 2874: 2870: 2866: 2860: 2852: 2825: 2805: 2802: 2799: 2796: 2793: 2773: 2770: 2764: 2761: 2758: 2752: 2732: 2729: 2723: 2720: 2716: 2709: 2686: 2673: 2670: 2663: 2659: 2643: 2634: 2621: 2615: 2607: 2603: 2596: 2590: 2587: 2581: 2573: 2569: 2548: 2541:surprisal of 2540: 2535: 2522: 2516: 2510: 2507: 2504: 2501: 2498: 2492: 2484: 2480: 2459: 2451: 2447: 2428: 2425: 2417: 2398: 2395: 2389: 2386: 2375: 2365: 2351: 2331: 2308: 2305: 2302: 2291: 2285: 2277: 2273: 2249: 2246: 2243: 2240: 2237: 2234: 2231: 2220: 2214: 2211: 2208: 2200: 2197: 2194: 2190: 2169: 2160: 2152: 2148: 2139: 2136: 2133: 2125: 2122: 2119: 2115: 2108: 2105: 2099: 2096: 2093: 2085: 2082: 2079: 2075: 2062: 2052: 2049: 2046: 2043: 2039: 2035: 2032: 2026: 2018: 1942: 1922: 1914: 1909: 1896: 1893: 1887: 1881: 1878: 1875: 1868: 1864: 1857: 1844: 1833: 1807: 1804: 1784: 1781: 1775: 1769: 1760: 1756: 1751: 1746: 1742: 1737: 1731: 1726: 1722: 1720: 1712: 1707: 1703: 1698: 1681: 1675: 1669: 1666: 1661: 1657: 1650: 1644: 1632: 1629: 1625: 1621: 1618: 1612: 1595: 1578: 1572: 1560: 1555: 1547: 1520: 1507: 1498: 1492: 1489: 1486: 1483: 1472: 1463: 1457: 1443: 1437: 1421: 1404: 1401: 1398: 1387: 1381: 1375: 1352: 1349: 1346: 1330: 1327: 1283: 1276: 1272: 1264: 1254: 1250: 1248: 1238: 1236: 1232: 1226: 1219: 1202: 1199: 1194: 1190: 1181: 1177: 1171: 1165: 1158: 1152: 1138: 1134: 1130: 1127: 1124: 1104: 1100: 1096: 1093: 1090: 1080: 1067: 1063: 1054: 1048: 1044: 1039: 1035: 1030: 1026: 1022: 1016: 1010: 990: 981: 975: 969: 964: 960: 956: 953: 947: 941: 921: 912: 910: 909: 900: 884: 864: 860: 851: 845: 841: 836: 832: 829: 806: 800: 777: 771: 748: 742: 722: 714: 710: 706: 704: 698: 696: 692: 681: 661: 655: 652: 649: 646: 630: 626: 622: 618: 617:combinatorics 614: 610: 605: 603: 599: 595: 591: 587: 583: 579: 575: 562: 558: 556: 552: 548: 544: 540: 539: 534: 530: 526: 510: 470: 464: 458: 455: 452: 446: 440: 428: 425: 421: 417: 414: 408: 377: 374: 371: 355: 352: 308: 299: 295: 291: 279: 274: 272: 267: 265: 260: 259: 257: 256: 251: 248: 246: 243: 241: 238: 236: 233: 232: 231: 230: 225: 222: 220: 217: 216: 215: 214: 209: 206: 204: 201: 199: 196: 194: 191: 189: 186: 184: 181: 179: 178:Joint entropy 176: 174: 171: 169: 166: 164: 161: 160: 159: 158: 154: 150: 149: 146: 143: 142: 139: 131: 128: 120: 117:February 2019 109: 106: 102: 99: 95: 92: 88: 85: 81: 78: – 77: 73: 72:Find sources: 66: 62: 56: 55: 50:This article 48: 44: 39: 38: 33: 19: 17105:Hutter Prize 17069:Quantization 17041: 16974:Compensation 16768:Quantization 16491:Compensation 16057:Shannon–Fano 15997:Entropy type 15932: 15927:Rosetta Code 15902: 15878:Online books 15868: 15834: 15819: 15793: 15784: 15773:, Springer, 15770: 15756: 15739: 15736:Thomas, J.A. 15707: 15706: 15685: 15678: 15666:. Retrieved 15641: 15637: 15627: 15615:. Retrieved 15588: 15578: 15569: 15560: 15550:20 September 15548:. Retrieved 15523:1509.05363v6 15513: 15503: 15491:. Retrieved 15485: 15472: 15460: 15431: 15425: 15413:. Retrieved 15401: 15388: 15376:. Retrieved 15364: 15351: 15308: 15304: 15294: 15284: 15254:. Retrieved 15250:the original 15239: 15227:. Retrieved 15202: 15198: 15188: 15155: 15151: 15141: 15122: 15118: 15108: 15099: 15093: 15076: 15043: 15039: 15016:. Retrieved 14996: 14992: 14979: 14961: 14956: 14930:. Retrieved 14921:. Santa Fe. 14914: 14907: 14898: 14889: 14870: 14830: 14810: 14803: 14787:. Springer. 14783: 14776: 14768: 14763: 14751:. Retrieved 14731: 14721: 14711:– via 14705:. Retrieved 14691: 14648: 14642: 14583: 14577: 14541: 14534: 14378:. We do see 14342: 14327:Typoglycemia 14140: 14126: 13999: 13993: 13933: 13927: 13924: 13911: 13908: 13798: 13438: 13316: 13312: 13308: 13302: 13298: 13294: 13180: 13175: 13171: 13112: 13108: 13104: 13099: 13095: 13089: 13081: 13071: 13066: 13059: 13053: 13047: 13044: 13037: 13033: 13026: 13022: 13016: 13009: 13005: 12949: 12796: 12790: 12784: 12776: 12772: 12765: 12758: 12754: 12747: 12740: 12560: 12549: 12545: 12542: 12437: 12433: 12426: 12414: 12406: 12377: 12365: 12361: 12319: 12315: 12308: 12298: 12287: 12281: 12275: 12271: 12265: 12255: 12245: 12242: 12081: 12075: 12069: 12062: 12058: 12054: 12050: 12046: 12042: 12036: 12026: 12020: 12011: 12009: 11930: 11927: 11825: 11819: 11813: 11805: 11801: 11795: 11789: 11787: 11765: 11760: 11758: 11598: 11321: 11021: 10893: 10792: 10788: 10762: 10759: 10752: 10748: 10742: 10735: 10732: 10724: 10710: 10704: 10701: 10543: 10539: 10533: 10511: 10463: 10017: 9814: 9638: 9632: 9626: 9585: 9580: 9577: 9448:entropy rate 9446: 9437: 9430: 9426: 9423: 9343:Markov model 9340: 9330:one-time pad 9327: 9322: 9285: 9273: 9264: 9240: 9232: 9221: 9214: 9210: 9205: 9204:its entropy 9202: 9191: 9181:entropy rate 9179: 9170: 9161: 9155: 9123: 9100: 9038: 9036: 9027: 9022: 9018: 9003: 8994:entropy rate 8984:. (See also 8966: 8932: 8923: 8913: 8903: 8897: 8892: 8888: 8875: 8869: 8846: 8787: 8770: 8766: 8757: 8745: 8743: 8732: 8650:to give the 8645: 8622: 8618: 8605: 8602: 8517: 8507: 8498: 8496: 8188:The entropy 8041: 8035: 7958: 7952: 7946: 7940: 7443: 7437: 7431: 7425: 7419: 7412: 7408: 7311: 7306: 7299: 7151: 7147: 7113: 7041: 6503: 6483: 6320: 6310: 6304: 6288: 6284: 6277: 6271: 6267: 6264: 6002: 5997: 5993: 5986: 5979: 5975: 5969: 5955: 5951: 5944: 5938: 5932: 5306: 5302: 5285:Continuity: 5276: 5271: 5267: 5260: 5255: 5245: 5241: 5237: 5232: 5228: 5225: 5211: 5207: 5205: 5199: 5192: 5185: 5179: 5121: 4893: 4232: 4157: 4061: 3954: 3947: 3939: 3931: 3919: 3907: 3896: 3890: 3886:equiprobable 3881: 3878: 3863: 3856: 3849: 3842: 3827: 3816: 3802: 3795: 3791: 3785: 3773: 3769: 3764: 3760: 3756: 3747: 3507:= 0.7, then 3503: 3497: 3493: 3487: 3481: 3478: 3216: 3209: 3183: 3176: 3161: 3123: 2918: 2635: 2538: 2536: 2371: 1910: 1831: 1761: 1754: 1744: 1740: 1729: 1718: 1710: 1696: 1596: 1558: 1521: 1422: 1261:Named after 1260: 1251: 1244: 1234: 1224: 1217: 1169: 1163: 1156: 1153: 1081: 913: 906: 715:of an event 712: 708: 701: 699: 694: 690: 687: 684:Introduction 606: 581: 571: 537: 293: 287: 203:Entropy rate 162: 138: 123: 114: 104: 97: 90: 83: 71: 59:Please help 54:verification 51: 17064:Prefix code 16917:Frame types 16738:Color space 16564:Convolution 16294:LZ77 + ANS 16205:Incremental 16178:Other types 16097:Levenshtein 15787:, Springer. 15732:Cover, T.M. 15668:16 December 15617:16 December 15415:31 December 15378:31 December 15256:27 November 15229:15 December 14322:Theil index 12382:along with 12300:Terence Tao 11601:log(Δ) → −∞ 9292:uncertainty 9006:compression 8970:typical set 8924:application 5479:permutation 4707:introducing 3808:information 1231:information 17169:Categories 17121:Mark Adler 17079:Redundancy 16996:Daubechies 16979:Estimation 16912:Frame rate 16834:Daubechies 16794:Chain code 16753:Macroblock 16559:Companding 16496:Estimation 16416:Daubechies 16122:Lempel–Ziv 16082:Exp-Golomb 16010:Arithmetic 15816:Weaver, W. 15710:PlanetMath 15018:2 November 14526:References 14306:Randomness 14276:Perplexity 14134:to obtain 13566:Note that 13185:and hence 12441:, we have 11326:, we have 9011:redundancy 8978:Lempel–Ziv 8629:microstate 7038:Discussion 6841:Symmetry: 5966:Discussion 5296:Symmetry: 5291:continuous 5289:should be 5087:, so that 4230:, so that 3166:(i.e. the 2702:such that 2662:set family 1368:such that 1257:Definition 598:losslessly 87:newspapers 17098:Community 16922:Interlace 16308:Zstandard 16087:Fibonacci 16077:Universal 16035:Canonical 15916:"Entropy" 15909:EMS Press 15899:"Entropy" 15658:2168-2887 15493:18 August 15335:1466-8238 15219:0018-8646 15068:204177762 14942:cite book 14707:5 October 14491:→ 14481:: 14449:⁡ 14408:⁡ 14394:→ 14357:⁡ 13876:≥ 13865:− 13851:− 13762:− 13736:− 13722:− 13662:∑ 13631:− 13617:− 13532:− 13523:⁡ 13504:− 13495:− 13483:⁡ 13467:− 13397:≤ 13370:≤ 13246:⁡ 13240:≤ 13222:∈ 13145:∈ 13111:+1, ..., 12979:∈ 12920:∈ 12871:∑ 12857:≤ 12835:… 12783:{1, ..., 12717:∈ 12698:… 12660:… 12630:− 12616:… 12482:∏ 12478:≤ 12470:− 12332:λ 12192:⁡ 12174:∫ 12135:⁡ 12129:∫ 12117:‖ 11971:⁡ 11948:∞ 11945:→ 11900:Δ 11881:⁡ 11861:∞ 11856:∞ 11853:− 11849:∫ 11722:⁡ 11702:∞ 11697:∞ 11694:− 11690:∫ 11686:− 11675:Δ 11672:⁡ 11661:Δ 11641:→ 11638:Δ 11558:⁡ 11538:∞ 11533:∞ 11530:− 11526:∫ 11522:→ 11490:⁡ 11484:Δ 11460:∞ 11455:∞ 11452:− 11442:∑ 11407:∞ 11402:∞ 11399:− 11395:∫ 11391:→ 11384:Δ 11360:∞ 11355:∞ 11352:− 11342:∑ 11302:Δ 11296:⁡ 11290:Δ 11266:∞ 11261:∞ 11258:− 11248:∑ 11244:− 11216:⁡ 11210:Δ 11186:∞ 11181:∞ 11178:− 11168:∑ 11164:− 11156:Δ 11124:Δ 11097:⁡ 11091:Δ 11067:∞ 11062:∞ 11059:− 11049:∑ 11045:− 11037:Δ 11004:Δ 10980:∞ 10975:∞ 10972:− 10962:∑ 10953:→ 10950:Δ 10918:∞ 10913:∞ 10910:− 10906:∫ 10858:Δ 10838:Δ 10831:∫ 10824:Δ 10774:Δ 10720:Boltzmann 10716:H-theorem 10663:⁡ 10637:∫ 10633:− 10612:⁡ 10606:− 10487:⁡ 10422:− 10381:∏ 10374:⁡ 10334:− 10307:⁡ 10277:∑ 10258:⁡ 10219:− 10192:⁡ 10159:∑ 10140:⁡ 10100:⁡ 10048:∑ 10044:− 10029:η 9988:⁡ 9948:⁡ 9896:∑ 9892:− 9851:η 9764:⁡ 9724:∑ 9695:∑ 9675:∑ 9671:− 9541:⁡ 9507:∑ 9487:∑ 9483:− 9396:⁡ 9380:∑ 9377:− 9323:guesswork 9196:.) Other 9138:dominance 9104:broadcast 9078:Broadcast 8990:checksums 8826:⁡ 8711:ρ 8708:⁡ 8702:ρ 8675:− 8658:in 1927: 8641:Boltzmann 8574:⁡ 8558:∑ 8545:− 8473:LogSumExp 8445:≤ 8442:λ 8439:≤ 8348:λ 8345:− 8312:λ 8309:≥ 8290:λ 8287:− 8265:λ 8142:≤ 8081:≤ 8045:, we have 7907:≤ 7383:⁡ 7370:≤ 7351:… 7263:… 7207:… 7086:… 7003:− 6970:→ 6926:… 6877:… 6810:… 6754:… 6540:≤ 6433:μ 6430:⁡ 6424:⋅ 6412:μ 6382:∩ 6355:⋅ 6346:∣ 6265:Choosing 6222:… 6140:∑ 6108:… 6047:… 5892:⏟ 5867:… 5796:⏟ 5779:… 5702:… 5669:≤ 5650:… 5585:Maximum: 5440:… 5362:… 5060:∈ 5034:≥ 5022:⁡ 4968:∈ 4951:for some 4930:⁡ 4909:⁡ 4832:− 4820:⁡ 4759:⁡ 4677:⁡ 4651:⁡ 4574:⁡ 4514:⁡ 4460:⁡ 4413:⁡ 4362:⁡ 4337:⁡ 4295:⁡ 4215:⁡ 4201:− 4114:⁡ 4093:⁡ 4036:⁡ 4030:− 4004:⁡ 3986:⁡ 3705:− 3699:⋅ 3693:− 3684:− 3678:⋅ 3672:− 3669:≈ 3650:⁡ 3634:− 3622:⁡ 3606:− 3584:⁡ 3568:− 3556:⁡ 3540:− 3448:− 3442:⋅ 3411:∑ 3407:− 3383:⁡ 3342:∑ 3338:− 3305:⁡ 3255:∑ 3251:− 3171:surprisal 3110:μ 3087:Σ 3079:μ 3038:μ 3023:⊆ 2998:μ 2891:μ 2878:∈ 2871:∑ 2853:μ 2803:∈ 2762:∩ 2753:μ 2721:⁡ 2717:∪ 2710:μ 2674:⊆ 2658:partition 2644:μ 2608:μ 2604:σ 2591:μ 2574:μ 2511:μ 2508:⁡ 2502:− 2485:μ 2481:σ 2450:surprisal 2432:Σ 2429:∈ 2399:μ 2393:Σ 2109:⁡ 2063:× 2053:∈ 2040:∑ 2036:− 1882:⁡ 1861:→ 1808:∈ 1797:for some 1706:logarithm 1667:⁡ 1633:∈ 1626:∑ 1622:− 1573:⁡ 1490:⁡ 1484:− 1458:⁡ 1341:→ 1200:⁡ 1036:⁡ 970:⁡ 957:− 899:logarithm 833:⁡ 709:surprisal 653:⁡ 647:− 525:logarithm 491:Σ 456:⁡ 429:∈ 422:∑ 418:− 366:→ 356:: 17084:Symmetry 17052:Timeline 17035:FM-index 16880:Bit rate 16873:Concepts 16721:Concepts 16584:Sampling 16537:Bit rate 16530:Concepts 16232:Sequitur 16067:Tunstall 16040:Modified 16030:Adaptive 15988:Lossless 15937:Archived 15919:Archived 15839:Archived 15755:(2003), 15738:(2006), 15662:Archived 15611:Archived 15544:Archived 15540:59361755 15406:Archived 15369:Archived 15343:85935463 15277:Archived 15223:Archived 15180:17870175 15009:Archived 14969:Archived 14932:4 August 14923:Archived 14747:Archived 14729:(2003). 14701:Archived 14680:Archived 14615:Archived 14313:(SampEn) 14161:See also 13077:, where 12398:Archived 12387:Archived 12371:per the 10738:bin size 9134:evenness 9130:richness 9050:exabytes 8943:Landauer 8794:equation 8643:(1872). 6283:= ... = 5992:+ ... + 5477:for any 5212:messages 5196:nats or 5155:for the 5144:for the 5130:for the 5049:for all 4816:′ 4775:′ 4755:′ 4673:″ 4647:′ 4570:″ 4510:′ 4456:′ 4409:′ 3835:I(1) = 0 3491:, where 3181:, where 3168:expected 3158:Entropy 2656:-almost 2539:expected 691:will not 547:hartleys 533:shannons 17042:Entropy 16991:Wavelet 16970:Motion 16829:Wavelet 16809:Fractal 16804:Deflate 16787:Methods 16574:Latency 16487:Motion 16411:Wavelet 16328:LHA/LZH 16278:Deflate 16227:Re-Pair 16222:Grammar 16052:Shannon 16025:Huffman 15981:methods 15934:Entropy 15911:, 2001 15818:(1949) 15313:Bibcode 15286:Science 15160:Bibcode 15119:Entropy 15060:1426210 14896:at the 14894:Entropy 14713:YouTube 13439:where 13320:. Then 13297:0 < 12771:, ..., 12753:, ..., 12558:in the 12554:is the 12259:is the 12249:is the 10714:in the 9256:program 9235:) = log 9067:Storage 9040:Science 8974:Huffman 8882:is the 8746:changes 8612:is the 8525:is the 8514:entropy 8499:entropy 8482:Aspects 8221:concave 6300:(1) = 0 5950:, ..., 5266:, ..., 5208:meaning 5175:(2) = 1 3942:) + log 3934:) = log 3196:6 bits. 3150:Example 1714:are 2, 1704:of the 1700:is the 1552:is the 1544:is the 1273:) of a 1241:Example 897:is the 609:entropy 566:‍ 553:of the 294:entropy 163:Entropy 101:scholar 17153:codecs 17114:People 17017:Theory 16984:Vector 16501:Vector 16318:Brotli 16268:Hybrid 16167:Snappy 16020:Golomb 15866:about 15849: 15826: 15801: 15777: 15763: 15746: 15693: 15656: 15603: 15538: 15448: 15341: 15333: 15217: 15178: 15085: 15066: 15058: 14877: 14818: 14791: 14753:9 June 14739: 14549: 14186:(ApEn) 14084:given 13093:. Let 13085:| 13079:| 13075:| 12950:where 12543:where 11820:Δ 11761:is not 11599:Note; 9767: 9758: 9630:given 9578:where 9516: 9424:where 9258:for a 9144:, the 9136:, and 8916:Jaynes 8847:where 8631:. The 8616:, and 8603:where 8243:, i.e. 7738:yields 7618:where 5984:where 5275:) = Η( 4860: 4843: 4801: 4784: 4694: 4625: 4608: 4488: 4447: 4329: 4252:Proof 3862:) + I( 3855:) = I( 3724:0.8816 2448:. The 2444:be an 2418:. Let 2182:where 1748:, and 1694:where 1548:, and 1267:Η 877:where 625:axioms 523:, the 483:where 292:, the 103: 96: 89: 82: 74: 16944:parts 16942:Codec 16907:Frame 16865:Video 16849:SPIHT 16758:Pixel 16713:Image 16667:ACELP 16638:ADPCM 16628:μ-law 16623:A-law 16616:parts 16614:Codec 16522:Audio 16461:ACELP 16449:ADPCM 16426:SPIHT 16367:Lossy 16351:bzip2 16342:LZHAM 16298:LZFSE 16200:Delta 16092:Gamma 16072:Unary 16047:Range 15536:S2CID 15518:arXiv 15409:(PDF) 15398:(PDF) 15372:(PDF) 15361:(PDF) 15339:S2CID 15176:S2CID 15064:S2CID 15056:JSTOR 15012:(PDF) 14989:(PDF) 14926:(PDF) 14919:(PDF) 14334:Notes 13301:< 12745:: if 12079:with 11605:Δ → 0 11324:Δ → 0 9586:state 9584:is a 9184:of a 9092:0.281 9084:1900 9062:2007 9009:less 9004:If a 8739:trace 8521:of a 6506:Aczél 5236:= Pr( 5198:0.301 5191:0.693 4894:This 3708:1.737 3687:0.515 2939:be a 2660:is a 2446:event 2414:be a 1843:limit 1829:0 log 1522:Here 1172:= 1/2 531:(or " 296:of a 108:JSTOR 94:books 16956:DPCM 16763:PSNR 16694:MDCT 16687:WLPC 16672:CELP 16633:DPCM 16481:WLPC 16466:CELP 16444:DPCM 16394:MDCT 16338:LZMA 16239:LDCT 16217:DPCM 16162:LZWL 16152:LZSS 16147:LZRW 16137:LZJB 15847:ISBN 15824:ISBN 15799:ISBN 15775:ISBN 15761:ISBN 15744:ISBN 15691:ISBN 15670:2021 15654:ISSN 15619:2021 15601:ISBN 15552:2023 15495:2014 15446:ISBN 15417:2013 15380:2013 15331:ISSN 15258:2008 15231:2021 15215:ISSN 15083:ISBN 15020:2022 14948:link 14934:2017 14875:ISBN 14816:ISBN 14789:ISBN 14755:2014 14737:ISBN 14709:2021 14677:here 14612:here 14547:ISBN 14287:for 13306:let 13107:−1, 12049:) = 9451:is: 9227:−log 9206:rate 9178:the 9169:the 9160:the 9073:295 9059:1986 9023:less 9019:more 8961:and 8896:= 1/ 8428:and 8039:and 7944:and 7423:and 6490:dual 5961:box. 5814:< 5251:and 5206:The 5153:bans 5142:nats 5128:bits 5098:< 4257:Let 4187:for 4172:> 4140:< 4129:for 4082:are 3768:log( 3727:< 3485:and 3204:and 3179:= 1) 2919:Let 2745:and 2537:The 2265:and 1979:and 1935:and 1757:= 10 1752:for 1750:bans 1738:for 1736:nats 1727:for 1725:bits 1702:base 1180:trit 1162:1 − 700:The 695:will 619:and 529:bits 80:news 17001:DWT 16951:DCT 16895:VBR 16890:CBR 16885:ABR 16844:EZW 16839:DWT 16824:RLE 16814:KLT 16799:DCT 16682:LSP 16677:LAR 16662:LPC 16655:FFT 16552:VBR 16547:CBR 16542:ABR 16476:LSP 16471:LAR 16456:LPC 16421:DWT 16406:FFT 16401:DST 16389:DCT 16288:LZS 16283:LZX 16259:RLE 16254:PPM 16249:PAQ 16244:MTF 16212:DMC 16190:CTW 16185:BWT 16157:LZW 16142:LZO 16132:LZ4 16127:842 15925:at 15646:doi 15593:doi 15528:doi 15436:doi 15321:doi 15207:doi 15168:doi 15156:106 15127:doi 15048:doi 15001:doi 14966:url 14901:Lab 14673:PDF 14661:hdl 14653:doi 14608:PDF 14596:hdl 14588:doi 14446:log 14405:log 14387:lim 14354:log 14259:in 14148:or 13914:+ 1 13514:log 13474:log 13243:log 13042:). 13020:in 12189:log 12132:log 12030:is 11968:log 11878:log 11790:not 11774:). 11768:→ ∞ 11719:log 11669:log 11634:lim 11603:as 11555:log 11487:log 11322:As 11293:log 11213:log 11094:log 10946:lim 10718:of 10660:log 10609:log 10478:log 10365:log 10298:log 10249:log 10183:log 10131:log 10091:log 9979:log 9939:log 9761:log 9538:log 9393:log 9307:127 9286:In 9268:log 9231:(1/ 9095:65 9081:432 9070:2.6 9028:all 8980:or 8937:). 8781:or 8508:In 8471:is 8219:is 7938:If 7876:so 7583:If 7374:log 7304:log 6963:lim 6463:log 6292:= 1 5538:of 5171:log 5160:log 5135:log 4927:log 4212:log 4111:log 4033:log 4001:log 3926:log 3914:log 3902:log 3825:in 3821:is 3759:−Σ 3696:0.3 3675:0.7 3656:0.3 3641:log 3637:0.3 3628:0.7 3613:log 3609:0.7 3575:log 3547:log 3374:log 3296:log 3192:log 3186:= 1 3175:Pr( 3124:all 3016:sup 2983:is 2943:on 2838:is 2561:is 2472:is 2452:of 2106:log 1879:log 1854:lim 1835:(0) 1732:= 2 1658:log 1556:of 1487:log 1271:eta 1227:= 1 1222:or 1220:= 0 1191:log 1176:bit 1151:). 1027:log 961:log 934:by 903:log 885:log 830:log 711:or 650:log 611:in 543:nat 511:log 453:log 288:In 63:by 17171:: 16819:LP 16650:FT 16643:DM 16195:CM 15907:, 15901:, 15814:, 15734:, 15660:. 15652:. 15640:. 15636:. 15609:. 15599:. 15542:. 15534:. 15526:. 15516:. 15512:. 15484:. 15444:. 15404:. 15400:. 15367:. 15363:. 15337:. 15329:. 15319:. 15309:12 15307:. 15303:. 15266:^ 15221:. 15213:. 15201:. 15197:. 15174:. 15166:. 15154:. 15150:. 15123:17 15121:. 15117:. 15062:. 15054:. 15042:. 15028:^ 15007:. 14997:25 14995:. 14991:. 14944:}} 14940:{{ 14843:^ 14745:. 14699:. 14659:. 14649:27 14647:. 14641:. 14626:^ 14594:. 14584:27 14582:. 14576:. 14561:^ 13986:. 13784:1. 13311:= 13065:Η( 12436:⊆ 12419:. 12375:. 12366:H' 12324:= 12306:. 12291:. 12274:= 12063:dx 12047:dx 11812:1/ 11042::= 10730:. 10722:. 9220:1/ 9132:, 9110:. 9013:. 8976:, 8823:ln 8796:: 8769:/ 8741:. 8705:ln 8571:ln 8505:. 8460:. 7155:: 7140:. 7111:. 6427:ln 6318:. 6275:, 6270:= 6006:, 6001:= 5281:. 5240:= 5162:10 5151:, 5149:ln 5140:, 5113:. 3932:mn 3897:mn 3841:I( 3815:I( 3810:: 3730:1. 3496:≠ 3461:1. 3214:. 3160:Η( 3146:. 2636:A 2505:ln 2364:. 2221::= 1897:0. 1845:: 1759:. 1743:= 1734:, 1562:. 1420:: 1388::= 911:. 604:. 415::= 17155:) 17151:( 15971:e 15964:t 15957:v 15853:. 15807:. 15716:. 15699:. 15672:. 15648:: 15642:4 15621:. 15595:: 15554:. 15530:: 15520:: 15497:. 15454:. 15438:: 15419:. 15382:. 15345:. 15323:: 15315:: 15260:. 15233:. 15209:: 15203:5 15182:. 15170:: 15162:: 15135:. 15129:: 15070:. 15050:: 15044:6 15022:. 15003:: 14950:) 14936:. 14899:n 14883:. 14824:. 14797:. 14757:. 14715:. 14686:) 14671:( 14669:. 14663:: 14655:: 14621:) 14606:( 14604:. 14598:: 14590:: 14555:. 14506:] 14503:1 14500:, 14497:0 14494:( 14486:X 14478:p 14458:) 14455:0 14452:( 14443:0 14423:0 14420:= 14417:) 14414:x 14411:( 14402:x 14397:0 14391:x 14366:) 14363:0 14360:( 14112:X 14092:X 14072:Y 14052:Y 14032:) 14029:X 14026:, 14023:Y 14020:( 14017:G 14014:I 13972:) 13969:n 13965:/ 13961:k 13958:( 13954:H 13950:n 13946:2 13934:k 13928:n 13912:n 13891:1 13888:+ 13885:n 13881:1 13871:q 13868:n 13862:n 13858:) 13854:q 13848:1 13845:( 13840:n 13837:q 13833:q 13826:) 13821:k 13818:n 13813:( 13781:= 13776:n 13772:) 13768:) 13765:q 13759:1 13756:( 13753:+ 13750:q 13747:( 13744:= 13739:i 13733:n 13729:) 13725:q 13719:1 13716:( 13711:i 13707:q 13699:) 13694:i 13691:n 13686:( 13677:n 13672:0 13669:= 13666:i 13637:q 13634:n 13628:n 13624:) 13620:q 13614:1 13611:( 13606:n 13603:q 13599:q 13591:) 13586:k 13583:n 13578:( 13541:. 13538:) 13535:q 13529:1 13526:( 13518:2 13510:) 13507:q 13501:1 13498:( 13492:) 13489:q 13486:( 13478:2 13470:q 13464:= 13461:) 13458:q 13455:( 13451:H 13424:, 13419:) 13416:q 13413:( 13409:H 13405:n 13401:2 13390:) 13385:k 13382:n 13377:( 13364:1 13361:+ 13358:n 13352:) 13349:q 13346:( 13342:H 13338:n 13334:2 13317:n 13315:/ 13313:k 13309:q 13303:n 13299:k 13274:| 13270:) 13267:A 13264:( 13259:i 13255:P 13250:| 13237:] 13230:i 13226:S 13219:j 13215:) 13209:j 13205:X 13201:( 13198:[ 13194:H 13183:) 13181:A 13179:( 13176:i 13172:P 13153:i 13149:S 13142:j 13138:) 13132:j 13128:X 13124:( 13113:d 13109:i 13105:i 13100:i 13096:S 13090:A 13082:A 13072:A 13067:X 13060:A 13054:A 13048:X 13038:i 13034:S 13027:i 13023:S 13017:j 13010:j 13006:X 12987:i 12983:S 12976:j 12972:) 12966:j 12962:X 12958:( 12935:] 12928:i 12924:S 12917:j 12913:) 12907:j 12903:X 12899:( 12896:[ 12892:H 12886:n 12881:1 12878:= 12875:i 12865:r 12862:1 12854:] 12851:) 12846:d 12842:X 12838:, 12832:, 12827:1 12823:X 12819:( 12816:[ 12812:H 12797:r 12791:d 12785:d 12777:n 12773:S 12769:1 12766:S 12759:d 12755:X 12751:1 12748:X 12726:. 12723:} 12720:A 12714:) 12709:d 12705:x 12701:, 12695:, 12690:1 12686:x 12682:( 12679:: 12676:) 12671:d 12667:x 12663:, 12657:, 12652:1 12649:+ 12646:i 12642:x 12638:, 12633:1 12627:i 12623:x 12619:, 12613:, 12608:1 12604:x 12600:( 12597:{ 12594:= 12591:) 12588:A 12585:( 12580:i 12576:P 12561:i 12550:i 12546:P 12527:| 12523:) 12520:A 12517:( 12512:i 12508:P 12503:| 12497:d 12492:1 12489:= 12486:i 12473:1 12467:d 12462:| 12457:A 12453:| 12438:Z 12434:A 12362:X 12347:) 12344:H 12341:+ 12338:n 12335:( 12320:H 12316:X 12288:m 12282:m 12276:m 12272:p 12266:m 12256:m 12246:m 12228:. 12225:) 12222:x 12219:d 12216:( 12213:m 12210:) 12207:) 12204:x 12201:( 12198:f 12195:( 12186:) 12183:x 12180:( 12177:f 12171:= 12168:) 12165:x 12162:d 12159:( 12156:p 12153:) 12150:) 12147:x 12144:( 12141:f 12138:( 12126:= 12123:) 12120:m 12114:p 12111:( 12105:L 12102:K 12097:D 12082:m 12076:f 12070:m 12065:) 12061:( 12059:m 12057:) 12055:x 12053:( 12051:f 12045:( 12043:p 12037:m 12027:p 12021:m 11980:) 11977:N 11974:( 11942:N 11931:x 11913:, 11910:x 11907:d 11903:) 11896:) 11893:x 11890:( 11887:f 11884:( 11875:) 11872:x 11869:( 11866:f 11845:= 11841:H 11826:x 11814:x 11808:) 11806:x 11804:( 11802:f 11796:x 11766:n 11744:, 11741:x 11738:d 11734:) 11731:x 11728:( 11725:f 11716:) 11713:x 11710:( 11707:f 11683:= 11679:) 11666:+ 11656:H 11650:( 11644:0 11630:= 11627:] 11624:f 11621:[ 11618:h 11580:. 11577:x 11574:d 11570:) 11567:x 11564:( 11561:f 11552:) 11549:x 11546:( 11543:f 11515:) 11512:) 11507:i 11503:x 11499:( 11496:f 11493:( 11481:) 11476:i 11472:x 11468:( 11465:f 11449:= 11446:i 11434:1 11431:= 11428:x 11425:d 11421:) 11418:x 11415:( 11412:f 11381:) 11376:i 11372:x 11368:( 11365:f 11349:= 11346:i 11308:. 11305:) 11299:( 11287:) 11282:i 11278:x 11274:( 11271:f 11255:= 11252:i 11241:) 11238:) 11233:i 11229:x 11225:( 11222:f 11219:( 11207:) 11202:i 11198:x 11194:( 11191:f 11175:= 11172:i 11161:= 11151:H 11128:) 11121:) 11116:i 11112:x 11108:( 11105:f 11101:( 11088:) 11083:i 11079:x 11075:( 11072:f 11056:= 11053:i 11032:H 11007:, 11001:) 10996:i 10992:x 10988:( 10985:f 10969:= 10966:i 10956:0 10942:= 10939:x 10936:d 10932:) 10929:x 10926:( 10923:f 10894:f 10879:x 10876:d 10872:) 10869:x 10866:( 10863:f 10855:) 10852:1 10849:+ 10846:i 10843:( 10835:i 10827:= 10821:) 10816:i 10812:x 10808:( 10805:f 10793:i 10789:x 10763:f 10753:n 10749:p 10743:n 10711:Η 10705:h 10687:. 10684:x 10680:d 10675:) 10672:x 10669:( 10666:f 10657:) 10654:x 10651:( 10648:f 10642:X 10630:= 10627:] 10624:) 10621:X 10618:( 10615:f 10603:[ 10599:E 10595:= 10592:) 10589:X 10586:( 10582:H 10557:X 10546:) 10544:x 10542:( 10540:f 10512:b 10496:) 10493:n 10490:( 10482:b 10449:. 10446:) 10441:) 10436:i 10432:x 10428:( 10425:p 10418:) 10412:i 10408:x 10404:( 10401:p 10396:n 10391:1 10388:= 10385:i 10377:( 10369:n 10361:= 10358:) 10353:) 10348:i 10344:x 10340:( 10337:p 10330:) 10324:i 10320:x 10316:( 10313:p 10310:( 10302:n 10292:n 10287:1 10284:= 10281:i 10273:= 10267:) 10264:n 10261:( 10253:b 10243:) 10238:) 10233:i 10229:x 10225:( 10222:p 10215:) 10209:i 10205:x 10201:( 10198:p 10195:( 10187:b 10174:n 10169:1 10166:= 10163:i 10155:= 10149:) 10146:n 10143:( 10135:b 10125:) 10122:) 10117:i 10113:x 10109:( 10106:p 10103:( 10095:b 10087:) 10082:i 10078:x 10074:( 10071:p 10063:n 10058:1 10055:= 10052:i 10041:= 10038:) 10035:X 10032:( 10003:. 9997:) 9994:n 9991:( 9983:b 9973:) 9970:) 9965:i 9961:x 9957:( 9954:p 9951:( 9943:b 9935:) 9930:i 9926:x 9922:( 9919:p 9911:n 9906:1 9903:= 9900:i 9889:= 9882:x 9879:a 9876:m 9872:H 9868:H 9863:= 9860:) 9857:X 9854:( 9825:X 9795:. 9792:) 9789:k 9786:( 9781:j 9778:, 9775:i 9771:p 9755:) 9752:k 9749:( 9744:j 9741:, 9738:i 9734:p 9728:k 9720:) 9717:j 9714:( 9709:i 9705:p 9699:j 9689:i 9685:p 9679:i 9668:= 9665:) 9660:S 9655:( 9651:H 9633:i 9627:j 9612:) 9609:j 9606:( 9601:i 9597:p 9581:i 9563:, 9560:) 9557:j 9554:( 9549:i 9545:p 9535:) 9532:j 9529:( 9524:i 9520:p 9511:j 9501:i 9497:p 9491:i 9480:= 9477:) 9472:S 9467:( 9463:H 9438:i 9431:i 9427:p 9409:, 9404:i 9400:p 9388:i 9384:p 9374:= 9371:) 9366:S 9361:( 9357:H 9303:2 9276:) 9274:n 9272:( 9270:2 9243:) 9241:N 9239:( 9237:2 9233:N 9229:2 9222:N 9215:N 9142:D 8907:B 8904:k 8898:W 8893:i 8889:p 8879:B 8876:k 8870:W 8855:S 8832:, 8829:W 8818:B 8814:k 8810:= 8807:S 8774:B 8771:k 8767:S 8761:B 8758:k 8718:, 8714:) 8699:( 8694:r 8691:T 8683:B 8679:k 8672:= 8669:S 8623:i 8619:p 8609:B 8606:k 8588:, 8582:i 8578:p 8566:i 8562:p 8553:B 8549:k 8542:= 8539:S 8518:S 8475:. 8448:1 8436:0 8414:2 8410:p 8406:, 8401:1 8397:p 8372:) 8367:2 8363:p 8359:( 8355:H 8351:) 8342:1 8339:( 8336:+ 8333:) 8328:1 8324:p 8320:( 8316:H 8306:) 8301:2 8297:p 8293:) 8284:1 8281:( 8278:+ 8273:1 8269:p 8262:( 8258:H 8231:p 8207:) 8204:p 8201:( 8197:H 8173:) 8170:Y 8167:( 8163:H 8159:+ 8156:) 8153:X 8150:( 8146:H 8139:) 8136:Y 8133:, 8130:X 8127:( 8123:H 8107:. 8095:) 8092:X 8089:( 8085:H 8078:) 8075:Y 8071:| 8067:X 8064:( 8060:H 8042:Y 8036:X 8015:. 8012:) 8009:X 8006:( 8002:H 7998:= 7995:) 7992:Y 7988:| 7984:X 7981:( 7977:H 7959:X 7953:Y 7947:Y 7941:X 7921:) 7918:X 7915:( 7911:H 7904:) 7901:) 7898:X 7895:( 7892:f 7889:( 7885:H 7860:, 7857:) 7854:) 7851:X 7848:( 7845:f 7841:| 7837:X 7834:( 7830:H 7826:+ 7823:) 7820:) 7817:X 7814:( 7811:f 7808:( 7804:H 7800:= 7797:) 7794:X 7790:| 7786:) 7783:X 7780:( 7777:f 7774:( 7770:H 7766:+ 7763:) 7760:X 7757:( 7753:H 7726:) 7723:) 7720:X 7717:( 7714:f 7711:, 7708:X 7705:( 7701:H 7680:0 7677:= 7674:) 7671:X 7667:| 7663:) 7660:X 7657:( 7654:f 7651:( 7647:H 7626:f 7606:) 7603:X 7600:( 7597:f 7594:= 7591:Y 7565:. 7562:) 7559:X 7556:( 7552:H 7548:+ 7545:) 7542:X 7538:| 7534:Y 7531:( 7527:H 7523:= 7520:) 7517:Y 7514:( 7510:H 7506:+ 7503:) 7500:Y 7496:| 7492:X 7489:( 7485:H 7481:= 7478:) 7475:Y 7472:, 7469:X 7466:( 7462:H 7444:Y 7438:X 7432:Y 7426:Y 7420:X 7415:) 7413:Y 7411:, 7409:X 7407:( 7398:. 7386:n 7378:b 7367:) 7362:n 7358:p 7354:, 7348:, 7343:1 7339:p 7335:( 7331:H 7314:) 7312:n 7310:( 7307:b 7300:n 7291:. 7279:) 7274:n 7270:p 7266:, 7260:, 7255:1 7251:p 7247:( 7242:n 7237:H 7232:= 7229:) 7226:0 7223:, 7218:n 7214:p 7210:, 7204:, 7199:1 7195:p 7191:( 7186:1 7183:+ 7180:n 7175:H 7152:X 7123:H 7097:n 7093:p 7089:, 7083:, 7078:1 7074:p 7051:H 7033:. 7021:0 7018:= 7015:) 7012:q 7009:, 7006:q 7000:1 6997:( 6992:2 6987:H 6978:+ 6974:0 6967:q 6951:. 6937:n 6933:p 6929:, 6923:, 6918:1 6914:p 6893:) 6888:n 6884:p 6880:, 6874:, 6869:1 6865:p 6861:( 6856:n 6851:H 6826:) 6821:n 6817:p 6813:, 6807:, 6802:1 6798:p 6794:( 6789:n 6784:H 6779:= 6776:) 6773:0 6770:, 6765:n 6761:p 6757:, 6751:, 6746:1 6742:p 6738:( 6733:1 6730:+ 6727:n 6722:H 6697:Y 6694:, 6691:X 6671:) 6668:Y 6665:( 6661:H 6657:+ 6654:) 6651:X 6648:( 6644:H 6640:= 6637:) 6634:Y 6631:, 6628:X 6625:( 6621:H 6609:. 6597:Y 6594:, 6591:X 6571:) 6568:Y 6565:( 6561:H 6557:+ 6554:) 6551:X 6548:( 6544:H 6537:) 6534:Y 6531:, 6528:X 6525:( 6521:H 6467:2 6442:) 6439:A 6436:( 6421:) 6418:A 6415:( 6388:) 6385:B 6379:A 6376:( 6373:P 6370:= 6367:) 6364:B 6361:( 6358:P 6352:) 6349:B 6343:A 6340:( 6337:P 6311:n 6305:n 6298:1 6296:Η 6289:n 6285:b 6281:1 6278:b 6272:n 6268:k 6250:. 6246:) 6238:i 6234:b 6230:1 6225:, 6219:, 6212:i 6208:b 6204:1 6198:( 6190:i 6186:b 6180:H 6172:n 6167:i 6163:b 6155:k 6150:1 6147:= 6144:i 6136:+ 6132:) 6126:n 6121:k 6117:b 6111:, 6105:, 6100:n 6095:1 6091:b 6084:( 6078:k 6073:H 6068:= 6064:) 6058:n 6055:1 6050:, 6044:, 6039:n 6036:1 6030:( 6024:n 6019:H 6003:n 5998:k 5994:b 5990:1 5987:b 5980:i 5976:b 5956:k 5952:b 5948:1 5945:b 5939:k 5933:n 5916:. 5911:) 5904:1 5901:+ 5898:n 5885:1 5882:+ 5879:n 5875:1 5870:, 5864:, 5858:1 5855:+ 5852:n 5848:1 5837:( 5830:1 5827:+ 5824:n 5819:H 5809:) 5802:n 5790:n 5787:1 5782:, 5776:, 5771:n 5768:1 5757:( 5750:n 5745:H 5732:. 5719:) 5713:n 5710:1 5705:, 5699:, 5694:n 5691:1 5685:( 5679:n 5674:H 5666:) 5661:n 5657:p 5653:, 5647:, 5642:1 5638:p 5634:( 5629:n 5624:H 5600:n 5595:H 5582:. 5570:} 5567:n 5564:, 5561:. 5558:. 5555:. 5552:, 5549:1 5546:{ 5526:} 5521:n 5517:i 5513:, 5510:. 5507:. 5504:. 5501:, 5496:1 5492:i 5488:{ 5464:) 5456:n 5452:i 5447:p 5443:, 5437:, 5430:2 5426:i 5421:p 5417:, 5410:1 5406:i 5401:p 5396:( 5390:n 5385:H 5380:= 5376:) 5370:n 5366:p 5359:, 5354:2 5350:p 5346:, 5341:1 5337:p 5332:( 5326:n 5321:H 5307:i 5303:x 5298:H 5287:H 5279:) 5277:X 5272:n 5268:p 5264:1 5261:p 5259:( 5256:n 5253:Η 5249:) 5246:i 5242:x 5238:X 5233:i 5229:p 5200:n 5193:n 5186:n 5180:n 5173:2 5137:2 5126:( 5101:0 5095:k 5075:] 5072:1 5069:, 5066:0 5063:[ 5057:p 5037:0 5031:) 5028:p 5025:( 5019:I 4999:0 4996:= 4993:c 4972:R 4965:c 4962:, 4959:k 4939:c 4936:+ 4933:u 4924:k 4921:= 4918:) 4915:u 4912:( 4906:I 4875:k 4866:, 4863:u 4848:0 4840:= 4835:k 4829:) 4826:u 4823:( 4813:I 4809:u 4789:0 4781:= 4772:) 4768:) 4765:u 4762:( 4752:I 4748:u 4745:( 4733:2 4729:p 4723:1 4719:p 4715:= 4712:u 4699:0 4691:= 4686:) 4683:u 4680:( 4670:I 4666:u 4663:+ 4660:) 4657:u 4654:( 4644:I 4633:2 4629:p 4613:0 4605:= 4600:) 4595:2 4591:p 4585:1 4581:p 4577:( 4567:I 4561:2 4557:p 4551:1 4547:p 4543:+ 4540:) 4535:2 4531:p 4525:1 4521:p 4517:( 4507:I 4496:1 4492:p 4476:) 4471:1 4467:p 4463:( 4453:I 4444:= 4439:) 4434:2 4430:p 4424:1 4420:p 4416:( 4406:I 4400:2 4396:p 4378:) 4373:2 4369:p 4365:( 4359:I 4356:+ 4353:) 4348:1 4344:p 4340:( 4334:I 4326:= 4321:) 4316:2 4312:p 4306:1 4302:p 4298:( 4292:I 4265:I 4233:x 4218:x 4208:/ 4204:1 4198:= 4195:k 4175:1 4169:x 4158:k 4143:0 4137:k 4117:u 4108:k 4105:= 4102:) 4099:u 4096:( 4090:I 4070:I 4048:. 4045:) 4042:p 4039:( 4027:= 4023:) 4017:p 4014:1 4008:( 3998:= 3995:) 3992:p 3989:( 3983:I 3963:I 3950:) 3948:n 3946:( 3944:2 3940:m 3938:( 3936:2 3930:( 3928:2 3922:) 3920:m 3918:( 3916:2 3910:) 3908:n 3906:( 3904:2 3891:m 3882:n 3869:) 3867:2 3864:p 3860:1 3857:p 3853:2 3850:p 3848:· 3846:1 3843:p 3828:p 3819:) 3817:p 3803:i 3796:i 3792:p 3786:i 3781:I 3777:) 3774:i 3770:p 3765:i 3761:p 3721:= 3711:) 3702:( 3690:) 3681:( 3659:) 3653:( 3645:2 3631:) 3625:( 3617:2 3603:= 3593:) 3590:q 3587:( 3579:2 3571:q 3565:) 3562:p 3559:( 3551:2 3543:p 3537:= 3530:) 3527:X 3524:( 3520:H 3504:p 3498:q 3494:p 3488:q 3482:p 3458:= 3454:) 3451:1 3445:( 3437:2 3434:1 3426:2 3421:1 3418:= 3415:i 3404:= 3391:2 3388:1 3378:2 3368:2 3365:1 3357:2 3352:1 3349:= 3346:i 3335:= 3324:) 3319:i 3315:x 3311:( 3308:p 3300:b 3292:) 3287:i 3283:x 3279:( 3276:p 3270:n 3265:1 3262:= 3259:i 3248:= 3241:) 3238:X 3235:( 3231:H 3194:2 3184:X 3177:X 3164:) 3162:X 3134:X 3090:) 3084:( 3074:H 3052:. 3049:) 3046:P 3043:( 3033:H 3026:M 3020:P 3012:= 3009:) 3006:M 3003:( 2993:H 2971:M 2951:X 2927:M 2905:. 2902:) 2899:A 2896:( 2887:h 2881:P 2875:A 2867:= 2864:) 2861:P 2858:( 2848:H 2826:P 2806:P 2800:B 2797:, 2794:A 2774:0 2771:= 2768:) 2765:B 2759:A 2756:( 2733:1 2730:= 2727:) 2724:P 2713:( 2690:) 2687:X 2684:( 2679:P 2671:P 2622:. 2619:) 2616:A 2613:( 2600:) 2597:A 2594:( 2588:= 2585:) 2582:A 2579:( 2570:h 2549:A 2523:. 2520:) 2517:A 2514:( 2499:= 2496:) 2493:A 2490:( 2460:A 2426:A 2402:) 2396:, 2390:, 2387:X 2384:( 2352:Y 2332:X 2312:] 2309:y 2306:= 2303:Y 2300:[ 2296:P 2292:= 2289:) 2286:y 2283:( 2278:Y 2274:p 2253:] 2250:y 2247:= 2244:Y 2241:, 2238:x 2235:= 2232:X 2229:[ 2225:P 2218:) 2215:y 2212:, 2209:x 2206:( 2201:Y 2198:, 2195:X 2191:p 2170:, 2164:) 2161:y 2158:( 2153:Y 2149:p 2143:) 2140:y 2137:, 2134:x 2131:( 2126:Y 2123:, 2120:X 2116:p 2103:) 2100:y 2097:, 2094:x 2091:( 2086:Y 2083:, 2080:X 2076:p 2068:Y 2058:X 2050:y 2047:, 2044:x 2033:= 2030:) 2027:Y 2023:| 2019:X 2016:( 2012:H 1989:Y 1965:X 1943:Y 1923:X 1894:= 1891:) 1888:p 1885:( 1876:p 1869:+ 1865:0 1858:p 1839:0 1832:b 1813:X 1805:x 1785:0 1782:= 1779:) 1776:x 1773:( 1770:p 1755:b 1745:e 1741:b 1730:b 1719:e 1711:b 1697:b 1682:, 1679:) 1676:x 1673:( 1670:p 1662:b 1654:) 1651:x 1648:( 1645:p 1638:X 1630:x 1619:= 1616:) 1613:X 1610:( 1606:H 1582:) 1579:X 1576:( 1570:I 1559:X 1550:I 1531:E 1508:. 1505:] 1502:) 1499:X 1496:( 1493:p 1481:[ 1477:E 1473:= 1470:] 1467:) 1464:X 1461:( 1455:I 1452:[ 1448:E 1444:= 1441:) 1438:X 1435:( 1431:H 1408:] 1405:x 1402:= 1399:X 1396:[ 1392:P 1385:) 1382:x 1379:( 1376:p 1356:] 1353:1 1350:, 1347:0 1344:[ 1336:X 1331:: 1328:p 1306:X 1284:X 1235:p 1225:p 1218:p 1203:3 1195:2 1170:p 1164:p 1157:p 1139:2 1135:/ 1131:1 1128:= 1125:p 1105:6 1101:/ 1097:1 1094:= 1091:p 1083:( 1068:. 1064:) 1058:) 1055:E 1052:( 1049:p 1045:1 1040:( 1031:2 1023:= 1020:) 1017:E 1014:( 1011:I 991:, 988:) 985:) 982:E 979:( 976:p 973:( 965:2 954:= 951:) 948:E 945:( 942:I 922:E 865:, 861:) 855:) 852:E 849:( 846:p 842:1 837:( 810:) 807:E 804:( 801:p 781:) 778:E 775:( 772:p 752:) 749:E 746:( 743:p 723:E 705:, 668:] 665:) 662:X 659:( 656:p 644:[ 640:E 538:e 471:, 468:) 465:x 462:( 459:p 450:) 447:x 444:( 441:p 434:X 426:x 412:) 409:X 406:( 402:H 381:] 378:1 375:, 372:0 369:[ 361:X 353:p 331:X 309:X 277:e 270:t 263:v 130:) 124:( 119:) 115:( 105:· 98:· 91:· 84:· 57:. 34:. 20:)

Text is available under the Creative Commons Attribution-ShareAlike License. Additional terms may apply.