Shannon's Entropy, Probability Changes and Average Energy

In a previous note (1) we considered dE (thermal)= Sum over i  ei dP(i) where P(i) is the probability to have ei. In other words, one has the constant vector {ei} forming a dot product with the differential operator { dP(i) }.We suggested one might wish to find a vector function { G(P(i) } and create the dot product S= -  Sum over i G(P(i)) P(i) such that   dS= - Sum over i  G(P(i)) dP(i). For small changes {dP(i)} the vector { G(P(i))} is a constant of the system with G=ln the logarithm function as a solution.

    In this note we compare this picture with that of information theory for which one writes Poverall = Product over i    P(i) [to the power NP(i)]  where N is large. This approach also leads to the form of Shannon’s entropy i.e. one does not need to find the function ln - it appears automatically, but one needs a large N limit to formulate the problem whereas no such assumption is needed in (1). We consider the situation of P(i) → P(i) + dP(i). We ultimately try to link dE thermal to the change in the product probability of P(i)’s in a gas with dS being linked to the change in the product probability which itself is linked to the overall number of arrangements.

  We also consider how the form ln is associated with converting P(i)P(j) into a sum i.e. ln(P(i)) + ln(P(j)). For example, for an ideal gas one may consider two body elastic collisions. Assuming that P(i)P(j) =P(k)P(l) and given ei+ej=ek+el one may take the ln of the former and associate ln(P(i)) with -ei/T, where T is a constant. Thus one uses the idea of ln converting a product to a sum and the idea of conservation which in this particular case is conservation of energy. One may ask how this links with the picture of   P(i)ln(P(i)) leading to a locally constant vector? We argue that one may consider two particles and consider a change in P(i) i.e. P(i)+dP(i). In such a case it is not ln(P(i))+ln(P(i)) which is linked with conservation, but one is interested in the conservation rule P(i)+P(j)=constant. We show that it is form P(i)ln(P(i)) which is then linked with a conservation law namely Sum over i dP(i)=0 where i=1,2 in this case. Thus there is a link between the derivation of the Maxwell-Boltzmann distribution strictly from P(i)P(j) and ei+ej=ek+el and Shannon’s entropy which is related to ln(P(i)), but is slightly different, namely P(i)ln(P(i)), in order to make use of conservation of probability.