Questions tagged [information-theory]
The science of compressing and communicating information. It is a branch of applied mathematics and electrical engineering. Though originally the focus was on digital communications and computing, it now finds wide use in biology, physics and other sciences.
2,480
questions
2
votes
0
answers
18
views
Finding the optimal distribution that maximises
Consider two discrete random variables $X$ and $Y$. Let $Q$ be a distribution over $X, W$ be a conditional distribution $Y$ given $X, U$ be a conditional distribution $X$ given $Y$, and $s$ and $r$ be ...
- 97
0
votes
0
answers
17
views
Is there a similar concept to 'utility' in information theory?
Recently began learning about information theory in earnest, and was tripped up for a while by the common heuristic definitions for information content. It seems to me like the concept of the ...
- 269
0
votes
0
answers
8
views
Parametric and non parametric probability distributions
Non-parametric Statistical Models on Finite and Infinite Measure Spaces
Consider the following sets of probability densities:
In Amari it says Consider a family $S$ of probability distributions on $X$...
- 181
0
votes
0
answers
23
views
Non-parametric and parametric statistical manifolds: Equivalence of score functions in tangent spaces [closed]
Below is the framework to give the manifold structure to the space $M_{\mu}=\{f \in L^{1}(\mu): f>0 , \mu a.e , \int f d\mu=1\}$
Statistical Model and its Topology:
The Statistical Model and ...
- 181
0
votes
0
answers
26
views
Shannon's finite state transducer (FST) entropy theorem
I am trying to make sense of the proof of Shannon's theorem that a finite state transducer cannot increase the entropy of its input.
I would love some sort of drawing or intuitive formulation of it, ...
- 51
1
vote
1
answer
53
views
Generate Max Number of Sequences Separated by Hamming Distance of 3
I'm interested in whether there is an algorithm for generating the maximum possible number of DNA sequences that are $7$ nucleotides long that differ by at least $3$...
- 11
4
votes
1
answer
88
views
Statistical Model and its Topology
Consider the following setup:
Let $(\Omega, \mathcal{B}, \mu)$ be a probability space and let us denote $\mathcal{M}_{\mu}$ the set of all densities of all probability measures equivalent to $\mu$:
$$
...
- 181
3
votes
1
answer
195
views
Generalizing a logarithmic inequality
Let $\{x_i\}_{i=1}^N$ and $\{y_i\}_{i=1}^N$ be real numbers in the interval (0,1). Define for each $i$
$$\alpha_i = x_i (1-x_i) \log^2 \frac{x_i (1-y_i)}{y_i (1-x_i)}$$
and
$$\beta_i = x_i \log \frac{...
- 400
0
votes
0
answers
21
views
bivariate normal mutual information minus infinity
Let $$ \left[\begin{array}{c}
X\\
Y
\end{array}\right]\sim\mathcal{N}\left(\left[\begin{array}{c}
0\\
0
\end{array}\right],\left[\begin{array}{cc}
\sigma^{2} & \rho\sigma^{2}\\
\rho\sigma^{2} &...
- 882
2
votes
0
answers
54
views
Kullback-Leibler divergence. Is it true that: $D(\mathbb{P}_{f \circ X},\mathbb{P}_{f \circ \tilde{X}}) \le D(\mathbb{P}_X,\mathbb{P}_{\tilde{X}})$? [duplicate]
Suppose that $(\Omega,\mathcal{F})$, $(\mathcal{X},\mathcal{F}_X)$ and $ (\mathcal{Y},\mathcal{F}_{\mathcal{Y}})$ are measurable spaces.
Suppose that $X,\tilde{X}$ are measurable maps from $(\Omega,\...
- 5,773
0
votes
1
answer
51
views
Mutual information expansion not justifiable
I have recently read a mutual information term,
$I(X;Y,Z)=E_{p(X,Y,Z)}\big[\log\frac{ p(X|Y,Z)}{p(X)}\big]$.
While this expansion does not make sense to me. Is it correct? My understanding (using ...
- 164
0
votes
0
answers
17
views
KL Divergence larger than Conditional KL Divergence
Let $q(z|y)$ and $r(z)$ be variational approximations of $p(z|y)$ and $p(z)$, respectively. If I know that $H(q(z|y))=H(r(z))$, where $H$ is the entropy, I'd like to know if it is true that:
\begin{...
- 13
0
votes
1
answer
44
views
How to derive resulting PDF of a random variable transformation involving discrete and continuous random variables?
This is a communication theory question, an example from the book Foundations of MIMO Communications. Suppose a symbol $s$ is drawn from a BPSK constellation (the book says distribution, but I do not ...
- 587
1
vote
1
answer
142
views
Computing a "Generalized" Sinkhorn distance between two discrete probability distributions: A bi-convex optimization model
Given two discrete probability distributions $\mu = (\mu_1, \mu_2, \ldots, \mu_m)$ on $X = \{x_1, x_2, \ldots, x_m\}$ and $\nu = (\nu_1, \nu_2, \ldots, \nu_n)$ on $Y = \{y_1, y_2, \ldots, y_n\}$, the ...
- 836
3
votes
1
answer
93
views
Maximising the difference between the ordering of pairs of elements in a collection of totally ordered sets
Suppose I have $M$ totally ordered sets, all of length $N$, who all share the same elements. For example, for $M=3$ and $N=4$, we can have $S_1 = \{A < B < C < D\}$, $S_2 = \{B < A < C &...
