pclim.comPapers and Essays Speech Recognition In Noisy Environments

| Search

INDEPENDENT COMPONENT ANALYSIS Independent Component Analysis (ICA) is a method for finding underlying factors or components from multidimensional statistical data. It is a very general-purpose statistical technique in which observed random data are expressed as a linear transform of components that are statistically independent from each other.

ICA is most commonly used to solve the Blind Source Separation (henceforth abbreviated as BSS)problem, which was described in chapter 1.

Throughout this chapter, in order not to distract from the flow of the `thought process', references to chapter 3 have been given, if more details are sought than what is presented in the more limited space available in this chapter.

2.1 Independence and Uncorrelatedness

As an intermediate step towards understanding ICA, it is instructive to first review some basic concepts from probability theory, namely the concept of statistical independence and its relation with uncorrelatedness.

For simplicity, in this section, we consider the case of just two different scalar random variables x and y.

Two random variables x and y are said to be uncorrelated, if their covariance is zero:

E{xy} - E{x}E{y} = 0 (2.1)

On the other hand, true statistical independence is a stronger property. Basically, 2 random variables x, y are said to be independent if information on the value of x does not give any information on the value of y, and vice versa. For example, x and y can be outcomes of two events that have nothing to do with each other, or random signals originating from two quite different physical processes that are in no way related to each other. Examples of such independent random variables are the value of a dice thrown and of a coin tossed, or of more relevance to the project, speech signal and background noise originating from a ventilation system at a certain time instant.

Mathematically, statistical independence is defined in terms of probability densities. The random variables x and y are said to be independent if the joint pdf of x and y is factorizable in the following way:

f_XY(x, y) = f_X(x)f_Y(y)

In other words, the joint density of x and y must factorize into the product of their marginal densities.

Statistical independence is a key concept that constitutes the foundation of ICA. Indeed, it is surprising that the independent components can be estimated from linear mixtures with no more assumptions than their independence! Not much more than this assumption is needed to ascertain that the model can be estimated. This is why ICA is such a powerful method with applications in many different areas. We will investigate other application areas in chapter 7.

(Indeed, the application to audio separation in this thesis is just one of many possible applications of ICA. Our supervisor agrees - originally, for example, he had intended to utilize ICA not to the area of speech recognition at all, but to biomedical applications!)

An important point to note is that independence is a much stronger property than uncorrelatedness.

To illustrate, let h₁ and h₂ be functions of x and y respectively. If x and y are independent, then the following property holds:

E{h₁(x)h₂(y)} = E{h₁(x)}E{h₂(y)} (2.2)

So equation(2.2) reveals that statistical independence always implies uncorrelatedness, i.e.equation(2.2) $\Rightarrow$ ( 2.1) but not the other way round. In fact, equation(2.1) is derived from equation(2.2) as a special case where both h₁(x) and h₂(y) are linear functions, and takes into account second-order statistics (correlations or covariances) only. Specifically, we can see that by setting h₁(x) = x and h₂(y) = y in equation(2.2), we get equation(2.1). We can thus always go from (2.2) to (2.1), but not the other way round.

Let's give a mathematical example to further show that independence is a much stronger property than uncorrelatedness. Assume that (x, y) are discrete-valued and follow such a distribution that the pair are with probability 1/4 equal to any of the following values: (0,1),(0,-1),(1,0), and (-1,0). Then x and y are uncorrelated, as can be simply calculated by applying equation(2.1): E{xy} = ${\frac{1}{4}}$(0 + 0 + 0 + 0) = E{x}E{y} = 0 ^. 0 = 0. On the other hand, if we apply equation(2.2) (setting h₁(x) = x² and h₂(y) = y²), we get E{x²y²} = 0 $\neq$ ${\frac{1}{4}}$ = E{x²}E{y²}.

What this implies is that uncorrelatedness in itself is not enough to separate the components in the aforementioned BSS problem. This is the reason why techniques like principal component analysis (PCA) cannot separate the signals: they give components that are uncorrelated, but little more.

Although PCA does not give components that are truly independent, nevertheless, it is very often used as a preprocessing step for ICA. The details of this will be described in section2.3.

Complete list of papers and essays