Kalman Filter for Film-Grain Noise Removal

Importance

Sampling-Based Unscented

Kalman Filter for Film-Grain Noise Removal

G.R.K. Sai Subrahmanyam, A.N. Rajagopalan, and Rangarajan Aravind Indian Institute of Technology Madras

Photographic film contains film-grain noise that translates to multiplicative, non-Gaussian noise in the exposure domain. A method based on the unscented Kalman filter can suppress this noise while simultaneously preserving edge information.

P

hotographic film is a widely used image-recording medium. Nega- tives contain tiny crystals of silver halide salts, which are light sensi- tive. When such a film is developed, these crystals turn into tiny filaments of metallic silver conventionally called grains. These ran- domly patterned grains, which constitute film- grain noise, are often visibly objectionable in photographic prints. While film grain can enhance the feel of a motion picture,¹ it renders the task of compression (transmission) difficult because of the noise. For the multi- media industry, reducing film-grain noise is important for scanning, digitization, and efficient storage.²

There is a well-known nonlinear relation- ship between the incoming light intensity (exposure) and the silver density deposited on the film. The domain in which the actual film formation takes place is referred to as the density domain and the domain in which the developed image is available for visual con- sumption is referred to as theexposure domain.

In the density domain, an image affected by film-grain noise can be modeled as a logarith- mic nonlinearity of the original scene cor- rupted by additive white Gaussian noise.^3,4 That is,r_d(m,n) 5 alog₁₀(s(m,n))+ b + v(m,n) where s(m,n) is the original scene, rd(m,n) is the degraded observation in the density domain, and v(m, n) denotes additive white Gaussian noise with zero mean and variance s²_v. Parametersaandbare the slope and offset derived from the film’s D 2 logE curve.

Alternatively, in the exposure domain, we can write

reðm,nÞ~s m,nð Þ10^{v m,nð Þ=a}

ð1Þ

but the noise becomes multiplicative and non-Gaussian. In Equation 1, re(m,n) is the degraded exposure domain image and is related to the density domain observation image asr_d(m,n)5alog₁₀(r_e(m,n))+b.

The unscented Kalman filter (UKF) can incor- porate multiplicative, non-Gaussian noise into the estimation procedure. Demonstrating this capability in the 2D domain, we propose an auto- regressive UKF for film-grain denoising. When modeled as an AR process, the image prior has limited capability to handle edges. We propose a novel methodology that reduces film-grain noise but preserves the edge information by judiciously incorporating a non-Gaussian prior within the (UKF) framework through importance sampling.

We achieve this by formulating a discontinuity- adaptive Markov random field (DAMRF) prior suited for recursive prediction and by employing importance sampling with a space-varying and heavy-tailed proposal density to estimate the DAMRF statistics.

State-space formulation and unscented transformation

The UKF framework is similar to that of the Kalman filter. Like the Kalman filter, the UKF is based on state-space formulation but differs in the way it represents and propagates Gaussianity through system dynamics. If the image is modeled as a 2D autoregressive (AR) process, the corresponding AR equation writ- ten in state transition form is

xðm,nÞ~f xðm,n{1Þ,uðm,nÞ

~Fxð_m,n{1Þzu_ðm,nÞ

Feature Article

Here,x_(m,n)is the state vector with dimension nx. If we consider a first-order, causal, non- symmetric half plane (NSHP) support, thennx

53,x(m,n)5[s(m,n),s(m,n21),s(m21,n)]^T andx_(m,n21)5[s(m,n21),s(m21,n),s(m21, n21)]^Twhere (m,n) denotes the current pixel position, and superscriptTrefers to transpose.

The state transition matrixFcontains the AR coefficients of the image. Vector u_(m,n) 5 (u_(m,n) 0 0)^T where u_(m,n) is state noise with variances²_u and dimensionn_u(51).

For the film-grain noise problem, the measurement equation in exposure domain (see Equation 1) is of the form

y_ðm,nÞ~h x_ðm,nÞ,vðm,nÞ

~Hx_ðm,nÞ10^vðm,nÞ=^a

wherey(m,n)5re(m,n) is a scalar measurement (with dimension ny5 1), v(m, n)5 v(m,n) is measurement noise of dimensionnv(51), and H5[1 0 0]. For the film-grain noise problem, the measurement model h is nonlinear.

Hence, we resort to a nonlinear counterpart of the Kalman filter, namely the UKF.

The UKF is a straightforward application of the unscented transformation (UT) for recursive minimum mean-square-error esti- mation.^5,6The UT is founded on the notion that with a fixed number of parameters, it’s easier to approximate a Gaussian distribu- tion than to approximate an arbitrary non- linear function or transformation.^5,7The UT is a deterministic sampling approach for calculating the statistics of a random vari- able that undergoes a nonlinear transforma- tion.

Consider propagating the random variable xthrough a nonlinear functiong:R^nx?R^ny to generate y 5 g(x). In our problem, the function g is equivalent to f for the state transformation, and h for the measurement transformation. Assume x has mean x¯ and covariance P. To calculate the first two moments of y using the scaled UT, we proceed as follows: First, we deterministically choose a set of 2n_x + 1 weighted samples or sigma points so that they exactly capture the true mean and covariance of the prior random variable x. A selection scheme that satisfies this requirement,^5,6is as follows:

X0~xx; w^{ð Þ}₀^m ~ l nxzl

ð Þ

Xi~xxz ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi nxzl

ð ÞP

p

i, i~1,. . .,nx; w₀^{ð Þc} ~w^{ð Þ}₀^m z1{a²_UTzb_UT

Xi~xx{ ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi nxzl

ð ÞP

p

i, i~nxz1,. . .,2nx; w_i^{ð Þm} ~w^{ð Þ}_i^c ~ 1

2ðnxzlÞ, i~1,. . .,2nx ð2Þ

Here, l~a²_UTðnxzkÞ{nx. Parameter a_UT controls the spread of the sigma points around x¯ and is usually set to a value between 0 and 1. The term b_UT is used to incorporate prior knowledge about x. Note that ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi

nxzl

ð ÞP

p

irepresents theith row (or column) of then_x3n_x-matrix square root of (nx + l)P.⁷ The addition and subtraction of these vectors from the mean value of the prior results in 2n_x symmetric sigma points.

Along with the prior meanX0we have 2n_x+ 1 sigma points. SubscriptionXrefers to the ith sigma point. The superscriptsm and con the weights refer to their use in mean and covariance calculations, respectively. The weight w^{ð Þ}_i^m associated with the ith point satisfies P2nx

i~0w^{ð Þ}_i^m ~1. Simple matrix calcu- lations can confirm that the first two mo- ments of these sigma points are exactly the same as that of the prior.

We instantiate each sigma point through the true nonlinearity to yield Y_i 5 g(X_i),i 5 0,1,…,2n_x. We estimate the mean and covari- ance ofyas

y

y~ X_2n

x

i~0w^{ð Þ}_i^mY_i and

Py~ X_2n

x

i~0w^{ð Þ}_i^cðY_i{yyÞðY_i{yyÞ^T

Positive semidefiniteness of the predicted covariance matrix P_yis guaranteed by choos- ingk $0.^6,7 These estimates of the moments of y are accurate to the second-order (third- order for symmetric priors) of the Taylor series expansion for any nonlinear function g(x), because the prior sigma points completely capture the prior distribution up to the second moment.⁷

April–June2008

Auto-regressive UKF

In this section, we propose a recursive filter, based on the UKF, that accounts for the image sensor nonlinearity in the observation model.

The UKF, based on the UT, provides a mathematically tractable way to propagate the first two moments even in the presence of multiplicative and non-Gaussian noise. We achieve this by augmenting the state-random variable with the (state and observation) noise random variables and applying UT in the prediction of state and observation.⁵We apply the scaled UT sigma point selection scheme to the new augmented vector x^a_ðm,nÞ~hx^T_ðm,nÞu m,nð Þv m,nð ÞiT

with dimen- sion na to calculate the corresponding aug- mented sigma point matrix X^a_ðm,nÞ of dimen- sionn_a3(2n_a+1). Heren_a5n_x+n_u+n_v. (For example, for a first-order NSHP support nx~3,X^a_ðm,nÞ will be of size 5 311 and each sigma point is of dimension 5).

We now propose the auto-regressive UKF (ARUKF), which sequentially estimates the in- tensity of the original image at each pixel (m,n) in a raster scan order from left to right. The filter updates the mean and covariance of the Gauss- ian approximation to the posterior distribution of the state as we describe in the following steps.

We start by initializing state statisticsx¯05 E[x0];P0=E[(x02x¯0)(x02x¯0)^T].

Step 1 augments state mean and covariance with noise statistics:

x

x^að_m,n{1Þ~hxx^Tð_m,n{1Þ 0 0iT

P^að_m,n{1Þ~

Pð_m,n{1Þ 0 0 0 s²_u 0 0 0 s²_v 2

66 4

3 77 5

For step 2, we calculate the augmented sigma point matrix (see Equation 2):

X^að_m,n{1Þ~

x

x^a_ðm,n{1Þxx^a_ðm,n{1Þ+ ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi nazl

ð ÞP^að_m,n{1Þ

h q i

In step 3, the state sigma points and state noise sigma points are propagated through the AR state model for prediction:

X^x_{ðm,nÞ=ðm,n{1Þ}~fX^x_ðm,n{1Þ,X^u_ðm,n{1Þ

~FX^xð_m,n{1ÞzhX^uTð_m,n{1Þ 0^T 0^TiT

where X^xð_m,n{1Þ consists of sigma points formed from the first nx rows of X^að_m,n{1Þ, whileX^uð_m,n{1Þis formed from then_urows that immediately follow the first nx rows in X^að_m,n{1Þ.

For step 4, we use the predicted state sigma points to determine the predicted mean and covariances:

x

x_ðm,nÞ=ð_m,n{1Þ~ X_2n

a

i~0w_i^{ð Þm}X^x_{i, m,nð Þ=}ð_m,n{1Þ

Pðm,nÞ=ðm,n{1Þ~ X_2na

i~0w^{ð Þ}_i^c

| X^x_{i, m,nð Þ=ðm,n{1Þ}{xx_ðm,nÞ=ð_m,n{1Þ

h i

| X^x_{i, m,nð Þ=}ð_m,n{1Þ{xxðm,nÞ=ðm,n{1Þ

h iT

whereirepresents theith sigma point that is theith column ofX^x_{ðm,nÞ=ðm,n{1Þ}.

In step 5, the sigma points corresponding to the measurement are predicted by propagating the (predicted) state sigma points and mea- surement noise sigma points through the film- grain nonlinearity in the exposure domain:

Y_ðm,nÞ=ð_m,n{1Þ~h X^x_ðm,nÞ=ð_m,n{1Þ,X^vð_m,n{1Þ

~HX^x_ðm,nÞ=ð_m,n{1Þ:10^:Xvðm,n{1Þ a

Here,X^v_ðm,n{1Þ is formed from the lastnvrows of X^a_ðm,n{1Þ. Operations .* and .⁽⁾ represent element-by-element operations (multiplica- tion and raised power, respectively). This results in measurement sigma point matrix Y_{(m,n)/(m,n21)}of dimensionn_y3(2n_a+1).

In step 6, we estimate the statistics required for updating using the predicted sigma points:

y

y_ðm,nÞ=ð_m,n{1Þ~ X_2n

a

i~0w^{ð Þ}_i^mY_{i, m,nð Þ=}ð_m,n{1Þ

Pyy~X_2na

i~0w^{ð Þ}_i^chY_{i, m,n}ð Þ=ðm,n{1Þ{yy_{ðm,nÞ=ðm,n{1Þ}i

| Y_{i, m,nð Þ=}ð_m,n{1Þ{yy_ðm,nÞ=ð_m,n{1Þ

h iT

P_xy~X_2na

i~0w^{ð Þ}_i^c X^x_{i, m,nð Þ=ðm,n{1Þ}{xx_ðm,nÞ=ð_m,n{1Þ

h i

| Yi, m,nð Þ=ðm,n{1Þ{yy_ðm,nÞ=ð_m,n{1Þ

h iT

where Y_{i, (m, n)/(m, n21)} represents the ith sigma point, which is the ith column of Y(m,n)/(m,n21).

IEEEMultiMedia

Step 7 performs updates, as in the Kalman filter:

The Kalman gainKðm,nÞ~PxyP^{_yy¹ x

xðm,nÞ~xxðm,nÞ=ðm,n{1Þ

zK_ðm,nÞ y_ðm,nÞ{yy_ðm,nÞ=ð_m,n{1Þ

P_ðm,nÞ~P_ðm,nÞ=ð_m,n{1Þ{K_ðm,nÞP_yyK^T_ðm,nÞ

Estimated pixel intensity at (m,n) issˆ(m,n) 5x¯(m,n)(1), the first component ofx¯(m,n). Non-Gaussian MRF prior

A homogeneous and linear AR model imposes a strong constraint on state evolution, which can hinder true transitions at image edges. Other research has addressed this issue.

For example, the research of Woods et al. uses spatially varying 2D AR parameters, estimated by windowing the observed image.⁸Jeng and Woods propose inhomogeneous AR image models using local statistics.⁹ Kadaba et al.

observe that Bernouli-Laplacian and Cauchy distribution are a better fit than the Gaussian for error residuals.¹⁰ Jeng and Woods use a compound Gauss MRF to model images.¹¹ Ceccarelli proposed edge-preserving MRFs.¹²

We base our approach to incorporating a non-Gaussian prior within our recursive esti- mation scheme on the fact that statistical dependence incorporated using an MRF model provides more flexibility in incorporating contextual constraints through a suitably derived prior. An MRF possesses a Markovian property—that is, the value of a pixel depends only on the values of its neighboring pixels.¹³ It’s important to realize that at discontinu- ities or edges, the notion of neighborhood dependency should be relaxed. Incorporating the smoothness constraint while simulta- neously preserving the edges is a challenging task.

For simplicity and analytical tractability, it is normal practice to model the original image sby a homogeneous Gaussian MRF. The issue with a homogeneous Gaussian MRF is that it imposes smoothness constraints everywhere, which inevitably leads to oversmoothing edg- es. A better way to handle the situation is to use a non-Gaussian conditional probability density function (PDF) to model the original image. Geman and Geman propose the use of line fields in which the smoothness constraint is switched off at points where the magnitude

of the signal derivative exceeds a certain threshold.¹⁴ However, the resulting poten- tial function becomes discontinuous. Li pro- poses a continuous adaptive regularizer model in which, unlike the line-field model, the interaction decreases as the derivative magni- tude becomes larger and is completely turned off at infinite magnitude of the signal deriva- tive.¹³

Following Li, we modelsas a non-Gaussian MRF with conditional PDF given by

P s m,nð ð Þ=^ss mð {i,n{jÞÞ~ 1

Zexp {clog 1z g²ðs m,nð Þ,^ssÞ c

ð3Þ

whereZis a normalization constant and g²ðs m,nð Þ,^ssÞ~

1 r²_ðm,nÞ

X

i,j ð Þ[R

s m,nð Þ{^ss mð {i,n{jÞ

ð Þ²

i²zj²

ð Þ

Here, (i,j)MR5{(i,j)I(1#i#M1,2M1#j# M1)<(0, 1#i#M1)}, andM1is the order of the NSHP support. Figure 1a (next page) illustrates NSHP support. Parameter r²_ðm,nÞ controls the allowable variation of the current pixel with respect to its neighboring values.¹⁵ Here, the potential function that accounts for the irregularities in the solution is given by clog 1z ^g2

c

. The plot in Figure 1b shows the degree of penalty dictated by the potential function. For this MRF model, the smoothing strength of the regularizer increases monoton- ically as g increases within the convex band Bc~ {pffiffiffic

,pffiffiffic

. Outside the band, smoothing decreases and becomes zero asgR‘. Because this feature preserves image discontinuities, it’s called a DAMRF model.¹³

Importance sampling

To incorporate the non-Gaussian prior given by the DAMRF model (see Equation 3) into the UKF framework, we need to estimate the mean and covariance of this prior. Because it is analytically impossible to compute these mo- ments, we resort to a Monte Carlo method known as importance sampling.¹⁶ This tech- nique determines a target PDF’s estimates using the samples from an easy-to-sample approxima- tion. Consider a PDFp(z) that is known up to a multiplicative constant but is difficult to esti-

April–June2008

mate its moments. From the functional form, we can estimate its support. Consider a distri- butionq(z), a reasonable approximation top(z), which is also known up to a multiplicative constant, is easy to sample, and the (non-zero) support ofq(z) includes the support ofp(z). Such a density q(z) is called the sampler density.

Figure 1c shows a typical plot that illustrates the target PDFpand sampler distributionq.

Mathematically, we can explain the princi- ple of importance sampling as follows. Sup- pose the densityq(z) roughly approximates the density of interest p(z), then the expected value of a functiongof a random variablezis

Ep½g zð Þ~ ð

g zð Þp zð Þdz~ ð

g zð Þ p zð Þ q zð Þ

q zð Þdz

~Eq g zð Þ p zð Þ q zð Þ

& 1 L

X^L

l~1

g z ^{ð Þl} p z ^{ð Þl} q zð ^{ð Þl}Þ

!

Here,z^(l)represents samples drawn fromq(z).

The basic idea behind importance sampling is to choose a distribution that emphasizes certain values of the input random variables that have more impact on the parameter being estimated than others by sampling them more frequently.

Our aim is to determine the estimates under the PDF p, using samples from q.

Because it’s difficult to draw samples from the target PDFp, we drawLsamples,z^{ð Þl}L

l~1

from the sampler PDF q—that is, instead of choosing samples from a uniform distribu- tion, they are now chosen from a distribution qthat concentrates on those points where the functionpis large. When we use samples from qto determine any estimates underp, in the regions where q is greater than p, the esti- mates are over-represented. In the regions where q is less than p, they are under- represented. To account for this, we use correction weights

w^l~ p z ^{ð Þl} q zð ^{ð Þl}Þ

in determining the estimates under p. For example, to find the mean of the distribution pwe use

^m_p~ P_L

l~1w^lz^{ð Þl} PL

l~1w^l

AsLR‘, the estimatemˆ_ptends to the actual mean value ofp. In our problem, the DAMRF distribution corresponds top. We choose the Cauchy distribution for the importance func- tionq, as it yields better estimates (due to its heavy tailed nature) compared to the Gauss- ian sampler.¹⁶

Edge-preserving UKF

We now extend the basic structure of the UKF to include a non-Gaussian prior. In the proposed strategy, we use knowledge of the conditional PDF to capture the image’s local physical characteristics. Doing so implicitly generalizes the state transition equation and Figure 1. (a)

Nonsymmetric half- plane support at a pixel (m,n). (b) The degree of penalty with distance imposed by the discontinuity- adaptive Markov random field model.

(c) Importance sampling:pis the target probability density function whose moments are to be estimated, whileqis the (Cauchy) sampler density.

IEEEMultiMedia

doesn’t restrict it to being linear. We base our proposed extension on the observation that the Kalman filter variants differ only in state vector propagation but have same update equations.⁵

Specifically, we formulate the state condi- tional density of the pixel to be estimated (given its neighbors) based on the DAMRF model discussed previously. We estimate the mean and variance of the non-Gaussian conditional prior, which constitutes the pre- dicted statistics of the state, by importance sampling. On the basis of these and known measurement noise statistics, we determine predicted sigma points and propagate them through the nonlinearity. The approach takes the sigma points to the update stage of the UKF and determines the final image estimate.

The steps involved in the proposed impor- tance sampling UKF (ISUKF) method follows.

For step 1, at each pixel, construct the state conditional PDF using the past pixels in the NSHP support, and the values ofr²_ðm,nÞandcin the DAMRF model. That is,

P s m,nð ð Þ=^ss mð {i,m{jÞÞ

~exp {clog 1z g²ðs m,nð Þ,^ssÞ c

If M1 5 1, then the neighbors considered in the first-order NSHP support are sˆ(m,n 2 1), sˆ(m21,n),sˆ(m21,n21), andsˆ(m21,n+1), which already are estimated. Because the parameterr²_ðm,nÞ of the DAMRF model charac- terizes the local dependency, we set it to P(m,n21), the covariance of the previous state.

For step 2, obtain the mean and covariance of the previous PDF using importance sam- pling. Draw samples z^{ð Þl}L

l~1 from a Cauchy sampler. Cauchy distributed samples with locationk₁and scales₁can be generated using k₁+ s₁tan(p(u 2 0.5)), where u is uniformly distributed from 0 to 1. To facilitate a close match with the DAMRF PDFp at each pixel, the location of the sampler density q(z) is varied as m_q~ ¹

4ð^ss m,nð {1Þz^ss mð {1,nÞz

^ss mð {1,n{1Þz^ss mð {1,nz1ÞÞ and the scale is chosen so that it is large enough to include the support ofpinq. The samples are weighted by correction weights

w^l~ p z ^{ð Þl} q zð ^{ð Þl}Þ

The mean mˆ_p and variance ^s²_p of p are computed as

m^_p~ P_L

l~1w^lz^{ð Þl} P_L

l~1w^l and

s^²_p~ P_L

l~1w^lz^{ð Þl} {m^_p2

PL l~1w^l

In step 3, we directly use these estimates to predict the one-step-ahead sigma points.

These estimates for the mean and covariance are augmented to deal with the nonadditive noise, as the following shows:

x

x_ðm,nÞ=ð_m,n{1Þ~ ^m_p, P_ðm,nÞ=ð_m,n{1Þ~s^²_p,

x

x^a_ðm,nÞ=ð_m,n{1Þ~hxx^T_ðm,nÞ=ð_m,n{1Þ 0iT

~h^m_p 0iT

,and P^a_ðm,nÞ=ð_m,n{1Þ~ Pðm,nÞ=ðm,n{1Þ 0

0 s²_v

" #

~

s^²_p 0 0 s²_v 2 4

3 5

Because we base prediction directly on the prior PDF, we need not augment with the state noise.

We use these augmented mean and covari- ances to determine the sigma point matrix (see Equation 2):

X^a_ðm,nÞ=ð_m,n{1Þ~ x

x^a_ðm,nÞ=ð_m,n{1Þ

h x

x^a_ðm,nÞ=ð_m,n{1Þ+ ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi nazl

ð ÞP^a_ðm,nÞ=ð_m,n{1Þ

q i

Then we propagate each of the 2na + 1 sigma points corresponding to state and measurement noise through the film-grain nonlinearity.

Y_ðm,nÞ=m,n{1Þ~h X^x_ðm,nÞ=m,n{1Þ,X^vð_m,n{1Þ

~X^x_ðm,nÞ=m,n{1Þ:10: ^Xvðm,n{1Þ

a

April–June2008

where X^x_{ðm,nÞ=m,n{1Þ} and X^v_ðm,n{1Þ are formed from the firstn_xrows ofX^a_ðm,nÞ=m,n{1Þand then_v rows following n_x, respectively. Here .* and.⁽⁾ are element-by-element operations.

Finally, we estimate the statistics ofy(from the sigma points) and update following steps 6 and 7 as in the auto-regressive UKF. We derive the estimated pixel intensity by sˆ(m,n) 5 x¯(m,n). Thus,sˆyields the estimated image.

Experimental results

Here, we present results obtained using the proposed ARUKF and ISUKF methods and compare their performance with the stand- ard modified Wiener filter (MWF)⁴ and with the recently proposed particle filter (PF) meth- od.¹⁷

The c and r_(m,n) parameters influence the DAMRF model. We have found experimentally thatc51.8 yields the best performance. From the D 2 logE characteristics of the film, we

chosea 5 5 for all experiments. We set the parameters asa_UT51,b_UT50, andk51. The UKF is robust to variations in these parame- ters.

Simulated degradation

For a quantitative comparison, we use the improvement in signal-to-noise ratio (ISNR), defined as

ISNR~10log₁₀ P

m,nðreðm,nÞ{s m,nð ÞÞ² P

m,nð^ss m,nð Þ{s m,nð ÞÞ²

! dB

Here, re, s, and sˆ represent the degraded, original and estimated exposure domain im- ages, respectively.

Figure 2a shows an image of a house.

Figure 2b shows the image degraded by film- grain noise. Figures 2c–f show images estimat- ed by MWF, PF, ARUKF, and ISUKF.

Figure 2. (a) Image of a house. (b) Degraded image (s²_v~_{0:05). (c)} Result using a modified Wiener filter (improvement in signal-to-noise ratio5 2.51 dB), (d) particle filter (ISNR5 3.48 dB), (e) auto- regressive unscented Kalman filter (ISNR5 3.40 dB), and (f) importance sampling unscented Kalman filter (ISNR5 4.55 dB).

The ARUKF preserves sharp details (such as grass and tree leaves) akin to MWF, but is comparatively less noisy. The performance of PF is comparable to that of ARUKF. However, the image estimated by ISUKF is the best. The noise has been effectively filtered while simul- taneously preserving fine details such as windowpanes, grass, and leaves. Note that qualitatively, the output of the proposed ISUKF method is strikingly good. This is also reflected in its high ISNR value.

Next, we tested their performance on a large data set. For this purpose, we took a data set of 25 images that included faces, natural scenes, movie frames, textures, and satellite images, and compared the performance of ARUKF and ISUKF with MWF and PF. We performed the experiments with both moderate- and high- noise levels in several trials. Figure 3 gives the results in terms of mean ISNR values. From the plots, we observe that the proposed ISUKF outperforms other filters. ARUKF performance is better than MWFs and comparable to PF. The PF and ARUKF require the AR parameters of the

original image, which is a difficult proposition in real situations.

Real degradation

We now demonstrate performance on im- ages degraded by real film-grain noise. Fig- ure 4a shows a cropped portion of an image frame from the old classic movieThe Testament of Dr. Mabuse. The film-grain noise shows up clearly on the white coat, the face, and so on.

Figures 4b–e show the output of MWF, PF, and the ARUKF and ISUKF methods. We observed that ARUKF performs on par with PF, but the ISUKF method is the most effective even in real situations. The overall noise level is quite low. The folds on the coat are best brought out in the ISUKF’s output.

Next, Figure 5a (next page) shows a face image with film-grain noise. Residual film-grain noise is visible in the MWF’s output (see Figure 5b). The PF result (see Figure 5c) is comparatively better. The ARUKF (see Figure 5d) is marginally better than PF. The ISUKF’s output (see Figure 5e) has all the sharp edges intact, and

Figure 3. Performance comparison of filters on different images over 20 Monte Carlo runs at: (a) moderate noise (sv²50.05) and (b) high noise (s_v² 50.15).

Figure 4. (a) Cropped portion of a frame from the movieThe Testament of Dr.

Mabuse. Output of (b) modified Wiener filter, (c) particle filter, (d) autoregressive unscented Kalman filter, and (e) importance sampling unscented Kalman filter. (Courtesy Criterion Collection.)

is least affected by grains. Finer details such as lips and eyebrows are well preserved. The image obtained using ISUKF is visually striking in quality and appears the most natural among them all.

Finally, in Figure 6a we show another image with real film-grain noise. Figures 6b–e depict the output from MWF, PF, ARUKF, and ISUKF. The result obtained using ARUKF is comparable to PF. The ISUKF output is again the best. The edges are well preserved and the grainy appearance in the original degraded image (see Figure 6a) is greatly mitigated.

Unlike the particle filter, in which we had to propagate about 200 samples throughout the filter in an attempt to faithfully represent the posterior distribution, for the ISUKF method we found that as few as 50 samples are sufficient to reliably estimate the first two moments of the non-Gaussian prior during prediction. At each pixel, the PF with S samples requires O(S) operations in weight calculations andO(Slog S) comparisons in the resampling step. The ISUKF requires O(L) operations in the impor- tance sampling step, ⁿ₆^3a z ^n2a

2 multiplications and additions for sigma point calculation, and O(2na+1) operations for updating. Table 1 (on page 62) gives a summary of the filters’

computational requirements. The MWF is

simple and requires only O(MN log MN) operations. On a Pentium 4 PC with 256 Mbytes of RAM and for a 2003200 image, PF takes 148 seconds (forS5200). In comparison, the ISUKF (forL5100) and the ARUKF execute in 18 seconds and 14 seconds, respectively.

Conclusions

In this article, we have discussed the application of the unscented Kalman filter for removal of film-grain noise in images. In addition to film-grain noise, old films and photographs also suffer from damage due to physical and chemical effects, scratches, stains, and digital drop-outs. Image inpaint- ing, a companion topic to film-grain noise removal, refers to the process of reconstructing such damaged portions in images.

One important extension of the UKF dis- cussed here is for joint recovery of images from scratches and film-grain noise. The UKF can be combined with image inpainting techniques to achieve this goal. Given the huge legacy of old movies captured on photographic film, there is enormous scope for application of the techniques we have discussed in this article. MM Figure 5. (a) A face

image with film-grain noise. Image estimated using (b) modified Wiener filter, (c) particle filter, (d) auto- regressive unscented Kalman filter, and (e) importance sampling unscented

Kalman filter.

Figure 6. (a) A building image. Output image obtained using a (b) modified Wiener filter, (c) particle filter, (d) autoregressive unscented Kalman filter, and (e) importance sampling unscented

Kalman filter.

Related Work

Here we provide a brief summary of various methods reported to suppress film-grain noise. Andrews et al. expand the nonlinear observation model into a Taylor series and derived an approximate filter for recovering the original image.¹ Using the film’s D 2 log E curve, and assuming Gaussian models for image and noise statistics, Hunt derives a maximum a posteriori (MAP) estimate of the restored image is derived.² The resulting nonlinear equations are solved by steepest descent using fast transform computa- tions. Naderi and Sawchuk proposed a linear filter that incorporates noise’s signal-dependent nature by adaptively changing the noise variance.³Its performance is better than the conventional Wiener filter. Froehlich, Walkup, and Krile consider image estimation in signal-dependent, film-grain noise based on different priors.⁴ These authors employ numerical integration and solve higher-order polynomials to find the MAP estimate. Tavildar, Guptha, and Guptha model the image as Gaussian and use a signal-independent transformation to reduce the MAP solution’s complexity.⁵ Tekalp and Pavlovic propose a modified Wiener filter in the exposure domain, which is tailored specifically to tackle film- grain noise.⁶By incorporating sensor nonlinearity into the restoration procedure, they demonstrate significant im- provements over the traditional wiener filter. Yan and Hatzinakos estimate the noise model parameters using higher-order statistics and propose a Wiener-type filter.⁷ They obtain filter coefficients by minimizing a cumulant criterion based on the extended Wiener-Hopf equations.

Stefano, White, and Collis propose an undecimated wavelet-domain technique for reducing film-grain noise.⁸ Using a set of training images, the wavelet coefficients are thresholded for different noise levels by optimizing a cost function related to visual quality. Argenti, Torricelli, and Alparone perform signal-dependent film-grain noise reduc- tion in both spatial and wavelet domains.⁹The detail wavelet coefficients are adaptively shrunk using the local statistics derived from the noisy image. The wavelet filter is shown to outperform its spatial counterpart. Recently, Sadhar and Rajagopalan have demonstrated good results using a recursive spatial domain particle filter (PF)-based approach to tackle film-grain nonlinearity.¹⁰ The PF arrives at the conditional mean of the posterior by propagating samples and their associated weights within a Bayesian framework.

However, the approach is computationally intensive and its performance is constrained by the assumption of a homo- geneous autoregressive state model.

Recursive estimation techniques have memory and implementation advantages and permit spatial adaptivity to be easily incorporated into the filter model.¹¹ The dynamic state-space formulation of the Kalman filter can

readily incorporate an image’s space-varying local statistics into the estimation procedure. However, the Kalman filter is limited to linear state and measurement models. To deal with nonlinear systems, the extended Kalman filter (EKF) works by linearizing the state and measurement models.

There have been recent successful works in 1D nonlinear filtering by employing the unscented Kalman filter (UKF) over the EKF.¹²The UKF, proposed by Julier and Uhlmann, not only outperforms EKF in implementation ease and accuracy but is also more stable.¹²

References

1. H.C. Andrews and B.R. Hunt.Digital Image Restoration, Prentice- Hall, 1977.

2. B.R. Hunt, ‘‘Bayesian Methods in Nonlinear Digital Image Restoration,’’IEEE Trans. Computers, vol. 26, no. 3, 1977, pp. 219-229.

3. F. Naderi and A.A. Sawchuk, ‘‘Estimation of Images Degraded by Film- Grain Noise,’’J. Applied Optics, vol. 20, no. 8, 1978, pp. 1228-1237.

4. G.K. Froehlich, J.F. Walkup, and T.F. Krile, ‘‘Estimation in Signal- Dependent Film-Grain Noise,’’Applied Optics, vol. 20, no. 20, 1981, pp. 3619-3626.

5. A.S. Tavildar, H.M. Guptha, and S.N. Guptha, ‘‘Maximum A Posteriori Estimation in Presence of Film-Grain Noise,’’Signal Processing, North-Holland, vol. 8, no. 3, 1985, pp. 363-368.

6. A.M. Tekalp and G. Pavlovic, ‘‘Restoration in the Presence of Multiplicative Noise with Application to Scanned Photographic Images,’’IEEE Trans. Signal Processing, vol. 39, no. 9, 1991, pp. 2132-2136.

7. J.C.K. Yan and D. Hatzinakos, ‘‘Signal-Dependent Film Grain Noise Removal and Generation Based on Higher-Order Statis- tics,’’Proc. IEEE Signal Processing Workshop on Higher-Order Statistics, 1997, pp. 77-82.

8. A.D. Stefano, P.R. White, and W.B. Collis, ‘‘Film Grain Reduction on Colour Images Using Undecimated Wavelet Transform,’’

Image and Vision Computing, vol. 22, no. 11, 2004, pp. 873-882.

9. F. Argenti, G. Torricelli, and L. Alparone, ‘‘MMSE Filtering of Generalised Signal-Dependent Noise in Spatial and Shift Invariant Wavelet Domains,’’Signal Processing, vol. 86, no. 8, 2006, pp. 2056-2066.

10. S.I. Sadhar and A.N. Rajagopalan, ‘‘Image Recovery Under Nonlinear and Non-Gaussian Degradations,’’J. Optical Society of America-A, vol. 22, no. 4, 2005, pp. 604-615.

11. E.S. Meinel, ‘‘Origins of Linear and Nonlinear Recursive Restoration Algorithms,’’J. Optical Society of America–A, vol. 3, no. 6, 1986, pp. 787-799.

12. S.J. Julier and J.K. Uhlmann, ‘‘A New Extension of the Kalman Filter to Nonlinear Systems,’’SPIE AeroSense Symp., 1997, pp. 182-193, http://www.cs.unc.edu/˜welch/kalman/media/

pdf/Julier1997_SPIE_KF.pdf.

References

1. D. Zou et al., ‘‘Data Hiding in Film Grain,’’Proc. 5th Int’l Workshop Digital Watermarking, LNCS 4283, Springer, 2006, pp. 197-211.

2. T.M. Moldovan, S. Roth, and M.J. Black, ‘‘Denois- ing Archival Films Using a Learned Bayesian Model,’’Proc. IEEE Int’l Conf. Image Processing, 2006, pp. 2641-2644; http://www.cs.brown.edu/

people/roth/pubs/icip2006.pdf.

3. H.C. Andrews and B.R. Hunt,Digital Image Restoration, Prentice-Hall, 1977.

4. A.M. Tekalp and G. Pavlovic, ‘‘Restoration in the Presence of Multiplicative Noise with Application to Scanned Photographic Images,’’IEEE Trans.

Signal Processing, vol. 39, 1991, pp. 2132-2136.

5. S.J. Julier and J.K. Uhlmann, ‘‘A New Extension of the Kalman Filter to Nonlinear Systems,’’SPIE AeroSense Symp., 1997, pp. 182-193; http://www.

cs.unc.edu/˜welch/kalman/media/pdf/

Julier1997_SPIE_KF.pdf.

6. R. van der Merwe et al.,The Unscented Particle Filter, tech. Report CUED/F-INFENG/TR 380, Engi- neering Dept., Cambridge Univ., 2000.

7. S.J. Julier and J.K. Uhlmann,A General Method for Approximating Nonlinear Transformations of Probabil- ity Distributions, tech. report, RRG, Dept. of Engi- neering Science, Univ. of Oxford, 1996.

8. H. Kaufman et al., ‘‘Estimation and Identification of Two-Dimensional Images,’’IEEE Trans. Automatic Control, vol. 28, no. 7, 1983, pp. 745-756.

9. F.C. Jeng and J.W. Woods, ‘‘Inhomogeneous Gaussian Image Models for Estimation and Restoration,’’IEEE Trans. Acoustics Speech and Signal Proc., vol. 36, no. 8, 1988, pp. 1305- 1312.

10. S.R. Kadaba, S.B. Gelfand, and R.L. Kashyap,

‘‘Recursive Estimation of Images Using Non- Gaussian Autoregressive Models,’’IEEE Trans. Im- age Processing, vol. 7, no. 10, 1998, pp. 1439- 1452.

11. F.C. Jeng and J.W. Woods, ‘‘Compound Gauss- Markov Random Fields for Image Estimation,’’IEEE Trans. Signal Processing, vol. 39, no. 3, 1991, pp. 683-697.

12. M. Ceccarelli, ‘‘Fast Edge-Preserving Picture Re- covery by Finite Markov Random Fields,’’ F. Roil and S. Vitulano, eds.,Proc. Int’l Conf. Image Analysis and Processing, Springer Verlag, 2005, pp. 277- 286.

13. S.Z. Li,Markov Random Field Modeling in Computer Vision, Springer Verlag, 1995.

14. S. Geman and D. Geman, ‘‘Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of Images,’’IEEE Trans. Pattern Analysis and Machine Intelligence., vol. 6, no. 11, 1984, pp. 721-741.

15. R. G. Aykroyd, ‘‘Bayesian Estimation for Homogeneous and Inhomogeneous Gaussian Random Fields,’’IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 20, 1998, pp. 533- 539.

16. D.J.C. Mackay, ‘‘Introduction to Monte Carlo Methods,’’Learning in Graphical Models, M.I.

Jordan, ed., NATO Science Series, Kluwer Aca- demic Press, 1998, pp. 175–204.

17. S.I. Sadhar and A.N. Rajagopalan, ‘‘Image Recovery Under Nonlinear and Non-Gaussian Degrada- tions,’’J. Optical Society of America-A, vol. 22, 2005, pp. 604-615.

G.R.K. Sai Subrahmanyam is a PhD candidate in the De- partment of Electrical Engineer- ing at the Indian Institute of Technology Madras. His re- search interests include image restoration, inpainting, and com- puter vision. Subrahmanyam has an MS in electrical engineering from the Indian Institute of Science, Table 1. Comparison of computational complexity at each pixel.

Filter

Log and exponential evaluations

Additions and multiplications

Inequality comparisons

Random number generations

Particle filter 2S O(S) O(SlogS) 2S

Auto-regressive UKF

2na+1 n³_a

6 z n²_a

2 zOð2naz1Þ

0 0

Importance sampling UKF

2L+2na+1

O Lð Þzn³_a 6 zn²_a

2 zOð2naz1Þ

0 L

IEEEMultiMedia

Bangalore. Contact him at saisubrahmanyam.

gorthi@gmail.com.

A.N. Rajagopalanis an asso- ciate professor in the Depart- ment of Electrical Engineering at the Indian Institute of Technology Madras. His re- search interests include shape from X, image restoration, super-resolution, and biometric recognition. Ra- jagopalan has a PhD in electrical engineering from the Indian Institute of Technology Bombay.

He is a Humboldt Fellow and an associate editor

of the IEEE Transactions on Pattern Analysis and Machine Intelligence. Contact him at raju@ee.iitm.

ac.in.

Rangarajan Aravindis a fac- ulty member in the Department of Electrical Engineering at the Indian Institute of Technology Madras. Aravind has a PhD in electrical engineering from the University of California, Santa Barbara. His research interests include image and video processing and compression. Contact him at aravind@ee.iitm.ac.in.

April–June2008