Dr.B.A.T.

University, Lonere

Image Cartoonize with Stegnography

CHAPTER 1 MULTIPASS BILATERAL FILTER – CARTOONIZER
1.1Introduction
This application applies Multipass Bilateral Filter to colour Images. Bilateral filtering is an Edge-preserving smoothing filter. This technique extends the concept of Gaussian smoothing by weighting the filter coefficients with their corresponding relative pixel intensities. Pixels that are very different in intensity from the central pixel are weighted less even though they may be in close proximity to the central pixel. This is effectively a convolution with a non-linear Gaussian filter, with weights based on pixel intensities. This is applied as two Gaussian filters at a localized pixel neighbourhood , one in the spatial domain, and one in the intensity domain.

1.1.1 Changes
V6 – Better Contour Algorithm; Luminance segmentation; Bilateral Filter optimization; Interface look.

1.1.2 Cartoon
To obtain cartoon-like results, these new effects have been implemented:

1.1.3 Contour
Contour, when used, is applied to the Bilateral Filter Output in the CieLAB colour space. It has 2 parameters: General Amount, and how much it is based on Luminance (L) or Hue (AB). It's created with a Sobel filter applied in separated channels: “L” and “A & B” (of the CieLAB colourspace).

Dept of Information Technology

1

Dr. (No Segmentation applied) 25% Presence means that the Luminance of the output picture is given by 25% of the Segmented Luminance and by 75% of the bilateral filter output luminance and so on 100% Presence means that the Luminance of the output picture is fully given by Segmented Luminance.3 Algorithm: It's built an histogram of the Luminance channel (L) of Bilateral filtered picture. To each segment it is assigned a value of Luminance given by the weighted average of the Histogram segment values.T. Then this histogram is divided by N segments (in a way so that in each segment there's the same number of intensities values). 0% Presence means that the bilateral filter output luminance has no variation.University. Dept of Information Technology 2 .B. Lonere Image Cartoonize with Stegnography 1.A.2 Luminance Segmentation This is a nice way to obtain a more cartoon-like output. (Fig.) 1. Example Yellow lines) Then these segments are merged with original “bilateral output” by a value called “Presence”. (Like contour this works with CieLAB colour space. The best way to have an idea of how this work is to tweak these parameters and to watch the output results. Note that we are talking only about Luminance values.

4. – To follow this path set Contour Amount = 0 and Segmentation Presence = 0 1. Lonere Image Cartoonize with Stegnography 1.T.4 Flow As we can see.A. The pre-Effext can be one of this: 1. (The Number of Segments is customizable) Dept of Information Technology 3 .University. Contrast & Saturation. before applying “Bilateral Filter” it's possible to apply a pre-Effect.3 Luminance Segmentation (blue) 0-1-3-4-6 [1-3] A “Luminance Segmentation” is applied to the Bilateral Output.Dr. Auto Histogram Equalize 2.4.Exposure Control 3.2 Add a Contour (red) 0-1-2-5-6 [1-2] A contour filter is applied to the Bilateral Output [1-5] Contour is applied to the bilateral output – To follow this path set Contour Amount > 0 and Segmentation Presence = 0 1.Brightness.1 Pure Bilateral (black) 0-1-6 The Output Picture is the Output of Bilateral.4. After the “Bilateral filter” there are 4 possible paths to follow: 1.B.

1 Colour Space This is the Colour space the bilateral filter will be applied to.6.B.5 Parameters Details Parameters are saved in the files “LastSettings.4. – To follow this path set Contour Amount = 0 and Segmentation Presence > 0. make a copy. R=1 means that for each pixel the computed pixel will be the result of ((R*2)+1)^2 = 9 neighborhood pixels R=2 means that for each pixel the computed pixel will be the result of ((R*2)+1)^2 = 25 neighborhood pixels Dept of Information Technology 4 .T. (The Number of Segments is customizable) [1-4] We merge the “Luminance Segmented” picture with the bilateral Output. -RGB Bilateral filter is computed from all channels Red.University.5 Pre-EFFECT As seen above it can be: Auto Equalize Exposure BCS Brightness. -CieLAB. 1. This merge is customizable with “Presence %” slidebar. [4-5] Contour is applied to the output of above merging.2 Radius Radius of Intensity and Spatial Domains.4 CARTOON-like: Contour and Luminance Segmentation (green) 0-1-2-3-4-5-6 [1-2] A contour filter is applied to the Bilateral Output [1-3] A “Luminance Segmentation” is applied to the Bilateral Output. -- 1.txt” for easy reading.4. 1. 1. – To follow this path set Contour Amount > 0 and Segmentation Presence greater > 0.txt” and files “LastSettingsEX.Green and Blue.Dr. This merge is customizable with “Presence %” slidebar.6. Here the bilateral filter is computed only from the Luminance (L) channel of the CieLab Colour Space faster. Lonere Image Cartoonize with Stegnography [1-4] The Output is the result of the merging of the “Luminance Segmented” picture with the bilateral Output.6 BILATERAL FILTER 1. Contrast.A. Saturation 1. Not to loose them.

8 LUMINANCE SEGMENTATION Segments Number of Luminance Segments Samples This are good settings for cartoon-like results PreEFFECT: Brightness:110 Dept of Information Technology 5 .6. 1.6.A.T.University.7 CONTOUR Amount Contour Darkness Lum/Hue How much the contour is based on Luminance or Hue. 1.6 Iterations How many times Bilateral Filter is applied.6.5 Spatial Sigma This is how fast the Spatial space decrease going far from central pixel.3 Intensity Mode This is the type of decay curve for the intensity space. Lonere Image Cartoonize with Stegnography R=3 means that for each pixel the computed pixel will be the result of ((R*2)+1)^2 = 49 neighborhood pixels R=4 means that for each pixel the computed pixel will be the result of ((R*2)+1)^2 = 81 neighborhood pixels R=5 means that for each pixel the computed pixel will be the result of ((R*2)+1)^2 = 121 neighborhood pixels 1.4 Intensity Sigma This is how fast the intensity space decrease 1.6. it can be 0 Gaussian1 1 Gaussian2 2 InvProportional 3 Linear 1.B. 1.Dr.

5 LUMINANCE SEGMENTATION: Segments: 4 Presence: 60% Dept of Information Technology 6 .Dr.355 Spatial Sigma: 1000 Iterations: 3 CONTOUR: Amount : 100 Lum/Hue: . Lonere Contrast :0 Saturation:100 Image Cartoonize with Stegnography BILATERAL FILTER: Colour Space: CieLAB Radius: 5 Intensity Mode : 1 Gaussian2 Intensity Sigma: 0.A.B.T.University.

There is no general agreement among authors regarding where image processing stops & other related areas such as image analysis& computer vision start. The area of image analysis (image understanding) is in between image processing & computer vision. Vision is the most advanced of our sensor. Dept of Information Technology 7 . However.1. They can operate also on images generated by sources that humans are not accustomed to associating with image.1 Background: Digital image processing is an area characterized by the need for extensive experimental work to establish the viability of proposed solutions to a given problem. y). 2. When x. The elements are called pixels. This characteristic implies that the ability to formulate approaches &quickly prototype candidate solutions generally plays a major role in reducing the cost & time required to arrive at a viable system implementation.2 What is DIP? An image may be defined as a two-dimensional function f(x.1 Digital Image Processing 2. Sometimes a distinction is made by defining image processing as a discipline in which both the input & output at a process are images. each of which has a particular location & value. An important characteristic underlying the design of image processing systems is the significant level of testing & experimentation that normally is required before arriving at an acceptable solution.A. we call the image a digital image. who are limited to the visual band of the EM spectrum imaging machines cover almost the entire EM spectrum.B.Dr. so it is not surprising that image play the single most important role in human perception. ranging from gamma to radio waves.University. y & the amplitude values of f are all finite discrete quantities. The field of DIP refers to processing digital image by means of digital computer. unlike humans.T. & the amplitude of f at any pair of coordinates (x.1. This is limiting & somewhat artificial boundary. y) is called the intensity or gray level of the image at that point. Lonere Image Cartoonize with Stegnography CHAPTER 2 COLOURIZATION USING OPTIMIZATION 2. Digital image is composed of a finite number of elements. where x & y are spatial coordinates.

y) is the intensity of the image at the point (x.University.level process is characterized by the fact that both its inputs & outputs are images. y) where x and y are spatial co-ordinates and the amplitude of „f‟ at any pair of coordinates (x. description of that object to reduce them to a form suitable for computer processing & classification of individual objects. A mid-level process is characterized by the fact that its inputs generally are images but its outputs are attributes extracted from those images. I(x. Lonere Image Cartoonize with Stegnography There are no clear-cut boundaries in the continuum from image processing at one end to complete vision at the other. A low. However. a]  [0.level processing involves “Making sense” of an ensemble of recognized objects.A. as in image analysis & at the far end of the continuum performing the cognitive functions normally associated with human vision.B. 1 Gray scale image: A grayscale image is a function I (xylem) of the two spatial coordinates of the image plane. Digital image processing.Dr. I (xylem) takes non-negative values assume the image is bounded by a rectangle [0. mid-. y) on the image plane. contrast enhancement & image sharpening. & high-level processes. one useful paradigm is to consider three types of computerized processes in this continuum: low-. a] [0. b]I: [0. b]  [0. Mid-level process on images involves tasks such as segmentation. info) Dept of Information Technology 8 . Low-level process involves primitive operations such as image processing to reduce noise. Finally higher. 2.1 What is an image? An image is represented as a two dimensional function f(x. y) is called the intensity of the image at that point. as already defined is used successfully in a broad range of areas of exceptional social & economic value.T.

Figure 2. R (xylem) for red.Dr.T. Digitizing the coordinate‟s values is called sampling.1: Given a grayscale image marked with some colour scribbles by the user (left). An image may be continuous with respect to the x and y coordinates and also in amplitude. G (xylem) for green and B (xylem) for blue.B. Lonere 2 Color image: Image Cartoonize with Stegnography It can be represented by three functions. our algorithm produces a colourized image (middle). Dept of Information Technology 9 .A. Converting such an image to digital form requires that the coordinates as well as the amplitude to be digitized. Digitizing the amplitude values is called quantization.University.

Lonere Image Cartoonize with Stegnography Abstract: Colourization is a computer-assisted process of adding colour to a monochrome image or movie. Our method is based on a simple premise: neighboring pixels in space-time that have similar intensities should have similar colours. as was succinctly pointed out by Earl Glick1 in 1984: .T. In our approach an artist only needs to annotate the image with a few colour scribbles. Colourization of classic motion pictures has generated much controversy which partially accounts for the fact that not many of these movies have been colourized to date. lies in the fact that it is an expensive and time-consuming process. The process typically involves segmenting images into regions and tracking these regions across image sequences.A major difficulty with colourization. colourization requires considerable user intervention and remains a tedious. nor accurate region tracking. The term is now used generically to describe any technique for adding colour to monochrome stills and footage. and the indicated colours are automatically propagated in both space and time to produce a fully colourized image or sequence. time-consuming. But for $50. For example. you can turn it into colour and have a brand new series with no residuals to pay. as evidenced by multiple colourization tutorials on the World Wide Web . while the financial incentives are substantial. segmentation 2.9 [Image Processing and Computer Vision]: Keywords: colourization. recolouring. CR Categories: I. Colourization of still images also appears to be a topic of considerable interest among users of image editing software.A. We formalize this premise using a quadratic cost function and obtain an optimization problem that can be solved efficiently using standard techniques.In this paper we present a simple colourization method that requires neither precise image segmentation.University. consequently. We demonstrate that high quality colourizations of stills and movie clips may be obtained from a relatively modest amount of user input. there are still massive amounts of black and white television shows that could be colourized: the artistic controversy is often irrelevant here.Dr.000 a segment. however. and expensive task.2 Introduction Colourization is a term introduced by Wilson Markle in 1970 to describe the computer-assisted process he invented for adding colour to black and white movies or TV programs [Burns]. However.You couldn't make Wyatt Earp today for $1 million an episode. Neither of these tasks can be performed reliably in practice.4.B. in order to colourize a still image an Dept of Information Technology 10 .

Our contribution. such as the boundary between a subject's hair and her face. an extremely useful operation in digital photography and in special effects. in addition. This colourization process is demonstrated in Figure 1. This assumption leads to an optimization problem that can be solved efficiently using standard techniques. 2. automatic segmentation algorithms often fail to correctly identify fuzzy or complex region boundaries. The underlying algorithm is based on the simple premise that nearby pixels in space-time that have similar gray levels should also have similar colours. Although not much is publicly known about the techniques used in more contemporary colourization systems used in the industry. Unfortunately. Colourization of movies requires.University. Lonere Image Cartoonize with Stegnography artist typically begins by segmenting the image into regions.B.3 Previous work: In Markle's original colourization process [Markle and Hunt 1987] a colour mask is manually painted for at least one reference frame in a shot. allowing colours to be automatically assigned to other frames in regions where no motion occurs. thus. Dept of Information Technology 11 . instead of tracing out its precise boundary. and then proceeds to assign a colour to each region. again requiring massive user intervention in the process. In addition to colourization of black and white images and movies. there are indications that these systems still rely on defining regions and tracking them between the frames of a shot.Dr. Using these user supplied constraints our technique automatically propagates colours to the remaining pixels in the image sequence. Colours in the vicinity of moving edges are assigned using optical flow. Thus. tracking regions across the frames of a shot. The technique is based on a unified framework applicable to both still images and image sequences.T. is a new simple yet surprisingly effective interactive colourization technique that drastically reduces the amount of input required from the user.A. The user indicates how each region should be coloured by scribbling the desired colour in the interior of the region. In this paper we describe a new interactive colourization technique that requires neither precise manual segmentation. which often requires manual fixing by the operator. the artist is often left with the task of manually delineating complicated boundaries between regions. Existing tracking algorithms typically fail to robustly track non-rigid regions. our technique is also applicable to selective recolouring. nor accurate tracking. Motion detection and tracking is then applied.

commonly used in video. where Y is the monochromatic luminance channel. encoding the colour [Jack 2001]. in our technique the artist chooses the colours directly.University. t) and outputs two colour volumes U(x. y. Does not explicitly enforce spatial continuity of the colours. the user must direct the search for matching pixels by specifying swatches indicating corresponding regions in the two images. provides the user with useful brushes and colour palettes. y. 2. The algorithm is given as input an intensity volume Y(x.B. Thus.g. and small when the two intensities are different. and in some images it may assign vastly different colours to neighbouring pixels that have similar intensities. This technique works well on images where differently coloured regions give rise to distinct luminance clusters. r. They examine the luminance values in the neighbourhood of each pixel in the target image and transfer the colour from pixels with matching neighbourhoods in the reference image. but the segmentation task is left entirely to the user. To simplify notation we will use boldface letters (e. Y(r) is the intensity of a particular pixel. Similar weighting functions are used extensively in image segmentation algorithms where they are usually referred to as affinity functions. t) and V(x. or possess distinct textures. we wish to minimize the difference between the colour U(r) at pixel r and the weighted average of the colours at neighbouring pixels: where wrs is a weighting function that sums to one. y. and is able to retune the results by scribbling more colour where necessary. In other cases. while U and V are the chrominance channels.T. While this technique has produced some impressive results. s) to denote (x. We Dept of Information Technology 12 . which we will refer to simply as intensity.A. It is also difficult to retune the outcome selectively in problematic areas. we wish to impose the constraint that two neighbouring pixels r. Thus. s should have similar colours if their intensities are similar.4 Algorithm: We work in YUV colour space. a commercial software for colourizing still images. t) triplets. In contrast. y. As mentioned in the introduction. Lonere Image Cartoonize with Stegnography Black Magic. note that the artistic control over the outcome is quite indirect: the artist must and reference images containing the desired colours over regions with similar textures to those that she wishes to colourize. t). large when Y(r) is similar to Y(s).Dr. describe a semi-automatic technique for colourizing a grayscale image by transferring colour from a reference colour image.

not to propagate colours through time. vy(x. Note that the optical flow is only used to define the neighbourhood of each pixel. The notation r belongs N(s) denotes the fact that r and s are neighbouring pixels.Dr. t +1) if: The flow field vx(x0). The correlation affinity can also be derived from assuming a local linear relation between colour and intensity. The simplest one is commonly used by image segmentation algorithms and is based on the squared difference between the two intensities: A second weighting function is based on the normalized correlation between the two intensities: where mr and sr are the mean and variance of the intensities in a window around r. this Dept of Information Technology 13 . we define two pixels as neighbours if their image locations are nearby.A. are nearby. y) denote the optical flow calculated at time t. Lonere Image Cartoonize with Stegnography have experimented with two weighting functions. let vx(x. t) is a neighbour of pixel (x1. it assumes that the colour at a pixel U(r) is a linear function of the intensity Y(r): U(r) = aiY(r)+bi and the linear coefficients ai. a simple elimination of the ai. and when the intensity is an edge the colour should also be an edge (although the values on the two sides of the edge can be any two numbers). More formally. y). we define two pixels as neighbours if their image locations. Since the cost functions are quadratic and the constraints are linear. Now given a set of locations ri where the colours are specified by the user u(ri) = ui.bi are the same for all pixels in a small neighbourhood around r. Then the pixel (x0.University.bi variables yields an equation equivalent to equation 1 with a correlation based affinity function. J(V) subject to these constraints. y1.T. v(ri) = vi we minimize J(U). In a single frame. While this model adds to the system a pair of variables per each image window. y0. after accounting for motion. Formally.B. Between two successive frames. vy(y0) is calculated using a standard motion estimation algorithm [Lucas and Kanade 1981]. This assumption can be justified empirically and intuitively it means that when the intensity is constant the colour should be constant.

one attempts to find the second smallest eigenvector of the matrix D -W where W is a n pixels matrix whose elements are the pair wise affinities between pixels (i. Dept of Information Technology 14 . which may be solved using a number of standard methods.e.B. The second smallest eigenvector of any symmetric matrix A is a unit norm vector x that minimizes xTAx and is orthogonal to the first eigenvector. In image denoising algorithms based on anisotropic diffusion one often minimizes a function similar to equation 1. our algorithm minimizes the same cost function but under different constraints. the quadratic form minimized by normalized cuts is exactly our cost function J. that is xT(D-W)x = J(x). but the function is applied to the image intensity as well. By direct inspection.T. In image segmentation algorithms based on normalized cuts [Shi and Malik 1997].A. Our algorithm is closely related to algorithms proposed for other tasks in image processing. the r..University.Dr. s entry of the matrix is wrs) and D is a diagonal matrix whose diagonal elements are the sum of the affinities (in our case this is always 1). Thus. Lonere Image Cartoonize with Stegnography optimization problem yields a large. sparse system of linear equations.

we used the original colour channels of each image when picking the colours. and then _ne-tune the colourization results by adding more scribbles. Lonere Image Cartoonize with Stegnography 2. but also on the similarity of their colours in the original image. very convincing results are generated by our algorithm even from a relatively small number of colour scribbles. For still images we used Matlab's built in least squares solver for sparse linear systems. Specifically. Even though the total number of colour scribbles is quite modest. and for the movie sequences we used a multigrid solver. the resulting colourization is surprisingly convincing. and from Chaplin's classic movie Modern Times. we minimize the cost (equation 1) under two groups of constraints. since colours are not propagated across intensity boundaries. First. for pixels covered by the user's scribbles. Figures 7 and 8 compare our method to two alternative methods. As can be seen.B.I Love Lucy. The mean and variance m. In figure 7 the alternative method is one where the image is first Dept of Information Technology 15 . for pixels outside the mask.T.s for each pixel were calculated by giving more weight to pixels with similar intensities. the run time was approximately 15 seconds per frame. Using the multigrid solver. or equivalently using the local linearity assumption). the artist first defines a rough mask around it and then scribbles inside the orange using the desired colour.Dr. The threshold T in equation 4 was set to 1 so that the window used was 3*3*3. The bottom row of the figure shows another example. The original clips were obviously in black and white. Visually similar results were also obtained with the Gaussian window (equation 2). Since automating the choice of colours was not our goal in this work. our algorithm does not recolour the other orange in the image. Note that unlike global colourmap manipulations. To change the colour of an orange in the top left image to green. Figure 3 demonstrates such a progression on a still image. the final colour should be the colour of the scribble. the colour should be the same as the original colour. Figures 5 and 6 show selected frames from colourized movie clips. Second. In this application the affinity between pixels is based not only on similarity of their intensities. We have also successfully colourized several short clips from the television show . so in these examples we did not have a colour reference to pick the colours from. the artist may want to start with a small number of colour scribbles.5 RESULTS: The results shown here were all obtained using the correlation based window (equation 3.University. Figure 4 shows how our technique can be applied to recolouring. Typically. Our technique is then used to propagate the green colour until an intensity boundary is found. All other colours are automatically determined by the optimization process. Figure 2 shows some still gray scale images marked by the user's colour scribbles next to the corresponding colourization results.A.

A. Figure 8 compares our method for colourizing image sequences to an alternative method where a single frame is colourized and then optical flow tracking is used to propagate the colours across time. either using automatic segmentation or using tracking to propagate colours across time. for the correct colourization but the colourization can be quite good even when these hints are wrong. Segmentation is a very difficult problem and even state-of-the-art methods may fail to automatically delineate all the correct boundaries. if the automatic segmentation had been perfect then flood filling segments would have produced perfect results. the colourization achieved with this alternative method (figure 7b) is noticeably worse than the one computed by our method (figure 7c). the same colour scribbles were used. In both cases. or the low contrast boundary between the lips and the face. such as the intricate boundary between the hair and the forehead. state-of-the-art algorithms still do not work perfectly in an automatic fashion. Yet despite many years of research in computer vision. Likewise. Lonere Image Cartoonize with Stegnography segmented automatically and then the scribbled colours are used to “flood fill” each segment.University. if dense optical flow had been perfect then propagating colours from a single frame would have also worked perfectly. Dept of Information Technology 16 . In other words.Dr.T. An advantage of our optimization framework is that we use segmentation cues and optical low as hints. it is much more robust to tracking failures.B. Since our method uses optical flow only to define the local neighbourhood. Distinctive colours were deliberately chosen so that flaws in the colourization would be more apparent. Figure 7a shows the result of automatic segmentation computed using a version of the normalized cuts algorithm [Shi and Malik 1997]. In both cases. the results could be improved using more sophisticated algorithms. Consequently.

A. Top row: the input black-white image with scribbled colors.B. Lonere Image Cartoonize with Stegnography Figure 2: Still image colorization examples.T. Bottom row: resulting color image (a1) (b1) (c1) (a2) (b2) (c2) Dept of Information Technology 17 .Dr.University.

Next. By adding colour scribbles on the table cloth and on the wall (b1) these problems are eliminated (b2). Note that the table cloth gets the same pink colour as the girl's dress. Note that it was not necessary to mark each and every bead (a) (b) (c) Figure 4: A comparison with automatic segmentation.University. For visualization purposes disctinctive colors were used. some colour is bleeding from the cyan pacifier onto the wall behind it. (c) Our result. Segmenting fuzzy hair boundary is a difficult task for typical segmentation methods. yielding the final result (c2). which yield the result in (a2). Lonere Image Cartoonize with Stegnography Figure 3: Progressively improving a colorization. the artist decides to change the colour of the beads by sprinkling a few red pixels (c1).T. Also. Dept of Information Technology 18 .Dr. The artist begins with the scribbles shown in (a1). (a) Segmented image. (b) Result of colouring each segment with a constant color.B.A.

Sign up to vote on this title
UsefulNot useful