Author Topic: Image background removal  (Read 5063 times)

Patrice Terrier

  • Administrator
  • *****
  • Posts: 951
    • zapsolution
Image background removal
« on: May 31, 2016, 03:01:04 pm »
Mike--

Have you ever used the GPU to perform canny edge detection and image background removal.

I think that there is a specific OpenGL shader that can be used to do it.

Several years ago i wrote a specific flood fill algo that could achieve this fast with the CPU, but that would be nice if i could rather use the GPU.

For example in the attached screen shot, i would like to remove/replace all the white parts around the shirt to keep just the object, while using a tolerance percentage to get rid of the grayed pixels on the right.

The hard thing, is that it must be an automated process (no user interaction), using the top left pixel color (coordinate 0, 0) for the seed color to remove. In my case, this seed color will use a transparent alpha channel in order to save the result as a transparent PNG, like when doing a screen shot in ObjReader.

Any suggestion?

...




« Last Edit: May 31, 2016, 05:53:53 pm by Patrice Terrier »
Patrice
(Always working with the latest Windows version available...)

Michael Lobko-Lobanovsky

  • Administrator
  • *****
  • Posts: 777
Re: Image background removal
« Reply #1 on: May 31, 2016, 04:53:03 pm »
Hi Patrice,

No I haven't seen a ready-made shader for this but judging by your description, the task and solution should be pretty straight-forward as long as the background color/image to remove is artificially monochrome rather than a real-world colorized background like e.g. a wallpaper.

Moreover, you can speed up processing in the plain C language 3 to 3.5 times making the filter multithreaded spawning 4 threads on a quadcore CPU whereby each worker thread would process one pixel wide scanline at a time until the entire image is processed.

Note that the technique will guarantee 100% correct conversion of lossless image formats only because e.g. JPEG noise will generate colorized spekles and background pixel recognition will fail. There are techniques to process real-world backgrounds and lossy image formats too but they are far from 100% foolproof and are usually academic, i.e. presented as research paper annotations without actual programmatic implementations given.

Once the algo is developed and debugged in C, it can be ported to GLSL that can make it even faster. However, the existing ObjReader's GLSL setup won't be able to save the result because the technique presupposes writing the resultant image into the backbuffer/memory texture rather than into the on-screen window.
Mike
(3.6GHz Intel Core i5 Quad w/ 16GB RAM, nVidia GTX 1060Ti w/ 6GB VRAM, Windows 7 Ultimate Sp1)

Patrice Terrier

  • Administrator
  • *****
  • Posts: 951
    • zapsolution
Re: Image background removal
« Reply #2 on: May 31, 2016, 05:47:34 pm »
Here is an example of canny edge detection, performed in real time, using GLSL.

https://www.youtube.com/watch?v=fJtOdJ9MHsA
Patrice
(Always working with the latest Windows version available...)

Michael Lobko-Lobanovsky

  • Administrator
  • *****
  • Posts: 777
Re: Image background removal
« Reply #3 on: May 31, 2016, 06:05:59 pm »
Thanks Patrice, that's most spectacular, but that's not background removal, that's edge detection. Your task is much simpler and it doesn't require such an elaborate edge detection scheme, does it? Simple discrimination between colored and monochrome pixels would do, wouldn't it?


P.S. Do you want me to try my hand at separating your shirt from its gradient background? BTW the gradient isn't black and white; this must be a JPEG re-saved in the PNG format.
« Last Edit: May 31, 2016, 06:18:53 pm by Michael Lobko-Lobanovsky »
Mike
(3.6GHz Intel Core i5 Quad w/ 16GB RAM, nVidia GTX 1060Ti w/ 6GB VRAM, Windows 7 Ultimate Sp1)

Patrice Terrier

  • Administrator
  • *****
  • Posts: 951
    • zapsolution
Re: Image background removal
« Reply #4 on: May 31, 2016, 07:06:41 pm »
Quote
Simple discrimination between colored and monochrome pixels would do,

No, because i could have black and white clothes, or only monochrome pictures.

Indeed, canny edge detection can be very handy to create a mask to remove the unwanted region.

I already have written code to perform color removal, and or flood fill with adjacent pixels, just like with the magic wand in Photoshop. However GLSL canny edge detection would be great, because of the use of the GPU, and the better edge detection.

There is an example on Code Project using a shader and the SOBEL algo.
http://www.codeproject.com/Articles/94817/Pixel-Shader-for-Edge-Detection-and-Cartoon-Effect

Canny edge detection in C#
http://www.codeproject.com/Articles/93642/Canny-Edge-Detection-in-C
« Last Edit: May 31, 2016, 07:15:55 pm by Patrice Terrier »
Patrice
(Always working with the latest Windows version available...)

Michael Lobko-Lobanovsky

  • Administrator
  • *****
  • Posts: 777
Re: Image background removal
« Reply #5 on: May 31, 2016, 09:55:07 pm »
I can't make out how to connect edge detection with background. Exactly which edge should we take for the background borderline? Can you formalize it? We won't be able to automate borderline recognition if we can't formalize the criteria...

Human interaction in determining the object/background borderline is required because we use a yet different mechanism of human visual pattern recognition based on our real-life experience to determine the borders of the object we wish to crop, or even to reconstruct that border when it becomes invisible against, and thus indistinguishable from, a similarly colored background pattern.

As deducted from your OP, I thought you would like to crop out monochromatic backgrounds with possible gradients within certain tolerances as seen and done in the alpha channel when reconstructing transparency masks for 24-bit images, whereby pixel monochromaticity, albeit artificial, could be a very solid criterion to differentiate between the background and foreground. [0,0] pixel coloration with a given tolerance might be usable too. But both would fail as soon as you come across similarly colored areas in the foreground object unless you set forth, and/or reconstruct where necessary, the exact borders within which the discrimination rules are inapplicable because the pixels would in fact belong to the object that you wish to preserve.

You seem to want to generalize the task to fully automatic any-image-against-any-background technique, in which case pattern recognition would be a must and would complicate the task beyond any reason for an indie project. Automated flood filling/color replacement are commonplace but pattern recognition isn't trivial at all. I used to develop red eye removal filters for my graphics effect libraries and I also used to work on a captcha breaker freelance project once for money at the now defunct RentACoder dot com site; I'm familiar with the problems. :)
« Last Edit: June 01, 2016, 09:47:57 am by Michael Lobko-Lobanovsky »
Mike
(3.6GHz Intel Core i5 Quad w/ 16GB RAM, nVidia GTX 1060Ti w/ 6GB VRAM, Windows 7 Ultimate Sp1)

Patrice Terrier

  • Administrator
  • *****
  • Posts: 951
    • zapsolution
Re: Image background removal
« Reply #6 on: June 01, 2016, 11:04:45 am »
The technic to remove the color will be the same than the one being used for flood fill, starting from a seed point.
In this specific case the seed point will be always X,Y coordinates 0,0, but should work also from any other seed coordinates.

My zDraw application does that already, however without using the canny edge detection and with traditional CPU coding.

My idea was to use the canny edge detection to create a black and white mask, then using the pixel color at coordinate 0, 0 that in my case will be most of the time from the black color, to perform the flood fill removal until the white edges.

I have attached the zDraw.chm help file (written in PB long ago), to show you what it could do, and look at the Examples section named "Tutorial files"

The attached image is an example of the process i want to automate.

And you know what, my zDraw project (inspired by Photoshop), was at the origin of GDImage.

...
 
« Last Edit: June 01, 2016, 11:13:42 am by Patrice Terrier »
Patrice
(Always working with the latest Windows version available...)

Michael Lobko-Lobanovsky

  • Administrator
  • *****
  • Posts: 777
Re: Image background removal
« Reply #7 on: June 01, 2016, 06:34:34 pm »
Congrats Patrice,

This piece of SW, however obsolete it might be, does look good judging by its brief description in its help file. :)

But!

It is obviously human-assisted just like Photoshop effect gimmicks are. They do require human brain to supply a criterion to differentiate between foreground and background pixels, or at least some artificial intelligence as e.g. in Google's in-built trainable neuronal-network image/object/pattern recognition system, to be able to reconstruct intelligently the main object's outlines in borderline cases that occur every so often.

Please have a look below at how miserably both flood fill and edge detection fail given the [0,0] coord color and 15%/30% tolerances, respectively, in the absence of solid criteria to restore the exact outlines of the foreground object. Similarly, the canny edges of the moving objects detected in the youtube video you pointed me to aren't solidly closed (a.k.a. "water-tight") and thus aren't suitable for foolproof automatic flood filling or background cropping.

I'm sorry to say but I still think what you're trying to do is simply not feasible in the scope of an indie project, GLSL or no GLSL.
Mike
(3.6GHz Intel Core i5 Quad w/ 16GB RAM, nVidia GTX 1060Ti w/ 6GB VRAM, Windows 7 Ultimate Sp1)

Patrice Terrier

  • Administrator
  • *****
  • Posts: 951
    • zapsolution
Re: Image background removal
« Reply #8 on: June 01, 2016, 07:27:00 pm »
Mike, thanks for the feedback.

Here is the best i could do with my zDraw flood fill in automated mode, with a tolerance of 50%, and without using the edge detection.

As you can see i couldn't get rid of the grayed pixels on the right, because with a higher tolerance i remove also pixels from the shirt itself.

Some of the cany edge detection i have seen are able to produce full closed area, hence the reason why i thought it could be used for automation.

...
« Last Edit: June 01, 2016, 07:29:09 pm by Patrice Terrier »
Patrice
(Always working with the latest Windows version available...)

Michael Lobko-Lobanovsky

  • Administrator
  • *****
  • Posts: 777
Re: Image background removal
« Reply #9 on: June 01, 2016, 07:43:31 pm »
Here is the best i could do with my zDraw flood fill in automated mode, with a tolerance of 50%, and without using the edge detection.
That's because the object boundaries are generally high contrast and not borderline cases as in my test case above.

Quote
Some of the cany edge detection i have seen are able to produce full closed area, hence the reason why i thought it could be used for automation.
Unfortunately not in this case as you can see in the snapshot below...
Mike
(3.6GHz Intel Core i5 Quad w/ 16GB RAM, nVidia GTX 1060Ti w/ 6GB VRAM, Windows 7 Ultimate Sp1)