No matter what you’re shooting—landscapes, travel, wildlife, street photography, portraiture… Literally anything you might be photographing, there’s always an opportunity to make selections on the image when retouching it in Photoshop.
For me, I make a lot of selections when retouching portraits to separate subjects from backgrounds, and I use a variety of different tools to do it. But still, every once in a while I learn something new that take my retouching to the next level.
Enter Nathaniel Dodson and his Tutvids YouTube channel. Dodson has a great video outlining a bunch of great ways to make accurate selections, and then to make them even better with the right adjustments.
He covers a variety of selection methods including AI tools like select sky, select color range, object selection too, focus area and more. Then he uses tools like refine edge, select and mask, smudge tool, channel-based selections, decontaminate colors and others to improve those selections and make them blend more seamlessly.
For instance, there’s one unlikely tool for cleaning up edge artifacts where background color is sneaking through in fine details, complex areas of edges and corners. This occurs often with things like feathers and hair and foliage. Dodson suggests using the Smudge tool to clean up a mask and make for a smoother, cleaner edges that camouflage the shortcomings of Photoshop—or more to the point, the shortcomings of we amateur retouchers—when it comes to very fine, complex selections.
“This is like a super power,” Dodson explains. “Using this tool, which I use for almost nothing in photoshop, will help clean up all these edges. There’s really no tool in Photoshop that gets rid of that stuff very well [edge selection artifacts]… It just doesn’t do it very well. So if you select the layer mask itself you can use the smudge tool. I keep the strength right around 50 and I reduce the brush size down. We are working on the mask. I’m just gonna paint and I’m going to push white in my mask up over the bits that are green, and it’s going to wipe them away. What it’s done is it’s given us a very realistic edge.”
Dodson next goes on to use the Select and Mask tool, changing view modes and zooming in for close-up views of fine selection details. He suggests feathering the pixels with the Select and Mask controls in order to clean up mask edges, and to easily see where the shortcomings are by changing backgrounds to red or black to better view the finest details of the selection.
He uses my favorite edge-refining tool for selections, too: Refine Edge.
“The refine edge brush is a little bit tricky,” he says. “It’s a bit temperamental. One of the things that I’ve found works best with this brush is A, use it with a soft edge, and B, make it as small as you can get away with. You just want it to straddle the edge that you’re looking at. You don’t want to take a giant brush and stretch it out over the whole thing. It just doesn’t seem to work very well.”
It also works on defringing edges around masks. For instance, if you cut an image from a light background and paste it on a dark background, you’ll end up with a bit of a halo, or a lighter fringe along the edge. Dodson uses a paint brush with a special mode setting to paint on the mask and improve the edge.
“In order to make this easy,” he says, “the cheat code is to go to Mode and choose the Soft Light blend mode. Make your brush a little bigger and just paint up to the edges. And what this Soft Light blend mode is making the whites a little bit whiter, pushing the whites a little bit further. Or if you paint with black it’s going to push black a little bit further and that is going to do things like allowing you to paint right up to an edge and remove a bit of that halo.”
He goes on to discuss converting elements to smart objects to help repair halo effects along edges, as well as blend modes and brightness along the layer’s edge to help blend it more seamlessly into the background.
Finally, Dodson wraps up by discussing ultra-complex selections with Channels. This is applicable for difficult subjects, such as selecting and masking all of the foliage and branches on a tree and allowing the sky and background to be masked away—even as it appears in small glimpses through the center of the tree. To make such an ultra-complex selection, he suggests a combination of object selection tools and manual lassoing to adjust the selection before masking based on the color channel of the subject. Then he fine-tunes the edges of a complex selection by using the levels controls to adjust the black and white areas of a layer mask. Because the tones in the mask are either black or white, levels adjustments effectively expand or contract that mask. This is a great way to tighten up a mask around the edges in order to eliminate halos and fuzzy edges. Check out the full video at the Tutvids YouTube channel, or watch it below!