How to hide Ghost light shadows(4.22)

hasu0703

I wanted to use the new Iray visble to Primary Rays feature in 4.22 to create clothes that would only show up if they were in a reflection.
However, when I tried this, the clothes disappear, but the shadow appears where the clothes were and does not have the desired look.
Adding Emission to the clothes removes the shadows, but the color changes from other places and the reflections look too bright.
Is there any way to avoid shadows in Iray visble to Primary Rays?