I don't recall hearing such a term. I have seen the concept expressed as "Everything I touch turns to sh*t."

For the specific realm of gardening, the opposite to having a "green thumb" is having a "brown thumb." Login to read more.