Setting setIsRecyclable to false causes ghosting

I have a ViewHolder containing two view types. I can alternate between those viewTypes via a click. I would like for one of those views to be nonrecyclable, and the other one to be recyclable. For this purpose I set setIsRecyclable(false) when changing from one viewType to the other, and then again setIsRecyclable(true) when changing the viewType the other way around.

public void onClick(View v) {
            int position = getLayoutPosition();
            if (mSelected.contains(position)) {
                setIsRecyclable(true);
                mSelected.remove(position);
            } else {
                mSelected.add(getLayoutPosition());
                setIsRecyclable(false);
                itemView.refreshDrawableState();
            }
            mListener.onItemClick(v);
        }

This almost works as expected, however I do get ghosting effects (artifacts) of the view which used to be recyclable. How can I avoid this effects?