He just dismissed James Woods as a shitty person, which he 100% was, but Ginger never saw it that way and Sam didn't try to connect with that).
Ultimately he never really tried to understand Ginger and what her real concerns were (read: her relationship with James Woods.