When analyzing a mental health conversation between a counselor and his/her client, one can examine the semantics underlying the utterances of conversation to understand if the counselor has practiced the appropriate psychotherapy techniques at different points of the conversation. Despite the many breakthroughs in solving NLP tasks, state-of-the-art large language models (LLMs) still perform poorly on this utterance label prediction task. While a simple supervised learning architecture combining an utterance encoder with a linear softmax layer can yield better accuracy, the trained classifiers still suffer from poor quality ground truth labels assigned by human annotators. Motivated by this observation, we propose a quality-aware framework that derives quality weights of ground truth utterance labels, trains a target classifier in two stages, and evaluates the target classifier with quality weights. Our experiments on three mental health conversation datasets show that target classifiers trained using our framework yield significantly improved accuracy over classifiers trained not using quality weights, even outperforming the strong LLMs using direct prompting.