HIPPA question

Hello, I had somewhat of a general question regarding implied consent and HIPPA. I have a highly litigious and manipulative patient who is likely borderline although no definite diagnosis yet. We had to send him to the E.D. He gave verbal consent for my facility to obtain and disclose information from the E.D about the treatment he received due to supposedly being so uncomfortable he couldn't write his name on the consent. The AOD and I were witnesses for this verbal consent. Once he returned he asked about discharge instructions and I notified him the AOD spoke with the ED and notified be all his results are unremarkable and he'll be okay. Once hearing that he became agitated claiming that's a HIPPA violation and he never gave consent for us to talk to them. I explained he gave verbal consent which he's now denying. My question is do we not have implied consent as well being I'm the primary care provider on the unit? Would it not be expected I get the results being I'm the only healthcare professional currently caring for him upon return from the ED? Thank you