I suggest you build a cabinet inside your closet.
That's what I did... and it enables you to control the transfer of heat, and the light leaks.
((light leaks are 100% unacceptable.))

Another option that crosses my mind, is that you can extend the wall towards the rail of the closet's door... then close the door up to that new wall. In this new wall, you can cut and build your airflow ducts.

Here's my thread, where I built a cabinet inside a closet just like yours, yet smaller in size.
Cabinet, 1000W Hydro-DWC SCRoG White Widow

Here's another one
Closet design... suggestions?

Post up your ideas, plans, doubts, etc. and I'll try to help as much as I can.