I just checked with DHS (Private.Aircraft.Support@DHS.gov). A lady there replied almost instantly... what a great service.

Anyway, it seems I was mistaken. Non-US citizens/permanent residents can depart from any airport/place in the US.