Unfortunately, I don't believe that there's a way to include the buyer note on the invoice currently. The only note field for invoices is under My Account -> Post Sale Management, but that will apply to all invoices.
I will pass along your request for evaluation as a future improvement.