Paul, I don't think they issue a S264 notice to conduct an audit, except on rare occasions.
If they ask for records (in the absence of a 264) and you refuse, what are they going to do? Knock out your deductions? If they do, lodge an objection with copies of your receipts. End of story.
Of...