Gareth Ainsworth said Gillingham fans had every right to be angry, after admitting his side did not turn up against Oldham.