That could possibly arise from your power settings.
If it does then that might point to the cpu underperforming. How many of the 4 mic position samples have you loaded up? If more than one mic position, try it with one position only to see if that improves matters.
It's not too bad a cpu so perhaps it's surprising you're getting problems with such a large buffer - there's certainly no point in increasing the buffer size since latency must already be noticeable or close to it.
