Just like the pixel density of an image, beyond a certain value, makes no difference to the human eye, perhaps the touch screen response time also gets perceived only up to a certain speed. Any faster and it makes no difference. Outliers with above average visual capability do perceive things differently and maybe you have a similar ability with respect to noticing touch screen response times. That might explain why most people don't care so much. Just theorising here, but if anybody has any numbers, it would be good.
These latency numbers are around the border of minimum perceptible visual latency, which is somewhere around 100ms (http://stackoverflow.com/a/2547903/547213) A difference between between 55ms and 123ms is enough to matter.