Oh I agree, and I am certain that is exactly why but it still caused a head canon moment for me, where Holmes leant forward and rather bluntly told the cabby that he was a fool and that he would not be paying for all the detours as a result of incompetence/ignorance.

The problem is Scotland Yard is nowhere near the Tower of London. At all.

I loved the episode!