The 29-year-old suspect, who was believed to be armed with a knife, was confronted two Ministry of Defence Police officers at around 8.50am, police said.
Taser was deployed and the man was restrained by officers.
The “incident” saw the busy London street closed between Trafalgar Square and Parliament Square.
He was arrested on suspicion of attempted murder and possession of an offensive weapon.
The man remains in custody at a central London police station.
There were no reports of any injuries. The incident is not being treated as terror-related. An investigation is under way.
Met officers will continue to liaise closely with the Ministry of Defence Police as it progresses.
A brief CCTV video clip, via Transport for London, shows police vans blocking off the street.
Westminster MPS tweeted about an incident just after 11.30am.
Horse Guards Parade is the ceremonial parade ground in St James’s Park.
Anyone who witnessed the incident, or who has other information which could assist officers, is asked to call 101, providing the reference 1730/18APR.
Information can also be provided to Crimestoppers, anonymously, on 0800 555 111.