Around 1480 Walter Innes (2nd son of Robert Innes 11th of that Ilk) got the lands of Invermarkie from the Earls of Atholl and Huntly. Water's son - Robert Innes - was appointed Lieutenant-General of the kingdom, and did well enough to be created constable of the fortress of Ethirdurver in the Black Isle. The Black Isle was part of the Earldom of Moray, Robert and James Earl of Moray became great friends. When James died in 1544, he left the lands of Coxton to Robert in his will - even though it was technically not his to give him (James sold Redhall, Stynie and Blackhills to the 1st Laird of Cromey in 1523, also technically illegal). The R.I. on the panel over the door refers to this Robert. Robert married Elizabeth Stewart, who was sister to the Duke of Atholl and niece to King James II. Elizabeth's father was given Balvenie in 1454, which later was owned by the Innes's. Robert died in 1547. He had four sons (Robert, Walter of Achlunkart, Alexander and John Innes of Edingight).

Water Innes's fouth son, Patrick of the keam of Duffus, married Christian Cockburn. It was their son Alexander, who obtained the lands of Coxton from the 3rd laird of Invermarkie in 1571 (Charter 9/3/1571). Robert the third baron of Invermarkie, was grandson of Robert Innes the 2nd baron. John Innes 17th of that Ilk, was going to die without heirs, so he transfered the title of clan chief and estates to Alexander Innes 3rd Laird of Cromey. Robert, being the richest branch of the family, thought he should be Chief, so resolved to murder Cromey. Robert didn't object to Cromey being heir, only the way he took over the title and lands of John before his death. Both John and Alexander lived at Kinnairdy, and Robert Innes proceeded to feed John lies about Cromey. His Chance came in 1580 when Cromey's son fell ill while attending college in Aberdeen. Robert, John and other co-conspirators entered the house, Robert shot Cromey, and got the whole lot involved with the murder, by getting every one to plunge a dagger into the dead Cromey, including John Innes the ex-clan chief. John Innes (Son of Alexander of Coxton and Janet) was roused out of bed (He was also at school in Aberdeen), to plunge the reeking dagger into the body of his murdered chief. This caused alot of arguments, and in September 1584, firstly Robert the 3rd baron of Invermarkie was himself murdered (by Alexander Innes of Cotts and Leuchars), and then the laird of Innes burnt the newly build place of Coxton to the ground, probably due to John Innes of Coxton's involvement with the murder. This created a long legal wrangle with Robert Innes 4th Baron of Invermarkie and Robert the laird of Innes, so it was a long time before the laird of Innes paid Alexander the sum of 7,000 merks and 2 termes for damages (1 merk = £1/3 = 1s 4d !!). Invermarkie later changed his title to Balvenie, as the name Invermarkie was always linked to the murder of Cromey.

The A. I. on the panel over the door refers to Alexander Innes and his two wives - J.R. is Janet Reid and K.G. is Kate Gordon. Alexander died on 12 October 1612 and his son John Innes was granted in 1618, a royal charter for the barony of Coxton, thus confirming the ownership of the land given by the Earl of Moray. John had two wives, firstly to Margaret Kinnaird of Culbin (15/12/1586) and then to Margaret Gordon of Gight, daughter of William Gordon. John started to rebuild the tower in 1621. After John died in 1635 the place of Coxton was again sacked. James Innes of Halton, the eldest son of John (who had already died in about 1620) had married Marjorie Innes of Leuchars against his fathers wishes. His grandfather and father then excluded him from any inheritance (As well as Walter Kinnaird of Culbin, Patrick Grant of Caron, Robert Innes of Invermarkie and other friends), so in 1635 James's son Alexander (While still a minor), broke down the Tower's doors and carried off into the night the charter chest after a bloody fight. Innes of Leuchars and other members of the clan, were ordered to pay 1,000 merks to restore Coxton. In spite of the disinheritance of his father, Alexander Innes became the 2nd Baron of Coxton, after intervention of his uncles (Walter who married Jane Gordon; and John from whom the Inneses of Newseat are descended. The 2nd lairds daughters were Jean; and Barbara - who married William Brodick of Orton, by whom she had 24 children, before marrying Arichbald Innes of Westestone. Barbara was charged with slaughter of her own servant).

Alexander Innes first married has cousin, Mary Gordon of Gight. She was taken ill in 1646 and died the following year, not helped by the minister of the kirk, Alexander Anderson, insisting that she attended every Sunday service. This was the same minister who earlier had to take refuge in the tower, when Montrose's army invaded Moray (Montrose lived at Fyvie in 1644). Sir Alexander Innes (as he now was, having being knighted by Charles II, probably when he came to Speymouth on 23rd June 1650), then married Mary Mackenzie, sister of the Baronet of Coul. (Her grandfather was Colin "Càm" Mackenzie, clan chief until 1594) The shield above the south window of the ground floor refers to this union, and is unusual heraldry as marshalling of arms is normally achieved in Scotland by division Palewise, whilst this shield is parted fesswise. While still living, Alexander gave the tower to his nephew, Sir Alexander Innes (son of John Innes of Culdrain and Helen Gordon).
Sir Alexander Innes, was known as one of the gentleman of Scotland, Prior to becoming 4th laird of Coxton in 1677, he had built a town house in Elgin (Where No. 53 High street now stands), which probably sealed the redundancy of the Tower later. Sir Alexander had two Wives - Jean Rollo of Bannockburn and Jane Meldrum. In 1688 he became provest of Elgin and in 1689 he represented Moray in Parliament. The 4th Laird, 3rd Baron and 1st Baronet (Granted in 1686) of Coxton was a staunch Jacobite. Calverhouse, Viscount of Dundee visited on 24th April 1689, during the campaign that ended at Killiecrankie. Although they won that battle, Calverhouse's death took the heart out of the uprising. Alexander Innes and General Buchan were the last Jacobite leaders to submit the following year. He was also among those who planned the projected uprising of 1708 (After the Act of Union and the death of James II's last descendent, both the previous year). Sir Alexander Innes died in 1708.

Sir George Innes, the 2nd Baronet of Coxton inherited alot of his fathers debts. He was also "A mighty weak and indolent man, much given to the bottle". Sir George Innes married Elizabeth Gordon, heiress to the Towie-Barclay estate. In 1714 he sold the tower, a transaction he celebrated at a drunken feast. Sir George died after the battle of Sheriffmuir in 1715. The Innes Baronetcy of Coxton continued (Sir George has sold the barony along with Coxton). The Baronetcy was though to have died out after the death of the 8th Baronet in 1886, but it was revived in 1973 and Sir David Innes is currently the 12th Baronet of Coxton.

Mr. William Duff of Dipple (Which he had bought from Robert Innes 21st of that Ilk in 1684), bought the Tower from Sir George Innes for what was considered a high price. After hearing about George Innes's celebrations he said "Poor fellow, he is as well pleased at parting with his inheritance as I at getting it." Later on William discovered that he had taken on all the lairds debts, but managed to get George to sign a list of his creditors. William paid these without any fuss. But the signed list was £20,000 Scots short of the lairds debts. So when the ones not on Sir George's list, turned up on after his death, William refused to pay. These creditors then sued William in the Edinburgh Court of Session, and were awarded the monies. William Duff was not happy with their decision, so he took the case to the House of Lords, who reversed the judgement, saying William was only liable for Sir George's signed list.

William died in 1722, leaving Coxton and a fortune to his only surviving son, William who became Lord Braco in 1735, then Viscout Macduff and Earl of Fife (Irish Honours) in 1759. Lord Braco did erect a stone stair to the first floor, otherwise Coxton was rather neglected, as no one lived there. William married Lady Janet Ogilvie and four years later Jean Grant from which he has 14 children (William, Anne, Janet, James, Alexander, Jane, George, Lewis, Patrick, Helen, Sopia, Catherine, Arthur and Margaret). William died in 1763.

The 2nd Earl of Fife was James (William had died in London in 1753), He nearly doubled the lands owned by the Duffs, including purchase of Innes, Inchbroom, Dunkinty and Leuchars from the Innes family. He married Lady Dorothea Sinclair, only child of the ninth Earl of Caithness. James was MP for Banffshire 1754-1784 and of Morayshire 1784-1790. James lived at Duff House and died in 1809. The Earldom then passed to his Brother Alexander, who married Mary Skene. Alexander died in 1811.

The 4th Earl was Alexander's Son James, who was MP for Banffshire 1818-1826. He had married Maria Caroline Manners, but she died after becoming ill, following a Newfoundland dog who bit / scratched her in 1805. He died in 1857

In around 1840 Benjamin Penichuik came with his wife, 3 daughters and son to Coxton from Dunning in Perthshire, to become the Earl of Fife's forester. As there was no place for them to live they moved into the tower, while they were waiting for the nearby cottage to be built. They hauled their furniture up to the square turret with ropes, to get the furniture in, and put in a new stair to the first floor. Their fourth daughter Rachael was born in the Tower, before they moved into the Cottage. They later moved to Lhanbryde, Where Benjamen Penichuik, who was a herbalist, brewed a tea to cure Lhanbryde of an epidemic of Jaundice in about 1852. Of his children - Madaline ran the kindergarten in Lhanbryde; Margaret married Torquil Patterson and ran the Wool/Blanket mill in Lhanbryde; Kate moved to Perth; Alexander went to work with Findlays in Rangoon, and helped to set up the Burma Oil Company; and Rachael went for an extended holiday in Japan. No one has lived in the Tower since them.

The 14th laird of Coxton - Ralph Christie, nephew of Thomas, died in 1991