From the very founding of the United States, absorbing Canada has been a part of the plan.  Consider the Articles of Confederation, #11:

• “Canada acceding to this confederation, and joining in the measures of the United States, shall be admitted into, and entitled to all the advantages of this union: but no other colony shall be admitted into the same, unless such admission be agreed to by nine states.”

The Articles were agreed by the states in 1781.  These articles were among the founding documents before the US Constitution (1789) (in which Canada is quite reasonably not mentioned).

I think a case can be made that the cultural/political breech pre-dated the US  Declaration and War of Independence – and long-predated the arrival of the United Empire Loyalists.  To the founding states of the US, Canada was just another colony: of course they would want to join.  But Canada was distinct in many ways: in Lower Canada une société distincte, in Upper Canada and the Territories an area largely controlled by a corporate empires (Hudson’s Bay Company, Northwest Company).  There were issues with the government of Britain, to be sure, but they were quite different from the issues between the founding American states and Britain.

Canada was substantially different from the founding United States well before 1776.  It remains substantially different from the 50 States today.