In order for it to be an ethnic migration, you’d expect at least some Cagots somewhere to have preserved a language or dialect, like the Roma or the Irish Travelers do.

Right, additionally I don't even know who would be able to invade the area and leave a population demo there but not make it into any history books over it.

Also my original comment was gonna be "They're gypsies, CMV" before I started to work on something more serious.

I do think most likely explanation is they received some kind of social protections (perhaps they were a kind of indirect social protection) while the Moors were in power that were no longer available after Reconquista; this stance subject to change as I look into it further.