Yeah the party began to be seen as weak on fascism and just an arm of Moscow, which was obviously not popular among most American Communists. The extremely quick about face when Barbarossa happened just cemented the idea that working for the CPUSA was just being a pawn for Russian interests rather than working for liberation in the US. One of the many unfortunate consequences of the geopolitical realities at the time wrt Communism.