But while on the nest hub it's completely frictionless, here you need to change user on the touchscreen before speaking, which makes it completely pointless because in that time you could do the action manually.

Example turn off a light. You speak the order and it "doesn't understand" because it's on the kid profile. So you go over there, change profile, speak again, wait for the response... In the same time you could have done that action manually much more faster