I'm a software engineer for an IoT company. I don't have smart locks because I live in a shitty apartment and can't replace them, but I would have them otherwise.

I think the stuff that's really sketchy is the wifi-enabled stuff. That I might not trust, but I think the main exposure that would come from a zigbee/z-wave lock (beyond someone just picking it) is via your account with the IoT hub provider, which is as secure as any other internet account, but will let someone open your door.

There are solutions that can air-gap that stuff, the downside being then that you can only control it if you're on your home wi-fi, but it's possible.

When it comes down to it, someone can always just throw a brick through your window.