Timing the start of something is not a simple process. Punishing people for being late disproportionately impacts poor and disabled people. Not all of us can drive a car or even have a car to drive. We might have to wait on other people, or use public transportation, and the more steps we add to the process the more likely something is to go wrong. Punishing people for being late is systematic oppression towards these groups. This punishment can include starting without people, especially if that itself is framed as a punishment.
“Let the late ones be late and miss out (they can read the minutes), and reward the prompt ones by not wasting their time”
From the rusty's rules of order, something the IWW uses to organize. They are ableist.
On the other hand, waiting too long to start can impact people with limited time or energy. Not everyone can stay awake an extra hour just to wait for something to start.
This means that there is not one singular solution for how to start things (although obviously don’t do punishment). In small groups the best solution is to talk things over with everyone and get an idea for what everyone wants to happen, what can go wrong, and plans to mitigate any potential issues. If public transportation is running late, maybe someone with a car can go pick you up.
For large groups, most things do not need a strict starting time. If it is a large group and it requires strict attendance then you brought hierarchy into it long ago and ableism and such was always the conclusion you were going to get anyways.
I don't know about punishment, but I can certainly understand the frustration if you're time poor.
My chosen social groups of hippies and goths are sometimes frustrated by me arriving early or on time, but not to the point of uninviting me to stuff or yelling at me.
As a punk, I know shows start on punk time which is at least an hour and a half after whenever the flyer says and everyone is a half hour late to everything at least.