You will lose audience if you can't commit to delivering content on a dependable schedule. Even the best show concepts can get torpedoed by erratic publishing schedules. Think of subscribing to a magazine, back in the day. You knew that your favorite magazine would be in your mailbox within the same general window every month.
It was something to look forward to.
Now imagine if that magazine didn't show up for several days ... weeks ... maybe a month. You'd wonder what was going on, because the content was meaningful to you. It would be a concern for a little while, but if the magazine consistently fell off schedule, you'd lose interest. Eventually, the magazine would be replaced by other distractions.
The window of forgiveness tends to be narrower with digital audio. Once you're up and running, there's an expectation you'll deliver content on a consistent schedule. When you get lazy or lose focus, you're telling the audience it's okay for them to feel the same way.