Hide users without tasks on the workflow board

We have many users (20+) with a view role and sometimes assign them to one or more tasks. I would like to use the workflow board by choosing all users, but see only those with at least one task.