Currently you can ADD a series to a chart, but there isn't a way (or it isn't intuitive) to remove it to go back to a chart without series. I can delete the text but the series remains checked. I would recommend adding a "Clear value" option at the top of the list of fields like is available in feature, etc. records for optional fields.
Attached are screenshots of:
Initial add of series
Attempt to remove series
Series still being there
Field where a "Clear Value" option exists (recommendation)