Time of day is is a field like say 10:25 am, ie when you look at our watch, what time is it.
That is not a Duration field. A duration field type expresses how long something took. For example if it took 15 minutes and 20 seconds to handle a call, that is a duration. I suspect that you are using a time of day field to hold a duration and that is why the chart is not allowing you to measure the time of day.
If this is the case and you already have a lot of data entered, then that data will need to be converted once we understand who you have been entering the data.