Agreed, although in a lot of systems of early notation the rhythm is sort of implied by the text and contextual clues such as knowing whether a particular piece was a certain sort of dance and therefore the likely range of tempi and main beat stress patterns etc. I think it was around 1300 BCE when western music notation included both pitch and rhythm (with mensural notation).