First, the concept of matter and energy are entagled since Einstein. Most people have an intuition on the word energy as some untouchable fuild with destructive power,mainly due to hollywood. But energy and mass have a much more specific, mathematical meaning in physics. So instead, let me rephrase the question as "does it makes sense to talk about time in the abscence of "sutff"?"
That´s certainly a good question. We do physics by specifying positions of stuff (like in classical mechanics) in space and time, or value of a "stuff" called fields in space and time (like electromagnetism). Mostly all known physics work by assuming a background and describing how things (like tensor fields, spinor fields,etc) evolve in this background. There may even be a relationship between the background and stuff that lives in it (like in general relativity, where geometry is determined by the stress energy tensor). But ultimately the concepts of stuff and background are independent.
It makes sense to talk about the geometry of empty space-time in general relativity, and quantum field theory even tells us that there is no empty space-time at all.
So according to our current most accepted theories, space and time have a meaning even in the abscence of any "material" thing such as clocks, atoms, fields, etc. And we´ve got to this point, where we invented the concepts of space and time to explain the motion of things, and now they "have a life by themselves".
But could we recast physics in such way that time gets meaning only upon the motion of stuff? Ernst Mach used to think that
It is utterly beyond our power to measure the changes of things by time. Quite the contrary, time is an abstraction at which we arrive by means of the changes of things.
This kind of thought has inspired people like Einstein to develop general relativity. But as we can see he hasn´t completely succedded in implementing Mach´s ideas, as it still makes sense to talk about empty space in relativity. There is however some very interesting steps in this direction, mainly due to Julian Barbour and collaborators where time is an emergent concept from "change". In the causal dynamical triangulation approache to quantum gravity, the whole 4d space-time arena is recovered from "something else". It may be that theories like this end up giving completely new testable physics.