You can look at the reflectance curves of the metal in the 380-750 nm range. In white light it will be "colored" by the wavelengths that are strongly reflected. In practice this is a slight coloration for most metals, but you could imagine an analog of looking along the mirror glass where you look at light reflected multiple times through a tube of the metal.
Here is an example set of curves from PV Lighthouse:

Copper is the most obvious colored metal, reflecting more yellow and red light than blue, producing the copper tone. Chromium on the other hand is (slightly) bluish, while molybdenum is a bit "greener" and tin nearly "white".
In practice the surface matters a lot. Oxides and surface finish can change the color strongly. Because of the Fresnel equations, reflectance can in principle be polarization- and angle-dependent, especially if the refractive index changes a lot with wavelength. This is not the case for metals as far as I know, so the contribution is minimal, but some of the dielectric oxide layers this can matter (not to mention when the layer itself has thin-film color).
So the general answer is that perfectly pure metals usually have very weak color, but it exists. In practice it is more the surface that determines what they look like. Pure cobalt and samarium are much more silvery than my cubes below, due to oxidation.
