Would it be as easy as just calculating the mass of the pushed object as Mass*wheelCoefficent if that object has wheels?

Pretty much yes.

A more physically accurate version of that equation would be: Weight * FrictionCoefficient.

And from this we can see that you are basically trying to say that having wheels means a reduction in the amount of kinetic friction and therefore makes it easier to push.

Would differences in wheels make a significant enough difference to make the coefficent specific to a type of wheel or would this just be unnecessary data clutter?

That's up to you really. It depends how complete and accurate a model you are building. Potentially there's a lot more to modelling wheels to take into account (diameter, material, tread, axle friction, temperature, tyre pressure - to name a few).