Connes' non-commutative geometry appears naturally in physics in the quantization of classical phase space and in string theory in the presence of constant two-form potentials. String theory predicts the existence of a second type of non-commutative geometry where coordinates are promoted to U(N) valued matrices; this non-commutative geometry, coined D-geometry, is the one seen by a cluster of N particle-like non-perturbative objects, D0-branes. We investigate how D-geometry can be made to obey the requirements of diffeomorphism invariance. The associated physics question is how the D0-branes couple to gravity. We construct an normal-coordinate based algorithm, consistent with a number of physics- and mathematics-based consistency conditions defining D-geometry, that allows us to determine the line-element constructed out of U(N)-valued coordinates order by order. We discuss some mathematical implications and connections between D-geometry and Connes' non-commutative geometry as well as physical implications. In particular we will give evidence for the existence of a gravitational Myers' effect.