If average weight of a human is 80kg (just pulled that out of thin air), then the combined weight of the colonists would be 400,000,000 kg ->
400,000 tons + infrastructure = 525,000 tons of additional mass. In my uneducated opinion, that addition wouldn't affect Deimos at all, as
Deimos current mass, in tons = 1,480,000,000,000
Deimos with additions, in tons = 1,480,000,525,000
Then you would probably have to add furniture, equipment, food stuffs, drinking and septic water and all that jazz, so maybe the weight of infra would double. Would be interesting if someone had the math to actually calculate this through, to find at which point the orbit starts to decay enough that it would plummet to Mars in human life time.