Accept immigrants due to colonial guilt?

Some people say we should accept mass immigration due to the crimes committed by our Empire in the past. I think this is irrational as there have been many empires throughout the world who have committed multitudinous crimes and atrocities but their ancestors are not held accountable in the same way we are. The British Empire was also one of the least ferocious empires there were. This is not to excuse the crimes but instead to state a fact. People have to realise that nobody came out of history clean and why should people have their quality of life diminished purely based on what their ancestors did a very long time ago.