His inaction has killed many more Americans with Covid than would have died otherwise. He will also be killing poor Americans once he withdraws Obamacare. Yes, he didn't do it directly but he is certainly responsible.
Very few leaders of authoritarian dictatorships have personally killed people, certainly after taking office, the point being you can make the laws and rules which sees them killed by someone (or something) else. Hitler didn't gas a single Jew in his lifetime, it doesn't mean he shouldn't be held accountable for the Holocaust.
(P.S. I am not comparing Trump to Hitler, I am making a point based on the most well known example of genocide in existence)