"Might be a little late in life for you to start giving a shit about how I feel." Because it's not like it's ever made a difference to Souji before, that Hijikata's tried and failed to understand Souji's perspective, that Hijikata still wakes up every day feeling like he and Kondou didn't do right by Souji and that they should've been able to give him something better than the miserable lot in life they all got.

And maybe he ought to be able to give Souji better now, but that doesn't seem like it's going to happen, either, so the least Hijikata can do is not run out and leave him to it alone.