Here's the good news: so long as the 4 inch group is centered on the point of aim, you limit your shots to 100 yards, and you aim at the center of the vital triangle, all animals bigger than a woodchuck will be dead.
I'm not saying that you shouldn't work on improving, but a sure way to worsen your shooting is to get worried about it. You'd start yanking trigger, forgetting about natural point of aim, muscling shots, or holding your breath.
Many people think a 1 inch group at 100 yards is important. It is nice, but not important. What is important is knowing at what range you can make a shot and having the maturity not to push it. You know that on your worst day, off sticks, you can hit a cocktail napkin at 100 yards. Aim right and the animal will not know the difference.
I've got a rifle that I have never, under any circumstances, on my best day, with my best ammo, gotten to shoot less than 2 inches at 100 yards, off bags. Someday I may get something more accurate to replace it, but I'm putting it on the plane Friday for Namibia. It didn't let me down last year, so it's going again this year. I've got more accurate rifles, but this one is light and comfortable and gets the job done, so it goes.