If you gave a Worker the boot, that means they're headed off to greener pastures. While you won't be able to get that specific Worker back, they'll leave you a nice little note with a sweet reward! You'll also see a confirmation window before booting your bud, which makes it more challenging to perform this action accidentally.
Don't worry -- if you are in need of more workers, just keep an eye on your bus stop. There's always more showing up, and there's definitely a chance you'll see that worker come by for another visit to your farm!