Similar experience. One of them came to my door at 9:30 pm on a Saturday night. I live in a rural area. Demanded I open the door and answer his questions for the census. He was about 50 years old, out of shape and drove a real beater. I  never opened the door, just yelled at him that he had 30 seconds to leave.I had my gun in a holster at my side, and let him see that. He left. I filed a complaint with the census management in my state. Told them I would sue for targeted harassment. Never heard anything else. But, yes, they want to build section 8 here, too, and will likely take property from people like me. I will fight as long as I can. Hope you can, too.