You seem unfamiliar with the reference. In the moments following this tense scene the character played by Ed Norton does indeed stand up for himself. He counters the jewish dinner guest’s assertions on the causes of race riots, before politely informing him that his kind isn’t particularly welcome at his family’s dinner table, removing his shirt to reveal a swastica chest tattoo for added emphasis.