This type of accusation can only come from one literary form's view of the world as opposed to the other form, which is genre fiction which includes science fiction, fantasy, and horror. It is a debate that has been continuing for a long time and my answer may appear to be a small ripple.
I was initially inspired to write this article as a response to a video by the YouTuber CuteFuzzyWeasel, who posted a video about how his friend ended their friendship after differences in opinion about literary and genre fiction. He then referenced an article his friend sent him that supposedly aided his argument about the importance of literary fiction, though CuteFuzzyWeasel could not remember the title and rather just said the keywords "literary fiction Huffington Post." As soon as I entered those words in the search engine, the only article that appeared was by Steven Petite and was bluntly titled "Literary Fiction vs. Genre Fiction."
If that was the article he was talking about, then I think he was misinformed throughout the latter half of the video, because he rants about how Petite thought science fiction was inferior to literary fiction because it does not speak to the human condition. Petite actually wrote that genre and literary fiction could not be measured on superiority or inferiority, rather they were different in how they viewed the world. While genre fiction tends to be macrocosmic, focusing on what is out there; literary fiction is microcosmic, focusing on what is inside. The only moment, I thought, Petite, did show bias was when he wrote that the writing style was simpler in conventional genre fiction, but other than that, I did not detect any favoritism towards literary fiction.
Although there continue to be courses dedicated to genre fiction (such as a course about vampires), there was a bias towards literary fiction when I was pursuing my Bachelor's in English (and pursuing my Master's in English). It is perhaps for the reason that literary fiction offered insights into various real-world topics, such as Caribbean multi-racial identity and religion as a form of guidance in colonial America. Since literary fiction comes in the form of journals, novels, and essays written by people who come from all wealth backgrounds, I would say that it does more to unite people, than divide them.
Though, I want to be fair to this YouTuber, since he sarcastically suggested three interesting prompts that anyone might write. I think he truly underestimates their value because those prompts can actually speak to the human condition because they are either relatable or possible and there would be complicated emotions playing about. There definitely would be people willing to purchase and read stories dedicated to those prompts, because there are chances that those buyers may have been suicidal and diagnosed with cancer; or they were homosexual office workers entering a relationship with their bosses or young women sleeping with older men to help their ailing marriages. If such topics can relate to people who were given the hardest or strangest treatment by life itself, then surely it would not be considered elitist, since elitist literature would involve characters who were aristocrats or wealthy people in general.
For that reason, literary fiction was originally thought to be more connected to the reader since it develops a sense of empathy on behalf of the reader and the writer, however, there have been new breakthroughs which came to the conclusion that a lifetime of reading literature actually increases empathy. So in that way, literary fiction is not that separate from genre fiction when trying to appeal to human sensibility.
Although it is impossible to make the outright claim that literary fiction is elitist, it does tend to be viewed as being above petty commercial demand, therefore it does not have to be adjusted to suit a large number of people. Of course, if the romance/erotica genre is considered literary fiction, then there is an entire business model dedicated to marketing towards women, specifically from such publishing houses as Harlequin Enterprises. The label of literary fiction does tend to develop a murky definition as it would need to broaden itself as far as it can reach. If the criteria of literary fiction is that it takes place in the real world, then not only would romance be considered literary fiction, but also crime noir novels (though that genre also has a specific business model).
So based on this observation, it can be fair to say that literary fiction can market itself as much as fantasy, science fiction, and horror; so it cannot be called elitist. And just like genre fiction, the sub-genres can comingle. However, this can only go so far in genre fiction, for the world-building is what distinguishes them. If the setting of the story is set on a distant planet like Arrakis inhabited by humans with the assistance of technology and science, then it would become science fiction. If it takes place in a fictional world such as Middle-Earth where magic is the governing force and there are mythical creatures such as elves, dwarves, and orcs, then it would qualify as fantasy. Whereas in literary fiction, there really is no distinction made between the setting and how it operates, since it would mostly take place in the real world.
As for me personally, a literary fiction novel can be "not an escape from reality, but into reality." Although there are many ways to pinpoint what exactly qualifies as literary fiction, what can make it relatable to the common folk is the subject matters. However, it really is important to just note that there really is no difference between those two forms of literature, since they both demand the same care and attention from the readers.