When faced with behavior that adults consider childish (like obsessing over Star Wars or watching anime), westerners will commonly spit out "grow up" or "get a life". In the case of the first statement, the idea is that certain activities are fine for kids (i.e. - reading comics) but that you're supposed to grow out of them (the exceptions being "kid activities" like soccer, football and baseball) as you mature and gain more experience.
So, the question becomes, "how would a Japanese adult react here"? Because the same "growing up" pressure exists in Japan as it does in the west. The answer is, Japan is more tolerant of non-mainstream behavior, so most adults wouldn't say anything directly to your face. Example: An adult male that obsesses over pop idols. His adult male compatriots would probably sympathize and ask to look his collection over. Women of the same age would roll their eyes and slink away from him. Older women would whisper snide remarks quietly behind his back. He still wouldn't really fit in to the "adult" world like everyone else, but no one would say that out loud to him.
But, why? What's the pressure to move away from "childish activities"? In part, employment. Japanese men still put in 60 to 80 hour work weeks, even though they're supposed to be cutting back to avoid "karoushi" (death from over work). Plus, salaried workers (i.e. - "salarymen") are expected to mingle with co-workers and management in after-work activities like karaoke, golf and drinking binges. (It's not quite the same for women, who generally hold 40-hour-per-week secretarial positions, and spend their after-work hours either at home or in the clubs. Most women don't expect to work after age 25, instead they plan on getting married and raising a family. Some women, though, may plan to work as late as up to age 30. But, there's still not the amount of job-related after-work mingling with co-workers over one's lifetime as there is with men.) This means that the average adult just doesn't have the time for manga and anime anymore. That is, the pressure to "grow up" is simply the demand to fit into the workforce or to raise a family.
To the outside world, the reactions our pop idol fan receives from those around him are due to his obviously not knuckling down and concentrating on 60-hour work weeks, golf and karaoke in order to cement his place in society.
The exception being college students. More on that in Part 3.