Ray Romano Says Everybody Loves Raymond’s ‘Abrupt’ End Caused Anxiety for His Kids: ‘A Little Guilty’ (Exclusive)

Saying goodbye to Everybody Loves Raymond was a tough pill for Ray Romano to swallow, but what he didn’t anticipate was how difficult it would also be for his family.

In the upcoming documentary Romano Twins, Ray’s twin sons, Matt and Greg, 31, open up about how growing up with a famous father has shaped — and in some ways, hindered — their lives.

The project, which premiered at the Fort Lauderdale International Film Festival on Nov. 9, also features interviews with their mother, Anna, their siblings Alexandra and Joseph, and Ray himself, who reflects on the end of his nine-season sitcom and its impact on his kids — particularly Matt, who struggled with anxiety afterward.

“It came out of nowhere. It took us by surprise,” Ray, 66, tells PEOPLE about the ending of the show, which ran from 1996–2005. “The big concern was, ‘How am I going to handle it?’ I never thought for a second about it having such an effect on these guys.”

Elaborating on his kids’ involvement in the sitcom — in which he starred as a sports columnist with a “dysfunctional” family with twins of their own — Ray adds, “They would come once a week to the show. We would take turns. Matt would come one week to the taping, the filming of the show, and I would say hello to the audience, and they would run out. I’d do a little shtick with them. Every week, we shared this experience.”

The eldest of the four Romano siblings, Ally, 34, vividly recalls the end of the sitcom. “I think the show affected all of us,” she tells PEOPLE. “We were all bawling, crying on that last night. We all grew up on that set.”

As someone who has “been through the ups and downs” and admittedly has dealt with anxiety himself, Ray says the way Everybody Loves Raymond affected his kids left him feeling “a little guilty.” He adds, “This was their world from 3 to 12, and then, it was just stopping abruptly. It really caught us off guard that it would affect them that much, and then, that effect would stay with Matt a little bit.”

“You feel a little guilty about it. It’s hard. It’s hard for any parent no matter what the situation is. And then, I was also dealing with myself, this life change.”

Being thrust into the spotlight for nine years felt like “being in a submarine,” Ray explains — and it left him feeling disoriented. “Before the show, I lived in New York, and then it starts. I’m in a submarine for nine years, because it’s work. And all of a sudden, it ends, and I’m in L.A. I’m like, ‘What the hell is this?’ ” he says. “It was a big transition for the whole family.”

Looking back nearly 20 years later, Ray says as hard as it was to say goodbye to the show, the Romanos were able to get through it together — as a family.

“It’s like anything in life. People who don’t have this situation go through the same thing in their own way,” he says. “We dealt with it, and we got to the other side.”