Good question!
Consider these shorter phrases:
We are the only ones (ser)
vs
We are in the restaurant (estar)
The verb is expressing "being" about "being the only people" which is independent of what follows (which happens to be a location). You could write the sentence in a longer form, We are the only people [who are] in the restaurant to see there are two 'being' verbs implied, in fact.
It might be tempting to still think estar is the right verb choice, since 'being the only ones' is temporary state, but, in fact, 'only' is a modifier and as such a red-herring too! Really, that sentence boils down to talking about people: We are the [optional modifer] people...
I think this is the closest rule:
https://progress.lawlessspanish.com/revision/grammar/use-ser-not-estar-to-introduce-people
I'll double check with one of our Spanish native linguists to be sure though.