Who are "the rest"?

The 'random for today' -

http://www.biblegateway.com/passage/?search=Acts%205&version=ESV

and

http://www.biblegateway.com/passage/?search=Nehemiah%2011&version=ESV

In Acts 5-

Many Signs and Wonders Done

¹² Now many signs and wonders were regularly done among the people by the hands of the apostles. And they were all together in Solomon's Portico. ¹³ None of the rest dared join them, but the people held them in high esteem.

I was struck by the phrase "the rest" wondering exactly what that meant.  I think it means believers apart from the apostles staying away so they wouldn't be arrested along with the apostles.  What do you think?