When I was reading The Fifth Witness by Michael Connelly yesterday, I came across this line similar to what we were discussing:
P.43 "I didn't think so. So all I'll say then is be careful."
The author did not use "to be careful".
I still don't have a clear explanation to that, but I tentatively believe, by dropping "to", it would give the statement a warning, or an admonishing or a demanding undertone.