Fixed packet stream destroy crashes when closing connections.

Take a reference of the packet stream and then destroy it.  The final
reference is freed in the timeout callback.  Any operation in the mean
time will detect the stream is destroyed and won't do anything.