Improve performance in the case where ip_output() returns an error.

When this happens, we know for sure that the packet data was not
received by the peer. Therefore, back out any advancing of the
transmit sequence number so that we send the same data the next
time we transmit a packet, avoiding a guaranteed missed packet and
its resulting TCP transmit slowdown.

In most systems ip_output() probably never returns an error, and
so this problem is never seen. However, it is more likely to occur
with device drivers having short output queues (causing ENOBUFS to
be returned when they are full), not to mention low memory situations.

Moreover, because of this problem writers of slow devices were
required to make an unfortunate choice between (a) having a relatively
short output queue (with low latency but low TCP bandwidth because
of this problem) or (b) a long output queue (with high latency and
high TCP bandwidth). In my particular application (ISDN) it took
an output queue equal to ~5 seconds of transmission to avoid ENOBUFS.
A more reasonable output queue of 0.5 seconds resulted in only about
50% TCP throughput. With this patch full throughput was restored in
the latter case.

Reviewed by:	freebsd-net

sys/netinet/tcp_output.c

@@ -864,6 +864,21 @@ send:
 	    (so->so_options & SO_DONTROUTE), 0);
     }
 	if (error) {
+
+		/*
+		 * We know that the packet was lost, so back out the
+		 * sequence number advance, if any.
+		 */
+		if (tp->t_force == 0 || !callout_active(tp->tt_persist)) {
+			/*
+			 * No need to check for TH_FIN here because
+			 * the TF_SENTFIN flag handles that case.
+			 */
+			if (flags & TH_SYN)
+				tp->snd_nxt--;
+			tp->snd_nxt -= len;
+		}
+
 out:
 		if (error == ENOBUFS) {
 	                if (!callout_active(tp->tt_rexmt) &&