std::io::Read::read_to_end does not specialize for &[u8] or Take<&'a [u8]>

4a35bf2
Opened by Paul Masurel at 2023-09-26 04:41:23

read_to_end currently includes some logic to do exponentially large .resize / .read operations.

For &[u8] or Take<&'a [u8]>, the problem is much simpler as we know the size that will be required :

v.extend_from_slice(&data);
let n = v.len();
data = &data[n..];

The manual implementation is typically between 3 and 5 times faster on my computer.

Assuming what seems a non-pathological case : no resize required, the Vec has a sufficient capacity to begin with, and running the following benchmark https://gist.github.com/fulmicoton/a1fb87c3f3578f118552917636c95933 yields the following result :

test tests::bench_read_manual ... bench:           9 ns/iter (+/- 1)
test tests::bench_read_std    ... bench:          35 ns/iter (+/- 6)

  1. A simple solution would be to specialize .read_to_end for those types.

    Alternatively, we could also expose a public trait similar in spirit to TrustedLen for iterators, and implement the specialization for this new trait. It seems a bit overkill to expose another Trait for this though.

    Currently, libstd does not enable on the #![feature(specialization)] feature. Other crates of rust (liballoc for instance), already enable it. Is there a case against doing it yet?

    Paul Masurel at 2017-09-25 00:45:12

  2. No, there's no reason we can't enable specialization in libstd.

    I don't think specialization is required for the &[u8] case though.

    Steven Fackler at 2017-09-25 02:59:54

  3. @sfackler By which you mean it is not used / not worth it?

    Paul Masurel at 2017-09-25 03:09:39

  4. No - you can just add a read_to_end implementation in the impl<'a> Read for &'a [u8] { ... }.

    Steven Fackler at 2017-09-25 03:43:01

  5. Oh yes that is true :)

    Paul Masurel at 2017-09-25 04:07:04