Tracking Issue for RFC2509: concat_bytes!()

Opened by Mara Bos at 2025-02-21 22:41:03

Feature gate: #![feature(concat_bytes)]

This is a tracking issue for https://github.com/rust-lang/rfcs/pull/2509

Public API

#[macro_export]
macro_rules! concat_bytes { .. }

Steps / History

[x] RFC: https://github.com/rust-lang/rfcs/pull/2509
[ ] Implementation
[ ] Final comment period (FCP)
[ ] Stabilization PR

Unresolved Questions

Should additional literal types be supported? Byte string literals are basically the same thing as byte slice references, so it might make sense to support those as well (support &[0, 1, 2] in addition to [0, 1, 2]).
What to do with string and character literals? They could either be supported with their underlying UTF-8 representation being concatenated, or rejected.

Feature gate: #![feature(concat_bytes)]

This is a tracking issue for https://github.com/rust-lang/rfcs/pull/2509

Public API

```
#[macro_export]
macro_rules! concat_bytes { .. }
```
Steps / History

- [x] RFC: https://github.com/rust-lang/rfcs/pull/2509
- [x] Implementation: #87599
- [ ] Final comment period (FCP)
- [ ] Stabilization PR

Unresolved Questions

- Should additional literal types be supported? Byte string literals are basically the same thing as byte slice references, so it might make sense to support those as well (support &[0, 1, 2] in addition to [0, 1, 2]).
- What to do with string and character literals? They could either be supported with their underlying UTF-8 representation being concatenated, or rejected.
Mara Bos at 2022-01-21 14:57:00
I'm working on an implementation at Smittyvb:concat_bytes.
Smitty at 2021-07-29 13:27:12
Implementation PR: #87599
Smitty at 2021-07-29 17:39:01
Regarding the unresolved questions - can we reject regular string slices if we accept &[] since str::as_bytes() is const?
Jeremiah Senkpiel at 2021-07-30 17:10:19
@Fishrock123 Macro arguments are not a constant evaluation context, so calls to const fns in macros result in the tokens representing the function call getting passed to the macro, not the result of evaluating the function call.

For example,
```
const fn add_one(x: u8) -> u8 { x + 1 }
macro_rules! demo {
    ($x:expr) => { let x = $x; }
}
trace_macros!(true);
demo!(add_one(2));
```
gives the output
```
note: trace_macro
 --> src/main.rs:8:1
  |
8 | demo!(add_one(2));
  | ^^^^^^^^^^^^^^^^^^
  |
  = note: expanding `demo! { add_one(2) }`
  = note: to `let x = add_one(2) ;`
```
The add_one(2) call never gets expanded to 3 (although LLVM can optimise that) , since it's not used in a constant evaluation context. It is treated no different then any other function.
Smitty at 2021-07-30 18:09:28
@m-ou-se should the implementation box be checked off now? It seems the PR to implement it was merged.
Octavia Togami at 2022-01-19 03:23:12
Was FCP started?
Michael Pankov at 2022-07-28 09:57:47
👋 , just ran into this trying to use it on a stable-codebase I'm working on. I'm curious is there any blockers/open-questions with this issue that aren't reflected here. Could we start stabilizing this?
Cynthia Coan at 2022-10-23 03:11:48
Hi. I have just implemented this in my library: const_str::concat_bytes. It supports const-evaluation, more powerful than std::concat_bytes.
Nugine at 2022-10-26 12:44:05
Is this feature still unstable?
Ratan Kaliani at 2023-01-20 18:27:57
I just ended up needing this macro for the first time. I needed to include a certificate in my binary. And my dependency's X509 type requires the byte slice to end with null (they later pass it on to FFI, and re-allocating just to add a null at runtime is a waste, especially in embedded).
```
static CERTIFICATE: X509 = X509::pem_until_nul(concat_bytes!(
    include_bytes!("../certs/server_certificate.pem"),
    b"\0"
));
```
I switched over to const_str::concat_bytes for now, since it's stable. But I personally think the std macro should adopt any advantages that it has and stabilize it. This type of macro belongs in core IMHO. Given how it's related to concat! and given how it's related to pretty fundamental include-style stuff that I would expect the language to have built in.
Linus Färnstrand at 2023-08-26 21:24:40
I think the only issues that need to be resolved before stabilization are:
- deciding if we want to allow unnested u8s: currently concat_bytes!(b'A', b'B') and concat_bytes!([65,66]) are ok but not concat_bytes!(65,66)
- deciding if we want to allow chars and &strs to be concatenated (currently they can't be)
I think the status quo is fine. Does anyone think concat_bytes! should behave differently?
Smitty at 2023-08-30 02:48:13
Not that I currently need it, but supporting const evaluation of non-literals seem very handy. Not being able to currently do concat_bytes!(b'a', "foo".as_bytes()) like const_str supports is limiting.

I have two questions:
1. Would it be hard to just make std::concat_bytes! as powerful as const_str::concat_bytes!? If not, let's just do it. I don't see any downsides.
2. Is allowing strictly more input to the macro really breaking? Otherwise we can stabilize it as is and improve it later.
Linus Färnstrand at 2023-08-30 06:14:02

Are the remaining questions irreversible decisions? For example, if concat_bytes! were stabilized without support for non-nested u8 or &str, could that be changed in a backwards-compatible way later?

If the answer is "yes" then I think it clearly should be stabilized. Combining &[u8] literals is important for no_std programs that need to embed complex data.

Also, I've been using this implementation of concat_bytes! as a placeholder -- it can be dropped in as-is and doesn't have any dependencies.

macro_rules! concat_bytes {
	($( $chunk:expr ),+ $( , )?) => {{
		struct Chunk<T>(T);
		#[allow(dead_code)]
		impl<const N: usize> Chunk<[u8; N]> {
			const fn get(&self) -> &[u8] { &self.0 }
		}
		#[allow(dead_code)]
		impl<const N: usize> Chunk<&[u8; N]> {
			const fn get(&self) -> &[u8] { self.0 }
		}
		#[allow(dead_code)]
		impl Chunk<&[u8]> {
			const fn get(&self) -> &[u8] { self.0 }
		}
		const fn bytes_len(chunks: &[&[u8]]) -> usize {
			let mut len = 0;
			let mut ii = 0;
			while ii < chunks.len() {
				len += chunks[ii].len();
				ii += 1;
			}
			len
		}
		const fn chunks_concat<const N: usize>(chunks: &[&[u8]]) -> [u8; N] {
			let mut buf = [0u8; N];
			let mut ii = 0;
			let mut buf_idx = 0;
			while ii < chunks.len() {
				let mut jj = 0;
				while jj < chunks[ii].len() {
					buf[buf_idx] = chunks[ii][jj];
					jj += 1;
					buf_idx += 1;
				}
				ii += 1;
			}
			buf
		}
		const CHUNKS: &[&[u8]] = &[$( Chunk($chunk).get() ),+];
		const BYTES_LEN: usize = bytes_len(CHUNKS);
		const BYTES: [u8; BYTES_LEN] = chunks_concat(CHUNKS);
		BYTES
	}};
}

John Millikin at 2023-10-22 02:10:19

I just ended up needing this macro for the first time. I needed to include a certificate in my binary. And my dependency's X509 type requires the byte slice to end with null (they later pass it on to FFI, and re-allocating just to add a null at runtime is a waste, especially in embedded).
```
static CERTIFICATE: X509 = X509::pem_until_nul(concat_bytes!(
    include_bytes!("../certs/server_certificate.pem"),
    b"\0"
));
```
I switched over to const_str::concat_bytes for now, since it's stable. But I personally think the std macro should adopt any advantages that it has and stabilize it. This type of macro belongs in core IMHO. Given how it's related to concat! and given how it's related to pretty fundamental include-style stuff that I would expect the language to have built in.
That feels like something better for an include_c_str!(...) than this. You can always c_str.to_bytes_with_null() if you need to include the null. (Very much poor API design for them not to just expose a X509::pem_c_str(&CStr) here, but that's just my personal opinion.)
Claudia Meadows at 2023-10-30 21:39:30

Since it's been a while, anyone else in the peanut gallery looking at what's out there, if your use case is like mine and you have a fixed size array as your destination, const functions are pretty powerful these days (but still awkward). No unsafe, no complex code, just a nested for loop:

const fn concat_bytes_to_array<const N: usize>(sources: &[&[u8]]) -> [u8; N] {
    let mut result = [0u8; N];
    let mut dst = 0;
    let mut src = 0;
    while src < sources.len() {
        let mut offset = 0;
        while offset < sources[src].len() {
            result[dst] = sources[src][offset];
            dst = dst
                .checked_add(1)
                .expect("constant calculation must be correct");
            offset = offset
                .checked_add(1)
                .expect("constant calculation must be correct");
        }
        src = src
            .checked_add(1)
            .expect("constant calculation must be correct");
    }
    assert!(dst == N, "did not fill the whole array");
    result
}

const MAGIC_COOKIE: [u8; 16] = concat_bytes_to_array(&[&42u64.to_le_bytes(), &13u64.to_le_bytes()]);

Tv at 2025-02-21 22:41:03