Why does the compiler understand ASCII by default? (piefed.blahaj.zone)

submitted 2 months ago* (last edited 2 months ago) by akunohana@piefed.blahaj.zone to c/c_lang@programming.dev

17 comments fedilink hide all child comments

Environment\
Compiler Clang on Termux on Samsung's One UI 7 on a Samsung Galaxy Tab A9+

The title is an assumption on it's own, so feel free to correct it/me!

I was experimenting with sanitizing user input, read to a character array with fgets. Specifically, I was trying to have a for loop remove (skip) certain input. Here is the code:

for (n = strlen(input) - 1; n >= 0; n--) { if (input[n] >= 0x30 && input[n] <= 0x39 || input[n] == ' ' || input[n] == '\t') { input[n] = 0x18; } }

While the program does behave as I want it to, I don't understand why it seemlingy by default understands that the various hex codes refer to the character encoding as per the ASCII table. I cated my tablet's filesystem encoding at /sys/fs/f2fs/dm-44/encoding, which yielded UTF-8. If I understand it correctly, the first 128 code points of Unicode are the same as ASCII's. But according to this article on Wikipedia, there are no hexadecimal references in Unicode, only octal and decimal.

If the underlying filesystem uses UTF-8, and Unicode code points are not referred to by hex, how then does my compiler (Clang) understand what ASCII code points I'm referring to?

Is there some conversion going on under the hood that I am not aware of? I did find a libxml2/libxml/encoding.h, which contains comments about some conversion to and from UTF-8. Is this it? I can't make head or tails of it because of my limited C knowledge...

you are viewing a single comment's thread
view the rest of the comments

[-] avidya@programming.dev 2 points 1 month ago* (last edited 1 month ago)

You can actually change this.

The compiler has an 'execution' character set that describes how characters should be interpreted at run time, and a 'source' character set, and they need not be the same.

Check this out

$ cat test_ebdic.c ; gcc -fexec-charset=EBCDIC-US test_ebdic.c ; ./a.out | xxd
#include <stdlib.h>
#include <stdio.h>
int main(void)
{
        char data[] = "Hello";
        printf("%s\n", data);
        return EXIT_SUCCESS;
}
00000000: c885 9393 960a                           ......

The file was typed in UTF-8, and the compiler automatically translated the literal to EBCDIC-US to emit at run time. My terminal can't display this, so I view it with a hex editor.

[-] akunohana@piefed.blahaj.zone 1 points 1 month ago

Sic!!! Thank you so much! I'll try this as soon as my new Gentoo system is up and running! Just to have another excuse for me to set it up already. 🤣

Is it gcc or the OS that determines what encodings I can pass as an option to the compiler?

[-] avidya@programming.dev 2 points 1 month ago

GCC uses libiconv to figure out how to transcode the bytes.

this post was submitted on 07 Jun 2026

7 points (100.0% liked)

C Programming Language

1340 readers

15 users here now

Welcome to the C community!

C is quirky, flawed, and an enormous success.
... When I read commentary about suggestions for where C should go, I often think back and give thanks that it wasn't developed under the advice of a worldwide crowd.
... The only way to learn a new programming language is by writing programs in it.

irc: #c

🌐 https://en.cppreference.com/w/c

founded 3 years ago

MODERATORS

snowe@programming.dev

thomas@lemmy.douwes.co.uk

modev@programming.dev