begriffs

Inside the C Standard Library

January 19, 2019
Newsletter ↳

Cover of the Standard C Library After diving into the C language through K&R, and then studying portability (see C Portability Lessons from Weird Machines), my next challenge was to take a systematic look at the standard library. To do this I worked through P. J. Plauger’s book The Standard C Library (ISBN 978-0131315099) where he examines an implementation of all the functions. It has a chapter for each header, with background information, an excerpt from the C89 standard, tips on use, and full implemention with tests. The author was on the X3J11 committee that defined ANSI C.

As I worked through the book – trying first to write the examples myself, then comparing his code to mine, and finally running the examples – I kept notes with questions about portability, rationale, and C behavior. By cross-referencing the following books, asking questions on IRC, and browsing StackOverflow and the comp.lang.c archives, I found satisfactory answers.

  • “The C Standard: Incorporating Technical Corrigendum 1” by The British Standards Institution (ISBN 978-0470845738). This is the C99 standard itself (rather than C89 like Plauger’s book), and it includes an entire first half devoted to the rationale behind language and library choices. This is helpful for understanding C semantics.
  • “Portable C Software” by Mark R. Horton (ISBN 978-0138680503). Written after ANSI C was standardized, but early enough where it wasn’t fully adopted. He provides early history of each standard library function, as well as some functions that are now defunct.
  • “Portable C” by Henry Rabinowitz (ISBN 978-0136859673). Great for illustrating the design decisions of the language as it relates to diverse hardware.
  • “The CERT C Coding Standard” by Robert C. Seacord (ISBN 978-0321984043). Illustrates potential insecurity with, among other things, the standard library. Lists real code that caused vulnerabilities.
  • “C Programming FAQs” by Steve Summit (ISBN 978-0201845198). I can see why these were historically the most frequently asked questions. I asked many of them myself.

This article is not a comprehensive explanation of the standard by any means. It’s just things that were new or interesting to me. Some of it may be old news to you, and conversely I may have omitted something that seemed basic to me but would have been useful to mention. The focus is C89, with comparisons to the later standards C99 and C11 when relevant.

(Note: updated several sections based on feedback from Lobste.rs, HN, and Reddit. Thank you for the review, everyone.)

Brief History of the Library

Functions in the library grew organically from communities of programmers sharing ideas and implementations. Many groups of people used C on Unix throughout the 70s, across multiple architectures. They wrote compilers with extra features, and experimented with additions to Unix. By February 1978 core C practice had stabilized to the point where Kernighan and Ritchie codified it in the first edition of their book The C Programming Language (ISBN 978-0131101630).

By 1980 C users formed the “/usr/group” organization to combine their library experience into an informal standard, which they released in 1984. Meanwhile in 1983, the American National Standards Institute (ANSI) formed a committee, X3J11, to establish a standard specification of C and officially standardize the library. The committee reviewed the work of /usr/group, K&R 1st edition, and various compiler extensions. They deliberated from 1983 to 1989 to produce the C89 standard (“ANSI C”).

“Design by committee” may not have pleasant associations for some people, but in this case the committee drew on a lot of experience, and often declined to speculatively innovate, working to clarify existing practice instead. The result was a small, tight language and standard library.

Compared with libraries in other languages, the standard C library is lean. It doesn’t have much in the way of general algorithms or containers. This helped the language port easily and more widely. The library has basic facilities for time, math, I/O etc, and operates on simple types. It also provides portable facilities to do non-portable things, like variadic arguments, and non-local gotos.

We will restrict attention to the C standard library only, and not extensions such as POSIX. Thus no functions from unistd.h, fcntl.h etc.

assert.h

A simple way to halt a program with debugging information if an assumption doesn’t hold:

#include <assert.h>
int main(void) { assert(1 == 0); }

/* outputs
Assertion failed: (1 == 0), function main, file assert.c, line 2.
Abort trap: 6
*/

Because it echoes the statement under test and includes a filename and line number, it’s useful for a quick and dirty test suite. As a hack to include a little extra information, logically conjoin a string:

assert(1 == 0 && "Checking this because...");

Furthermore, assert() calls abort() rather than exit(), which causes the program to quit and dump core if permitted by the operating system. If the binary was compiled with debugging support you can load that core file in a debugger and inspect the program’s full state at the time the assertion was made:

gdb -c /path/to/corefile

# ^^^ inspect variables, backtrace, etc in the debugger

This means that littering functions liberally with assertions can be a good way to debug problems. It provides richer information than debug print statements in situations where it’s OK to terminate the program. It also adds no overhead to the final release, because the assertions will be removed by the preprocessor when compiled with NDEBUG defined. This can be done by adding -DNDEBUG to CFLAGS or by adding a regular #define NDEBUG in the code.

All headers in the standard library are idempotent, except assert.h. By including it multiple times in a file you can enable or disable the assert macro.

#define NDEBUG
#include <assert.h>

/* Now assert() won't do anything... */

#undef NDEBUG
#include <assert.h>

/* Now assert() works again */

It’s OK to include assert.h twice in a row because it causes what C calls a “benign” redefinition of the assert macro.

/* no harm, this is a benign redefinition: */
#define FUN 1
#define FUN 1

/* not benign and not allowed: */
#define FUN 1
#define FUN 2

A properly designed assert macro should work in any context, even somewhere weird like for (i = 0, assert(n<10); i < n; ++i). Because I didn’t think of this, my own attempt at writing assert used an if statement. In fact this is what Mark Horton shows in his Portable C Software book:

/* incorrect definition */

#ifndef NDEBUG
#define assert(p) (if(!(p)) ... )
#else
#define assert(p) ;
#endif

This can be improved. Plauger uses the ternary operator, and substitutes the result ((void)0) when the predicate holds. He also keeps the code (to print the error and exit) in a real function which is defined in a separate source file. That’s because headers in the standard library are not allowed to include each other. It’s a self-imposed discipline.

The last trick I found interesting was how to delay preprocessor evaluation into two steps using these helpers:

/* a "thunk" to evaluate __LINE__ */
#define _STR(x) _VAL(x)
#define _VAL(x) #x

The string can then be built as

"Assertion failed: " #p ", file " __FILE__ ", line " _STR(__LINE__) "\n"

Finally, some implementations tolerate an arbitrary scalar expression as the argument to assert, but the ANSI committee decided to require int expressions for correct operation.

Given that it was the first header I saw in Plauger’s book, this is where I learned that the standard reserves names like _Foo starting with underscore and a capital letter, for the standard library. Don’t use that naming convention in regular code.

ctype.h

Although I learned a few things from this header, it was ultimately kind of a letdown. It’s not sufficient for international text processing because, while the functions operate on the int type, they specify that its value should fall in the range of unsigned char or the negative value EOF. So while non-English speakers can use a new codepage for their locale, it will be unable to hold more than 255 symbols (plus \0, plus EOF), which is too few for east asian languages. C99 introduced wctype.h to operate on wide characters, but that has its own problems. (More about that in the stdlib.h section below.)

Even for languages which fit in an 8-bit codepage, the ctype functions don’t always suffice. For example the German letter ß uppercases into two letters, SS. The tupper() function in ctype.h can’t handle it, replacing one character with another. The greek letter Σ ordinarily lowercases to σ, except at the end of a word where it should be ς. The tolower() function doesn’t have enough context to pick the correct form.

I still learned some interesting techniques by studying this header. Plauger implements all the isxxxxx() functions with a lookup table of bit-packed shorts. The table itself is specific to EOF (from stdio.h) being -1, and relies on a certain size of unsigned char. The code uses a nice trick to fail early on a system where this assumption is incorrect:

#include <limits.h>
#include <stdio.h>

#if EOF != -1 || UCHAR_MAX != 255
#error WRONG TABLES IN CTYPE.H
#endif

Although the code is non-portable, it fails in the most honest and upfront way possible at compile time. Also as a historical note, the isxxxxx() functions used to be defined only for 7-bit characters, but ANSI requires them to handle all values for unsigned char.

One advantage of using a lookup table is that the value to be looked up requires only one evaluation. Thus the lookup function can be a macro without danger of executing code with a side effect more than once.

/* evaluates c at least twice
 * (also ASCII-specific, but don't focus on that)
 */
#define isalpha(c)  \
  (((c) >= 'a' && (c) <= 'z') || ((c) >= 'A' && (c) <= 'Z'))

/* evaluates c only once */
#define isalpha(c)  (_Ctype[(int)(c)] & (_LO|_UP))

/* consider how this would behave */
if (isalpha(c = getchar()))
  ...

Using macros for these little functions is good for performance. However, every library function in the standard library (unless specifically noted otherwise) must be represented as an actual function too, in case a program wishes to pass its address as a parameter to another function. How can we define a function called “isalpha” if that name is also recognized by the preprocessor? I learned you can just enclose the name in parens:

int (isalpha)(int c) {
  return (_Ctype[(int)(c)] & (_LO|_UP));
}

/* now &isalpha will be defined if needed */

The trick is used throughout the standard library but I first saw it in ctype. Conversely, every library function is a candidate for redefinition as a macro, provided that the macro evaluates each of the arguments exactly once and parenthesizes them thoroughly. (Well, getc is an exception but more on that later.)

Another trick is to perform an array lookup using a pointer that is shifted forward in the original array.

static const short ctyp_tab[257] = {0, /* ... */};
const short *_Ctype = &ctyp_tab[1];

The shifted pointer allows EOF (-1) to be looked up easily without undefined behavior. The expression _Ctype[EOF] means _Ctype[-1] which is the same as *(ctyp_tab+1-1), which does not attempt to dereference – or even point to – memory before a primary data object. Pointers are allowed to be assigned only addresses either inside, or one space to the right, of a data object. (Data objects are arrays, structures, or regions allocated from the heap. See Rabinowitz’s book for a good discussion of this.)

Character codes 128-255 are interpreted as negative numbers when considered as signed char. C does not specify whether char is signed or unsigned (it varies by platform). When char is signed, beware of converting it to int for the ctype functions. The integral promotion will “sign-extend,” creating a negative int value. This is not suitable for ctype functions, which require int values that are either storable in unsigned char or are the special EOF value. To avoid sign extension, cast char values to unsigned char in calls to ctype functions.

errno.h

Errno is the mechanism everyone loves to hate. The X3J11 committee wanted to remove it but decided not to make such a radical innovation on existing practice. In the early drafts of the standard it was kept it in stddef.h, but they decided that stddef should exist on freestanding environments and they split errno.h off into its own header.

Not much to say about it. The errno global variable is set to zero at the start of program execution. Library functions can set it to nonzero values but will never set it to zero themselves. To use it, explicitly set it to zero yourself, call a library function, then check whether errno changed.

One interesting tidbit is that errno is sometimes not a global variable at all, but a macro for (*_Error()). This allows different program threads to see a different value in a multi-threaded application. Even before threading considerations, having to set a real data object immediately after performing hardware floating point ops would break the FPU pipeline. Allowing the check to be deferred until requested with this _Error() function doesn’t break the pipeline.

float.h, limits.h, and math.h

Both float.h and limits.h are inventions of the committee. You can generate them with the enquire program written by Steven Pemberton. It performs runtime checks on the data types to find information about them, then generates the desired header file. It’s very (but not completely) portable, as it “only works if overflows are ignored by the C system or are catchable with signal().” It also detects and outputs information about the representation of base types, like endianness.

I decided not to slow down to study these headers in depth because I lack the knowledge of floating point representation necessary to understand the internals of the functions inside. I do know the definitions and parameters in float.h were recommended by numerical analysts on the Committee. The set was chosen so as not to prejudice an implementation’s selection of floating-point representation.

The math functions are written carefully to avoid overflow and underflow. I’ll revisit the topic after studying Michael Overton’s short book, “Numerical Computing with IEEE Floating Point Arithmetic.”

locale.h

Changing a program’s locale tells it how to handle local customs. There are multiple locale categories that can be adjusted independently. They control different things, like the codeset used by ctype, the date or monetary formatting desired, or alphabetical sort order. Typically all categories are set to the same locale. The alternative – a so called “mixed locale” – is less common.

Some of the settings are meant for interpretation by your own program, and others automatically affect the C standard library. For example, when parsing a double, strtod checks what the current locale uses for a decimal point symbol. Even if you want to avoid ctype and use a third-party Unicode library, some of the locale information is still useful for your program.

By default C programs use the “C” locale, which is ASCII text and American formatting. The most respectful thing, though, is to accept the locale in all categories as set by system environment variables. This is indicated by the empty string for locale name.

#include <locale.h>
#include <stdio.h>
#include <stdlib.h>

int main(void)
{
  if (setlocale(LC_ALL, "") == NULL)
  {
    fputs("Unable to select system locale", stderr);
    return EXIT_FAILURE;
  }

  /* ... */
}

To see what locales are known on your local system, run

locale -a

There are 203 installed on MacOS. The list begins with:

  • en_NZ
  • nl_NL.UTF-8
  • pt_BR.UTF-8
  • fr_CH.ISO8859-15
  • eu_ES.ISO8859-15
  • en_US.US-ASCII
  • af_ZA

They are in the format [language[_territory][.codeset]]. OpenBSD has chosen to support only the “C” (aka “POSIX”) and “UTF-8” codesets, but supports many languages and territories in the other locale categories. Unicode subsumes all those partial character encodings, so BSD just wanted to eliminate a source of complexity.

One way to see locales in action is by setting environment variables and using Unix tools. We can change sort order using LC_COLLATE.

cat <<EOF >cn.txt







EOF
LC_COLLATE=C sort <cn.txt LC_COLLATE=zh_CN.UTF-8 sort <cn.txt

Setlocale() doesn’t play well with threads because it updates an internal global variable. Set the locale before spawning threads.

One other random nugget of wisdom from the book, unrelated to locales but mentioned in that chapter, is to use a tool to visualize the call tree in a C program. This can help you understand a new codebase. Try cflow. See also Steve Summit’s C FAQ question 18.1.

setjmp.h

The setjmp/longjump functions create a “saveable goto” statement to return to places you’ve already been. It allows jumping from one function into another. They are like the C programmer’s version of exception handling. Setjmp and longjmp are very tricky inside, as they have to save and restore variables, arguments, registers etc to resume execution in another location.

#include <setjmp.h>
#include <stdio.h>

void dostuff(void);

jmp_buf target;

int main(void)
{
  if (setjmp(target) == 0)
  {
    puts("Saved the target, continuing on.");
    dostuff();
  }
  else
    puts("I feel like I've been here before...");
  return 0;
}

void dostuff(void)
{
  longjmp(target, 42);
}

To indicate that the jmp_buf was set, setjmp returns 0. When execution is jumped back to this point, setjmp returns value passed as the second argument to longjmp, for us 42. The same if statement is evaluated again but with a different result, like waking up in an alternate universe.

As simple as this looks, it’s easy to break. The statement containing setjmp must be very simple. Calling setjmp in an if statement or switch statement are fine, but you should not save the return value like n = setjmp(...). An assignment statement is too complicated and can disturb the sensitive machinery.

The function containing setjmp should be as simple as possible. Only variables declared as volatile are guaranteed to be restored. Generally it’s best to execute the real processing in another function called when setjmp returns 0.

Longjmp should not be called from an exit handler (i.e., a function registered with the atexit function). Finally it’s undefined behavior to attempt to longjmp to a function that has since returned. Thus longjmp’s usual use case is like exceptions in other languages, going back up the call chain, skipping intermediate functions.

The jmp_buf type is actually an array behind a typedef, which is why it’s typically passed to setjmp without an ampersand. The standard forbids jmp_buf to be implemented as a scalar or struct.

Given all these caveats, the committee considered requiring that compilers recognize calling setjmp as a special case. Then the function could work in all types of statements. However they decided against it for consistency because they don’t require any other function to be a special case (although they allow compiler writers to make special cases as desired).

signal.h

Signals are a UNIX technique for interprocess communication that causes a process to asynchronously call a handler function, or else take a default action. Programs also receive “synchronous” signals for their own logical exceptions like division by zero, segmentation faults, or floating point problems.

The ANSI committee decided to standardize a weakened portable version of signal functionality. Portable signal handlers can do very little. Here’s a typical example of what a handler can do safely:

/* a global */
volatile sig_atomic_t intflag = 0;

/* SIGINT handler */
void field_int(int sig)
{
  signal(SIGINT, &field_int);
  intflag = 1;
  return;
}

It does as little as possible, simply setting a global variable which regular code can check at leisure. One thing to note is that it re-installs itself in the very first line with the signal() function. That’s because on some platforms (like Linux), as soon as a signal is handled it reverts to its default handler, which in the case of SIGINT terminates the program. Other systems such as BSD leave a handler installed when called.

On Linux this would be an unwise thing to do:

void field_int_badly(int sig)
{
  /* open a window where a repeat signal could
   * hit the default handler before we reinstall */
  sleep(1);

  signal(SIGINT, &int_catch);
  intflag = 1;
  return;
}

Even our earlier technique of calling signal() right away in the handler isn’t completely safe. The CERT C Coding Standard warns that this leaves a tiny window open for a race condition. They suggest not to use the C library signal functionality at all, but the equivalent POSIX functions instead. POSIX allows you to specify persistence of the handler during initial registration.

Another thing to note is the type declaration of the shared flag (that we called intflag in the example). Exception handlers should read and write only volatile variables. For asynchronous exceptions, volatile alone isn’t even enough. The variable should be small enough that it can be read or written atomically by the processor.

The C standard provides the sig_atomic_t typedef for this. Each standard library implementation defines it as an alias for a suitably small integral underlying type. If you’re writing portable code, don’t assume that sig_atomic_t is anything bigger than a char, and don’t assume its signedness. Thus the portable value range is 0…127, although C99 added macros to determine its min and max values.

The standard says not to call any standard library functions from a signal handler except abort(), exit(), longjmp() or signal(). Certainly avoid any functions that interact with state, like those performing I/O or else stdio streams can become corrupted.

Even though the C standard says it’s OK to call longjmp from a signal handler, CERT gave an example where doing so caused a vulnerability [VU #834865] in Sendmail because it allowed attackers to time a race condition in main() by timing signals.

A program can raise signals for itself with the raise() function. (It used to be called kill.) However longjmp() is less tricky than raising signals for yourself and should be preferred.

The standard library defines this limited list of signals:

  • SIGABRT abnormal termination, such as is initiated by the abort function
  • SIGPPE an erroneous arithmetic operation, such as zero divide or an operation resulting in overflow
  • SIGILL detection of an invalid function image, such as an illegal instruction
  • SIGINT receipt of an interactive attention signal
  • SIGSEGV an invalid access to storage
  • SIGTERM a termination request sent to the program

Using other signals makes a program less portable.

stdarg.h

On the PDP-11 it was easy to walk function arguments with a pointer. The memory layout of arguments was well known, the size of pointers was the same as the size of int, and the original C language could not pass structures by value.

However when spreading to other architectures, C benefited from creating a portable way to access variable numbers of arguments. Non-PDP architectures had complex calling conventions. Using a library for variadic arguments makes code clearer too, whether or not portability is an issue.

Pre-ANSI C on UNIX used <varargs.h>, which required a final “dummy argument.” ANSI C got more picky about arguments matching a declaration, and introduced the “…” token to take the place of the dummy argument, which breaks varargs. The “…” also signals to a compiler that it may want to change the function calling convention.

The committee turned varargs.h into stdarg.h, and generalized the macros to the extent that all known C implementations would be able to handle them with little modification. This functionality was important for other functions in the standard library like printf and scanf.

Using the library is pretty easy, just initialize the list based on the final fixed argument, loop through the args, and release the list.

/* add a list of n numbers */
int sum(int n, ...)
{
  va_list ap;
  int s;

  va_start(ap, n);

  for (s = 0; n > 0; --n)
    s += va_arg(ap, int);

  va_end(ap);
  return s;
}

It is undefined behavior to call va_arg() more times than there are arguments, so the function will need to determine the number of arguments by other means. Our function above consults the n parameter for that information.

Speaking of the n variable, we’re not actually passing its value to va_start(), much as it may look. The va_start macro manipulates “n” as merely a name, so it can calculate the address of the next argument. Both va_start and va_arg must be implemented as macros, not as functions.

These are the portable assumptions for using stdarg.h:

  • The variadic function must declare at least one fixed argument
  • The function must call va_end before returning (for cleanup on some architectures)
  • va_arg can deal only with those types that become pointers by appending “*” to them. Thus register variables, functions, and arrays can’t be returned by va_arg
  • If a type widens with default argument promotions, then va_arg should request the widened type

The last point requires some explanation. When functions had no prototypes in pre-ANSI C, the compiler would promote smaller types to wider ones when sending them to functions. That’s because doing so is essentially free – it costs more to put a byte in a register than to put a word in.

Although ANSI requires functions to have prototypes, the promotion rule still applies to variadic arguments. Char and short get expanded to int, and float gets promoted to double. Thus to accept a char argument, ask for its value with va_arg(ap, int) and then cast to char. Don’t do va_arg(ap, char).

To pass a va_list to another function for continued processing either a) memcpy it if you want to consume all arguments in the current function or b) pass a pointer to it if the called function should consume some or all of the arguments. C99 added va_copy() for the first scenario.

The implementation of stdarg.h macros is gnarly and entirely platform specific. On my system they resolve to builtin compiler functions.

stddef.h

Stddef.h is a catchall place for definitions, sort of like stdlib.h. Why make two headers rather than combine them into one? It’s because C can be compiled in either a “hosted” or a “freestanding” environment. The latter is for embedded programming where there isn’t enough room for the entire standard library. An implementation must include all the standard library headers to be considered a hosted environment, while a freestanding environment must include only float.h, limits.h, stdarg.h, and stddef.h.

The committee deliberated about putting the things in stddef into the C language itself, but decided the need to extend C is not quite there.

Stddef provides ptrdiff_t, size_t, wchar_t, NULL, and offsetof(structname, attrib). Only ptrdiff_t and offsetof are unique to this header. Other headers usually contain duplicates of the other definitions.

I used to think that NULL was a reserved word in C, but have come to learn that the constant 0 is actually the crux. The compiler treats 0 specially in a pointer context and transforms it to whatever value represents the NULL pointer on the given architecture (which needn’t be bitwise zero). Thus NULL is typically a macro for ((void *)0). Ordinarily function pointers and data pointers cannot be mixed, but the standard makes an exception for the null pointer constant, allowing it to be assigned to even a function pointer: int (*foo)(void) = NULL.

The typedef size_t is an unsigned integral type big enough to hold the size of the largest possible object in memory. In some systems that might not be very large, for instance only 64k in the segmented memory model on the Intel 80286. Related rule of thumb: if a variable is going to index an array, it should be type size_t.

The typedef ptrdiff_t is a signed integral type of the result of a pointer subtraction. It’s signed because if (p - q) is positive then (q - p) will be negative. Note that if size_t is already the largest integer type, then ptrdiff_t can be no larger, yet the latter loses one bit to hold the sign. So it’s possible to make an array with cells too far apart for ptrdiff_t to measure. (Assuming there is room in memory for such a large single array.)

C99 provides a macro SIZE_MAX with the maximum value possible in size_t. C89 doesn’t have it, although you can obtain the value by casting (size_t)-1. This works in any number representation (C89 section 6.2.1.2, C99 section 6.3.1.3 signed and unsigned integers), because the value (-1) is converted by “repeatedly adding or subtracting one more than the maximum value that can be represented in the new type until the value is in range of the new type.”

The offsetof() macro can determine the offset in bytes of a member within its structure. This cannot easily be determined otherwise due to structure padding. The C99 rationale talks about using it to provide “generic” routines that work from descriptions of the structures, rather than from the structure declarations themselves. On many platforms it is defined as:

#define offsetof(type, field) ((size_t)(&((type *)0)->field))

That’s undefined behavior, but that’s what the standard library is for: a portable way to do sometimes non-portable operations.

stdio.h

UNIX I/O was clean and simple compared with other systems of its day. It was 8-bit clean, and used a consistent line terminator. At the edges ioctl() would translate the simple I/O streams for the idiosyncrasies of attached devices. The kernel would hold file control state internally and give programs a simple integer descriptor for to use when reading and writing.

Leaving UNIX, C ran up against the complexity of I/O on other systems. The X3J11 committee talked with vendors and came away with a sharper understanding of the I/O model they wanted to support. They had to distinguish text and binary modes. While UNIX used ‘’ for line endings, DOS used \r\n and Mac classic used ‘. These endings have to be normalized to’’ in text mode but not binary mode. UNIX ignores binary mode, but you better enable it for portability when necessary.

UNIX also had an unusually faithful representation of files. You could put bytes in and expect to read them out unchanged. When doing fully portable I/O keep these caveats in mind:

  • A final line without a terminating newline (in text mode) can confuse some systems, and they may drop the line or append a newline.
  • Don’t count on the system preserving trailing space in a line. Some systems strip it out. Conversely, some systems add a space to a blank line so the line “has something in it.”
  • The maximum fully portable line length is 254 characters.
  • Implementations are free to pad the end of binary files with a block of NUL characters to make the files match certain disk block sizes.

Another difference between the C standard I/O and UNIX is buffering. In UNIX, people often wrote their own buffering code to reduce the number of relatively costly I/O system calls. The X3J11 committee decided to include this buffering functionality in stdio.

Buffering is an optimization that can be tailored to expected patterns of I/O. The standard library provides the setvbuf() function to change the size and location of a stream’s buffer, as well as choosing between line or block buffering. By default, stdin and stdout are line and block buffered respectively, and stderr is unbuffered. Setvbuf() must be called immediately after a stream is opened, before I/O happens, to have any chance of working.

(stdio.h) opening and closing

Perhaps surprisingly, there is a lot to learn about just opening files. First there may be a limit on how long a filename can be on a system. Stdio provides the FILENAME_MAX macro with this limit. If the system imposes no practical limit then the macro is just a suggested size. This value could be both too short, or paradoxically too long. If it is set very large then you might end up wasting memory or causing problems if allocating on the stack.

Similarly L_tmpnam is the size needed for an array large enough to hold a temporary filename generated by tmpnam(). This function is a security hazard (though it can be useful for generating entropy). It introduces a Time of Check, Time of Use (TOCTOU) race condition because another program or thread could obtain the same temporary file name and create the file first. Use the tmpfile() function instead which actually creates the file, and registers it for removal on normal program exit().

Another common TOCTOU happens with fopen() when trying to create but not replace a file. Programs first check existence, right after which an evildoer can create a symlink of the same name in time for the fopen with “w” mode to overwrite another file with possibly elevated permissions.

/* dangerous */

FILE *fp = fopen("foo.txt","r");
/* <-- attacker gets busy here */
if (!fp)
{
  fp = fopen("foo.txt","w");
  ...
  fclose(fp);
}
else
  fclose(fp);

C11 fixes this with an “x” (exclusive) mode modifier. If exclusive mode is specified (“wx”), the fopen fails if the file already exists or cannot be created. In C89 you can either go beyond the standard library, using the POSIX open() function with the O_CREAT | O_EXCL flags, or just try to keep the time between check and write as small as possible.

Once you have opened a file to your liking, or have been given a FILE pointer, treat the pointer as totally opaque. Don’t even try to make a copy of the FILE structure because some implementations rely on magic memory addresses for some of them. The CERT standard (FIO38-C) says that the following can cause a crash on some platforms if you try to use my_stdout:

/* don't do this */
FILE my_stdout = *stdout;

There’s a related function called freopen(), but it’s not used very often. The main use is converting a big program from reading stdin to reading a named file. It’s the simplest way to do that, whereas a new program should just directly fopen whatever file it wants.

During a normal program exit, all open files will be closed. Still, it’s useful to explicitly call fclose() on file handles. It helps avoid exceeding the FOPEN_MAX limit of files that can be open at once. Also, failing to properly close files may allow an attacker to exhaust system resources and can increase the risk that file buffers will not be flushed in the event of abnormal program termination.

Speaking of flushing buffers, fflush() might force items in the buffer to be processed, but there is no guarantee. For convenience, fflush(NULL) flushes all streams, which is useful in preparation for possible loss of program control, like going into a dangerous section, or telling the user to turn off the computer.

Two other quirks. Some operating systems will not actually create a file that you fopen() and fclose() unless you write something. Also you can close stdout or stderr, and there are sometimes reasons to do so!

(stdio.h) file navigation

Stdio.h has two similar pairs of functions to move around in a file: fgetpos/fsetpos and ftell/fseek. Why the duplication? The second pair represents position as a long integer. When the file is opened in binary mode, this long is the number of bytes from the start of file. This is useful because you can do arithmetic on the integer to jump to particular places. The drawback is that on some systems a long is only 32 bits, so cannot support large files.

The fgetpos/fsetpos pair works using a special structure that can represent positions in huge files. You must treat treat this structure as a magic bookmark. It’s only obtainable from a call to fgetpos, you cannot construct your own to point to a position you haven’t already been.

Stdio also includes a rewind() function, but don’t use it. It actively clears the error indicator for a stream. Instead do a fseek(stream, 0L, SEEK_SET).

These navigation commands were interesting to me, so I created a project called randln to experiment with different ways of picking a random line from a text file. You might find it interesting to look at the pros and cons of each method as explained in the readme.

You can actually inspect I/O as it happens. The trick is walk the program through a debugger while tracing its system calls in another terminal. To use that randln program as an example, First start it in the debugger. Enable tracing for the debugger’s I/O and that of its children.

ktrace -i -ti gdb randln

Put a breakpoint in the line-finding function and start the program. In another terminal find the PID of randln that was launched by gdb, and start tailing the ktrace dump. This command will show the first 20 bytes of data in each I/O request:

 kdump -m 20 -l -p <pid>

Now step through the program in the debugger and watch the consequences of each statement.

A final note about stream navigation and the shell. I noticed when doing foo <bar the program can perform fseeks on bar, but cat bar | foo cannot. It’s another reason not to abuse cat.

(stdio.h) reading and writing

The foundation of all stream input is fgetc(). The other standard library input functions must be implemented as if they call it repeatedly, even if they don’t. It pulls a character out of a stream (or a character that was pushed back by ungetc() if such exists), and refills the stream buffer if needed. Some platforms allow ungetc() to push back a whole stack of characters, but the portable assumption is that it can store only one.

While fgetc is a function, getc() is a macro that avoids incurring a function call just to get a character. The downside is that getc() is allowed to evaluate its argument more than once, so don’t do anything with a side effect there. Now, fgetc() is allowed to be a macro too, but may not be as efficient because it has less freedom in its implementation. It is not permitted to evaluate its argument twice.

Moving up the food chain we come to gets(), which reads characters into a string until it encounters the NUL character – and it’s a buffer overflow waiting to happen. It was removed in C11. The fgets() function is better because you can specify a max length. However if it fails, the contents of the array being written is indeterminate. It is necessary to reset the string to a known value to avoid errors on subsequent string manipulation functions.

The fread/fwrite functions work in chunks. My only notes about them are

/* prefer this */
fread(buf, 1,    size*n, stream)
/* over this   */
fread(buf, size, n,      stream)

The second form is worse because you can’t detect whether it read an extra size-1 characters past what it reports. Also some implementations of fread (fwrite) simply call fgetc (fputc) in a loop, whereas others are more optimized. Doing a straight UNIX read (write) can be faster.

The standard library doesn’t allow alternating reads and writes without intervening operations to flush or explicitly position the stream:

  • after writing, call fseek(), fsetpos(), rewind(), or fflush() before reading
  • after reading, call fseek(), fsetpos(), or rewind() before writing unless the file is at EOF: a read that hits EOF can be followed immediately by a write

The last thing I wanted to mention about reading and writing is a freak consequence of weird machines. On some digital signal processors (or more generally on the DeathStation 9000), both char and int are 32 bits. This causes a vulnerability in the common pattern:

int c;
while ((c = getchar()) != EOF)
  ...;

There is no extra room in the int type to hold EOF as distinct from a valid character code, so a valid character can make the loop stop early. It’s a high severity bug, a variation of which caused a nasty vulnerability in the bash shell, CA-1996-22.

Fine, you say, you don’t plan to target such machines!

#include <limits.h>

#if UCHAR_MAX == UINT_MAX
#error "Your machine is weird."
#endif

Also you’ll be careful not to indicate a parse failure with an unsigned code that casts to a signed value of EOF. Well the same logic can still trick you in C99 with wide characters if you’re not careful.

It works like this: the fgetwc(), getwc(), and getwchar() functions return a value of type wint_t. This value can represent the next wide character read, or it can represent WEOF, which indicates end-of-file for wide character streams. On most implementations, the wchar_t type has the same width as wint_t, and these functions can return a character indistinguishable from WEOF. For this situation be sure to check after the loop for feof() and ferror(). If neither happened then you’ve received a wide character that resembles WFEOF.

It’s yet another place where wide character implementations are half baked.

(stdio.h) formatting

Scanf and printf have formatting options I hadn’t seen before. First of all they can limit the length of strings they read or write.

/* limit to 10 characters */
printf("%.10s", big_string);

/* or limit to n characters */
printf("%.*s", n, big_string);

/* limit to 10 characters */
scanf("%10s", input);

I used to think scanf was very unsafe, but this limiting helps. Scanf also supports “scansets” to match strings containing specific characters. Here is how to match up to ten vowels: %10[aeiou]. Scanf also allows you to match but not capture, using *. E.g. %*d. The %n option saves the number of characters read so far in the scan.

Finally, in C99 printf has modifiers for size_t (%zu) and ptrdiff_t (%td). Because they are typedefs which change by architecture there is otherwise no way to specify them portably.

stdlib.h

Stdlib is a hodgepodge. It has six categories of functions inside:

  • algorithms (search, sort, rand)
  • integer functions
  • number parsing
  • multibyte conversions
  • storage allocation
  • environmental interactions

(stdlib.h) random numbers

Let’s start with an interesting topic: random numbers. The standard library provides a rand() function to generate a pseudorandom sequence starting from a seed specified by srand(). The numbers range from 0 to RAND_MAX. The first problem is that RAND_MAX can be very small (~ 65535) on some platforms. Second problem is that the quality of rand() is not generally very good on most platforms. On Mac OS it is horrible. Changing the random seed by only a small amount (such as seeding by the epoch), leads to similar initial random values.

seed 1st rand
1500000000 1189467867
1500000001 1189484674
1500000002 1189501481
1500000003 1189518288
1500000004 1189535095
1500000005 1189551902
1500000006 1189568709
1500000007 1189585516
1500000008 1189602323
1500000009 1189619130

Rather than rely on whatever implementation of rand you get on a given architecture, it’s just as easy to define your own xorshift rand function.

static unsigned long g_rand_state = 0;

unsigned long defensive_rand()
{
  uint32_t x = g_rand_state;
  x ^= x << 13;
  x ^= x >> 17;
  x ^= x << 5;
  /* restrict to 32-bits */
  x &= 0xFFFFFFFF;
  return (g_rand_state = x);
}

The constants 13, 17, 5 are specific to 32-bit numbers, so we mask the result to keep it within that range. There are also good 64-bit versions you can use with the “long long” type in C99. See also Chris Wellons’ discussion of creating good rand() functions through an exhaustive search.

Now we have a good rand() function, but how should we seed g_rand_state with an initial value? The typical way is to gather the epoch from time(NULL), but it will mean the application will use the same seed if run more than once per second.

There are two other sources of entropy available from the standard library. One is hashing the path generated by tmpnam(), which in many implementations consults the process ID or a higher precision clock. The second is hashing the address of the main function, which is often fairly unpredictable due to address space layout randomization (ASLR). Do not rely on ASLR alone, however, as not all platforms use it. The address of main() should always be mixed with other sources of entropy.

Using the address of main numerically is a little tricky. C99 has a type called intptr_t which can hold pointer values as an integer, but this type is for data pointers only, not function pointers which on some architectures have a different size. We might consider casting a pointer to main as (char *) and reading the bytes, but for the same size reason this isn’t feasible.

What we can do is create a function pointer to main, point another pointer at it and read the value out byte by byte. A pointer-to-function-pointer is just a data pointer and can be cast to (void *).

int (*p)(int, char**) = &main;
unsigned char bytes[sizeof(p) + 1] = { 0 };

memcpy(bytes, (void*)&p, sizeof(p));

The bytes array is a NUL terminated string and can be hashed. To see these techniques in action and how to combine the entropy, check out rand.c.

If you’re willing to use functions beyond the C standard library, POSIX provides a random() function that is higher quality, and the BSDs/macOS/illumos (or Linux with libbsd) provide arc4random() which returns crypto-grade randomness.

(stdlib.h) integer functions

In C89 the rounding of integer division isn’t fully specified. When one of the numerator or denominator are negative it may either round toward zero or downward. It matches whatever the underlying hardware does. The committee didn’t want to introduce overhead on any system going against the hardware convention. In C99 they changed their mind, reasoning that Fortran (known for numerical programming) defines the rounding.

C99 rounds toward zero. To match this behavior in C89, use the div and ldiv functions. They return structures div_t and ldiv_t that contain both the quotient and remainder of the division. The only reason to use these functions in C99 is for efficiency, because the functions may be implemented with a compiler builtin that can compute the quotient and remainder together in a single assembly instruction.

A note about the ato{i,l,f} functions – their behavior is undefined if the input cannot be parsed correctly. These functions need not set errno even. Except for behavior on error, atoi is equivalent to (int)strtol(nptr, (char **)NULL, 10). The strto{d,l,ul} functions should always be preferred, because they provide proper error reporting and choice of base.

The implementation of strtod in Plauger’s book was very careful about floating point overflow. It processed digits in groups of eight and then combined them later into a final result. It also consulted the decimal_point attribute set by the LC_MONETARY part of the current locale.

(stdlib.h.) memory

An interesting bit of history: malloc used to be considered a low-level UNIX function, while calloc (“C alloc”) was conceived as the portable C version. However the committee standardized malloc too because it has less overhead (doesn’t zero the memory). Nowadays it seems malloc is more popular. There also used to be a cfree, but it was identical to free and didn’t make the cut for ANSI C.

The most flexible allocation function is realloc because it can simulate both malloc and free. When passed NULL for the original pointer it behaves like malloc, and when passed 0 for requested size it acts like free.

(stdlib.h) termination and execution

Stdlib contains the EXIT_FAILURE and EXIT_SUCCESS macros with implementation defined integer values to indicate success or failure on exit. These values would be returned by main or passed to exit(). The C standard actually treats return 0 in main or exit(0) specially, and maps it to whatever the system specific success code is. (Similar to how 0 in a pointer context is treated specially as the NULL value the way we talked about earlier.) Thus you don’t need to include stdlib.h just to return successfully.

EXIT_FAILURE is necessary for portably indicating failure, though. The raw value 1 is considered successful on some platforms.

Speaking of exit(), it causes “normal” termination. It closes all open file handles, deletes files created by tmpfile(), and calls any handlers registered by atexit() in reverse order of their registration. The abort() function, on the other hand, causes immediate and “abnormal” termination. It needn’t flush buffers, remove temp files, or close open streams. It can be canceled by catching SIGABRT. Aborting can be useful because it will cause a core dump if the OS is configured to save one. Thus the assert() function calls abort() to produce a core file to debug assertion failure.

Stdlib.h provides the system() function to run a command in the shell. If the command is a NULL pointer, system() will return non-zero if the command interpreter is available, and zero if it is not. The CERT standard says system() is a security violation and flat out says not to call it. It’s easy to make a mistake with system(). Commands that are not fully pathed can be overridden, and even relative paths can be tricky if the attacker can control the current directory. And of course unsanitized input can attack through the shell. CERT suggests using execve() in POSIX to call fully pathed executables.

(stdlib.h) wchar_t

OK, this is going to get complicated. The takeaway is that the C standard library cannot portably handle Unicode. You’ll need a good third-party library. Let’s see why this is.

Before locales and all that, C was designed for 7-bit ASCII text. The committee endorsed the use of locales to specify a codepage that reassigned the meaning of extra characters using all eight bits (the minimum allowed size of char). However some languages use more than 255 symbols (+ NUL). There were two ways to handle bigger alphabets: use bigger chunks of memory to hold each character code, or define special sequences of single byte characters to stand for extended characters.

These approaches are called “wide characters” and “multibyte characters.” Generally because networking equipment and disk storage is byte oriented, programs use multi-byte character encoding for communication with the outside world and storage. However for in-memory use people felt that using wide characters would be cleaner, because each cell in an array would map to exactly one character, like in the old ASCII string days.

The C89 standard is not very helpful with multibyte or wide character encoding. It merely set the stage with a wchar_t type for wide characters, and functions in stdlib.h to convert multibyte to wchar_t (mbstowcs) and the reverse (wcstombs) based on the locale. The committee was waiting to see how people wanted to work with international characters before standardizing it. K&R 2nd edition does not list these functions but Plauger and the C89 spec have them.

The whole idea was that locale-specific multibyte to wide character conversion was supposed to be more general than any particular encoding system. However, nowadays Unicode has proven to be more popular than other encodings. The multibyte UTF-8 encoding is the standard interchange on the web, and the UCS-4 character set is up to the task of handling all the world’s languages and then some. Should be no problem, right?

Well, at the time C99 was being standardized, the Unicode consortium (actually the contemporaneous European ISO committee) was endorsing the more limited Universal Coded Character Set UCS-2. The codepoints for UCS-2 are 16 bits, so that’s the minimum width that C99 requires for wchar_t. Sadly the committee made their decision shortly before four byte UCS-4 was proposed (ISO 10646).

Vendors like Microsoft jumped into Unicode right away and implemented wchar_t as 16 bits. It’s deep in their APIs and they’re now stuck with that size for backward compatibility. Even Mac OS and iOS use 16 bit wchar_t for whatever reason.

UTF-16 combines the worst of multibyte characters and wide characters. a) Characters outside the “base multilingual plane” require two UTF-16 codepoints (called a surrogate pair) to represent them. This breaks the one-character-one-codepoint assumption. b) It’s wasteful of memory because even ASCII characters take two bytes.

C99 provides wide character versions of the ctype functions, in <wctype.h>, but they simply cannot work properly with surrogate pairs. For example (as pointed out here):

  • 0xD800 0xDF1E = U+1031E is a letter (iswalpha should be true)
  • 0xD800 0xDF20 = U+10320 is not a letter (iswalpha should be false)
  • 0xD834 0xDF1E = U+1D31E is not a letter (iswalpha should be false)
  • 0xD834 0xDF20 = U+1D320 is not a letter (iswalpha should be false)
  • 0xD835 0xDF1E = U+1D71E is a letter (iswalpha should be true)
  • 0xD835 0xDF20 = U+1D720 is a letter (iswalpha should be true)

Neither the first nor the second element of the pair alone can predict whether the resulting Unicode character is alphabetic. There is no way that a system can provide this information through a function ‘iswalpha’ that takes a single wchar_t argument.

C99 does guarantee a macro will be present when the current environment is ISO 10646 compliant, meaning wchar_t can hold every UCS-4 codepoint. We can blow up for the other platforms:

#ifndef __STDC_ISO_10646__
#error "Your wide characters suck."
#endif

Even assuming we restrict the code to ISO 10646 systems only, the wctype functions are too crude to deal with the subtleties of international languages. Because of how Unicode characters can join together, it’s infeasible to use pointer arithmetic to calculate “string length” or parse “words” with iswspace() robustly.

Some parts of programs can continue to operate on text as an opaque series of bytes. However for other parts that must inspect the characters themselves, you should use a sophisticated Unicode library like ICU or utf8proc. This has the advantage of working with C89, so you won’t be forced to upgrade to C99 just because of text processing.

With a good Unicode library we don’t need wide characters. We can use UTF-8 everywhere, even in program memory. That’s the school of thought behind utf8everywhere.org.

(stdlib.h) lessons from the code

While reading Plauger’s implementation for stdlib, I noticed some tricks worth sharing.

The comma operator can be used to group multiple small assignments together under an if statement without needing to add curly braces.

if (condition)
  foo = 1, bar = 2;

Also an if statement with no statements can be used to shrink a big expression in another if statement:

if (cond)
  ;
else if (big cond)
  foo;

C89 allows bare blocks inside a function to segregate variable declarations near the code that uses them. The variables are not accessible outside the block. It can help readability to know when variables are no longer needed, although you might argue that it suggests the function as a whole is too big.

#include <stdio.h>

int main(void)
{
  puts("Hello.");

  {
    int i = 0;
    printf("%d\n", i);
  }
  /* error: use of undeclared identifier 'i' */
  printf("%d\n", i);
}

The trick is even useful in C99 to reuse and reinitialize variables across repetitive test cases in a single function.

Speaking of brackets, Plauger uses Whitesmiths style indentation, with brackets indented to the level of their code:

if (cond)
  {
  foo();
  while (cond)
    {
    bar();
    baz();
    }
  }

I find it difficult to read (probably just unfamiliar), but it does have an internal consistency. The brackets are wrapping up multiple statements into a single unit, and this unit is indented the same way that a single statement would be. Still not going to indent this way, but just saying.

Another interesting trick is negating an unsigned number. Plauger does that to consolidate code for signed and unsigned numbers in the same function.

unsigned long _Stoul(const char *, char **, int);

#define atoi(s)                  (int)_Stoul(s, NULL, 10);
#define atol(s)                  (long)_Stoul(s, NULL, 10);
#define strtoul(s, endptr, base) _Stoul(s, endptr, base);

This _Stoul() function negates its unsigned long value if the string it is parsing has a negative sign. This operates bitwise on an unsigned value the same as it would on signed, and after casting for atoi and atol it will be negative as expected. I didn’t know it was “allowed,” but C doesn’t care.

string.h

This header is divided between functions starting with str- and those starting with mem-. The former work with NUL terminated strings, and the latter operate with explicit lengths. Some of the str- functions have a modifier to limit length (strncat, strncmp, strncpy) or a modifier to work backward (strrchr). However the header doesn’t have all permutations. For instance no strnlen or memrchr.

(string.h) widened types

Why does strchr take an int rather than a char for its search character? Same with memset, it takes an int value, but converts it to unsigned char internally. The spec dictates this in section 7.21.6.1. That misleading int signature is for backward compatibility with pre-ANSI code which lacked function prototypes and promoted char arguments to int. Under default argument promotions any integral type smaller than int (or unsigned int) always converts to int.

All standard library functions are specified in terms of the “widened” types. It’s just that string.h contains many functions where it is especially apparent. The widened types ensure that most library functions can be called with or without a prototype in scope. Legacy code doesn’t use prototypes, and ANSI C did not want to break backward compatibility. Even if people were willing to update all the legacy code, any legacy modules distributed as compiled object files rather than source would not link properly against functions with changed argument types.

Similar rationale lies behind the standard’s guarantee that char pointers and void pointers share the same representation and alignment requirements. Relying on this guarantee allows old code that to work with the void* malloc in place of the original char* malloc. (See the C99 Rationale section 7.1.4, Use of Library Functions.)

(string.h) signed vs unsigned

We’re not done with the type mysteries in string.h. Why is it that strchr casts its int argument to char internally and memchr casts to unsigned int? Well given that strchr is searching through char*, it makes sense to match the type. But memchr is searching through void*. Why not compare each memory location with char rather than unsigned char?

It turns out that the C standard makes special guarantees about unsigned char that make it an ideal type to represent arbitrary binary information.

  1. Unsigned char is guaranteed to have no padding bits. All bits contribute to the value of the data. Other types on some architectures include things like parity bits which don’t affect the value itself but do use space.
  2. No bitwise operation starting from an unsigned char value, when converted back into that type, can produce overflow, trap representations or undefined behavior. It can be freely manipulated. Trap representations are certain bit patterns that are reserved for exceptional circumstances, like the NaN value in floating point numbers.
  3. Accessing parts of a larger data object ith an unsigned char pointer will not violate any “aliasing rules.” The unsigned char pointer will be guaranteed to see all modifications of the data object.

In “Portable C,” Rabinowitz says the char type has a few distinct uses: codeset characters, units of storage, small numbers, and small bit patterns. He recommends regular char for codepoints and small numbers, and unsigned for the others. (In C99 wchar_t might be considered the proper way to represent codeset characters, but we’ve already seen the difficulty there.)

Note that Rabinowitz doesn’t specify signed char and unsigned char, but rather plain char and unsigned char. That’s because C does not specify whether plain char is signed or unsigned. We would get a warning trying to pass a definitively signed char* to a function in string.h if the default was unsigned char for that platform.

(string.h) mem- implementations

The next functions that taught me something are memcpy and memmove. The first one blasts bytes from one part of memory to another, possibly taking advantage of machine instructions to do large block copies. It doesn’t check for an overlap between the source and destination, in which situation the results are undefined. C99 marks the source and destination pointers with the restrict qualifier to allow the compiler to optimize under that assumption.

Memmove is the slower more careful brother. It works correctly even if the source and destination areas overlap. It is specified to act as if the source memory is first copied to a separate buffer, then copied into the destination. When I tried writing it that is exactly what I did, but Plauger has a faster way:

void *(memmove)(void *d, const void *s, size_t n)
{
  unsigned char *ds = d;
  const unsigned char *ss = s;

  if (s > d)
    while (n-- > 0)
      *ds++ = *ss++;
  else
    for (ss += n, ds += n; 0 < n; --n)
      *ds-- = *ss--;

  return d;
}

This compares the pointer positions to see which occurs before the other in memory, then does the copy from left to right or right to left depending on which comes before the other. This is very fast and uses no extra space.

However, isn’t comparing random pointers undefined behavior? C allows you to compare pointers within the same primary data object (like the addresses of different cells in the same array), but not any random pointers. Objects could be in different segments of a segmented memory architecture, or even in totally different memory banks such as in a Harvard architecture.

But with memmove why would you call it when copying one data object into a totally different one? There would be no danger that they overlap, unless you are copying too many bytes, which is already its own big problem. Thus we can assume that the pointers to memmove are in the same data object and thus comparable. What this also tells me is not to indiscriminately use memmove all the time in order to be “safe” or something, because some implementations like Plauger’s would then cause undefined behavior.

One other thing to note about the memmove implementation is that it uses unsigned char pointers to do the work rather than void pointers. Doing pointer arithmetic on void* is a GNU-ism not permitted in portable C.

(string.h) strncpy

No discussion of string.h is complete without talking about the dangers of that “hybrid” function strncpy(). I call it a hybrid because in some ways it acts like a mem- function operating on blocks of memory. Let’s look at the history.

The function was originally used to copy filenames into fixed width fields. Two fields were considered equal when each and every character of one matched the other. The comparison didn’t stop at \0 but blindly compared the full length of both memory regions. To represent a short file name, the remainder of the memory was padded with \0 just to normalize it. A long filename that used every available character wouldn’t have a \0 at all because the length was already known.

So while strncpy might look like a “safe” strcpy that is guaranteed not to overflow a destination buffer, it’s actually pretty weird and not safe. Consider this snippet:

char dst[BUFSZ], *src = /* ... */;

strncpy(dst, src, BUFSZ);

It copies the first BUFSZ characters of src into dst. If BUFSZ is shorter than src (meaning \0 is not one of the first BUFSZ characters) then \0 doesn’t get written to dst! The result is not NUL terminated. Also, strncpy does a bunch of potentially useless work adding extra NUL padding if BUFSZ is larger than strlen(src).

The proper way to handle strncpy depends on how you want to treat string truncation. In some cases it would be an error to truncate the source string, such as if it holds a URL or filename. In this case the code should check for truncation.

if (strlen(src) < BUFSZ)
  strcpy(dst, src);  /* a regular strcpy is safe */
else
  /* handle truncation error */

If truncation is permissible, then be sure to include a NUL character in the destination.

/* either of these techniques work in C89: */

/* adding the NUL character manually */
strncpy(dst, src, BUFSZ);
dst[BUFSZ-1] = '\0';

/* or truncating dst and using (safe) strncat */
dst[0] = '\0';
strncat(dst, src, BUFSZ);

/* C99 provides snprintf that can do this in one line */

snprintf(dst, BUFSZ, "%s", src);

The C11 standard in Annex K contains functions with “bounds-checking” interfaces. This includes strncpy_s which acts more safely. The functions in Annex K are safe for other reasons as well, but covering this goes too far afield from the C89 focus of this article. One thing to note is that it provides a memset_s which is guaranteed not to be removed by an optimizing compiler. Sometimes memset is optimized out if it operates on memory that the compiler can tell is not read afterwards.

Those are the notes I wanted to share for this header. Also C99 introduces wide character versions of these functions in wchar.h.

time.h

First, the C standard is very lenient about this header. It has functions to do all kinds of conversions, but the bigger picture is that implementations are allowed to make their “best approximation” to the date and time. Some of them might do a bad job and yet still conform to the standard. Many C environments do not support the concepts of daylight savings or time zones. Both notions are defined geographically and politically, and thus may require more knowledge about the real world than an implementation can support.

There are a lot of details in this header that I don’t want to regurgitate, but it is useful to see how the functions convert between the data types. I made a graph to make it clearer.

functions in time.h

One thing I didn’t know was that the header provides a clock() function to measure the CPU time elapsed since program start. Also that it provides a resolution of CLOCKS_PER_SEC which is often higher than one. The rest of the library is limited to nothing smaller than seconds of precision.