Concept

How can strtok() split a string into tokens?

ComputerScienceOne / Tokenizing

"Recall that tokenizing is the process of splitting up a string along some delimiter. For example, the comma delimited string, \"Smith,Joe,12345678,1985-09-08\" contains four pieces of data delimited by a comma. Our aim is to split this string up into four separate strings so that we can process each one. The C string library provides a tokenizing function:\n\nchar *strtok(char *str, const char *delim);\n\nThe first argument is the string that you want to tokenize and the second contains the delimiter that you want to split along. The second argument is actually a string and allows you to specify more than one delimiter, but we’ll restrict our attention to single character delimiters. The function returns a pointer to the first token in the string. To get the second and all subsequent tokens, we call strtok() again, but we pass it NULL as the first argument to continue parsing the same string. If we pass a new string as the first argument to strtok() the tokenization process will start over on the new string. Note that the first argument does not have the const keyword. This is because strtok() will make changes to the string during the tokenization process. If the string needs to be preserved, tokenization should be performed on a deep copy of the string. When there are no more tokens in the string, strtok() returns NULL to indicate no more tokens are in the string. This logic can be used to write a while loop to iterate over each token.\n\nConsider the following example.\n\nchar data[] = \"Smith,Joe,12345678,1985-09-08\";\n\nchar *token = NULL;\n\n// make the initial call to strtok:\n\ntoken = strtok(data, \",\");\n\nwhile(token != NULL) {\n\nprintf(\"token: %s\n\n\", token);\n\n// get the next token:\n\ntoken = strtok(NULL, \",\");\n\n}\n\nThis outputs the following:\n\ntoken: Smith\n\ntoken: Joe\n\ntoken: 12345678\n\ntoken: 1985-09-08"

Related Ideas

How can strtok() split a string into tokens? | ComputerScienceOne | Bifalgorithm | Bifalgorithm