Bifalgorithm post · Computer science one · Tokenizing

How can strtok() split a string into tokens?

How can strtok() split a string into tokens? (Computer science one · Tokenizing) — "Recall that tokenizing is the process of splitting up a string along some d…

Ideas in this flowchart

  • Tokenizing strings with strtok()
  • Tokenizing splits a string along a delimiter.
  • The comma-delimited string "Smith,Joe,12345678,1985-09-08" contains four data pieces.
  • The goal is to produce separate strings so each piece can be processed.
  • The C string library provides strtok(char *str, const char *delim).
  • The first argument is the string to tokenize, and the second specifies the delimiter.
  • The delimiter argument is a string, although this example uses single-character delimiters.
  • strtok() returns a pointer to the first token and returns NULL when no tokens remain.
  • Passing NULL continues parsing the same string; passing a new string starts tokenization over.
  • strtok() modifies the input string, so a deep copy is needed when the original must be preserved.
  • A while loop can repeatedly process tokens until strtok() returns NULL.
  • The example initializes strtok(data, ",") and then calls strtok(NULL, ",") for subsequent tokens.
  • The example outputs Smith, Joe, 12345678, and 1985-09-08 as separate tokens.
Feed/

How can strtok() split a string into tokens?

No cover

Channel

Computer science one

1 page

"Recall that tokenizing is the process of splitting up a string along some delimiter. For example, the comma delimited string, \"Smith,Joe,12345678,1985-09-08\" contains four pieces of data delimited by a comma. Our aim is to split this string up into four separate strings so that we can process each one. The C string library provides a tokenizing function:\n\nchar *strtok(char *str, const char *delim);\n\nThe first argument is the string that you want to tokenize and the second contains the delimiter that you want to split along. The second argument is actually a string and allows you to specify more than one delimiter, but we’ll restrict our attention to single character delimiters. The function returns a pointer to the first token in the string. To get the second and all subsequent tokens, we call strtok() again, but we pass it NULL as the first argument to continue parsing the same string. If we pass a new string as the first argument to strtok() the tokenization process will start over on the new string. Note that the first argument does not have the const keyword. This is because strtok() will make changes to the string during the tokenization process. If the string needs to be preserved, tokenization should be performed on a deep copy of the string. When there are no more tokens in the string, strtok() returns NULL to indicate no more tokens are in the string. This logic can be used to write a while loop to iterate over each token.\n\nConsider the following example.\n\nchar data[] = \"Smith,Joe,12345678,1985-09-08\";\n\nchar *token = NULL;\n\n// make the initial call to strtok:\n\ntoken = strtok(data, \",\");\n\nwhile(token != NULL) {\n\nprintf(\"token: %s\n\n\", token);\n\n// get the next token:\n\ntoken = strtok(NULL, \",\");\n\n}\n\nThis outputs the following:\n\ntoken: Smith\n\ntoken: Joe\n\ntoken: 12345678\n\ntoken: 1985-09-08"

Flowchart preview
👍
0
0 comments0 shares

Sign in to view and post comments.