Merge remote-tracking branch 'Screwtapello/rewrite-expansions-doc'

5 files changed, 264 insertions, 142 deletions

diff --git a/doc/interfacing.asciidoc b/doc/interfacing.asciidoc
index 58766add..5ab5f878 100644
--- a/doc/interfacing.asciidoc
+++ b/doc/interfacing.asciidoc
@@ -91,7 +91,8 @@ line.column[+len]@timestamp candidate1|desc1|menu1 candidate2|desc2|menu2 ...
 the first element of this string list specify where and when this completion
 applies, the others are a triplet `<completion text>|<docstring>|<menu text>`
 
-The menu text is a markup string, so it can contain `{face}` directives.
+The menu text is a markup string (see <<faces#markup-strings,`:doc faces
+markup-strings`>>), so it can contain `{face}` directives.
 
 To effectively use that completion option, it should get added to the completers
 option.
diff --git a/doc/pages/commands.asciidoc b/doc/pages/commands.asciidoc
index 17a8c5d6..a373ef72 100644
--- a/doc/pages/commands.asciidoc
+++ b/doc/pages/commands.asciidoc
@@ -166,7 +166,7 @@ of the file onto the filesystem
 
     *-markup*:::
         expand the markup strings in *text* (See
-        <<expansions#markup-strings,`:doc expansions markup-strings`>>)
+        <<faces#markup-strings,`:doc faces markup-strings`>>)
 
     *-debug*:::
         print the given text to the *\*debug** buffer
diff --git a/doc/pages/expansions.asciidoc b/doc/pages/expansions.asciidoc
index d1579b47..4a105a3d 100644
--- a/doc/pages/expansions.asciidoc
+++ b/doc/pages/expansions.asciidoc
@@ -1,199 +1,299 @@
 = Expansions
 
-== Strings
+While parsing a command (see <<command-parsing#,`:doc command-parsing`>>),
+Kakoune recognises certain patterns and will replace them with their
+associated value before executing the command. These patterns are called
+expansions.
 
-\'strings'::
-    uninterpreted strings, use a backslash (\') to escape the separator
+Every expansion consists of a `%`, followed by the expansion _type_ (one
+or more alphabetic characters), a nestable punctuation character (`(`, `[`,
+`{`, or `<`), and then all the text up to and including its matching opposite
+(`)`, `]`, `}`, or `>`). It doesn't matter which character is used, but
+`{}` are most common. For example, the command `echo %val{session}` will
+echo the ID of the current Kakoune session.
 
-"strings"::
-    expanded strings, % strings (c.f. next section) contained are expended,
-    use a backslash (\\%) to escape the separator
+Expansions are processed when unquoted and anywhere inside double-quoted
+strings, but not inside unquoted words, inside single-quoted strings, or
+inside %-strings or other expansions. So:
 
-%\{strings\}::
-    these strings are very useful when entering commands
+* `echo %val{session}` echoes the current session ID
 
-    * the '{' and '}' delimiters are configurable, any non alphanumeric
-      character can be used
+* `echo x%val{session}x` echoes the literal text `x%val{session}x`
 
-----------------------------------------------------------
-e.g. %[string], %<string>, %(string), %~string~, %!string!
-----------------------------------------------------------
+* `echo '%val{session}'` echoes the literal text `%val{session}`
 
-    * if the character following '%' is one of '{[(<', then the closing
-      one is the matching '}])>' and the delimiters are not escapable but
-      are nestable
+* `echo "x%val{session}x"` echoes the current session ID, surrounded by `x`
 
------------------------------------------------------------
-e.g. %{ roger {}; } is a valid string, %{ marcel \} as well
------------------------------------------------------------
+* `echo %{%val{session}}"` echoes the the literal text `%val{session}`
 
-== Typed expansions
+* `echo %sh{ echo %val{session} }"` echoes the literal text `%val{session}`
 
-%\{strings\} can have an expansion type between the *%* and the opening
-character. They will be written *%<type>\{<content>\}*. They will be
-expanded according to the given *<type>* using *<content>* as its
-parameter:
+Like "variable expansion" and "command substitution" in shell programming,
+Kakoune expansions can expand to multiple "words" - that is, separate
+arguments on the resulting command-line. However, unlike shell programming,
+Kakoune expansions cannot _accidentally_ expand to multiple words, even if
+the expansion contains whitespace or other special characters. While in
+shell-programming it's good practice to always wrap expansions in
+double-quotes, in Kakoune it's perfectly safe to leave expansions unquoted.
 
-*sh*::
-    shell expansion, similar to posix shell '$(...)' construct (c.f. next
-    section)
+== Argument expansions
 
-*reg*::
-    register expansion, will expand to the strings stored in the register
-    named by *<content>*. See <<registers#,`:doc registers`>>
+Expansions with the type `arg` can only be used inside the "commands" parameter
+of the `define-command` command (See <<commands#declaring-new-commands,`:doc
+commands declaring-new-commands`>>).
 
-*opt*::
-    option expansion, will expand to the value of the option named by
-    *<content>*. See <<options#,`:doc options`>>
+The following expansions are available:
 
-*val*::
-    value expansion, will expand to the value of the environment variables
-    available to shell expansion. *<content>* shall be the name of that
-    variable without the *kak_* prefix.
+*%arg{n}*::
+     (where _n_ is a decimal number) +
+     expands to argument number _n_ of the current command
 
-*arg*::
-    argument expansion, expand to the arguments of the current
-    command, *<content>* can be a number, or @ for all arguments
+*%arg{@}*::
+    expands to all the arguments of the current command, as individual words
+
+== Option expansions
+
+Expansions with the type `opt` expand to the value associated with the named
+option in the current scope (See <<options#,`:doc options`>>).
+
+For example, `%opt{BOM}` expands to `utf8` or to `none`, according to the
+current state of the `BOM` option.
+
+== Register expansions
+
+Expansions with the type `reg` expand to the contents of the named
+register. For registers named after symbols (like the search register
+`/`), the expansion can use either the symbol or the alphabetic name (See
+<<registers#,`:doc registers`>>).
+
+For example, `%reg{/}` expands to the content of the `/` register, and so does
+`%reg{slash}`.
 
 == Shell expansions
 
-The '%sh{...}' expansion replaces its content with the output of the
-shell commands in it. The following environment variables are used to pass
-informations about Kakoune's state:
+Expansions with the type `sh` are executed as shell-scripts, and whatever
+the script prints to standard output replaces the expansion. For example,
+the command `echo %sh{date}` will echo the output of the `date` command.
 
-*kak_selection*::
-    content of the main selection
+TIP: If a shell expansion writes to standard error, that output is appended to
+Kakoune's `\*debug*` buffer. If you're trying to debug a shell expansion,
+check the debug buffer with `:buffer \*debug*` to see if anything shows up.
 
-*kak_selections*::
-    quoted list of the contents of the selections.
+Because Kakoune does not expand expansions inside the text of an expansion,
+you can't use normal expansions inside `%sh{}`. Instead, Kakoune can export
+expansions as environment variables to make them available to the shell.
+Here's how expansion patterns map to variable names:
 
-*kak_selection_desc*::
-    range of the main selection, represented as anchor,cursor; anchor
-    and cursor are in this format: line.column
+*%arg{n}*::
+    (where _n_ is a decimal number) +
+    becomes `$_n_`. For example, `%arg{3}` becomes `$3`.
 
-*kak_selections_desc*::
-    unquoted list range of the selections.
+*%arg{@}*::
+    becomes `$@`
 
-*kak_bufname*::
-    name of the current buffer
+*%opt{x}*::
+    becomes `$kak_opt_x`
 
-*kak_buffile*::
-    full path of the file or same as kak_bufname when there’s no
-    associated file
+*%reg{x}*::
+    (where _x_ is the alphabetic name of a register) +
+    `$kak_reg_x` contains all the selections in register _x_ +
+    `$kak_main_reg_x` contains only the main selection
 
-*kak_buflist*::
-    quoted list of the currently opened buffer names
+*%val{x}*::
+    becomes `$kak_x`
 
-*kak_buf_line_count*::
-    the current buffer line count
+When turned into environment variables, list-type options, `$kak_reg_x`, and
+"quoted list" values will be shell-quoted so the shell doesn't get confused
+about how many items the list contains. You will need to apply `eval` to get
+back the original values. For example, if you want to process the contents
+of each selection, you can do something like:
 
-*kak_timestamp*::
-    timestamp of the current buffer, the timestamp is an integer value
-    which is incremented each time the buffer is modified
+----
+eval set -- $kak_selections
+while [ $# -gt 0 ]; do
+    # ... do a thing with $1 ...
+    shift
+done
+----
 
-*kak_history_id*::
-    history id of the current buffer, the history id is an integer value
-    which is used to reference a specific buffer version in the undo tree
+The `eval` command will take the expanded `$kak_selections` and unquote them,
+then execute the resulting `set` command, which sets the shell's argument
+variables to the items from `$kak_selections`. The `while` loop with `shift`
+iterates through the arguments one by one.
 
-*kak_runtime*::
-    directory containing the kak support files, determined from kakoune's
-    binary location.
+Only variables actually mentioned in the body of the shell expansion will
+be exported into the shell's environment. For example:
 
-*kak_config*::
-    directory containing the user configuration
+----
+echo %sh{ env | grep ^kak_ }
+----
 
-*kak_version*::
-    version of the current Kakoune server (git hash or release name)
+... will find none of Kakoune's special environment variables, but:
 
-*kak_count*::
-    count parameter passed to the command, defaults to 0 if no count given
+----
+echo %sh{ env | grep ^kak_ # kak_session }
+----
 
-*kak_register*::
-    register parameter passed to the command
+... will find the `$kak_session` variable because it was mentioned by name
+in a comment, even though it wasn't directly used.
 
-*kak_opt_<name>*::
-    value of option *name*
+TIP: These environment variables are also available in other contexts where
+Kakoune uses a shell command, such as the `|`, `!` or `$` normal mode commands
+(See <<keys#,`:doc keys`>>).
 
-*kak_reg_<r>*::
-    quoted list of value of register *r*
+== Value expansions
 
-*kak_main_reg_<r>*::
-    content of register *r* associated with the main selection.
+Expansions with the type `val` give access to Kakoune internal data that is
+not stored in an option or a register. Some value expansions can only be used
+in certain contexts, like `%val{hook_param}` that expands to the parameter
+string of the currently-executing hook, and is not available outside a hook.
 
-*kak_session*::
-    name of the current session
+The following expansions are supported (with required context _in italics_):
 
-*kak_client*::
-    name of the current client
+*%val{buffile}*::
+    _in buffer, window scope_ +
+    full path of the file or same as `%val{bufname}` when there’s no
+    associated file
 
-*kak_client_pid*::
-    process id of the current client
+*%val{buf_line_count}*::
+    _in buffer, window scope_ +
+    number of lines in the current buffer
 
-*kak_client_list*::
-    unquoted list of the names of clients connected to the current session
+*%val{buflist}*::
+    quoted list of the names of currently-open buffers (as seen in
+    `%val{bufname}`)
 
-*kak_source*::
-    path of the file currently getting executed (through the source
-    command)
+*%val{bufname}*::
+    _in buffer, window scope_ +
+    name of the current buffer
 
-*kak_modified*::
-    buffer has modifications not saved
+*%val{client_env_X}*::
+    _in window scope_ +
+    value of the `$X` environment variable in the client displaying the current
+    window (e.g. `%val{client_env_SHELL}` is `$SHELL` in the client's
+    environment)
 
-*kak_cursor_line*::
-    line of the end of the main selection
+*%val{client_list}*::
+    unquoted list of the names of clients (as seen in `%val{client}`)
+    connected to the current session
 
-*kak_cursor_column*::
-    column of the end of the main selection (in byte)
+*%val{client}*::
+    _in window scope_ +
+    name of the client displaying the current window
 
-*kak_cursor_char_value*::
-    unicode value of the codepoint under the cursor
+*%val{client_pid}*::
+    _in window scope_ +
+    process id of the client displaying the current window
 
-*kak_cursor_char_column*::
-    column of the end of the main selection (in character)
+*%val{config}*::
+    directory containing the user configuration
 
-*kak_cursor_byte_offset*::
-    Offset of the main selection from the beginning of the buffer (in bytes).
+*%val{count}*::
+    _in `map` command <keys> parameter_ +
+    current count when the mapping was triggered, defaults to 0 if no
+    count given
 
-*kak_window_width*::
-    width of the current kakoune window
+*%val{cursor_byte_offset}*::
+    _in window scope_ +
+    offset of the main cursor from the beginning of the buffer (in bytes)
 
-*kak_window_height*::
-    height of the current kakoune window
+*%val{cursor_char_column}*::
+    _in window scope_ +
+    column of the main cursor (in characters), the fourth component of
+    `%val{selection_desc}`
 
-*kak_hook_param*::
-    filtering text passed to the currently executing hook
+*%val{cursor_char_value}*::
+    _in window scope_ +
+    unicode value of the codepoint under the main cursor
 
-*kak_hook_param_capture_N*::
-    text captured by the hook filter regex capture N
+*%val{cursor_column}*::
+    _in window scope_ +
+    column of the main cursor (in bytes)
 
-*kak_text*::
-    the text entered by the user at a `prompt` command, not available in other
-    contexts
+*%val{cursor_line}*::
+    _in window scope_ +
+    line of the main cursor, the third component of `%val{selection_desc}`
 
-*kak_client_env_<name>*::
-    value of the *name* variable in the client environment
-    (e.g. *$kak_client_env_SHELL* is the SHELL variable)
+*%val{history_id}*::
+    _in buffer, window scope_ +
+    history id of the current buffer, an integer value which refers to a
+    specific buffer version in the undo tree (see also `%val{timestamp}`)
 
-*kak_user_modes*::
-    unquoted list of user modes.
+*%val{hook_param_capture_n}*::
+    _in `hook` command <command> parameter_ +
+    text captured by capture group _n_, if the executing hook's filter regex
+    used capture groups
 
-Quoted lists are separated by spaces, and each element is surrounded by
-`'` with contained `'` doubled. Unquoted lists are simply separated by
-spaces and is used for values that will not contain whitespaces.
+*%val{hook_param}*::
+    _in `hook` command <command> parameter_ +
+    the complete parameter string of the executing hook
 
-Note that in order for Kakoune to pass a value in the environment, the
-variable has to be spelled out within the body of the expansion.
+*%val{modified}*::
+    _in buffer, window scope_ +
+    `true` if the buffer has modifications not saved, otherwise `false`
+
+*%val{register}*::
+    _in `map` command <keys> parameter_ +
+    current register when the mapping was triggered
+
+*%val{runtime}*::
+    directory containing the kak support files, determined from Kakoune's
+    binary location
+
+*%val{selection}*::
+    _in window scope_ +
+    content of the main selection
+
+*%val{selections}*::
+    _in window scope_ +
+    quoted list of the contents of all selections
+
+*%val{selection_desc}*::
+    _in window scope_ +
+    range of the main selection, represented as `a.b,c.d` where _a_
+    is the anchor line, _b_ is the anchor column, _c_ is the cursor
+    line (like `%val{cursor_line}`), _d_ is the cursor column (like
+    `%val{cursor_char_column}`), and all are 1-based decimal integers
+
+*%val{selections_desc}*::
+    _in window scope_ +
+    unquoted list of the ranges of all selections, in the same format as
+    `%val{selection_desc}`
+
+*%val{session}*::
+    name of the current session
+
+*%val{source}*::
+    _in `.kak` file_ +
+    path of the file currently getting executed (through the source command)
+
+*%val{text}*::
+    _in `prompt` command <command> parameter_ +
+    the text entered by the user in response to the `prompt` command
+
+*%val{timestamp}*::
+    _in buffer, window scope_ +
+    timestamp of the current buffer, an integer that increments each time the
+    buffer is modified, including undoing and redoing previous modifications
+    (see also `%val{history_id}`)
+
+*%val{user_modes}*::
+    unquoted list of user modes.
+
+*%val{version}*::
+    version of the current Kakoune server (git hash or release name)
 
-Those environment variables are available in every context where
-Kakoune use a shell command, such as the `|`, `!` or `$` normal
-mode commands (See <<keys#,`:doc keys`>>).
+*%val{window_height}*::
+    _in window scope_ +
+    height of the current Kakoune window
 
-== Markup strings
+*%val{window_width}*::
+    _in window scope_ +
+    width of the current Kakoune window
 
-In certain contexts, Kakoune can take a markup string, which is a string
-containing formatting informations. In these strings, the {facename}
-syntax will enable the face facename until another face gets activated,
-or the end of the string is reached.
+Values in the above list that do not mention a context are available
+everywhere.
 
-Literal '{' characters shall be written '\{', and a literal backslash ('\')
-that precedes a '{' character shall be escaped as well ('\\').
+A value described as a "quoted list" will follow the rules of Kakoune string
+quoting (See <<command-parsing#,`:doc command-parsing`>>). An "unquoted list"
+cannot contain any special characters that would require quoting.
diff --git a/doc/pages/faces.asciidoc b/doc/pages/faces.asciidoc
index 68f8e19c..45f4f7a6 100644
--- a/doc/pages/faces.asciidoc
+++ b/doc/pages/faces.asciidoc
@@ -128,3 +128,21 @@ The following faces are used by builtin highlighters if enabled.
 
 *Whitespace*::
     face used by the `show-whitespaces` highlighter
+
+== Markup strings
+
+In certain contexts, Kakoune can take a markup string, which is a string
+containing formatting informations. In these strings, the {facename}
+syntax will enable the face facename until another face gets activated,
+or the end of the string is reached.
+
+For example, the following command displays the text "default" in the
+Default face, and "error" in the Error face:
+
+----
+echo -markup 'default {Error}error{Default} default'
+----
+
+Inside a markup string, a literal `{` character is written `\{`, and a
+literal backslash (`\`) that precedes a '{' character is escaped as well
+(`\\`).
diff --git a/doc/pages/options.asciidoc b/doc/pages/options.asciidoc
index 0f924dc9..e76641f0 100644
--- a/doc/pages/options.asciidoc
+++ b/doc/pages/options.asciidoc
@@ -108,7 +108,9 @@ are exclusively available to built-in options.
     a list of `<text>|<docstring>|<menu text>` candidates,
     except for the first element which follows the
     `<line>.<column>[+<length>]@<timestamp>` format to define where the
-    completion apply in the buffer. Markup can be used in the menu text.
+    completion apply in the buffer. Markup (see
+    <<faces#markup-strings,`:doc faces markup-strings`>>) can be used in the
+    menu text.
     `set -add` adds a new completion to the list
 
 *enum(value1|value2|...)*::
@@ -248,7 +250,8 @@ are exclusively available to built-in options.
 *modelinefmt* `string`::
     A format string used to generate the mode line, that string is
     first expanded as a command line would be (expanding '%...{...}'
-    strings), then markup tags are applied (See <<expansions#,`:doc expansions`>>)
+    strings), then markup tags are applied (see
+    <<faces#markup-strings,`:doc faces markup-strings`>>)
     Two special atoms are available as markup:
 
         *`{{mode_info}}`*:::