Before and after: docstrings¶

Everything on this page is executed when the documentation is built. The "before" output is what pydoc shows you today, produced by inspect.getdoc; the "after" is the same docstring through simplemark. Nothing is retyped by hand.

The width problem¶

Start with a docstring that has no markup in it at all: textwrap.wrap from the standard library. Its author wrapped it at about 68 columns, which was a reasonable choice for the source file.

Here is what pydoc prints, on a 56-column terminal:

python -c "
import inspect, textwrap
print(inspect.getdoc(textwrap.wrap))
"

Wrap a single paragraph of text, returning a list of wrapped lines.

Reformat the single paragraph in 'text' so it fits in lines of no
more than 'width' columns, and return a list of wrapped lines.  By
default, tabs in 'text' are expanded with string.expandtabs(), and
all other whitespace characters (including newline) are converted to
space.  See TextWrapper class for available keyword args to customize
wrapping behaviour.

Every line overflows. Your terminal will wrap them at arbitrary points, or you will scroll.

Now the same docstring, reflowed:

python -c "
import textwrap, simplemark
print(simplemark.render_text(simplemark.parse(textwrap.wrap.__doc__), width=56, style='none'))
"

Wrap a single paragraph of text, returning a list of
wrapped lines.

Reformat the single paragraph in 'text' so it fits in
lines of no more than 'width' columns, and return a list
of wrapped lines.  By default, tabs in 'text' are
expanded with string.expandtabs(), and all other
whitespace characters (including newline) are converted
to space.  See TextWrapper class for available keyword
args to customize wrapping behaviour.

And on a wide terminal, where the "before" would have wasted a third of the screen:

python -c "
import textwrap, simplemark
print(simplemark.render_text(simplemark.parse(textwrap.wrap.__doc__), width=100, style='none'))
"

Wrap a single paragraph of text, returning a list of wrapped lines.

Reformat the single paragraph in 'text' so it fits in lines of no more than 'width' columns, and
return a list of wrapped lines.  By default, tabs in 'text' are expanded with string.expandtabs(),
and all other whitespace characters (including newline) are converted to space.  See TextWrapper
class for available keyword args to customize wrapping behaviour.

Note that this docstring was not modified. It contains no simplemark markup whatsoever. It is one of the 93.0% of standard library docstrings that already parse clean, and it gets reflow for free.

Note also the two spaces after each full stop. Reflow preserves the author's spacing rather than normalising it.

Reflow with exemptions¶

Prose that flows is only half of it. The other half is the content that must not flow: examples, doctests, and hand-aligned tables. Here is a docstring with all three.

exemptions.py:

def convert(source, strict=False):
    """Convert `source` between formats.

    The recognised escapes are:

        "\\n"     newline
        "\\t"     tab, expanded to the next multiple of 8
        "\\\\"     a single backslash

    Set *strict* to raise on an unknown escape instead of copying it
    through unchanged, which is what the default does.

    Example:

        >>> convert("a\\\\tb")
        'a<TAB>b'
    """

At 48 columns:

python -c "
import exemptions, simplemark
print(simplemark.render_text(simplemark.parse(exemptions.convert.__doc__), width=48, style='none'))
"

Convert 'source' between formats.

The recognised escapes are:

    "\n"     newline
    "\t"     tab, expanded to the next multiple of 8
    "\\"     a single backslash

Set strict to raise on an unknown escape instead
of copying it through unchanged, which is what
the default does.

Example:

    >>> convert("a\\tb")
    'a<TAB>b'

The prose rewrapped to 48. The escape table did not; it is still column-aligned, still wider than 48, and its backslashes are intact. The doctest is untouched and doctest would still run it.

That is the "free flow with exemptions" the whole format exists to provide.

What markup adds¶

The two examples above had no markup, or almost none. Here is what you get when you use it: the same source rendered for a terminal that supports styling, and for one that does not.

marked.py:

def connect(host, port=443, timeout=None):
    """Open a connection to `host`.

    Waits at most *timeout* seconds.  Raises `TimeoutError` if the
    connection is not established in time, and `OSError` for every
    other failure.  See <https://example.com/protocol> for the wire
    format.
    """

With no styling available (a pipe, or TERM=dumb):

python -c "
import marked, simplemark
print(simplemark.render_text(simplemark.parse(marked.connect.__doc__), width=58, style='none'))
"

Open a connection to 'host'.

Waits at most timeout seconds.  Raises 'TimeoutError' if
the connection is not established in time, and 'OSError'
for every other failure.  See
<https://example.com/protocol> for the wire format.

Code spans became 'quoted'; emphasis lost its markers. The distinction that carries meaning, this word is an identifier, not prose, survives on the dumbest terminal there is. Decoration does not, because it was decoration.

With ANSI styling, the same spans are bold and the URL is underlined instead:

python -c "
import marked, simplemark
out = simplemark.render_text(simplemark.parse(marked.connect.__doc__), width=58, style='ansi')
print(out.replace('\033[1m', '<b>').replace('\033[4m', '<u>').replace('\033[0m', '</>'))
"

Open a connection to <b>host</>.

Waits at most <u>timeout</> seconds.  Raises <b>TimeoutError</> if the
connection is not established in time, and <b>OSError</> for
every other failure.  See <u><https://example.com/protocol></>
for the wire format.

The escape sequences are shown as tags here so they are visible on the page.

The lines break in different places from the unstyled version, and for a reason worth understanding: with no styling, a code span is rendered 'TimeoutError' with quotes, which is two visible characters wider than the bold TimeoutError. The text being laid out is genuinely different, so the wrapping is too.

The escape sequences themselves do not change the layout. Width arithmetic counts visible characters only, so a styled line is never pushed over the limit by its own decoration:

python -c "
import re, marked, simplemark
out = simplemark.render_text(simplemark.parse(marked.connect.__doc__), width=58, style='ansi')
visible = [len(re.sub(r'\033\[[0-9;]*m', '', line)) for line in out.split('\n')]
print('longest visible line:', max(visible), '<= 58:', max(visible) <= 58)
print('longest raw line:    ', max(len(line) for line in out.split('\n')))
"

longest visible line: 58 <= 58: True
longest raw line:     74

The raw lines are up to 74 characters. Every one of them fits in 58 columns on screen. That is the bug every ANSI-aware wrapper eventually ships, and it is why simplemark does not use textwrap.

The one that needed fixing¶

Not every docstring is clean, and this is the case to understand before migrating anything. The old GNU quoting style, `foo', is neither Markdown nor reST, and 264 standard library docstrings use single backticks in some form.

legacy.py:

def dispatch(line):
    """Interpret the argument as a command.

    A command `foo' is dispatched to the method `do_foo'.
    """

python - <<'EOF'
import legacy, simplemark
print("default:", simplemark.check(legacy.dispatch.__doc__))
for d in simplemark.check(legacy.dispatch.__doc__, compat=True):
    print("compat: ", d)
EOF

default: []
compat:  3:11: L005 code span looks like legacy `quoted' text

Nothing at all in default mode, because a single backtick is simply a code span. Only --compat spots it; the text actually renders as:

python -c "
import legacy, simplemark
print(simplemark.render_text(simplemark.parse(legacy.dispatch.__doc__), width=60, style='none'))
"

Interpret the argument as a command.

A command 'foo' is dispatched to the method'do_foo'.

The two opening backticks paired with each other. Everything between them became one code span, and the apostrophes ended up in the wrong places. Markdown and reST would both do the same thing; simplemark is not being unusual here, it is being faithful.

Run --compat before adopting simplemark on an old codebase

A single backtick means "code span" to simplemark, to CommonMark and to GitHub. None of them mean "opening quote". This is what compat mode is for: it finds the constructs that predate the format, so you fix them before opting in rather than after. L005 is the check that catches this one.

With an odd number of them you get a real error instead, because the last one never closes:

python -c "
import simplemark
for d in simplemark.check(\"A command \`foo' is dispatched.\"):
    print(d)
"

1:11: E001 unclosed code span

This is why compat mode exists, and why the corpus report is part of the design rather than an afterthought: the 3.6% of standard library docstrings containing an error are mostly this.

Next¶

argparse help: the same treatment for command-line tools
Diagnostics: every code and its fix