Category Archives: Technology

Docs as Code: Mermaid inline diagrams

Less is more!

Docs as Code is an important aspect of Rust (doc-book), but integrating graphics/images as SVG or PNG into source files or markdown files has been a pain so far, 1) requiring external tooling to create/update these graphics/images, 2) requiring additional manual steps to link these images from the rust-docs rs-files or markdown files (README.md); AFAICS Rust-docs lacked support for easy to use inline graphics and diagrams.


Therefor the support for the Mermaid diagram tool (Rust module aquamarine) is a great enhancement regarding expressiveness of rust-docs and the project documentation overall! Mermaid is a tool that allows us to create flowcharts, graphs, diagrams, sequence diagrams, state machines, Gantt charts and class diagrams, and many more.

For example diagrams can be added to the documentation of functions’ rust-doc, being displayed in HTML output of cargo doc 

#[cfg_attr(doc, aquamarine::aquamarine)]
/// A function showcasing aquamarine defaults
///
/// With aquamarine it's possible to embed Mermaid diagrams into your Rust documentation using the code snippets
/// 
/// ```mermaid
/// graph LR
///     s([Source]) --> a[[aquamarine]]
///     r[[rustdoc]] --> f([Docs w/ Mermaid!])
///     subgraph rustc[Rust Compiler]
///     a -. inject mermaid.js .-> r
///     end
/// ```
///
/// The diagram is going to be located in place of the code snippet
///
/// Dark mode is automatically enabled if `dark` or `ayu` rustdoc theme is selected.
///
/// You might need to reload the page to redraw the diagrams after changing the theme.
pub fn example() {}

Probably you noticed, Mermaid is similar to PlantUML/Graphviz but the Mermaid rendering engine is JavaScript based, and if used in web-browser context it doesn’t need any additional interpreters or remote rendering services. Mermaid can be integrated straightly into various frameworks such as Rust-Docs, github, gitlab, Reveal.JS etc. Usually the mermaid diagram is delivered to the browser and rendering is performed by JavaScript in the Web-Browser, off-loading the web-server.

So far project teams are using a zoo of graphic tools, for example developers tent to use tools such as graphviz (or similar) but management is preferring PowerPoint, LibreDraw, Visio, or Enterprise Architect. The charts/diagrams have not been shared between groups in the project team. Graphs and diagrams have been re-created again and again with huge effort, but prettified versions never made it back from management layer back to the docs of the source code.

Well, using mermaid, now charts and diagrams can be shared between team members, no matter which position or role in the project, being improved time over time.

For example, using the crate aquamarine inline diagrams can be added to Rust source code docs directly or adding diagrams to README.md being rendered by github (github adding the mermaid JS rendering engine to the README.md HTML page and the final graph rendering happens in the Web-browser)

The following frameworks are supporting mermaid (incomplete list):

Rust-Docs: Aquamarine plugin embedding the mermaid rendering functionality into the generated HTML pages. Off-loading the rust-doc-server, rendering the graphs in user’s web-browser!

See the corresponding Rust doc

Reveal.JS: Reveal.JS is a HTML Presentation Framework. The plugin reveal.js-mermaid-plugin may be used to embed nice graphs into presentations easily, and sharing these graphs with rust-docs is possible!

markdown: github and gitlab are providing support for inline mermaid diagrams, for example the project’s README.md or the issue tickets. Sharing these mermaid graphs with rust-docs (rs-files) is possible!

For example the project’s README.md file 

## Markdown Mermaid Inline 
Browsing github/gitlab the following diagram in README.md would be rendered in your browser.
```mermaid
 graph LR
     s([Source]) --> a[[aquamarine]]
     r[[rustdoc]] --> f([Docs w/ Mermaid!])
     subgraph rustc[Rust Compiler]
     a -. inject mermaid.js .-> r
     end
```

command line: mermaid-cli command line tool mmdc rendering mermaid mmd-files to SVG, PNG, etc. The command line tool can be used to convert the mermaid text format into SVG/PNG to be integrated into PowerPoint or Keynote presentations. 😉

VueJS UI: vue-mermaid

Sphinx: The Sphinx Document Generator provides mermaid support

AsciiDoc: AsciiDoc is a plain text markup language for writing technical content, supporting mermaid format as well. AsciiDoc may be used for larger documents being divided into multiple files. The mermaid charts/diagrams may be shared with markdown documents. AsciiDoc is “PDF First”! So, in constrast to markdown, the asciidoc compiler will render mermaid diagrams immediately and embed them as SVGs into the PDF or HTML.

For example user-guide.adoc

[mermaid]
....
 graph LR
     s([Source]) --> a[[aquamarine]]
     r[[rustdoc]] --> f([Docs w/ Mermaid!])
     subgraph rustc[Rust Compiler]
     a -. inject mermaid.js .-> r
     end
....

WordPress: a plugin providing mermaid support in this blog as well

flowchart LR

A[This] -->|Blog| B(is)
B --> C{using}
C -->|mermaid| D[as]
C -->|well| E[happily]

Atlassian/Jira+Confluence: mermaid-charts-diagrams-for-jira

crates.io: ?? Sadly, so far not supporting mermaid (21.12.2022) 🙁 Support for mermaid would be really nice! Please support my request for this feature!

Conclusion

The documentation of a Rust project is done at various locations, 1) via rust docs in rs-files (for example main.rs), 2) but also via md-files (such as README.md), not to mention confluence etc. The former being processed during the build-process invoking cargo doc, whereas md-files being processed by the markdown rendering pipeline of github or gitlab.

Now, as Mermaid graphs are supported in many use cases now, it is possible to share/re-use graphs in rs-files, md-files, wiki and slides (reveal.js). Using mermaid, the source code documentation in rs-files, the project documentations in markdown files, wiki, and the slides of management meetings (using reveal.js) may benefit from each other’s graphs in a cyclic process (give and take).

Note! When refactoring code or implementing new features, one should grant the required time to update the documentation as part of the task/story, including graphs and class diagrams.

Sadly crates.io (the Rust package registry) does not support mermaid graphs yet (21.12.2022). Therefor, information of embedded diagrams may get lost when crates.io is extracting the crate-description from corresponding markdown file README.md; the description of a crate may become irritating when referring to the lost diagrams in the text. Supporting mermaid graphs would be a nice feature of crates.io! Please support the request for this feature! It would not add stress/load to the server infrastructure as the rendering would happen in web-browser of the user!

Using a combination of markdown and mermaid, all groups of the project team – requirements engineers, software architects, software developers, technical writers and managers – may have common ground now to share architecture/API documentation and graphs (watch), and keeping them in sync with each other using version control (git). Technical graphs/diagrams may be shared between each other and edited, without the trouble about licenses for editors, such as Enterprise Architect or Rational Rose. So far these WYSIWYG tools have failed! Either too complex or the license model too restrictive!

Last but not least, one step further! If combining markdown/mermaid documentation now with Swagger/OpenAPI as the REST-API documentation (what is swagger UI?) this sounds like a good starting point for a Rust project going for docs as code.

PS(protocol): mermaid cannot illustrate the packet structure of a protocol. In this case use https://github.com/luismartingarcia/protocol and embed its plain text output into your md-files and rs-files, for example the following rust-doc in main.rs would show up in the generated api-docs as follows. Don’t forget to add the command parameter also, if someone wants to modify it later.

/// Example
///
/// $ protocol "Source:16,TTL:8,Reserved:40"
/// ```text
///      0                   1                   2                   3
///      0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
///     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
///     |             Source            |      TTL      |               |
///     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+               +
///     |                            Reserved                           |
///     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
/// ```
fn example() {}

PS(TeX): For the illustration of a protocol header the TeX bytefield package may create more sophisticated SVG/PNG output, but must be linked into rs-files and md-files as image. Pretty, but avoid the hassle! The compilation of this TeX document will produce the following astonishing diagram:

\documentclass[border=10pt,png]{standalone}
\usepackage{bytefield}


\begin{document}

    \begin{bytefield}[bitwidth=1.1em]{32}
        \bitheader{0-31} \\
        \begin{rightwordgroup}{RTP \\  Header}
            \bitbox{2}{V=2} & \bitbox{1}{P} & \bitbox{1}{X}
            & \bitbox{4}{CC} & \bitbox{1}{M} & \bitbox{7}{PT}
            & \bitbox{16}{sequence number} \\
            \bitbox{32}{timestamp}
        \end{rightwordgroup} \\
        \bitbox{32}{synchronization source (SSRC) identifier} \\
        \wordbox[tlr]{1}{contributing source (CSRC) identifiers} \\
        \wordbox[blr]{1}{$\cdots$} \\
        \begin{rightwordgroup}{RTP \\ Payload}
            \wordbox[tlr]{3}{MPEG-4 Visual stream (byte aligned)} \\
            \bitbox[blr]{16}{}
            & \bitbox{16}{\dots\emph{optional} RTP padding}
        \end{rightwordgroup}
    \end{bytefield}

\end{document}

PPS: a good overview of tools https://kroki.io/examples.html

Java – Wait for Multiple Processes

In Java, the function Process.onExit() creates a CompletableFuture as exit handler and this is the enabling feature in Java to wait for any termination of a set of processes. 

public Process anyProcTermination(List<Process> procs)  {
    CompletableFuture<Process>[] exitHandlers =
            procs.stream()
                    .map(p -> p.onExit())
                    .toArray(CompletableFuture[]::new);
    CompletableFuture<Object> exitHandle = CompletableFuture.anyOf(exitHandlers);
    return  (Process) exitHandle.join();
}

IPv6 resolving hostnames faster

The title of this post might be misleading. IPv6 is not faster than IPv4, but nowadays applications assume IPv6 being the default, having an implication onto the time to establish a connection to a remote host.

Before an application is able to connect to a specific host, first its hostname has got to be resolved via DNS to get to know the corresponding remote IP address.

By default the resolver tries to resolve the corresponding IPv6 address first. If this does not succeed within specific timeout, the resolver is falling back to IPv4.

Thus, if you want to speed up the connection times of your applications, make sure both, your local network and also your internet provider, are supporting IPv6.

If your internet provider does not support native IPv6, it might be better to disable IPv6 in your local network as well, to prevent your local applications from using IPv6 at all.

Do not contract an internet provider for your home area network without IPv6 support!

From the other side of the table, in case you are providing an internet service, your customers might try to connect to your IPv6 service endpoint first, just falling back to IPv4 later. So, your IPv6 endpoint will provide a better usability for your customers.

Go To Statement Considered Harmful (discussed)

Talking with other developers about control-flow patterns, after a while someone will cite Edgar Dijkstra’s paper

“Go To Statement Considered Harmful” Communications of the ACM 11, 3 (March 1968), pages 147-148

Usually those developers argue, that go to statements should be abolished completely.

Just, it is important to note, that this paper is about loop-control-structures, comparing the expressiveness of recursive function calls, while/repeat-statements and goto-statements with each other (preferred pattern mentioned first).

And Dijkstra does not oppose the usage of go to statements in general, but mentioning a use case for go to statements:

“The remark about the undesirability of the go to statement is far from new. I remember having read the explicit recommendation to restrict the use of the go to statement to alarm exits, but I have not been able to trace it; presumably, it has been made by C.A.R. Hoare.”

My interpretation of this statement is, that go to statements can be tolerated for so called “alarm exits” as being used widely in C-code of OS-Kernels and embedded systems.

An “alarm exit” may look like this, realizing a single exit point for the success case and a single one for the error case, instead of spreading multiple exit-points all over the function.

/**  foo reading a file and returns with SUCCESS, otherwise with ERROR
  */
int foo(const char* filename) {
    int fd = INVALID_FD;
    if (filename==INVALID_STRING)  { 
         goto alarm_exit;   
    }
    fd = open_file(filename);
    if (fd == INVALID_FD)    { 
         goto alarm_exit;   
    } 
    /* do something, release resource and exit with success */
    close_file(fd);
    return SUCCESS_CASE;

alarm_exit:
    /* release resources if allocated */
    if (fd!=INVALID_FD) close_file(fd);
    return ERROR_CASE;
}

So, next time having a discussion with fundamental developers, abolishing all occurances of go to statements (even for the “alarm exits” use case), I will be happy to cite Dijkstra and Hoare 😉

Rust – Arrays? Make chains, not concat!

If your application needs to iterate over a bunch of items from different sources or arrays, someone with C/C++ background might copy all items into a single vector and iterate this vector.

This strategy will cause high costs in terms of allocating heap memory for the consecutive vector buffer.

Instead, keep the data where it is, and chain it together to form an iterator over a virtual array.

The following Rust code demonstrates the chaining of multiple arrays, forming a single iterator, without any additional allocation of vector buffer in heap.

Note, the zip operation in the following code snippet pairs each item with a slot in the buffer. The zip will stop if either end is reached, the end of chained_byte_sequence or the end of therequest buffer, whichever comes first. Just, the length of chained_byte_sequence might be unknown and only a single pass shall be performed. So, how do we know all items have been written, and the amount? The solution is to borrow the iterator via chained_byte_sequence.by_ref(), iterating the chain and finally verifying if any item remained in the sequence chained_byte_sequence.next()!=None. 

  let req_method_get = b"GET ";
    let req_method_head = b"HEAD ";
    let uri = b"/infotext.html HTTP/1.1";

    let mut request = [0 as u8; 128];

    // chaining various byte-arrays together, for example two here:
    let mut chained_byte_sequence =  req_method_get.iter()
        .chain(uri.iter());

    // take a ref and zip against the destination buffer and 
    // while iterating via fold, each element is counted.
    let  nwritten =
        chained_byte_sequence
        .by_ref()
        .zip(request.as_mut())
            .fold(0, |cnt, (item, slot) | {
                *slot = item.clone(); cnt+1
            });

    // finally, verify the iterator of the chained_byte_sequence is empty
    if chained_byte_sequence.next()!=None {
        /* slice too short */ panic!();
    } else {
        println!("{:?}", &request[0..nwritten].as_ref());
    };