Alternative command implementation

[AljoschaMeyer]

These are just some thoughts about an alternative way to implement commands, no actual implementation done. It might be overkill, result in a weird API, or plain unnecessary, but I might as well post this here. Idea came after reading this introduction to frp. I hope this is not just incoherent mumbling, but actually understandable.

Basic concept: Commands communicate via frp streams. Each instance of a command has an input stream and an output stream. When a user types and activates a command, an instance of the command is created, based on the given options. The actual argument is not used in the creation of the command instance. Then, the argument(s) are passed to the command's input stream. The command does some stuff, then writes the output through it's output stream. The output stream is listened to by vorpal, and by default the output is printed into the console. When the command is done, it ends the stream, which vorpal interprets as if the command's callback in the current implementation was called, returning focus to the user.

Command piping could easily be implemented by piping the output stream of the first command into the input stream of the second one. Unlike the current implementation, commands would run asynchronously. This way, two commands separated by ; could be started simultaneously.

Commands could also emit errors. This would allow implementing && and || for command chaining. One could even go further and use 2>> to only pass on errors etc.

Parsing a command string from the user could be done in two stages: First, separate actual commands and chaining operators via regexes. Second, create command instances and correctly chain the streams together. By separating this, parsing the command options could be delegated to one of the existing node modules.

edit: Parsing using regexes is not a good idea...

Command definition would be done by giving methods handling arguments from the input stream, errors from the input stream, and end of the input stream (vorpal could provide a sensible default for errors and end of stream), instead of command.action().

Pros:

• asynchronous commands
• well structured command parsing
• clear implementation of various piping and chaining possibilities
• well testable

Cons:

• slightly more complex API
• API breakage? (The stream based implementation can simulate the current one, so maybe not)
• Vantage integration?
• Complete rewrite of a lot of code needed

I currently don't have time to spend on this, but I wanted to get the idea out there. And if this is not just garbage, I could start working on this later - no need to hurry...

[dthree]

Thanks and received! I'll have to spend some more time thinking this over, but this is possibly something we can implement in the future.

[AljoschaMeyer]

Further thoughts: Instead of using the streams of a FRP library, this should instead use simple node streams. The implementation could actually mirror that of UNIX pipes, with a stdout, a stderr and an exit code. Then, a bridge could be built to Node's Child Process, which could enable running normal commands from vorpal.

This implementation of commands with std-streams and exit code should probably go into its own module, which Vorpal then could use. The module would also do the command parsing, handling the piping logic itself, and delegating options to Commander/Minimist/whatever proven library exists.
Since this would be rather low-level, Vorpal would probably provide a more convenient API.

Still lacking the time to implement this (those pesky real life obligations...), but I'll definitely get around to doing this some time. Even if it might be overkill for Vorpal, I find this much too fascinating not to do it.

[dthree]

Yeah, those darn real-life grumble grumble grumble... :)

I'm totally down for streaming. It's kind of embarrassing that Vorpal piping doesn't use streams.

Whenever you get around to it, throw together some concepts and then we'll go over it.

[AljoschaMeyer]

Basic API sketch (without asynchronous support): The functions defined here would be passed to some constructor.

// A fizzbuzz command.
var init = function () {
  this.fizzCount = 0;
};

var onInput = function (input) {
  var number = parseInt(input);
  if (Number.isNaN(number)) {
    // method provided by library
    this.stdErr("invalid input, expected a number");
  } else {
    var fb = "";
    if (number % 3 === 0) {
      fb += "Fizz";
      this.fizzCount++;
    }
    if (number % 5 === 0) {
      fb += "Buzz";
    }
    // method provided by library
    this.stdOut(fb === "" ? number : fb);
  }
};

var onInputExit = function (code) {
  this.stdOut("fizzCount: " + this.fizzCount);
  // method provided by library
  this.exit(0);
};

Expected outputs:

$ fizzbuzz 2
=> 2
=> fizzCount: 0

$ fizzbuzz 15
=> FizzBuzz
=> fizzCount: 1

$

// A counter command.
var init = function () {
  var delta = 1;
  // options object provided by library
  if (this.options.from > this.options.to) {
    delta = -1;
  }

  for (var i = this.options.from; i === this.options.to; i += delta) {
    if (i !== 12) {
      this.stdOut(i);
    } else {
      this.stdErr("Nah, no 12.");
    }
  }

  this.exit(0);
};

Expected outputs:

$ counter --from 3 --to 6
=> 3
=> 4
=> 5
=> 6

$ counter --from 13 --to 11
=> 13
=> Nah, no 12. # evil red error message
=> 11

$

Combining them:

$ counter --from 3 --to 6 | fizzbuzz
=> Fizz
=> 4
=> Buzz
=> Fizz
=> fizzCount: 2

$ counter --from 13 --to 11 | fizzbuzz
=> 13
=> Nah, no 12. # evil red error message
=> 11
=> fizzCount: 0

$

Behind the scenes, each command has three streams, stdin, stdout, and stderr (stdin and stdout can optionally be object streams). stdout is piped to the next command's stdin (or to the higher-level stdout). stdin is listened on to call onInput. stderr goes directly to the higher-level management code.

Stuff to consider: How to handle asynchronous actions.

• init would be given a callback to signal that initialization is done and onInput may be called.
• onInput can be async as well. May keep track of a queue of onInput calls, each call signals via callback that it is done. Exiting the command can then wait for async input handling to terminate.

Substitute "promise" for "callback" as preferred.

The code managing the command lifecycles should be able to interrupt commands (analogy to SIGINT). Maybe an event system is needed to notify the commands.

Anyways, doing more pesky real life grumbling for now =)

[AljoschaMeyer]

Any ideas for the SIGINT-equivalent? The framework should be able to stop a running command, but forcing the interruption does not sound like a good approach. Maybe add a shouldTermintate method to the object bound to init, onInput and onInputExit, returning whether interruption was requested?
The average command is not complex enough and can safely ignore that.
Or do you think a more forceful interruption is needed?

Also I'd gladly take better naming suggestions for the functions (or the module itself for that matter) - you are far better with (English) words than I am...

[AljoschaMeyer]

First of all, I'm not expecting immediate feedback on this, I suppose cash keeps you pretty busy right now. I'm just sharing some of the involved decisions. Because why not?

Thoughts on objectMode for the streams:

The low-level implementation can do either via options, but perhaps Vorpal should enforce one or the other.

• Object streams allow convenient implementation of commands interacting with specific other commands.
• Strings/Buffer streams would allow all commands to interact uniformly. Especially important if plugins are added, but this already applies for extensions.

Alternatively, commands could define whether they expect objects or strings. Vorpal could pass the required format (e.g. by using JSON.stringify when appropriate).

In either case, the developer using vorpal should not have to worry about handling strings as well as objects in the same command.

Similiary, should the error stream use strings, Error objects, both? Again the low-level implementation should allow both, but Vorpal might want to restrict this for consistency and a more convenient API.

[AljoschaMeyer]

This turned out to be more work than I anticipated, but here is a first implementation. Working version of the above example is here. Most of the complexity will be abstracted away, the end user would only pass three lifecycle methods (init, onInput, cleanup) with access to some properties to a constructor.

As a next step I'll write something which parses strings like 'foo || bar', passes 'foo' and 'bar' to a factory method returning a CommandInstanceand properly connects them. Then I'll try do build an example factory which executes the string as a ChildProcess and wraps that childprocess into a CommandInstance. That way I can do some sweet tests where I simply compare the input of e.g. ls | grep foo in an actual shell to that of the javascript version. And that code would make a useful vorpal extension as well.

The parser will be independently extendable with redirection, command substitution, here documents etc. Also, tee could be done as a vorpal extension/plugin.

A realistic command factory would be given command definitions similiar to the ones currently used in vorpal, parsing the string for options and passing the options for the created instances. The factory would probably be implemented by vorpal (but it can utilize commander or yargs or someting similiar), but the parser and the CommandInstance class would be self-contained modules.

All in all still a lot of work to do, but the result should be worth it.

[dthree]

You're overwhelming me with good ideas! Great work, will try to take a thorough look at this as soon as I can.

First priority currently is closing existing issues, so we have a clean base for some redesigns.

[dthree]

Okay, I looked at this a bit, it actually looks really good.

The most important thing though, as you said, very simply abstracting it for the user, and making it behave similarly to existing Vorpal commands.

For example,

• this.stdin could simply be this.log and this.stderr could be this.error, which would match console.log and console.error.
• As cleanup would be optional for a user, it should be an optional method on command:

vorpal.command('foo')
  .action(action)
  .cleanup(cleanup);

• Piping of commands should just work like streams people are used to. Perhaps:

vorpal.command('foo')
  .action(function () {
    this.on('data', function() {
      // piped data.
    });
   });

---

Can we discuss the end-user API on this?

[AljoschaMeyer]

Abstraction Hierarchy

I'll outline a full abstraction hierarchy for implementing all this stuff in this comment. See comments below for API discussion.

Command Instance

Not going to summarize this again, just check the readme.

Command Chain

The CommandChain takes CommandInstances, chains them together and initializes them. It has a stream.Readable input stream, a stream.Writable output stream and a stream.Writable error stream. If no stream redirection is requested, the input of the chain is piped into the stdin of the first CommandInstance, the output is piped to by the last CommandInstance's stdout and all stderrstreams are piped into the error stream of the CommandChain. This is basically the top right diagram here.

The CommandChain provides some methods which operate on command-instance level and implement different control structures. E.g. chain.pipe(commandInstanceA, commandInstanceB) (pipe stdout of A into stdin of B), or chain.conditionalAnd(commandInstanceA, commandInstanceB) (only run B if A exists with code 0) and so on.

Stream redirection belongs on this level as well.

The implementation might need to not only operate on CommandInstances, but on some sort of command blocks or command groups, but we can get to that later...

Parser

The Parser takes a string and contructs a corresponding CommandChain from it (or rejects it as syntactically invalid). It distinguishes between command strings and control structures. We can start by implementing only simple control structures (since these have to be implemented in CommandChain as well), but here is a more complex example:

ping -c1 8.8.8.8 || {echo 'no internet connection'; ls /etc/netctl | grep wlp2s0}

The command strings in here are "ping -c1 8.8.8.8", "echo 'no internet connection'", "ls /etc/netctl/profiles" and "grep wlp2s0". The control structures form a tree hierarchy with command strings as leaves.

Coding all this will take some work, but we can start by just implementing piping - that's all vorpal currently does as well. But by writing a parser which builds a syntax tree, all of this becomes doable. And the implementation is abstracted away in the CommandChain, which idealy has exactly one method corresponding to exactly one constrol structure node.

The parser needs to deal with command strings, because the CommandChain expects CommandInstances. For this, the parser delegates to a CommandInstanceFactory. Such a factory has a very simple interface: It takes a command string and returns a corresponding CommandInstance. The CommandInstanceFactory is supplied to the parser, so it does not care about the implementation itself.

Vorpal Implementation

Vorpal lets the parser build CommandChains from user input. Vorpal interacts with these chains by supplying input for interactive commands, and displaying (or at least handling) the output stream, the error stream and exit codes and messages. The parser is given a CommandInstanceFactory by vorpal, which is constructed at vorpal's initialization (but could be modified at runtimes if commands are added/removed/edited at runtime).

Vorpal's command function would essentially be a builder for a CommandInstanceFactory, which then constructs the correct CommandInstance. So the information it needs to provide are:

• How to parse a command string (anything between the control structures like |, ;, && etc):
  • determine which command to instantiate
  • set options
  • set arguments (operands)
• Implementation for the lifecycle functions of the command:
  • init
  • onInput
  • cleanup

The information from all commands is pooled to create the factory. The factory determines which command is going to be constructed and then dispatches to some objects/functions constructed via the command function.

This would create a bare-bones factory, but vorpal can enhance it with automatic help etc.

[AljoschaMeyer]

CommandInstance discussion

I'll need to adjust handling initialization and the 'ready' event a bit. Currently commands need to be started in the reverse order, which calls init in reverse order, which is pointless.

I could switch the implementation to use callbacks instead of promises. Promise resolution and rejection are handled the same anyways, they just indicate that the lifecycle method is done with its potentially async action. Errors need to be signaled with this.stderr and if unrecoverable exit(code, msg) anyways. Any preferences?

Handling Operands

Still unsure about how to deal with operands. The current implemention locks in the command options, and then expects all operands to be provided via stdin. So mkdir foo would be translated to:

• create a mkdir CommandInstance with no options
• initialize mkdir (no-op)
• pipe in 'foo' (triggering mkdir's onInput, which hopefully makes a directory)
• cleanup mkdir (no-op)

This is equivalent to echo foo | mkdir, which would not work with the unix mkdir command, but would work in vorpal.

The big advantage of this is that a CommandInstance can treat operands given via the command line and piped operands exactly the same. Or more accurately, it doesn't even know that an operand might have come from the command line instead of from another CommandInstance (or really any stream whatsoever). I like this behavior a lot.

However: What happens with a command like echo foo | grep bar? While 'bar' looks like an operand, it actually isn't, the real operand is 'foo'. 'bar' would need to be translated into an option. Which is completely doable, I'm just not sure how to expose this to command implementors.

I'd really appreciate thoughts on or suggestions for this.

[AljoschaMeyer]

Vorpal Command API Discussion

Everything below is of course just a suggestion. Point of view is that of the CommandInstanceFactory. The API methods modify the factory (or rather they modify a command object which is passed to the factory once it is done).

To recap what information we need:

• parsing a command string:
  • determine which command to instantiate
  • parse and set options
  • read in operand(s)
• lifecycle implementation
  • init
  • onInput
  • cleanup

Determining which command to instantiate:

Strategy:

Interpret the first part of the command string as the name. Dispatch which CommandInstance to create based on a map of names. Fail if no command of this name is found.

API:

• vorpal.command('name'): Vorpal's command creating method(s) should all take the command name.
• cmd.name('name'): Explicitely set the name for a command
• cmd.alias('name'): Set another key for the map which is used by the dispatcher of the CommandInstanceFactory.

Parse and set options

Strategy:

Define which options the command takes, and what kind of options they are. Pass the parsing of the command string and the actual creation of an options hash to library like commander or yargs.

API:

Can stay the same.

Read in operands

Strategy:

Pass the operands of a command to its stdin after its initialization

API:

None, this happens implicitely.

Lifecycle:

Strategy:

Allow the user to pass a function, or supply a default no-op one (e.g. function () {return Promise.resolve();}).

API:

• cmd.init(fn): Set the given function as the init lifecycle method.
• cmd.action(fn(input)): Set the given function as the onInput lifecycle method.
• cmd.cleanup(fn): Set the given function as the cleanup lifecycle method.

fn mus always return a promise (or this could be callback-based instead).

Example Usage

Trivial command:

vorpal.command('name');

A simple command:

vorpal.command('reverse').action(function (input) {
  // don't reverse strings like this at home!
  this.log(input.split('').reverse().join(''););
  return Promise.resolve();
});

The FizzBuzz command from the CommandInstance example:

vorpal.command('fizzbuzz')
  .init(function () {
    this.fizzCount = 0;
    return Promise.resolve();
  }).action(function (input) {
    // nobody wants to read another FizzBuzz implementation
    // returns a promise
    return doTheFizzBuzz(input);
  }).cleanup(function () {
    this.log('fizzCount: ' + this.fizzCount );
    return Promise.resolve();
  });

Example with options:

vorpal.command('repeat')
  .option('-a, --amount <int>', 'How often to repeat.')
  .action(function (input) {
    for (var i = 0; i < this.options.amount; i++) {
      this.log(input);
    }
    return Promise.resolve();
  });

Always returning Promise.resolve() for these synchronous commands is rather annoying though, could Vorpal detect and wrap this?

Stream handling

An interesting question is how vorpal should handle the stream options. Commands could output Buffers, Strings in various encodings, these could be automatically decoded, or object mode could be used. Vorpal coul either restrict this to one, or default to one but allow usage of others. No idea for the API for that though...

[dthree]

However: What happens with a command like echo foo | grep bar? While 'bar' looks like an operand, it actually isn't, the real operand is 'foo'. 'bar' would need to be translated into an option. Which is completely doable, I'm just not sure how to expose this to command implementors.

Foo would be stdin, and bar is the operand. Don't see why this is a problem.

onInput

Can we just refer to onInput as data, to match stream syntax? I think this will be clearer in the long run. As well, cleanup could be called end. These are standard stream events.

---

Okay, I'm really liking all of this. Here's the deal: I want to model Vorpal into a full bash interpreter (as you are saying). This would mean we would match all syntax and functionality of bash (including &&, ||, #, etc.) and properly implement stdin, stdout as you've gone over.

It is very possible that Cash is going to be packaged into NPM as a cross-platform Unix shell, and this update would make this possible.

I think this should release as Vorpal 2.0.

If I added you as a collaborator to Vorpal, and created a 2.0 branch, do you think we could start rolling on all of this? Is this something you could dedicate some time to? I would ideally love to do this as soon as possible.

[dthree]

Okay you have access to the repo now, and I created a 2.0 branch. You're free to push whatever you want to it (if you need to push something to 1.0, just ping me first).

[dthree]

Always returning Promise.resolve() for these synchronous commands is rather annoying though, could Vorpal detect and wrap this?

Sure. Additionally, I would like commands to support sync, async (callbacks and Promises. Right now async and Promise are supported, with sync supported through vorpal.execSync, however it should all be available through the same method.

---

Another thing is that I would like actions to return status code (0, 1), etc. Perhaps all stdout and stderr are sent through logs, and only status codes are returned? Vorpal could then abstract the this all to a standard interface.

[dthree]

btw, what time zone are you in (I'm PST). We seem to have flipped schedules, which is going to suck.

[AljoschaMeyer]

Foo would be stdin, and bar is the operand. Don't see why this is a problem.

Because it would be very elegant to handle operands by piping them into stdin after initialization. Disregarding exceptions like grep, most commands want to handle stdin as if it just contained additional operands, which happen to be supplied asynchronously. That way, our user has to define only one function which handles operands as well als stdin.

This is different from the way UNIX commands work. E.g. echo foo | mkdir won't create a foo directory. But a vorpal implementation of mkdir might as well get that behaviour for free.

However, if we actually try to copy bash, this would have to be handled differently. And since commands like grep require an operand as well as stdin, the current CommandInstance implementation will need to be changed anyways.

I suppose CommandInstance should just mimic UNIX commands and vorpal has to figure out how to provide a convenient API for command definition.

---

Can we just refer to onInput as data, to match stream syntax? I think this will be clearer in the long run. As well, cleanup could be called end. These are standard stream events.

Sure, sounds reasonable.

I want to model Vorpal into a full bash interpreter (as you are saying). This would mean we would match all syntax and functionality of bash (including &&, ||, #, etc.) and properly implement stdin, stdout as you've gone over.

A full bash iterpreter is going to be a lot of work. Also a pretty interesting challenge. I'd suggest writing a jison grammar with the most basic features and then gradually adding to it.

---

Another thing is that I would like actions to return status code (0, 1), etc.

The current CommandInstance implementation already has these, exit code and message are given witht the 'exit' event. These are a must for bash syntax like && and ||.

---

btw, what time zone are you in (I'm PST). We seem to have flipped schedules, which is going to suck.

I'm in CET, but look on the bright side: 24h/day of combined vorpalness... Ok yeah, this sucks.

Is this something you could dedicate some time to? I would ideally love to do this as soon as possible.

It is something I want to dedicate some time to. Currently writing my bachelor's thesis tough, so working on open source projects obviously is a lower priority. Then again, I don't spend all my waking hours on the thesis anyways, so I will get some coding done. And I will put aside some other projects/ideas of mine for this. So yes, count me in.

[AljoschaMeyer]

Also, we can move the CommandInstance specific discussion to its repo, and focus on the vorpal API here.

[AljoschaMeyer]

As for the bash reimplementation, I found these two projects, which may or may not be helpful.

The bash manual looks is surprisingly readable.

Do you think a full reimplementation is necessary and realistically possible? Bash essentially provides a whole programming language. A bash-inspired custom syntax or a strict subset of bash might be the way to go.

---

Just read most of the bash manual, these are my impressions:

• full clone is completely overkill, the key is to select a usefull subset of functionality
• the manual clearly defines terminology, processes and their order, so doing this might not be too difficult - just a lot of work
• we don't want to parse whole scripts, which means we can ignore many features

A first suggestion for the subset to implement:

Section 3.1.1 of the manual describes the general process of what we need to do. The numbers refer to the manual sections.

Core features:

• command building blocks
  • quotation (single quotes and double quotes) and escapes (backslash): 3.1.2.1 - 3.1.2.3
  • commands: 3.2.1
  • pipelines: 3.2.2
  • lists (;, &, ||, &&): `3.2.3
  • command groups ((), {}): 3.2.5
• shell expansions: 3.5
  • brace expansion (a{b,c}d and [0..9]): 3.5.1
  • tilde expansion: Replacing ~ with $HOME is easy. But what about ~+ and ~-? Do we want vorpal to provide a cwd abstraction, with an api that allows commands to set the cwd? 3.5.2
  • parameter expansion (${HOME:-foo}): Maybe start with a partial implementation and add stuff as requested (i.e. most likely never). 3.5.3
  • command substitution: 3.5.4
  • arithmetic expansion: 3.5.5
  • word splitting: 3.5.7
  • filename expansion (*.js): 3.5.8
  • quote removal: 3.5.9
• redirections: Not all of this is needed, but in any case, we need to write to the actual filesystem here. 3.6
• command execution: we can ignore some cases here (variables, functions). Do we need to provide an execution environment (3.7.3, 3.7.4) for our commands? Might be a bit too much, but I don't know how powerful this needs to be.

These core features still ignore much of what bash can do, but they should suffice for realistic non-scripting usage. Not only for vorpal, but also for advanced cash usage. And while this will definitely take some work, it is completely doable.

I'd argue to leave out compound commands (basically control structures) 3.2.4. These add a lot of complexity and should not be needed for common one-liners.

Also worth a look are the built-in commands, since some of these are mirrored in vorpal's api.

[dthree]

Wow - it is surprisingly readable. How pleasantly rare.

---

I think you nailed it on what's needed, I was about to say the same things.

Getting through the manual myself, will update you in a bit.

[dthree]

As a sequence of actions, I think I'm going to start working on the shell parser / AST. If you continue on the control flow (CommandInstance), I think we can move along pretty well in parallel.

[AljoschaMeyer]

Ok, my next step after polishing CommandInstance (which mirrors 3.2.1 simple commands) is to build a CommandGroup, which represents one of some of the the subsections of command from 3.2 shell commands, i.e. simple commands, pipelines, lists and groups. Of course extensible to include the full 3.2.4 compound commands in the future.

A CommandGroup instance will have a superset of the CommandInstance API, so that all the command group functionality can be defined recursively and eventually boils down to simple CommandInstances. It will additionally expose an API to create new CommandGroups from existing ones, representing pipes, lists and groupings. This API would be called from the parser / AST.

At a later point, the CommandGroup will also provide abstractions for 3.6 redirections, again accesible for the parser.

But before coding another line of javascript I have to take care of some other stuff first, which might take a few days. I'll still read everything and try to respond, though.

If you haven't built a parser/ast before, I recommend taking a look at this tutorial for a nice explanation of the manual process in js, and jison for automatic parser generation from a grammar (which is the approach I'd prefer). Or you might know this stuff already =)

[dthree]

Thanks.

js-shell-parse is very close on its AST, so it looks like I'm going to use it for now. He stopped maintaining it, so I'm going to fork it into the Vorpal org, and I got the nodeos guy to agree on working on it with me to finish.

Let me know when your time frees up a bit!

[AljoschaMeyer]

Sweet! I spent some time tinkering with the babel api, and I'm tempted to try the following for a basic shell implementation:

• add a plugin to babel which parses command-expressions like foo | bar
• use babel to compile the whole thing back to js
• execute js in a vm

I'm not sure whether that is a very practical approach, but the idea is cool enough to build a prototype for. For vorpal, building on js-shell-parse should be the better plan.

As for useful stuff, I'll soon upload a process class with an API similiar to node's ChildProcess to represent parts of a command chain.

[dthree]

Okay, awesome. Maybe we should focus on the plan in line with Vorpal, just because its already a huge amount of work, and then once we have a proof of concept we can turn it into tons of powerful offshoots, like the babel one (which sounds like a cool idea!).

My thought is:

vorpaljs/bash-parser: Parses into AST
vorpaljs/bash: Takes parsed AST and does low-level, context-independent actions such as all of the expansions, blowing things up into full commands.
vorpal will then take a high level rendition of the command and throw it into CommandInstance for execution with proper flow control.

[AljoschaMeyer]

Just uploading this before going to bed, without finished tests or documentation: pseudo-process.

This could represent commands internally, with an API similiar to node's ChildProcess. Instances of PseudoCommand have methods to pipe and to redirect input and output. Also has a function to construct PseudoCommands from actual ChildProcesses.

[dthree]

Very nice!