[Monitoring] [Logstash] Add Config View #18597

justinkambic · 2018-04-26T03:43:49Z

Changes

This feature will add a new visualization of Logstash Pipelines that mimics what the user will see when they look at the corresponding configuration file. Its purpose is to address a problem users encounter with the Pipeline Viewer when visualizing relatively large or complex pipelines.

The overall goal is to resolve #18423.

Review & Testing

Configure Logstash

If you need to run this PR against a stack with Logstash running please let me know and I can grant you access to a demo box we have set up for that express purpose.

I've moved the old instructions to the bottom of this section.

~~From a functional standpoint these changes do what we want for the initial version of this feature. At the moment we are looking for designer input.~~

You can see this view by selecting a pipeline from the Pipelines view in Logstash monitoring. Try defining multiple pipelines to see how the component will render each one. I've included an example configuration that I've used for development purposes. It doesn't do anything practical but it does model all of the if/else shapes.

Old configuration instructions:

Ensure you have Java 8 installed and set as the Java in your shell. If you don't, check out SDKMAN, it makes managing that a snap.
java -version should output java version "1.8.0_161"
Download Logstash 6.2.4
Unzip/untar LS (tar xofp logstash-6.2.4.tar.gz)
From Logstash root directory, install x-pack with ./bin/logstash-plugin install x-pack
Overwrite the existing logstash.yml with this one: logstash.yml.zip
NOTE: if you're using Security on your local stack, you'll have to open that file and un-comment the username/password keys and plug in appropriate values.
Set up a test pipeline by running ./bin/logstash -e 'input { stdin { } } output { stdout { } }' from the LS top directory. You'll see Logstash pop up in the Monitoring Overview page in your Kibana, and you should be able to drill into your pipeline to see the viewer.
From here you can use example.conf.zip with ./bin/logstash -f {PATH_TO_EXAMPLE_CONF}/example.conf
Tip: when killing Logstash, send it a second CTRL+C to kill it right away. Part of its shutdown procedure on the first kill command is to empty its queue, but we don't care about those events for our purposes.

Testing

Run various logstash pipelines shapes.
For each, navigate to the Logstash Monitoring/Pipelines page and click your pipeline.
Evaluate the rendered configuration view and ensure that what you see is an accurate representation of your config file.
For if/else branching, make sure that you're able to collapse and expand the elements, and the styling behaves as you'd expect.

Motivation

As mentioned, the primary goal of this view is to address the Pipeline Viewer's lack of usability for pipelines beyond a certain level of complexity, as discussed in #18423. To that end, we are using a more tree-like layout to express the structure of the pipeline configuration, rather than visualizing the actual flow of data through the user's pipeline.

Followup

Additional work to refactor/remove unneeded code will follow this one. Breaking this work up is a conscious choice driven by a desire to segregate the scope of the changes. Tracked here: [Logstash] [Pipeline Viewer] Remove old pipeline viewer code #19736
Close [Pipeline Viewer] Narrow Down New Design Options #18020
We need to update the top-level nav, but it lives in Angular, so that will be a separate PR to Reactify that piece of the view, tracked here [Monitoring] [Logstash] Update pipeline top nav #18976
The data the model classes give this view may end up changing, or we may need to modify the way the view behaves when displaying new pipelines, or dead pipelines [Logstash Monitoring] Pipeline Viewer shows 0s in the early life of a pipeline version #19429
A proposed design element is the highlighting of nested statements, probably on a delayed hover. This would lower the opacity of outside statements. It's tracked here: Add outline highlighting to Config View #19463
Refactor the CollapsibleStatement class to behave as a wrapper for props.children [Monitoring] [Pipeline Viewer] Refactor collapsible statement component to wrap props.children #19844
Remove bare <img>s in favor of EuiIcons where possible Swap ad-hoc imgs for EUIIcons in Detail Drawer and Config View sub-components #20021

elasticmachine · 2018-04-26T04:42:24Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

elasticmachine · 2018-04-26T16:26:16Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

elasticmachine · 2018-05-01T18:36:45Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

justinkambic · 2018-05-01T18:40:10Z

Notes from discussion with @snide and @cchaos:

Indent size will likely change
Explore possibility of filtering for slow/time-consuming plugins (if we did this it'd likely be a subsequent PR)
Update top-level nav (this would likely be a separate PR)
Mentioned it would be useful if we could somehow annotate pipelines
Removal of hard borders: represent the existing plugin data differently (❤️ this type of feedback)
Addition of color to enhance view's expressiveness

cc @ycombinator

elasticmachine · 2018-05-04T15:57:06Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

elasticmachine · 2018-05-09T21:13:57Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

elasticmachine · 2018-05-10T23:52:37Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

elasticmachine · 2018-05-15T21:10:54Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

elasticmachine · 2018-05-18T14:40:27Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

elasticmachine · 2018-05-23T23:13:35Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

…etrics.

justinkambic · 2018-06-14T19:32:08Z

@ycombinator thanks, sounds great!! Just as a heads up, I updated the jest snapshots for some of the DetailDrawer's test. I reviewed all the failing snapshots and they all seemed to be in keeping with the changes Dave made to the component.

I just wanted to get a nod from you since these were updated after you concluded your review. The snapshots are updated in a91dc56, and if you want to reference Dave's changes, they are all in 1f667cc.

elasticmachine · 2018-06-14T20:41:10Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

ycombinator · 2018-06-16T06:06:13Z

I just wanted to get a nod from you since these were updated after you concluded your review.

Still LGTM.

mattapperson · 2018-06-18T19:25:38Z

.../components/logstash/pipeline_viewer/views/__test__/__snapshots__/detail_drawer.test.js.snap

+        component="div"
+        grow={false}
+      >
+        <img


Maybe it is better to use an EuiIcon here? Along with the other areas using ad-hoc omg tags? But that's not a blocking item IMHO and is perhaps better in a followup PR anyway.

I'll create an issue to track it. These changes are indirect to the scope of this PR, it's based on some cleanup that Dave did after he updated the styles.

EDIT: #20021

* Patch ConfigView changes from x-pack-kibana to OSS Kibana. * Remove old css. * Update style for queue statement. * WIP modifying based on designer feedback. * Add flatten function and list class. * Rename functions to be more descriptive. * WIP checkin. Modify components to handle flattened object list. * Finish moving flatten logic into classes, add tests. * Simplify flattening, remove non-native dependency. Add more tests. * Add defaults to simplify function call. * Refactor two blocks into a function. * Begin adapting components for list. * Add collapse functionality. * Add expand functionality. * Nested collapsed elements remain collapsed on parent expansion. * Style section headers. * Update Plugin statement styles. * Add DetailDrawer support. * Update statement formatting. * Add stats to plugin element rows. * Resolve conflicts. * Remove obsolete code. * Reorganize code. * Remove warnings. * Add PropTypes. Add keys to arrays and iterables. * Update color for borders/buttons. * Add stat class. Clean up code. * Convert plugin statement component from class to function. * Update queue metrics message. * Update style to make view more responsive. * Change section heading size. * Remove gutter from metrics group. * Change name "stat" to "metric". * Remove obsolete export line, simplify declaration, based on PR feedback. * Add new functional component in place of model class. * Add PropTypes to several components. Rename a function. Add keys to metrics. * Convert stateless classes to functional components. * Prefer ES6 syntax over bindings for Component methods. * Do not render statement section if there are no statements. * design cleanup for pipeline viewer * Update CSS to add min-height to page. * Rename "elements" to "statements". Delete unused LESS variables. * Revert naming of "statements" to "elements" for StatementList component. * Update jest snapshots for DetailDrawer.

justinkambic · 2018-06-19T03:45:19Z

Backported to:
6.x/6.4.0 b01dc53

* Patch ConfigView changes from x-pack-kibana to OSS Kibana. * Remove old css. * Update style for queue statement. * WIP modifying based on designer feedback. * Add flatten function and list class. * Rename functions to be more descriptive. * WIP checkin. Modify components to handle flattened object list. * Finish moving flatten logic into classes, add tests. * Simplify flattening, remove non-native dependency. Add more tests. * Add defaults to simplify function call. * Refactor two blocks into a function. * Begin adapting components for list. * Add collapse functionality. * Add expand functionality. * Nested collapsed elements remain collapsed on parent expansion. * Style section headers. * Update Plugin statement styles. * Add DetailDrawer support. * Update statement formatting. * Add stats to plugin element rows. * Resolve conflicts. * Remove obsolete code. * Reorganize code. * Remove warnings. * Add PropTypes. Add keys to arrays and iterables. * Update color for borders/buttons. * Add stat class. Clean up code. * Convert plugin statement component from class to function. * Update queue metrics message. * Update style to make view more responsive. * Change section heading size. * Remove gutter from metrics group. * Change name "stat" to "metric". * Remove obsolete export line, simplify declaration, based on PR feedback. * Add new functional component in place of model class. * Add PropTypes to several components. Rename a function. Add keys to metrics. * Convert stateless classes to functional components. * Prefer ES6 syntax over bindings for Component methods. * Do not render statement section if there are no statements. * design cleanup for pipeline viewer * Update CSS to add min-height to page. * Rename "elements" to "statements". Delete unused LESS variables. * Revert naming of "statements" to "elements" for StatementList component. * Update jest snapshots for DetailDrawer.

justinkambic added review v7.0.0 Team:Logstash v6.4.0 Team:Monitoring Stack Monitoring team labels Apr 26, 2018

justinkambic requested review from ycombinator and andrewvc April 26, 2018 03:43

justinkambic requested a review from snide April 26, 2018 15:27

justinkambic mentioned this pull request May 1, 2018

[Monitoring] Reactify icons in Pipeline Viewer #18159

Closed

justinkambic force-pushed the monitoring/add-config-view branch from f3edcef to 86fa15e Compare May 1, 2018 15:35

justinkambic force-pushed the monitoring/add-config-view branch from 86fa15e to 01a35e5 Compare May 4, 2018 14:51

justinkambic force-pushed the monitoring/add-config-view branch from 01a35e5 to fcdb44d Compare May 9, 2018 19:57

justinkambic mentioned this pull request May 10, 2018

[Monitoring] [Logstash] Update pipeline top nav #18976

Closed

justinkambic force-pushed the monitoring/add-config-view branch from fcdb44d to 10a8b17 Compare May 10, 2018 22:49

justinkambic force-pushed the monitoring/add-config-view branch from 10a8b17 to 257a905 Compare May 15, 2018 19:47

justinkambic force-pushed the monitoring/add-config-view branch from 257a905 to a6ae20b Compare May 18, 2018 13:52

justinkambic force-pushed the monitoring/add-config-view branch from a6ae20b to 522a6b9 Compare May 23, 2018 22:04

justinkambic force-pushed the monitoring/add-config-view branch 3 times, most recently from 4f17978 to 127882f Compare May 24, 2018 19:41

justinkambic and others added 13 commits June 14, 2018 15:28

Remove gutter from metrics group.

a3b9bed

Change name "stat" to "metric".

0ab6b51

Remove obsolete export line, simplify declaration, based on PR feedback.

56d77f1

Add new functional component in place of model class.

fcb2452

Add PropTypes to several components. Rename a function. Add keys to m…

c580e5e

…etrics.

Convert stateless classes to functional components.

c7a92de

Prefer ES6 syntax over bindings for Component methods.

4536b27

Do not render statement section if there are no statements.

d845795

design cleanup for pipeline viewer

1f667cc

Update CSS to add min-height to page.

da426f3

Rename "elements" to "statements". Delete unused LESS variables.

1689b94

Revert naming of "statements" to "elements" for StatementList component.

c991bb1

Update jest snapshots for DetailDrawer.

a91dc56

justinkambic force-pushed the monitoring/add-config-view branch from 25f65f1 to a91dc56 Compare June 14, 2018 19:29

justinkambic requested a review from mattapperson June 15, 2018 13:36

snide approved these changes Jun 17, 2018

View reviewed changes

mattapperson approved these changes Jun 18, 2018

View reviewed changes

justinkambic mentioned this pull request Jun 19, 2018

Swap ad-hoc imgs for EUIIcons in Detail Drawer and Config View sub-components #20021

Open

justinkambic merged commit 6f7dfcf into elastic:master Jun 19, 2018

justinkambic deleted the monitoring/add-config-view branch June 19, 2018 03:45

justinkambic mentioned this pull request Jun 20, 2018

[Pipeline Viewer] Initial version follow-up work #20085

Open

ycombinator mentioned this pull request Aug 17, 2018

[Accessibility] Logstash Pipelines #18065

Closed

justinkambic mentioned this pull request Oct 17, 2018

[Logstash] [Pipeline Viewer] Improve LS monitoring accessibility #24169

Merged

4 tasks

ycombinator mentioned this pull request Nov 14, 2018

R&D: Service map graph visualization #25539

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Monitoring] [Logstash] Add Config View #18597

[Monitoring] [Logstash] Add Config View #18597

justinkambic commented Apr 26, 2018 •

edited

elasticmachine commented Apr 26, 2018

elasticmachine commented Apr 26, 2018

elasticmachine commented May 1, 2018

justinkambic commented May 1, 2018

elasticmachine commented May 4, 2018

elasticmachine commented May 9, 2018

elasticmachine commented May 10, 2018

elasticmachine commented May 15, 2018

elasticmachine commented May 18, 2018

elasticmachine commented May 23, 2018

justinkambic commented Jun 14, 2018

elasticmachine commented Jun 14, 2018

ycombinator commented Jun 16, 2018 •

edited

mattapperson Jun 18, 2018

justinkambic Jun 19, 2018 •

edited

justinkambic commented Jun 19, 2018 •

edited

[Monitoring] [Logstash] Add Config View #18597

[Monitoring] [Logstash] Add Config View #18597

Conversation

justinkambic commented Apr 26, 2018 • edited

Changes

Review & Testing

Configure Logstash

Testing

Motivation

Followup

elasticmachine commented Apr 26, 2018

💚 Build Succeeded

elasticmachine commented Apr 26, 2018

💚 Build Succeeded

elasticmachine commented May 1, 2018

💔 Build Failed

justinkambic commented May 1, 2018

elasticmachine commented May 4, 2018

💚 Build Succeeded

elasticmachine commented May 9, 2018

💚 Build Succeeded

elasticmachine commented May 10, 2018

💚 Build Succeeded

elasticmachine commented May 15, 2018

💔 Build Failed

elasticmachine commented May 18, 2018

💚 Build Succeeded

elasticmachine commented May 23, 2018

💔 Build Failed

justinkambic commented Jun 14, 2018

elasticmachine commented Jun 14, 2018

💚 Build Succeeded

ycombinator commented Jun 16, 2018 • edited

mattapperson Jun 18, 2018

Choose a reason for hiding this comment

justinkambic Jun 19, 2018 • edited

Choose a reason for hiding this comment

justinkambic commented Jun 19, 2018 • edited

justinkambic commented Apr 26, 2018 •

edited

ycombinator commented Jun 16, 2018 •

edited

justinkambic Jun 19, 2018 •

edited

justinkambic commented Jun 19, 2018 •

edited