GSoC: New Bowtie Test Case Widgets

[Julian]

Project title

New Bowtie Test Case Widgets

Brief Description

Bowtie is a JSON Schema tool for comparing implementations with each other, and reporting on when or whether they have behavior which does not match with behavior expected by the JSON Schema specification.

It renders this information on a website at bowtie.report.

A major part of showing this information is communicating:

• a specific JSON Schema
• a specific JSON instance (piece of data)
• the expected result for whether this schema considers the instance valid
• the actual result received from 15 or more implementations which were tested
• a description of this specific test case suitable for human readers

How to present these 5 bits of information (and a few more) can get challenging when showing a large number of test cases.

Bowtie's website currently contains a crude pair of UI widgets for doing this -- specifically it renders a long list of collapsible sections:

and then when expanding one, shows the schema, instance, and results in a table:

But we can do lots better! Help rewrite Bowtie's test UI components so that we can cleanly filter through hundreds or thousands of test cases in a clear and efficient way.

Expected Outcomes

• New UI components for test case lists as well as instance/schema pairs
• Integration of these components with Bowtie's site

Skills Required

• React / TypeScript expertise
• An eye for design
• Basic knowledge of JSON Schema

Mentors

@Julian

Expected Difficulty
Medium

Expected Time Commitment
175 Hour

[SimonDMC]

Hello! I have a few ideas for this project and I'd love to discuss potentially working on it as a part of GSoC, but it seems that the slack link is broken. Here's the error I'm getting:

[Siddharth-Singh-2004]

A few questions from my side to understand this requirement better:

1. Will UI developed by this project be net new completely replacing the existing UI or should we focus only on the parts highlighted in the project description, which is mainly about making it easy to navigate through the test case results?
2. Dialect Reports, for eg., draft2020-12.json: should we expect a change in format of the report as a part of any other project running in parallel with this project ? If yes, where may I access the details of the new format?
3. Would we like the UI to render differently for implementers vs maintainers (org/providers)? Implementers in the dev-test-fix cycle have a different set of requirements as compared to maintainers or decision makers.
4. There is another requirement on bowtie long term trend reporting GSoC: bowtie-trend: Long-Term Reporting With Bowtie #607. Are we looking at persisting data in a datastore (separate table for each type of data) as a part of that requirement or would we be working by searching through files? If yes, then should this project also work off of the latest results from the data store rather than a file, except when the user chooses to work of local file during dev-test-fix cycle?
5. Are we expecting the work from GSoC: bowtie-perf: a Performance Tester for JSON Schema implementations #605 and GSoC: bowtie-trend: Long-Term Reporting With Bowtie #607 getting linked from this report so implementers have one place to look at trends and performance metrics? Are we expecting deeper integration (components to be integrated) into this project's UI in a composable dashboard or will those reports have dashboards which will be linked from this project's UI?
6. Will there be need to support sign-in and role based access control tying it back to point (1) above (different UI based on role)?
7. If we do point (6) above, then support Profile based personalisation, for eg: pick and choose schemas I'm interested in, and not be shown all of the supported schemas.

[Julian]

Will UI developed by this project be net new completely replacing the existing UI or should we focus only on the parts highlighted in the project description, which is mainly about making it easy to navigate through the test case results?

Always focus on what an issue says :)

Dialect Reports, for eg., draft2020-12.json: should we expect a change in format of the report as a part of any other project running in parallel with this project ? If yes, where may I access the details of the new format?

Maybe, but if so the UI parsing code would be updated to handle it.

Would we like the UI to render differently for implementers vs maintainers (org/providers)? Implementers in the dev-test-fix cycle have a different set of requirements as compared to maintainers or decision makers.

Sounds like potentially a nice set of functionality -- but not very related to this issue, which is about "what's the clearest possible way to present the test result information". Unless you think the answer even for that question differs for these two groups (I don't think I do).

Are we looking at persisting data in a datastore (separate table for each type of data) as a part of that requirement or would we be working by searching through files?

Considering we're still using GH Pages, likely not, just files. Perhaps someday.

Are we expecting deeper integration (components to be integrated) into this project's UI in a composable dashboard or will those reports have dashboards which will be linked from this project's UI?

A dashboard and integration certainly sounds nice -- keep in mind though it's not necessarily the case that both projects will go forward, we have a limited number of mentors!

Will there be need to support sign-in and role based access control tying it back to point (1) above (different UI based on role)?

No.

[skushagra9]

I have a good experience with open source
Hey, I am interested to take this issue, should I start contributing?

[AgniveshChaubey]

Hey @skushagra9, since this is a project idea for GSoC and not just a project issue, at present maintainers cannot assign this task specifically to someone. However, you can start contributing to this issue, as early contributions definitely serve as a strong point in your proposal. :)

[skushagra9]

okay, got it

[Swarnendu0123]

Hey @Julian, I'm thinking about an idea. Could we implement a component similar to GitHub-style progress bar to indicate whether tests have passed or not?

We could use green for passed and red for failed. Additionally, users could hover over each box to view the details of the specific tests.

on click also we can design an additional page to show what are the exact points the the specific test is failing/passing.

I am Swarnendu, a student from India, and I have a vast experience in React-TypeScript, and Styling, can I do it as my GSoC project, maybe we can discuss and generate a better idea?

[Julian]

Hi, welcome!
Seems like a nice idea, though that component in GitHub is used to communicate progress over time right? Perhaps that makes that kind of idea a better fit for #607, since otherwise I'm not sure I fully follow what the X and Y axis will represent. But maybe it relates to both!

[Swarnendu0123]

since otherwise I'm not sure I fully follow what the X and Y axis will represent. But maybe it relates to both!

Can we just categories our tests, if we can, then we can solve that issue?

Perhaps that makes that kind of idea a better fit for #607

I have another improved idea for #607: when dealing with large volumes of data, it's beneficial to visualize the information using diagrams such as graphs. We can utilize a graph similar to this:

[AgniveshChaubey]

Adding a graphical representation of tests seems like a great idea to me. How about representing the number of tests on the Y axis and implementation test results on X-axis?

Also grouping the implementation graphs of same the language would make it easier for users to select the best-performing implementation for their use.

If bar chart is used, then maybe we can use the bottom bar to represent the number of passing tests by that implementation. Then we could add bar on top of the base one to show the skipped tests (if any) and the top one could show the number of failing tests. (Maybe we categorize them with colors as well for better visual representation--- green for passing, gray or yellow for skipped and red for failing)

This is the quickest ugliest representation I could come up with for what I said above:

Since there are a lot of implementations registered to Bowie, the graphs could possibly overflow the parent container, so adding a horizontal scroll bar might be a good option.

@Julian what's your comment on this? If Julian agrees then @Swarnendu0123, you might find this as a good start for your further progress.

[Swarnendu0123]

Loved your idea @AgniveshChaubey! And Thanks for the demonstration.

What are your views on it @Julian ?

[Julian]

Now that definitely is the kind of idea this project was about yes! I had that idea in a slightly different way, where we'd show them as stacked horizontal bars, because that makes it easier to sort visually by compliance percentage ("fraction of green"). But yes I agree that's exactly the kind of thing we should think about for this issue!

[benjagm]

Thanks a lot for joining JSON Schema org for this edition of GSoC!!

Qualification tasks will be published as comments in the project ideas by Thursday/Friday of this week. In addition I'd like to invite you to a office hours session this thursday 18:30 UTC where we'll present the ideas and the relevant date to consider at this stage of the program.

Please use this link to join the session:
🌐 Zoom
📅 20124-02-29 18:30 UTC

See you there!

[Siddharth-Singh-2004]

@benjagm, Would there be a recording of the session for people who cannot join the session live, or may need the same for reference after the session as they work on the proposal?

[benjagm]

Would there be a recording

Of course. Well record the session and attach it to this issue:
#643

[Sahil-Shadwal]

Hey @Julian whats the project qualification criteria for this project ??

[Julian]

Hey there! Thanks for the interest.

For all 3 of the Bowtie tickets the qualification task is to get the project running, find a task labelled "good first issue" and try to tackle it (check first to ensure no one else is working on it). If you find there aren't any, simply finding a part of the UI that could use testing and adding a UI test is a general backup task!

[SimonDMC]

Hello everyone!

I'll be applying to tackle this project as a GSoC contributor, so I figured I'd introduce myself here! I'm Simon, a web developer from Czechia, starting university this year.

What interests me about this project - and the reason I chose it - is the combination of creative potential and web development. As opposed to having a clear set of requirements and only working on an implementation, this project requires the contributor to design the UI and only then implement it, which I find fun.

I'd say some of my best work comes from collaborating. I could probably complete the entire project alone, but having a mentor to discuss my ideas and designs with will really improve the end product. I find having someone like-minded who can provide a second viewpoint on my work leads to a much better result. At the end of the day, this is all about collaboration!

I have a four-month break between the end of May and the end of September, so I won't be working or studying in that time. If time allows for it, I might still work on my personal projects, but I'm equally happy for GSoC to take up most of my time.

As for my work preferences, I usually keep a Google doc or a simple text file where I write things that need to be done and check them off over time. I tried to get into using Notion but it never really suited me. Though I suppose this project will likely involve using Github Issues for tracking progress, so I'm willing to give that a try. For designs and mock-ups, I use Figma.

If I'm chosen for this project, here's how I would structure it:

• Week 1 - First design
  • A full mock-up of how I envision the UI
• Weeks 2-3 - Refining the design
  • Discussing the design with the mentor and potentially getting feedback from the community, talking through what would work and what wouldn't
• Week 4 - Search bar
  • Adding a search bar seems like an ideal way to start this project, shouldn't be too difficult and will improve the UI significantly
• Week 5 - Dialect pane
  • Move the dialects dropdown to a pane on the left side of the screen, allowing the user to switch between dialects easily
• Weeks 6-7 - Single-implementation view/mode
  • Add a dropdown to the top of the test case section for selecting a specific implementation, allowing for status indicator icons next to each test case
• Week 8 - Test case result info
  • Add a set of icons (or a percentage to show the success rate?) to each test case to show how it behaves against all tested implementations
• Week 9 - Sorting and filtering
  • For the single-implementation mode, add a filter button to only show the test cases with a set success state. Also, add some filtering options, like alphabetical, most successful, least successful, etc.
• Weeks 10-11 - Revamping the test case view
  • Move the test case details to a separate, extendable right pane and improve the table view
• Week 12 - Polishing
  • Make sure all the new changes work together seamlessly and finish any remaining work

The schedule will probably shift around a bit as I figure out what takes more time and what takes less, but these are the objectives I see fit for the task.

Cheers!

[benjagm]

🚩 IMPORTANT INSTRUCTIONS REGARDING HOW AND WHERE TO SUBMIT YOU APPLICATION 🚩

Please join this discussion in JSON Schema slack to get the last details very important details on how to better submit your application to JSON Schema.

See communication here.