on disable filter -> can log jump to same line?

[caifara]

I feel lnav is quite close to a flow I would like to implement while developing/debugging:

• filter log output, add a filter
• upon finding an interesting line, I want to disable the filter, and inspect all lines around the match

However It seems that, when a filter is disabled, I jump back to the first log line.

I saw some possibilities with marking the line, but I want this to be w/o any friction (like jumping around): the top line should still be the top line when disabling the filter.

As an aside and/or follow up if I missed something that can do the above: it would help tremendously if the filter can easily be used as a search as well. If it would be possible, I would "search", if too much, switch to "filter" on the same thing, and enable/disable these filters all the time.

Is this possible? Or am I just trying to do something lnav is not meant to do?

[tstack]

What version of lnav are you using? The intention is to preserve the location, so if it's not working, that's definitely a bug.

filter log output, add a filter

Are you talking about a :filter-in here and enabling/disabling that?

[caifara]

Ah: I seem to be in the TEXT view.

I'm viewing a Ruby on Rails log (in test environment).

[tstack]

Ah: I seem to be in the TEXT view.

In the LOG view, the timestamp of the log message is used to preserve the location when enabling/disabling filters. I haven't spent time trying to preserve location in the TEXT view since it can be inefficient/inaccurate.

I'm viewing a Ruby on Rails log (in test environment).

If the logs have timestamps, then the first thing to do would be to create a log format definition so that they are recognized as log files by lnav. If you want to try and do that yourself, take a look at the Defining a New Format section of the docs. There is some integration with https://regex101.com, so you can enter some samples there and come up with a regex that way. Then, you can import it into lnav. If that's too much for you, file an issue here with some sample logs and I can create the format for you.

Is it possible to remap in one of the two uses for it: quitting lnav/exiting filters?

I'll add ESC as the primary way to exit the Files/Filter panel.

[caifara]

Right, thanks for the help! I'll try it as it seems that it will be really helpful.

Just to note: changing the log format + adding a new format to lnav + adding scripts for a nice search->filter integration is adding friction to what seems like a very promising workflow with lnav. That must hurt adoption (I know, this is the 4th time I'm trying to give up tail -f in function of lnav). I guess lnav is less tailored to navigate logs like those (less structured) coming from a Rails environment.

Speaking of my use case (the developing a webapp use case I guess), it would also be very handy to be able to clear the buffer (the use case would be to clear the screen between two tests so lnav only contains the log of the test that I run next).

To be clear, I'm very thankful your help and the impressive project. And it's perfectly okay if my use case is just not something you would like to support directly with lnav. I'm of course not in any position to influence the course of your project in any way and I only try to explain what I'm looking for in a 'better tail -f'.

[tstack]

Just to note: changing the log format

lnav automatically detects the log format for files if it can find a match, it's not something the user has to do. Or, do you mean something else?

• adding a new format to lnav

lnav needs a log format in order to accurately determine the start/stop of a log message, the timestamp, and the log level. If lnav tried to guess, it would produce incorrect results, which is unacceptable. I have tried feeding some of the logs I work with into other tools that work by guessing where the timestamp is and they produce garbage. lnav does ship with 60+ log formats and you can search around github to find even more. For example, there is a ruby format here that might work or you can use as a starting point:

https://github.com/penntaylor/lnav-ruby-logger-format

And, as I said, I am happy to add more formats. Filing a bug with some sample log messages would be a quick way to get something integrated.

I have also tried to make it easier to add a new format to lnav. The regex101 integration can get you really far along. And, lnav itself will print out a lot of information if it detects errors in a format. For example, part of a format is supplying sample log messages. If the regexes don't match the sample, you will get a pretty error message that shows how much of the sample was matched. There are also numerous timestamp formats builtin and lnav will explain if there is an issue in finding a match there.

• adding scripts for a nice search->filter integration is adding friction to what seems like a very promising workflow with lnav.

I am in the process of adding support for default "suggestions" for some commands where the suggestion is shaded and pressing right-arrow will accept the suggestion. For the filter commands, the suggestion is the current search term. Here's a gif of it in action:

I think that goes a long way to making it easier to convert a search term into a filter. However, since there are already so many hotkeys/commands, I'm very hesitant to add more when there's a pretty reasonable way to do this with the existing functionality that doesn't require the user to really learn anything.

I guess lnav is less tailored to navigate logs like those (less structured) coming from a Rails environment.

I'm not sure what you mean by this... can you clarify?

it would also be very handy to be able to clear the buffer (the use case would be to clear the screen between two tests so lnav only contains the log of the test that I run next).

The :hide-lines-before command can be used to hide log messages before a timestamp. The easiest way to use it is to focus on the first line that you want to be visible and use here. In your case, move to the very end and then run:

:hide-lines-before here

You could also press m to bookmark a line between tests and then use u / Shift + U to jump forward/backward through those bookmarks.

[caifara]

Thanks again for your very thoughtful and elaborate answers!

Just to note: changing the log format

I meant that I need to add a timestamp to the Rails logs. I understand that this is used as an anchor, so it's not possible to just not use timestamps.

I just did that, and it wasn't that simple: I have quite a few tests that mocked the current time. This practice also influences the timestamp but I found a solution for that problem. After that, the log is recognised as a 'generic log', and if the timestamp is really granular (like "2024-03-02 14:41:18.442469", it actually makes toggling the filter work as expected. If less granularity is used (which would use less terminal real estate), disabling the filter ends up on the first line with the same timestamp.

I do have a block of 27 characters preceding each line only to let lnav have usable anchorpoints. They are absolutely of no practical use during all of development/testing.
I found that I can move the view somewhat to the right, but it does not seem possible to do move it the exact amount of characters needed. I searched for a way to just hide that part of the view, but didn't find a way. Did I miss something here?
I still wish I wouldn't need timestamps at all and an offset of some sorts could be used to handle disabling filters (which may or may not be practical/possible, I never made log viewers).

About the "less structured" Rails logs during testing: the default logs are just lines of text, some depict incoming requests, others the render of some template, still others are mostly sql queries and so on. There's no option to properly split it and that is also just fine (for the purpose). Others may even be some random log line by a developer.

All that said, I think I can do most of what I would like to do. At the moment it only feels like it could be easier. But I think that is also mostly solvable through some keymaps and scripts.