@timestamp vs timestamp

[iano123]

Hi jptoto,

We are having an issue. Elasticsearch/logstash is alwayse saving the documents with a timestamp field name: "@timestamp". This project does the same but ompits the @ sign naming it "timestamp".

This makes it difficult to combine results from 2 indexes. Is there a way to change your logger so that it will also save "@timestamp"?

Many Thanks.
Jan

[jptoto]

@iano123 there is also a @timestamp field in the properties array which you can see an example of here http://jptoto.github.io/log4net.ElasticSearch/ in the sample doc. That should always be there for each event.

I wish I had originally named the timestamp field "@timestamp" so I will look to make that easier to find in the future. Let me know if that doesn't solve your issue.

[iano123]

Hi JP,

The problem is that this is in the properties field. This means we would need to use properties.@timestamp in the index definition in kibana.

What we want to achieve is the folowing:

index:

system.iis.date
system.eventlogs.date
system.log4net.date

and we want to put all these in kibana as:
system.*

This way we aggregate them. But this only works if the field is called "@timestamp" and not "properties.@timestamp"

[jptoto]

@iano123 I'll see if I can add an option to place @timestamp at the root and if that makes sense. I'm trying to finish up a new Pluralsight course at the moment which is taking all of my free time but I'll check into this ASAP.

[iano123]

Many Thanks JP. I think I will probably not by the only person with this use case.

[jptoto]

@iano123 you're the first person who's asked for it :)

[jptoto]

Will check to see if Kibana can find the @timestamp field in the properties section. If it can, I don't think I will add @timestamp to the root or make an option for it as it seems to be redundant. I haven't had any other requests for this.

[jptoto]

This works fine in Kibana so I'm gonna go ahead and close this. Thanks for inquiring about it, I just don't think there is a big need for it and by changing the root field name it'll throw off existing users.

[iano123]

Hi JP<

yes it can see it, but it is not the same field. it is a property of something else.

This is problematic when you want it in one search index: EG:

Systemname.winlog
Systemname.log4net
Systemname.topbeat

and open the indes in kibana as Systemname.*

This does not work as it will not know which field to use for the time related data.

Will addign an optional property to rename @timestamp be a breaking change? I believe not.

[jptoto]

I suppose simply adding the additional field won't hurt. I can eventually deprecate the old one.

[johnqa]

Hi jptoto,

Do you have any updates on this issue? I am also facing problems because of this timestamp notation.

Thanks!

[tylerrbrown]

Hi jptoto,

Do you have any updates on this issue? I am also facing problems because of this timestamp notation.

Thanks!

I added a pull request with a new configuration option:

  <fieldNameOverride type="log4net.ElasticSearch.FieldNameOverride, log4net.ElasticSearch">
    <original value="timeStamp" />
    <replacement value="@timestamp" />
  </fieldNameOverride>

...that allows you to override any field name (I needed @timestamp too). Fixed a few bugs I ran into, and added a setting:

rollingIndexNameDateFormat value="yyyy-MM-dd"

...so I could get the exact index name I wanted.

Cheers!