Setting up own server + extra questions

[sokol1412]

Hi Tom!

Firstly I really appreciate your work you've done here.
I've tested the solution with one of my APIs, I love the way you're presenting the data.

I have some questions (pointed also on the screen below):

• Do you have any idea why I cannot see any locations?

• Should I see 0's at the response times pane? I assume it might be due to most of the requests being CURLs

• Is there an option/future plan to add an endpoint filtering mechanism? E.g. select endpoint X and see response times timeline only for that endpoint?

• I can see in the git repo, that there is some monitoring code. I assume it's not yet deployed to the www.apianalytics.dev server?

• I can see in the git repo that there is a full server code. Maybe you have a guide on how to start it up (ideally I would like to set up the server locally as a separate Docker container).

Please let me know your thoughts! I'm open to collaboration :)

[tom-draper]

Hey! Thanks for raising this feedback, I appreciate it.

Locations are IP-based, which means either the IP address of the incoming request is not being logged correctly or the IP addresses do not have a location (e.g. if you're sending requests from your local machine to a local server). Which web framework are you using so I can look into this?

The response times show the time between your server receiving the request and sending the response, so it will be lower than you expect since it doesn't capture network latency. But unless you're using a rust framework and simply returning static data, 0ms seems very unusual, especially in the upper quartile, I'll look into this too.

Filtering by endpoint is a great idea! I'll get working on that. Do you think selecting specific endpoints to add to the filter by clicking on them in the bottom left list would be a good approach?

The repo has some code for an upcoming separate monitoring service that regularly pings custom endpoints to check their status and response times. There are also other sub-projects confusingly labelled monitoring that are just self-built tools for monitoring my own services.

Self-hosting is definitely on my list of features to add. I'm aiming to eventually get everything into docker containers that makes it much easier for someone else to deploy. At the moment the service has lots of different parts and takes quite a bit of work to deploy, I could put together some instructions but they would be very long and complicated, so I don't think it's quite ready for self-hosting yet unfortunately. But I completely get that not everyone wants their logged request data stored on someone else's private server. I've been asked about self-hosting before so I'll try and look into how to dockerize everything soon.

[Wamy-Dev]

Self-Hosting would be awesome! I just stumbled across this like less than 30 minutes ago and I think this is totally awesome!

[sokol1412]

@tom-draper Good morning and thanks for quick response!

• Locations - I'm using FastAPI, API is running on my localhost.
• 0ms Response times - actually I think these numbers make sense. the 0ms response times come from Curl requests which are localhost requests, when I check on some Proper requests which involve internal database communication, the response times are >0 (e.g. 14 ms)
• Endpoints filtering functionality - I think it would be enough to just click single endpoint on the bottom left and have dynamic response times filtering based on it. Not sure possibility to select multiple endpoints is that useful. I would limit it to 1 active endpoint filter.

Regarding instructions for self-hosting server - that would be SUPER appreciated. I'm happy to volunteer to test the guide as soon as it's available and provide proper feedback :)

[sokol1412]

Hey @tom-draper . I can see that you added the endpoint filtering mechanism which is great!

Are there maybe any updates/plans for server's self hosting guide? That would be super appreciated.

[tom-draper]

Hi, sorry I'm looking to get around to it but I've not had the time and other higher priority issues keep creeping up. I'll try and get to it in the next couple of weeks.

[asyba]

@tom-draper Im getting No Locations Found, my API is FAST api in a docker container local machine but through a tunnel cloudflare exposed to the internet.
I call the endpoint with the public domain, I can see the domain in the Dashboard but does't display the country.
is there a way to debug to see what IP is coming ?

[tom-draper]

@asyba That must mean the client IP address isn't stored in the usual place on the request with your setup. If you navigate to your local copy of the Analytics class in the fastapi.py file of the package and inspect the request value (printing or breakpoint etc.) to see where in the request your IP address is being written to (currently it's getting it from request.client.host). I'll try to update the FastAPI library soon so that you can provide a custom function during setup that will take the request as an input and let you specify where the IP address is located within your request.

[asyba]

@asyba That must mean the client IP address isn't stored in the usual place on the request with your setup. If you navigate to your local copy of the Analytics class in the fastapi.py file of the package and inspect the request value (printing or breakpoint etc.) to see where in the request your IP address is being written to (currently it's getting it from request.client.host). I'll try to update the FastAPI library soon so that you can provide a custom function during setup that will take the request as an input and let you specify where the IP address is located within your request.

why UI looks like this today?:

debugging not like that but doing the curl
curl --header "X-AUTH-TOKEN: xxxx" "https://apianalytics-server.com/api/data?dateFrom=2023-11-25&dateTo=2023-12-01&status=200"

when calling from domain is showing the IP from the docker container: 172.28.0.2/32
when calling locally is showing local IP hostname": "192.168.1.114",

{
    "hostname": "xxxxxxxx.com.ar",
    "ip_address": "172.28.0.2/32",
    "path": "/data/status",
    "user_agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36",
    "method": 0,
    "status": 200,
    "response_time": 8,
    "location": "  ",
    "created_at": "2023-11-28T21:27:11.342533Z"
  },
  {
    "hostname": "192.168.1.114",
    "ip_address": "192.168.1.104/32",
    "path": "/gruv",
    "user_agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36",
    "method": 0,
    "status": 200,
    "response_time": 27,
    "location": "  ",
    "created_at": "2023-11-28T21:28:14.560562Z"
  },

doing this when calling from the internet is showing the real public IP.
so maybe you can do something to find the IP from this header?

app.route("/data/status")
def data(request):
    client_ip = request.headers.get("X-Forwarded-For", request.client.host)
    logger.debug("Client IP : "+ client_ip)

and if is calling from a local IP, the country from that IP can't be done, so maybe you can implement a "country" called "LOCAL"
so we can filter in the UI by something.

[tom-draper]

@asyba Oh weird, I've recently updated the website build, so you may need to refresh your cache with Ctrl+F5.
Great, that's all I need to know thanks. I will update the FastAPI library soon to allow you to access the IP address with request.headers.get("X-Forwarded-For", request.client.host)
And adding a local country is a good idea! I'll include that too. Thanks

[tom-draper]

@asyba If you update your FastAPI analytics package to version >=1.2.0 you can now provide custom configuration.

You will need to create new config object and provide it with a custom function that returns the IP address from headers.

from fastapi import FastAPI
from api_analytics.fastapi import Analytics, Config

config = Config()
config.get_ip_address = lambda request: request.headers.get('X-Forwarded-For', request.client.host)

app = FastAPI()
app.add_middleware(Analytics, api_key=<API-KEY>, config=config)

[asyba]

@asyba If you update your FastAPI analytics package to version >=1.2.0 you can now provide custom configuration.

You will need to create new config object and provide it with a custom function that returns the IP address from headers.

from fastapi import FastAPI
from api_analytics.fastapi import Analytics, Config

config = Config()
config.get_ip_address = lambda request: request.headers.get('X-Forwarded-For', request.client.host)

app = FastAPI()
app.add_middleware(Analytics, api_key=<API-KEY>, config=config)

Hey I'm back from vacations.

I got some errors:
pip show fastapi-analytics
Name: fastapi-analytics
Version: 1.2.0

INFO:     Started reloader process [39] using StatReload
Process SpawnProcess-1:
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.9/site-packages/uvicorn/_subprocess.py", line 76, in subprocess_started
    target(sockets=sockets)
  File "/usr/local/lib/python3.9/site-packages/uvicorn/server.py", line 61, in run
    return asyncio.run(self.serve(sockets=sockets))
  File "/usr/local/lib/python3.9/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.9/asyncio/base_events.py", line 647, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.9/site-packages/uvicorn/server.py", line 68, in serve
    config.load()
  File "/usr/local/lib/python3.9/site-packages/uvicorn/config.py", line 467, in load
    self.loaded_app = import_from_string(self.app)
  File "/usr/local/lib/python3.9/site-packages/uvicorn/importer.py", line 21, in import_from_string
    module = importlib.import_module(module_str)
  File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/app/fastAPI-ui/app/main.py", line 14, in <module>
    from api_analytics.fastapi import Analytics, Config
  File "/usr/local/lib/python3.9/site-packages/api_analytics/fastapi.py", line 15, in <module>
    class Config:
  File "/usr/local/lib/python3.9/site-packages/api_analytics/fastapi.py", line 37, in Config
    get_path: Callable[[Request], str] | None = None
TypeError: unsupported operand type(s) for |: '_CallableGenericAlias' and 'NoneType'

[tom-draper]

@asyba Ah sorry, updating again to 1.2.1 will make it work for Python 3.9 and lower.

[asyba]

@asyba Ah sorry, updating again to 1.2.1 will make it work for Python 3.9 and lower.

thanks, it works now.

[Wamy-Dev]

is self hosted still coming?

[tom-draper]

Self-hosting is still in-progress, but I'm afraid I don't yet have any kind of date or timeline I can provide, other issues keep taking priority