But the strange thing is that it seems to stop sending out ajax calls to the backend (Prometheus) to get its data. Asking for help, clarification, or responding to other answers. Wow, not sure that is optimal! In the services I see for Prometheus: prometheus-server, prometheus-alertmanager, prometheus-kube-state-metrics, prometheus-node-exporter, prometheus-pushgateway,prometheus-server. Thanks for contributing an answer to Stack Overflow! To see all available qualifiers, see our documentation. https://10.XX.XX.XX:8024/d/Axz4zq2Mz/report-dashboard-osp-and-rhel-15m?orgId=1&from=now-30d&to=now, https://10.XX.XX.XX:3090/d/Axz4zq2Mz/report-dashboard-osp-and-rhel-15m?orgId=1&from=now-30d&to=now, http: proxy error: context canceled issues, Data source type & version: Prometheus 2.20, Query results from the inspect drawer (data tab & query inspector), Panel settings can be extracted in the panel inspect drawer JSON tab, Dashboard JSON can be found in the dashboard settings JSON model view. However, when testing the new DataSource, only the red exclamation mark appears and the log contains the following entry: grafana logger=data-proxy-log userId=1 orgId=1 uname=admin path=/api/datasources/proxy/4//_mapping remote_addr= referer= **What you expected to happen**: The text was updated successfully, but these errors were encountered: Hello @SJ1988. You switched accounts on another tab or window. Grafana 7.2.0 (no custom plugins), Prometheus 2.21.0, FreeBSD 12.1, Firefox 81. There's workarounds and officially we still don't support more than 26 queries per panel so I'll close this. Very curious why this works in 7.1.5 and not 7.2. t=2021-07-20T14:34:43+0530 lvl=eror msg="Data proxy error" logger=data-proxy-log userId=1 orgId=1 uname=admin path=/api/datasources/proxy/1/api/v1/query_range The URL work via direct https and curl from both the grafana and the database server. When clicking "Save & Test" button the data source should be added successfully; What happened instead? You were right. Suggestions cannot be applied while the pull request is closed. I would expect an HTTP GET request in this case. We are trying to proxy a datasource request through grafana backend as described in Add authentication for data source plugins | Grafana Labs . This is a known issue, and it has beed reported at grafana/grafana#27957, and it is due to the use of multiple queries towards a Prometheus datasource. When you start Grafana you will see new timeout parameter in Loki datasource settings. The other that is broken has 159 queries. You are receiving this because you authored the thread. Already on GitHub? It could be connected directly (Grafana -> Loki) or through Nginx proxy (Grafana -> Nginx -> Loki). You signed in with another tab or window. => Datasource test should work when saving my new datasource What happened instead? Thanks for creating this issue! You just have to open CMD and write ipconfig /all and then look at the IPv4 Direction and then you will have your IP. Usually located at /var/log/grafana/grafana.log on Unix systems or in <grafana_install_dir>/data/log on other platforms and manual installations. And did you change the datasource timeout with the new option? Sorry for the noise. deny all; Have a question about this project? I'm also seeing a 403 Forbidden on FQDN/api/frontend-metrics and a screenshot for that is attached too. I have the feature toggle fullRangeLogsVolume enabled and when using the Explore (with Loki datasource) I got consistent timeouts trying to load the Log volume metric graph. We read every piece of feedback, and take your input very seriously. **: It is very hard to give any advice without a better description. You must change the existing code in this line in order to create a valid suggestion. location / { In my world that doesnt fix it, it just takes a standard URL request and sends it to Grafana over https. => Debian Jessie, What did you do? Already on GitHub? You do know Prometheus supports regex label matching? But unfortunately calling const routePath = '/example'; getBackendSrv () .datasourceRequest ( { url: this.url + routePath + '/foo/bar', method: 'GET', }); return 301 https://grafana.mydomain.com; In the chrome console in Grafana page I see: http://grafana.domain.com/api/datasources/proxy/1/api/v1/query?query=1%2B1&time=1554043210.447, Grafana version: Grafana v6.0.0 (commit: 34a9a62), Grafana data source settings for Prometheus: prometheus.<namespace>.svc.cluster.local Start by inspecting your panel query and response. Could you please provide commit hash with working timeout parameter so I can check also? Sorry, I ran curl on the wrong server: it does not work actually with curl either. Not sure if its always the case, but seems like it has always issues with graphs that contain +10 queries. There is also no other way to get things refreshed etc by just doing F5. Also changed timeout,keepalive parameters to 600 but no luck. Using Expore Grafana interface I select more than 1h time range and request fails with 60 second timeout and HTTP 504 error code. privacy statement. We are using Grafana 7.1.3 where multiple dashboards are configured to view the KPI data coming via prometheus. All data sources also are in same network. It's working now . Like below, if you are run Grafana and Prometheus on docker on your local machine and this will do for the datasource settings, Add host as {host.docker.internal} : {port}, example - http://{host.docker.internal}:9090. Proxying http datasource don't work if grafana is hosted on https, https://github.com/notifications/unsubscribe/AAEp-UKXJsUWrWLBYmkUqLaAzd9n8_fLks5qVO1rgaJpZM4JLShi, What Grafana version are you using? Dashboards load fine when downgrading to 7.1.5. 
Maybe: It's in a different namespace It's defined but with a different name If it's a different namespace you can try changing your config with the full FQDN of the service. I have wildcard DigiCert signed cert deployed across all the servers. I have Grafana and Prometheus set up on my k8s cluster. error="http: proxy error: context canceled" / "Data proxy error" for Setting the dataproxy timeout to something higher (120) than the default value (10) did the trick. Already on GitHub? What is your code doing when it receives the request (in the query function in your data source)? That is good to hear! local:9090 in the Http url field. Sign up for a free GitHub account to open an issue and contact its maintainers and the community. So I reset grafana to protocol=http and removed the certificates and redirected to an http version of grafana. I can curl the test URL successfully from the internal and external IPs from the internal servers (grafana server, database server and another test server) and the external IPs from the trusted external IPs, I suspect I have a missed a setting, hence the Unknown, I just dont know what is Unknown. HTTP: proxy error: context canceled - Grafana Documentation - CERN When I try to set up the Prometheus as a data source in Grafana I get HTTP Error Bad Gateway. - User OS & Browser: Safari, Chrome How long time does your queries take that gets cancelled? Data proxy: Log proxy errors using Grafana logger 1fb7155 Now any errors logged by http.ReverseProxy are forwarded to Grafana's logger and includes more contextual information like level (error), user id, org id, username, proxy path, referer and IP address. Have a question about this project? Suggestions cannot be applied while viewing a subset of changes. With Grafana Live, you can push event data to a frontend as soon as an event occurs. Grafana datasource proxy request not working @hugohaggmark : there are way more requests indeed. Well occasionally send you account related emails. Suggestion cannot be applied right now. I get a Bad Gateway when I make that call. You can enable more logging by changing log level in the Grafana configuration file. **How to reproduce it (as minimally and precisely as possible)**: Unable to create a new dashboard or save changes to existing dashboards following update to Grafana v8.3.5 Suggestions cannot be applied on multi-line comments. Well occasionally send you account related emails. aknuds1 approved these changes. The text was updated successfully, but these errors were encountered: Is your InfluxDB data source accessible from the grafana server? Prometheus - http: proxy error: context canceled #27957 - GitHub Executing long-running Loki queries from within Grafana (Explorer) runs into timeout after approximately 16 seconds, while the same query executed via logcli works fine and does not timeout. As part of 7.2 we also made Prometheus data source cancelable changing to an api using Observables, could it be that the requests get the same requestId and now those are really canceled? Yes quite a lot :) But on 7.1.5 it works smooth. The reason will be displayed to describe this comment to others. - Grafana version: 8.3.5 2017/03/02 06:40:49 http: proxy error: EOF Reply to this email directly, view it on GitHub 17 comments borisyukd on Aug 28, 2021 Grafana was updated to 8.1.2 Loki dataproxy has been configured with timeout 600 Grafana version: Version 8.1.2 (commit: 103f8fa094) Data source type & version: The issue is how make Grafana use a https request to the database source. By clicking Post Your Answer, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct. I am able to add a postgreSQL data source. Now I am faced with the challenge of setting up an Elasticsearch DataSource. privacy statement. After installation while creating a dashboard and selcting metrics on the screen seeing "BAD Gateway" error. The setup was done via helm and at first glance everything looks normal. - Go to save the dashboard into a folder Maybe this is the reason for the error code 502, because the rerferer parameter in the request uses the external addresses of Grafana. Please note that your post might be misread as a link-only answer (which risks being deleted). I have 2 graph panels in the same dashboard with A-Z amount of queries. You can change it in your config. This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. - User OS & Browser: Safari, Chrome Then proxy_pass to the grafana instance which is likely what youre doing. Background: Grafana: Datasource (Prometheus ) query 403 - Stack Overflow I created docker image from main branch, from 7.5.6 tag and got after 60 seconds: error="http: proxy error: context canceled", dataproxy.timeout = 300 I have an option to use proxy server, but Grafana do not have proxy settings. thank you. When running Grafana behind a proxy, you need to configure the domain name to let Grafana know how to render links and redirects correctly. Under non https every thing works fine, so it must just be a config issue. aurimasplu November 5, 2021, 11:43am 1 Hi, I have Grafana server in closed network without direct internet access. Site design / logo 2023 Stack Exchange Inc; user contributions licensed under CC BY-SA. ssl on; **Anything else we need to know? The text was updated successfully, but these errors were encountered: Could you post a link to the postgres data source that you are using, please. I had the grafana instance also serving as https (protocol = h20 and was serving on localhost with the same certificates setup in grafana. Have a question about this project? Searching on Google I cant anythng for certificate is valid for Unknown, The working non-ssl config has access : proxy in the Yaml file under provisioning, I have tried direct as well, what reverse proxy are you using? SSLEngine on Sign up for a free GitHub account to open an issue and contact its maintainers and the community. - Data source type & version: InfluxDB Do not open a github issue. Is this mold/mildew? what could cause you to need so many queries? 10 comments erkexzcx commented May 13, 2020 Grafana version: Grafana v6.7.3 ( a04ef6c) Data source type & version: InfluxDB (latest stable), 1.8.0 OS Grafana is installed on: Debian 10. However sometime we are getting below error while loading the dashboard. There was a change to reverse proxy settings in 8.3.5+, **What happened**: Datasource is Thanos Querier which is further connected with prometheus. Looking for story about robots replacing actors. Sign in that's a lot of network request, wish Prometheus had request batching :), I have heard of like 5 or ten queries in a single panel but never anything like this :). Yes, the port 18086 is correct, and curl can query data from there. Nothing shown. After installation while creating a dashboard and selcting metrics on the screen seeing BAD Gateway error. SSLCertificateChainFile pathto/DigiCertCA.crt, Changed Proxy from internal ip to internal FQDN got the proxy working but still Bad Gateway. @borisyukd server { I'm not seeing anything of particular use in the /var/log/grafana/grafana.log at INFO level, let me know if I should try debug level. After upgrading grafana from 7.1.5 to 7.2.0, some dashboards don't load completely anymore. Im so stupid I just noticed that in this environment an HTTPS_PROXY environment variable Is set, and of course I need to set the appropriate NO_PROXY (elasticsearch URL) configuration in this case as well, Powered by Discourse, best viewed with JavaScript enabled, Data proxy error" error="http: proxy error: Forbidden". Conclusions from title-drafting and question-content assistance experiments Kubernetes cannot access grafana and prometheus fron Google cloud platform, Grafana HTTP Error Bad Gateway and Templating init failed errors, Prometheus / Grafana setup on kubernetes (gcloud), Kubernetes ingress nginx grafana dashboard prometheus data was empty, Can not get Prometheus data in Grafana dashboard in Kubernetes, Error in setting Prometheus datasource in Grafana on Kubernetes, Grafana dashboard not working with Ingress, Issues with adding Prometheus as a datasource to Grafana, Prometheus datasource : client_error: client error: 403. **Anything else we need to know? By clicking Sign up for GitHub, you agree to our terms of service and I think loadbalancer might be the issue here as well, good call. You switched accounts on another tab or window. I had a similar issue to the one described here and I got here by googling the data proxy error, this might help someone in a similar situation. The issue exists in both cases. Proxy Error 502 "Reason: Error reading from remote server" with Apache We read every piece of feedback, and take your input very seriously. **: install Loki setup Loki as Grafana Data Source setup Grafana ' [dataproxy]' timeouts load enough data to Loki to get request execution with more than 60 seconds try to make a request via Grafana Explore interface Grafana version: Version 7.4.3 (commit: 010f20c1c8, branch: HEAD) Data source type & version: User OS & Browser: I could not find a way to make Grafana ignore any verification/certifcate issues on the JSON databsource connection, Powered by Discourse, best viewed with JavaScript enabled, dashboards following update to Grafana v8.3.5 In the Grafana configuration file, change server.domain to the domain name you'll be using: [server] domain = example.com Bash Restart Grafana for the new changes to take effect. The setup was done via helm and at first glance everything looks normal. I see the same errors, however it happens after 50 seconds instead of 60. t=2020-05-13T12:51:25+0530 lvl=info msg=Request Completed logger=context userId=1 orgId=1 uname=admin method=POST path=/api/datasources/proxy/1/render status=502 remote_addr= time_ms=0 size=0 referer=http://:3000/dashboard/new?panelId=2&edit&fullscreen&orgId=1, Powered by Discourse, best viewed with JavaScript enabled, Http: proxy error: unsupported protocol scheme \"\. The plugin is not available anywhere in the link . Already on GitHub? error="http: proxy error: context canceled" / "Data proxy error" for Loki Data Source, Failed to fetch / http: proxy error: context canceled / Data proxy error, load enough data to Loki to get request execution with more than 60 seconds, try to make a request via Grafana Explore interface, Grafana plugins: No additional plugins installed, Get Grafana 504 Gateway timeout response. I've had a similar question about what happens if one of the request in the queue fails and as far as I've tested the rest of the queue will continue to process. I've tried enabling http2 on my reverse proxy in front of Grafana, but it still gives me cancelled requests. To avoid having your issue closed in the future, please read our CONTRIBUTING guidelines. Both were installed thru helm using https://github.com/helm/charts/tree/master/stable. You signed in with another tab or window. - Others: None. If what you show is meant as code then please format it accordingly and add an explanation. to your account, Please prefix your title with [Bug] or [Feature request] Data proxy error" error="http: proxy error: Forbidden" - Grafana plugins: None - Log into Grafana instance and click on "+" button and create new dashboard to your account. Please check back later. You signed in with another tab or window. To learn more, see our tips on writing great answers. Using Chrome Developer Tools inspecting the requests being made from the dashboard, can you see if they're cancelled? listen 80; When you are trying to add a data source, like Prometheus it is a little bit confusing because they are asking for the http, but you have to put your IP adress. What you expected to happen: Data should get loaded in the dashboard. - Go to save the dashboard into a folder I have found the workaround: Now you can start adding data sources and dashboards to create your visualizations. Seems like everything hangs. Visit http://localhost:3000 and login as admin / admin. If I was to guess, then maybe you are returning an empty response? For question please check Support Options. Grafana should return valid data from Loki with HTTP 200 code. (Bathroom Shower Ceiling). let me take a look. Ability to create/modify dashboards in Grafana "http: proxy error: EOF" when adding datasource for PostgreSQL Grafana throws HTTP Error Bad Gateway for Prometheus data source Looking at your screenshot, do I get this correctly that beginning with your 5th request the requests are always cancelled or are there more requests before in this screenshot? That bit works fine for me and always has. Help build the future of open source observability software Open positions Check out the open source projects we support Downloads An image of the test execution data fully expanded. So the last part is going to Prometheus and in the URL you have to put: It seems to be (as far as I tested different cases), always triggered by graphs with +10 queries. Already on GitHub? minimalistic ext4 filesystem without journal and other advanced features. So closing this, but feel free to reopen if you think that problem is with Grafana. proxy_connect_timeout 600; Connect and share knowledge within a single location that is structured and easy to search. INFO[03-02|06:40:50] Request Completed logger=context userId=1 orgId=1 uname=admin method=POST path=/query status=502 remote_addr=xx.xx.xx.xx time_ms=4.709s size=0. Configure auth proxy authentication | Grafana documentation . Datasource timeout (loki) = 300. This could be notifications about dashboard changes, new frames for rendered data, and so on. Solution: 2 Answers Sorted by: 3 Sounds like it cannot find the prometheus service in the cluster. Have a question about this project? 592), How the Python team is adapting the language for an AI future (Ep. Grafana Labs Community Forums Bad Gateway http: proxy error: x509: certificate is valid for Unknown, not FQDN Grafana Configuration thefamouseric March 2, 2022, 10:54pm 1 I have everthing working happily under non-https but using https I cannot get passed to your account, What happened: Ok, so it is your own custom plugin. - Attempt to save fails, "Origin Not Allowed" error shows up in upper right corner ServerName grafana.oninit.com Attached is what I'm seeing in the console upon trying to save; looks like it is getting a 403 Forbidden for FQDN/api/dashboards/db and this is preventing dashboard from saving. You can configure Grafana to let a HTTP reverse proxy handle authentication. Making statements based on opinion; back them up with references or personal experience. @Kavinsp what did you do to resolve this? @marefr assigning this to backend team. Suggestions cannot be applied while the pull request is queued to merge. Changing the JSON to have unique refId's make everything work again! Have a question about this project? This causes the canceled requests. How to reproduce it (as minimally and precisely as possible): Grafana dataproxy configuration from grafana.ini. Can you explain more about it? SSLProxyEngine on Many are running Grafana behind https with no issues, so this sounds strange. Am using proxy for the data source settings. Successfully merging this pull request may close these issues. 3 comments SJ1988 commented on Sep 14, 2021 Grafana version: 7.1.3 Data source type & version: Prometheus 2.20 OS Grafana is installed on: RHEL 7 I have upgraded to Grafana version v8.4.2. Add a new proxied (access=proxy) InfluxDB datasource which url looks like http ://my.datasource while my grafana instance url looks like https ://my.grafana.instance What was the expected result? Browse other questions tagged, Where developers & technologists share private knowledge with coworkers, Reach developers & technologists worldwide, The future of collective knowledge sharing. Not sure if I understood. Is it appropriate to try to contact the referee of a paper after it has been accepted and published? privacy statement. **Environment**: And what grafana version? In an ideal world it would be mysql_com{which="alter_table"}, but its not After some debugging with @hugohaggmark we've found out the root cause :). It seems to be a firewall configuration issue on my side. **How to reproduce it (as minimally and precisely as possible)**: What might be wrong and how to debug/fix it? I modified dataproxy timeouts to 10 minutes. SSLCertificateFile pathto/star_oninit_com.crt Add this suggestion to a batch that can be applied as a single commit. not sure then what could be the cause, in the issue title you mention https, had you tried it without https and it worked? Namespace. Attached is what I'm seeing in the console upon trying to save; looks like it is getting a 403 Forbidden for FQDN/api/dashboards/db and this is preventing dashboard from saving. The fix was to make sure the SSL worked across all the servers. "http: proxy error: EOF" when adding datasource for PostgreSQL, https://github.com/notifications/unsubscribe-auth/ACtngcTktSgGmi5YBPcodZMB9-6hz7lzks5tJmJpgaJpZM4MQlLv, An image or text representation of your metric query, The raw query and response for the network request (check this in chrome dev tools network tab, here you can see metric requests and other request, please include the request body and request response). monitoring. { "datasource": "Prometheus", "expr": "100 - (avg(irate(node_cpu_seconds_total{instance="10.67.250.60:9100",mode="idle"}[5m])) * 100)", "interval": "", "legendFormat": "10.67.250.60 CPU", "refId": "717" }. Popular web servers have a very extensive list of pluggable authentication modules, and any of them can be used with the AuthProxy feature. Why are you doing a POST for a select query? Sign in Now I am faced with the challenge of setting up an Elasticsearch DataSource. Clicking the "Save" button prompts one of these errors in the console. Any idea? Share Improve this answer Follow answered Dec 9, 2020 at 6:13 Saikat Chakrabortty 2,478 4 22 39 What is the audible level for digital audio dB units? Questions? Find centralized, trusted content and collaborate around the technologies you use most. For data visualization issues: For authentication, provisioning and alerting issues, Grafana server logs are useful. and I see in the Prometheus that one resource is down: prometheus-server. In Grafana data source settings for prometheus database add prometheus service dns and service port. Set up Grafana Live | Grafana documentation Bad Gateway http: proxy error: x509: certificate is valid for Unknown Pinging @hugohaggmark as well since it may be related to the changes made in regards to browser request cancellation/maximum queries allowed at the same time. You signed in with another tab or window. This way the reverse proxy error logs are marked as error. elasticsearch, datasource mguggi April 4, 2022, 8:25am #1 I have upgraded to Grafana version v8.4.2. t=2020-05-13T12:51:25+0530 lvl=eror msg=Data proxy error logger=data-proxy-log userId=1 orgId=1 uname=admin path=/api/datasources/proxy/1/render remote_addr=<> referer=http://:3000/dashboard/new?panelId=2&edit&fullscreen&orgId=1 error=http: proxy error: unsupported protocol scheme Grafana throws HTTP Error Bad Gateway for Prometheus data source, https://github.com/helm/charts/tree/master/stable, What its like to be on the Python Steering Council (Ep. Maybe your connection between the browser and the grafana backend was interrupted while waiting for the proxied request to return. Note that if I change the "Access" value from "proxy" to "direct", it works. Sign in So it should look like: Grafana doesn't allow me to save it if I have prometheus-server. How to make Grafana to use proxy for specific data sources? Data proxy: Log proxy errors using Grafana logger, Learn more about bidirectional Unicode characters, Provide additional context to reverseproxy logger, Data proxy: Log proxy errors using Grafana logger (. ***> wrote: You signed in with another tab or window. svc. I have Installed latest Grafana 6.7.3 on a CENT OS machine. The text was updated successfully, but these errors were encountered: I have a similar issue. privacy statement. Same here with Grafana v7.5.1. to your account. You might be hitting query_timeout. listen 443 ssl http2; http: proxy error: context canceled Issue #39178 grafana/grafana The DB I used is the default SQL-Lite.
Maybe: It's in a different namespace It's defined but with a different name If it's a different namespace you can try changing your config with the full FQDN of the service. I have wildcard DigiCert signed cert deployed across all the servers. I have Grafana and Prometheus set up on my k8s cluster. error="http: proxy error: context canceled" / "Data proxy error" for Setting the dataproxy timeout to something higher (120) than the default value (10) did the trick. Already on GitHub? What is your code doing when it receives the request (in the query function in your data source)? That is good to hear! local:9090 in the Http url field. Sign up for a free GitHub account to open an issue and contact its maintainers and the community. So I reset grafana to protocol=http and removed the certificates and redirected to an http version of grafana. I can curl the test URL successfully from the internal and external IPs from the internal servers (grafana server, database server and another test server) and the external IPs from the trusted external IPs, 592), How the Python team is adapting the language for an AI future (Ep. Grafana Labs Community Forums Bad Gateway http: proxy error: x509: certificate is valid for Unknown, not FQDN Grafana Configuration thefamouseric March 2, 2022, 10:54pm 1 I have everthing working happily under non-https but using https I cannot get passed to your account, What happened: Ok, so it is your own custom plugin. - Attempt to save fails, "Origin Not Allowed" error shows up in upper right corner ServerName grafana.oninit.com Attached is what I'm seeing in the console upon trying to save; looks like it is getting a 403 Forbidden for FQDN/api/dashboards/db and this is preventing dashboard from saving. You can configure Grafana to let a HTTP reverse proxy handle authentication. Making statements based on opinion; back them up with references or personal experience. @Kavinsp what did you do to resolve this? @marefr assigning this to backend team. Suggestions cannot be applied while the pull request is queued to merge. Changing the JSON to have unique refId's make everything work again! Have a question about this project? This causes the canceled requests. How to reproduce it (as minimally and precisely as possible): Grafana dataproxy configuration from grafana.ini. Can you explain more about it? SSLProxyEngine on Many are running Grafana behind https with no issues, so this sounds strange. Am using proxy for the data source settings. Successfully merging this pull request may close these issues. 3 comments SJ1988 commented on Sep 14, 2021 Grafana version: 7.1.3 Data source type & version: Prometheus 2.20 OS Grafana is installed on: RHEL 7 I have upgraded to Grafana version v8.4.2. Add a new proxied (access=proxy) InfluxDB datasource which url looks like http ://my.datasource while my grafana instance url looks like https ://my.grafana.instance What was the expected result? Browse other questions tagged, Where developers & technologists share private knowledge with coworkers, Reach developers & technologists worldwide, The future of collective knowledge sharing. Not sure if I understood. Is it appropriate to try to contact the referee of a paper after it has been accepted and published? privacy statement. **Environment**: And what grafana version? In an ideal world it would be mysql_com{which="alter_table"}, but its not After some debugging with @hugohaggmark we've found out the root cause :). It seems to be a firewall configuration issue on my side. **How to reproduce it (as minimally and precisely as possible)**: What might be wrong and how to debug/fix it? I modified dataproxy timeouts to 10 minutes. SSLCertificateFile pathto/star_oninit_com.crt Add this suggestion to a batch that can be applied as a single commit. not sure then what could be the cause, in the issue title you mention https, had you tried it without https and it worked? Namespace. Attached is what I'm seeing in the console upon trying to save; looks like it is getting a 403 Forbidden for FQDN/api/dashboards/db and this is preventing dashboard from saving. The fix was to make sure the SSL worked across all the servers. "http: proxy error: EOF" when adding datasource for PostgreSQL, https://github.com/notifications/unsubscribe-auth/ACtngcTktSgGmi5YBPcodZMB9-6hz7lzks5tJmJpgaJpZM4MQlLv, An image or text representation of your metric query, The raw query and response for the network request (check this in chrome dev tools network tab, here you can see metric requests and other request, please include the request body and request response). monitoring. { "datasource": "Prometheus", "expr": "100 - (avg(irate(node_cpu_seconds_total{instance="10.67.250.60:9100",mode="idle"}[5m])) * 100)", "interval": "", "legendFormat": "10.67.250.60 CPU", "refId": "717" }. Popular web servers have a very extensive list of pluggable authentication modules, and any of them can be used with the AuthProxy feature. Why are you doing a POST for a select query? Sign in Now I am faced with the challenge of setting up an Elasticsearch DataSource. Clicking the "Save" button prompts one of these errors in the console. Any idea? Share Improve this answer Follow answered Dec 9, 2020 at 6:13 Saikat Chakrabortty 2,478 4 22 39 What is the audible level for digital audio dB units? Questions? Find centralized, trusted content and collaborate around the technologies you use most. For data visualization issues: For authentication, provisioning and alerting issues, Grafana server logs are useful. and I see in the Prometheus that one resource is down: prometheus-server. In Grafana data source settings for prometheus database add prometheus service dns and service port. Set up Grafana Live | Grafana documentation Bad Gateway http: proxy error: x509: certificate is valid for Unknown Pinging @hugohaggmark as well since it may be related to the changes made in regards to browser request cancellation/maximum queries allowed at the same time. You signed in with another tab or window. This way the reverse proxy error logs are marked as error. elasticsearch, datasource mguggi April 4, 2022, 8:25am #1 I have upgraded to Grafana version v8.4.2. t=2020-05-13T12:51:25+0530 lvl=eror msg=Data proxy error logger=data-proxy-log userId=1 orgId=1 uname=admin path=/api/datasources/proxy/1/render remote_addr=<> referer=http://:3000/dashboard/new?panelId=2&edit&fullscreen&orgId=1 error=http: proxy error: unsupported protocol scheme Grafana throws HTTP Error Bad Gateway for Prometheus data source, https://github.com/helm/charts/tree/master/stable, What its like to be on the Python Steering Council (Ep. Maybe your connection between the browser and the grafana backend was interrupted while waiting for the proxied request to return. Note that if I change the "Access" value from "proxy" to "direct", it works. Sign in So it should look like: Grafana doesn't allow me to save it if I have prometheus-server. How to make Grafana to use proxy for specific data sources? Data proxy: Log proxy errors using Grafana logger, Learn more about bidirectional Unicode characters, Provide additional context to reverseproxy logger, Data proxy: Log proxy errors using Grafana logger (. ***> wrote: You signed in with another tab or window. svc. I have Installed latest Grafana 6.7.3 on a CENT OS machine. The text was updated successfully, but these errors were encountered: I have a similar issue. privacy statement. Same here with Grafana v7.5.1. to your account. You might be hitting query_timeout. listen 443 ssl http2; http: proxy error: context canceled Issue #39178 grafana/grafana The DB I used is the default SQL-Lite.