Skip to main content

Graylog is restarting stuck with disk full

Graylog is restarting...
There is no Graylog web application running at the moment, please reload this page in a minute. It can take up to 1-2 minutes until all services are running properly. In case this is a permanent error, check the following:

Check if all services are running - sudo graylog-ctl status shows an overview of all running services
Check for errors in log files - Relevant services write log files here: /var/log/graylog/*/current
Ask for help - If there is no way to fix the issue ask for help:


I got this error on my Gray-log server, upon troubleshooting I found that the disk was 100% full and was unable to start elastic search mongodb and etcd while checking gray-log server status with command
#graylog-ctl status

Solution to this problem was obvious that I have to clean some disk space to get gray-log working again but what file should I delete was my next thought!

Upon googling I found that I could safely delete the old log files of elastic search to free up the space.

So I stopped gray-log server with

$sudo graylog-ctl stop

My gray-log installation path for elasticsearch logs was at

root@graylog:/var/opt/graylog/data/elasticsearch/graylog/nodes/0/indices#

Listed the files at this path

root@graylog:/var/opt/graylog/data/elasticsearch/graylog/nodes/0/indices# ls -al

drwx------ 7 graylog graylog 4096 Aug 12  2016 graylog_0
drwx------ 7 graylog graylog 4096 Aug  3  2017 graylog_1

I deleted one old log folder "graylog_0" which had consumed disk space of around 5 GB inside it.

root@graylog:/var/opt/graylog/data/elasticsearch/graylog/nodes/0/indices# rm -R graylog_0/

 After deleting the log folder I restarted the graylog server

 root@graylog:~# graylog-ctl start

Now I can access graylog server, all my configuration and dashboards are in place and working good. But I am getting an error for etcd (for clustering of node) of database corruption, a type of file "wal" is not accessible.

Since this is the only of my node and not a cluster configuration, I deleted the etcd folder and reconfigured the graylog server.

Delete the etcd folder here

root@graylog:~#/var/opt/graylog/data/rm -R etcd

root@graylog:~#/var/opt/graylog/data/graylog-ctl reconfigure

Now i can see the working status of all service with graylog as below

root@graylog:/var/opt/graylog/data/etcd/member# graylog-ctl status
run: elasticsearch: (pid 4437) 21s; run: log: (pid 876) 1059s
run: etcd: (pid 4272) 25s; run: log: (pid 891) 1059s
run: graylog-server: (pid 4490) 20s; run: log: (pid 857) 1059s
run: mongodb: (pid 4314) 23s; run: log: (pid 890) 1059s
run: nginx: (pid 4515) 20s; run: log: (pid 856) 1059s
Labels:

Comments

Post a Comment

Popular posts from this blog

How to import Putty Saved Connections to mRemoteNG

Just started using mRemoteNG and its being very cool to connect to different remote connection with different protocols e.g Window Remote Desktop, VNC to Linux, SSH, HTTP connection etc. from a single application. As new user I configured some remote desktop connection which was quite easy to figure out. But when I wanted to add SSH connections, it came in my mind to import all of the saved connections in the putty. But I couldn't figure it out how can it be done, though it was quite easy and here are the steps. Open your mRemoteNG Create a folder if you want segregation of multiple networks Create a new connection Enter the IP address of remote server under connection in Config pane Under the config pane, select protocol " SSH version 2 ".  Once you select protocol to SSH version 2 you are given option to import putty sessions, as shown in the snap below. In the above snap, I have imported CSR-AWS session from my saved sessions in Putty.
1 comment
Read more

BGP Soft Reconfiguration vs. Route Refresh: Key Differences and Best Practices

In BGP (Border Gateway Protocol), managing route updates and reapplying new policies can sometimes be challenging, especially if you want to avoid resetting the BGP session. Two methods allow you to update routing policies without tearing down the session: BGP Soft Reconfiguration and BGP Route Refresh . While both methods serve the same purpose, they work differently and have distinct impacts on your router's resources. This post explains the key differences between Soft Reconfiguration and Route Refresh , when to use each, and why Route Refresh is preferred in most modern networks. 1. What is BGP Soft Reconfiguration? BGP Soft Reconfiguration is an older method of applying new policies (like route maps, filters, or prefix lists) without resetting the BGP session. It works by storing a local copy of all the routes received from a BGP neighbor before applying inbound policies. This local route copy allows the router to reprocess the routes when a policy change occurs. How So...
Post a Comment
Read more

BGP Local Preference Controlling Outbound Traffic in BGP

In BGP, Local Preference is used to control the outbound traffic path. It helps you decide which egress point (exit point) should be used when you have multiple connections to external networks, such as ISPs. Local Preference is an attribute that is local to your AS and is shared with all iBGP peers but not with eBGP neighbors. Higher Local Preference = More preferred outbound path. Example Scenario : You have two external links: ISP1 (via CE1) and ISP2 (via CE2). You want traffic to prefer ISP1 for all outbound traffic. Network Topology : CE1 (connected to ISP1): 10.0.1.1/30 CE2 (connected to ISP2): 10.0.2.1/30 iBGP Router (Internal) connected to both CE1 (10.0.1.2/30) and CE2 (10.0.2.2/30). Configuration on CE1 (Higher Local Preference) : Create a route map to set the local preference to 200 for routes learned from CE1: route-map SET_LOCAL_PREF permit 10 set local-preference 200 In the BGP configuration for CE1, apply this route map to the neighbor: router bgp 65001 ne...
Post a Comment
Read more